Why Medical AI Tools Must Be Explainable, Not Just Accurate
Accuracy is necessary. It is not sufficient. The most dangerous AI systems in medicine are not the ones that are wrong, they are the ones that are wrong with confidence.
In 2018, a paper published in Nature Medicine described a deep learning system that diagnosed diabetic retinopathy from fundus photographs with accuracy exceeding that of ophthalmologists. The result was real, the methodology was sound, and the headlines were appropriately impressed. What the headlines did not spend much time on was the question of what a clinician was supposed to do with a diagnosis they could not explain, verify, or contest.
That question has not gone away. It has become more pressing as AI tools have moved from research papers into actual clinical and research workflows, into the systems that inform treatment decisions, guide research priorities, and generate the evidence on which medicine is based.
Accuracy is necessary. It is not sufficient. This post is about why.
What explainability actually means in a medical context
Explainability in AI is sometimes discussed as a single property, a system either is or is not explainable. In practice it is several related but distinct requirements, and which ones matter depends on how the system is being used.
Traceability means being able to identify the source evidence behind a claim. In a medical research context, if a system tells you that a particular drug has been associated with a specific adverse event in a specific population, you need to know which studies that claim is based on. Without traceability, you cannot verify the claim, assess the quality of the underlying evidence, or identify the limitations that might affect whether it applies to your situation.
Reasoning transparency means being able to follow the logical steps between input and output. A system that produces a risk score without explaining what factors contributed to it and how they were weighted is producing something that looks like information but cannot be interrogated. In a clinical decision support context, this is not a minor inconvenience, it is the difference between a tool that supports clinical judgment and one that substitutes for it without accountability.
Uncertainty quantification means being honest about confidence. A system that produces confident outputs across the full range of its inputs, including the inputs that are far from its training distribution, or where the underlying evidence is genuinely ambiguous, is a system that cannot be trusted in the cases where it matters most. The most dangerous AI systems in medicine are not the ones that are wrong. They are the ones that are wrong with confidence.
The accountability problem
Medical practice and medical research both operate within accountability structures that AI systems do not naturally fit. A clinician who makes a decision is accountable for that decision, to the patient, to their institution, to regulatory bodies. A researcher who publishes a finding is accountable for its methodology and its interpretation. These accountability structures exist for good reasons, and they depend on being able to examine the reasoning that led to a particular conclusion.
When an AI system contributes to a clinical or research decision, the question of where accountability sits becomes genuinely complicated. If a system recommends a treatment and the clinician follows that recommendation, who is responsible for the outcome? If a research hypothesis was generated by an AI tool that identified a pattern in the literature, and that hypothesis turns out to be based on a spurious correlation in the training data, who bears responsibility for the resources spent testing it?
These are not hypothetical questions. They are questions that regulators, ethics committees, and legal systems are actively working through. The AI tools that will be trusted in serious medical and research contexts are the ones that are designed with these accountability questions in mind, that keep the human in the loop, that make their reasoning visible, and that are honest about what they do not know.
Why black-box models persist despite their limitations
The honest answer is that black-box models, particularly deep neural networks, often outperform more interpretable alternatives on accuracy benchmarks. There is a genuine trade-off in many domains between predictive accuracy and interpretability, and accuracy is easier to measure and to market.
The trade-off is also less absolute than it was five years ago. Techniques for extracting explanations from complex models, attention mechanisms, saliency maps, SHAP values, counterfactual explanations, have improved significantly. Inherently interpretable models have also improved in performance on many tasks. The claim that you have to choose between accuracy and explainability is less defensible now than it once was, at least for many of the tasks that matter in medical research.
The persistence of black-box approaches in medical AI reflects partly the inertia of research communities that developed their tools in non-medical contexts, partly the genuine difficulty of the engineering problem, and partly a culture in which impressive benchmark results attract more attention than careful consideration of deployment requirements.
What we built around at NousLab
When we designed NousLab, explainability was a constraint, not a feature we added later. Every claim the system makes about the literature is traceable to specific source papers. Every synthesis is accompanied by the evidence it is based on. Uncertainty is surfaced rather than suppressed, when the literature is genuinely conflicted on a point, the system says so rather than picking a side.
This is not altruism. It is a response to what the researchers we work with actually need. A researcher who cannot verify a claim cannot use it. A system that produces plausible-sounding outputs without provenance is not a research tool, it is a liability. The researchers and institutions doing serious medical work do not want a system that tells them what to think. They want a system that helps them think better, with full visibility into the basis for what it is showing them.
This means NousLab is sometimes less impressive in a demo than systems that produce fluent, confident-sounding outputs without attribution. We are comfortable with that trade-off. The demos are not the point. The research is.
The regulatory direction is clear
Regulatory bodies in the US, EU, and UK are all moving toward requiring greater explainability and auditability in AI systems used in clinical and research contexts. The EU AI Act classifies medical AI as high-risk and imposes transparency requirements that black-box systems will struggle to meet. The FDA's evolving guidance on AI in drug development emphasises the need for documentation of how AI outputs were generated and validated.
The direction of travel is not ambiguous. Explainability is becoming a regulatory requirement, not a design preference. Tools built without it will face increasing friction in the markets where medical AI matters most.
The researchers and institutions evaluating AI tools today would do well to ask not just "how accurate is it?" but "can I see why it thinks that?" The second question is the one that determines whether the tool is usable in a context where the answer actually matters. See how NousLab is built around transparency and traceable reasoning.
