AI in Healthcare: Promise and Pitfalls
Clinical AI has documented capability and documented failure modes. Read why validation, data provenance, and accountability decide whether it works.
Healthcare is the domain where the promise of AI is most compelling and the cost of failure highest. A system that reads images, predicts deterioration, or drafts clinical documentation could relieve real pressure on clinicians. The same system applied carelessly can harm a patient, and the harm is difficult to detect because the model's confidence is not evidence of correctness.
Direct answer:
Clinical AI succeeds or fails on three things that are not the model architecture: whether the evaluation reflects the population and workflow where it will be used, whether the data and the model's limits are documented, and whether a named clinician remains accountable.
Model cards call for "benchmarked evaluation in a variety of conditions" across relevant groups, and datasheets document a dataset's "motivation, composition, collection process, recommended uses". NIST's AI RMF is "intended for voluntary use and to improve the ability to incorporate trustworthiness considerations into the design, development, use, and evaluation of AI products". Those are the instruments that turn a promising result into a deployable system.

Where the promise is real
The capability is not hypothetical. Models can detect patterns in images and text at a scale that assists clinical work, and they can process routine documentation so clinicians spend less time on administration.
The value is clearest when the task has a verifiable output and a defined boundary. An image classification system that flags a suspicious region for a radiologist to review is assisting a decision, not making one. A model that drafts a discharge summary for a clinician to approve is reducing a repetitive task, not replacing judgement. In both cases the system's output is checked by a person who is accountable for the result.
This is the right frame for most clinical applications: augmentation with a clear review step, not autonomous action. The systems that fail are often the ones where the review step was assumed rather than designed — a clinician is nominally "in the loop" but has neither the time nor the information to check the model's output meaningfully.
The validation gap
Population shift. A model trained on images from one institution may not generalise to another, because the equipment, patient mix, and imaging protocols differ. The reported accuracy is a property of the evaluation set, not of the disease.
Workflow mismatch. Retrospective evaluation uses clean, complete data. Clinical practice is messy — missing fields, conflicting measurements, time pressure — and a model that requires inputs nobody records at the point of care will not be used, or will be used on guessed inputs.
Label uncertainty. The "ground truth" in clinical data is often a clinician's judgement, which varies, or a billing code, which is an administrative category rather than a diagnosis. A model evaluated against a noisy label can appear accurate for the wrong reason.
scikit-learn's guidance on data leakage is relevant: leakage occurs when "information that would not be available at prediction time is used when building the model". In clinical data this can be subtle — a note written after the outcome, a test ordered because the outcome was suspected — and it produces results that collapse in prospective use.
The discipline that addresses this is prospective validation in the actual workflow, with the same population and the same messy inputs the system will face. A retrospective result is a reason to run that study, not a reason to deploy.
Documentation is a clinical instrument
Model cards and datasheets are usually presented as governance paperwork, and in healthcare they have a more direct function: they are the only way a clinician can know what a system was trained on and where it is likely to fail.
A model card reports performance across relevant groups and conditions, along with the intended use and the evaluation procedure. A datasheet documents the dataset's composition, collection process, and recommended uses. Together they answer the questions a clinician should ask before trusting an output: was this validated on patients like mine, what was the outcome measured, and what is outside the system's scope?
Three properties matter:
- The documentation must travel with the model. A card filed separately from the deployed system will not be consulted when a clinician needs it.
- It must name the failures, including groups where performance was worse and conditions that were not evaluated. Absence of evidence is not evidence of safety.
- It must be dated and versioned. A model that has been retrained is a different system, and the previous card describes a model that no longer exists.
The NIST RMF's structure — governance, mapping, measurement, management — is a way of organising these artefacts into a programme rather than a folder. Its voluntary status is worth stating plainly: it is a framework for producing evidence, not a regulatory approval.
Who is accountable
A model cannot be responsible for a clinical decision. The accountable party must be a person or an institution with the authority to act on the system's output and to stop using it.
Three questions expose weak governance:
- Who reviews the output, and do they have time to do it properly? A review step that takes longer than the task saves is not a real control.
- What happens when the model and the clinician disagree? The escalation path has to be defined before deployment, not invented during an incident.
- Can the system be switched off? If the model is embedded in a workflow with no override, the organisation has ceded control of a clinical process.
This is an ethical and legal question as much as a technical one. For the broader treatment, see AI Ethics and Responsible AI.
A worked example of a validation failure
Suppose a model predicts sepsis risk from electronic health record data and reports 90% accuracy on a retrospective set. Before deployment, four questions decide whether that number means anything:
- What is the prevalence? If sepsis occurs in 2% of the set, a model that predicts "no sepsis" for everyone achieves 98% accuracy. The headline figure may be measuring the class balance rather than clinical skill.
- When were the inputs recorded? If a lab result that was ordered because sepsis was suspected is included as a feature, the model is reading the clinician's suspicion, not predicting it — a form of leakage.
- Which patients were in the set? If it excludes the elderly or the very young, performance on those groups is unknown, which matters because they are often at highest risk.
- What is the false-positive rate at the chosen threshold? A system that alerts frequently will be ignored, and an ignored alert system has no clinical value regardless of its metric.
The point is that the accuracy figure is the least informative part of the evaluation. Prevalence, timing, population, and the cost of errors are the substance, and all four require judgement about the clinical context.
A practical checklist
- Define the intended use and the review step before selecting a model.
- Check the data provenance with a datasheet: composition, collection, and known gaps.
- Audit for leakage with particular attention to time-ordered clinical events.
- Evaluate on the deployment population and report by relevant subgroup.
- State the error profile, not only the aggregate metric.
- Produce a model card with intended use, evaluation conditions, and limitations.
- Name the accountable clinician and define the override path.
- Validate prospectively before the system influences care.
For the evaluation discipline, see MLOps; for the documentation standard, see AI Governance.
A counterargument
The prescription is strong but leaves a tension unresolved: the cited sources are governance formats and a voluntary framework, not evidence of clinical safety.
Model cards call for benchmarked evaluation across groups and datasheets document provenance, while NIST's RMF is "intended for voluntary use" - none of them establishes that a system performs safely in a real workflow. Prospective validation, the article's own key control, is precisely what those artefacts cannot guarantee. Naming an accountable clinician does not supply the time or authority to override a system embedded in care, and the piece concedes that nominal review steps often fail because clinicians lack both. A reasonable critic would say the checklist produces evidence without ensuring it is acted on. The framing is incomplete about the gap between documenting limits and enforcing them.
References & Further Reading
- Margaret Mitchell et al., 2019. Model Cards for Model Reporting — The reporting template used to document intended use, evaluation, and limitations.
- Timnit Gebru et al., 2021. Datasheets for Datasets — The document format that makes a dataset's provenance and limits reviewable.
- NIST, 2023. AI Risk Management Framework | NIST — The governance framework referenced in AI risk and compliance work.