AI in Healthcare and Science

Medicine and science are where AI's promise and its failure modes are both at their sharpest. A model that reads chest X-rays, flags sepsis or predicts a protein's shape can save time and lives, but a model that quietly performs worse on one hospital's scanner or one group of patients can cause harm at scale. This lesson covers what AI does in these fields today, how it is regulated, and, most importantly, what evidence you should demand before trusting it.

This lesson is general education, not medical or regulatory advice. Regulatory details change; check the official sources listed before relying on them.

The intuition: a new instrument, not a new doctor

Think of a clinical AI model as a new laboratory instrument, like a blood analyser. Nobody asks whether a blood analyser is "intelligent". They ask: what exactly does it measure, how accurate is it, on which samples was that accuracy established, how does it drift over time, and what happens to patients when it is wrong? The same questions, asked with the same discipline, cut through most of the hype around medical AI. The instrument framing also explains why regulators treat much clinical AI software as a medical device: its output influences diagnosis or treatment, so it must show it is safe and effective for a stated purpose.

Science adds a second picture: AI as a very fast, very cheap hypothesis generator. AlphaFold proposes protein structures; graph networks propose stable crystals; learned weather models propose tomorrow's atmosphere. Each proposal is valuable only to the degree that it is checked against experiment or observation.

Where AI is used in healthcare

Medical imaging

Imaging is the most mature area because it fits supervised learning well: large archives of labelled images, a well-defined output (a finding, a measurement, a triage priority), and a clinician already in the loop. Typical functions are:

• Detection and triage: flagging suspected findings such as a brain bleed on CT or a lung nodule, and moving those studies up the reading queue.
• Measurement and segmentation: outlining organs or tumours and computing volumes, which is tedious and variable when done by hand.
• Screening support: acting as a second reader in programmes like mammography, or reading retinal photographs for diabetic retinopathy.

An important milestone was IDx-DR (now LumineticsCore), which the US Food and Drug Administration (FDA) authorised in 2018 as the first autonomous AI diagnostic system: it outputs a referral decision for diabetic retinopathy without a clinician interpreting the image. The pivotal study was prospective, ran in primary-care clinics and pre-specified its endpoints, which is exactly the kind of evidence that should be the norm.

Prospective trials of AI in screening are now appearing. The Swedish MASAI randomised trial (Lång et al., Lancet Oncology, 2023) reported in its interim safety analysis that AI-supported mammography screening found about 20% more cancers than standard double reading while cutting the radiologists' screen-reading workload by about 44%, with a similar false-positive rate. The trial's primary endpoint, interval cancers (cancers diagnosed between screening rounds), was reported in The Lancet in January 2026: about 12% fewer interval cancers in the AI-supported arm (82 versus 93 among roughly 106,000 women). The design (randomised, prospective, real screening population, a clinically meaningful endpoint) is the right one.

Clinical decision support and risk prediction

Clinical decision support (CDS) means software that gives clinicians patient-specific information or recommendations: drug-interaction alerts, deterioration scores, sepsis warnings, readmission risk. Risk models built on electronic health record (EHR) data are widespread, and they are where several cautionary tales come from:

• External validation failure. Wong et al. (JAMA Internal Medicine, 2021) evaluated a widely deployed proprietary sepsis model at Michigan Medicine and found an area under the ROC curve of 0.63, well below the developer's reported performance, with the model missing most sepsis cases while generating many alerts.
• Label bias. Obermeyer et al. (Science, 2019) showed that a commercial algorithm used to allocate extra care used healthcare cost as its training label. Because less money had historically been spent on Black patients with the same level of illness, the model systematically underestimated their needs. The algorithm was accurate at predicting its label; the label was the wrong proxy for need.

The lesson from both: a model can be technically sound and still be harmful when the data, label or deployment setting differs from what was validated.

Language models in clinical documentation

Large language models (LLMs) are moving fastest in administrative and documentation work, where errors are easier to catch than in diagnosis:

• Ambient scribes record (with consent) the clinician-patient conversation, transcribe it, and draft a structured note for the clinician to edit and sign.
• Draft replies to patient portal messages, discharge summaries, referral letters and prior-authorisation paperwork.
• Summarisation of long records and coding support.

The risks are specific: omissions (a symptom mentioned but not recorded), hallucinated content (an examination finding that never happened), wrong attribution (the patient's words recorded as the clinician's assessment), and automation complacency, where busy clinicians sign drafts they skimmed. Evidence so far focuses on clinician time, burnout and note quality; evidence on patient outcomes is thinner. A sound deployment measures error rates on audited samples, not just satisfaction surveys, and keeps the clinician as the accountable author.

Regulation in brief

United States

The FDA regulates software that meets the definition of a medical device, often called Software as a Medical Device (SaMD). Most AI devices reach the market through one of three pathways: 510(k) clearance (showing substantial equivalence to an existing "predicate" device), De Novo classification (for novel low-to-moderate-risk devices with no predicate), or premarket approval (PMA) for the highest-risk devices. Most AI devices have come through 510(k).

The FDA publishes a public list of AI-enabled medical devices authorised for marketing in the US and updates it periodically. The list grew past 1,000 devices by 2025, and radiology accounts for a large majority of entries, followed at a distance by cardiology and neurology. Check the FDA page for the current total; the count changes with every update.

Two regulatory ideas matter for engineers:

• Locked versus adaptive models. Traditionally a device's algorithm is fixed at authorisation, and significant changes need a new submission. That clashes with the ML habit of retraining.
• Predetermined Change Control Plans (PCCPs). The FDA finalised guidance on 4 December 2024 allowing manufacturers to describe in advance which model modifications they plan (for example, retraining on new data within defined bounds), how they will validate them, and how they will assess impact, so authorised changes can be made without a new submission.

Not all clinical software is a device. Some CDS that shows its basis to a clinician who can independently review it falls outside device regulation under criteria in the 21st Century Cures Act and FDA guidance. Where that line falls for LLM-based tools is an active question.

European Union

In the EU, medical device software falls under the Medical Device Regulation (MDR, Regulation (EU) 2017/745), which applied from 26 May 2021, and the In Vitro Diagnostic Regulation (IVDR) for diagnostic tests. Software that informs diagnosis or therapy is typically class IIa or higher under MDR's software rule, which means a notified body (an independent assessment organisation) must review it before it can carry the CE mark. MDR requires clinical evaluation, post-market surveillance and a quality management system.

The EU AI Act adds a second layer: AI that is a safety component of, or is itself, a product requiring third-party assessment under MDR or IVDR is classed as high-risk. As of November 2026, after the "Digital Omnibus" amendments (in force since 27 July 2026), the high-risk obligations for AI embedded in such regulated products apply from 2 August 2028 (see AI Law and Regulation).

Clinical validation: the evidence ladder

A useful way to judge a clinical AI claim is to ask how far up this ladder the evidence climbs:

Most published medical AI sits at level 1 or 2. Reporting guidelines exist to make studies comparable: TRIPOD+AI for prediction-model studies, CONSORT-AI and SPIRIT-AI for trials and trial protocols involving AI, and DECIDE-AI for early-stage clinical evaluation.

Metrics that matter, and why prevalence changes everything

Let sensitivity $= \frac{TP}{TP+FN}$ (the share of sick patients flagged), specificity $= \frac{TN}{TN+FP}$ (the share of healthy patients cleared), and positive predictive value $\text{PPV} = \frac{TP}{TP+FP}$ (the share of flags that are real). By Bayes' rule, with prevalence $\pi$, sensitivity $s_e$ and specificity $s_p$:

Sensitivity and specificity are properties of the test (if the population's case mix is similar); PPV depends heavily on prevalence.

Worked example. An AI screening tool has sensitivity 0.90 and specificity 0.95.

Screening population, prevalence 0.6%. Take 100,000 people. Diseased: $0.006 \times 100{,}000 = 600$. True positives: $0.90 \times 600 = 540$; false negatives: 60. Healthy: 99,400. False positives: $0.05 \times 99{,}400 = 4{,}970$; true negatives: 94,430.

Fewer than 1 in 10 flags is a real case. Negative predictive value is $94{,}430 / (94{,}430 + 60) \approx 0.9994$.

Specialist clinic, prevalence 20%. Diseased: 20,000; true positives 18,000. Healthy: 80,000; false positives 4,000. PPV $= 18{,}000 / 22{,}000 \approx 0.82$.

The same model goes from "mostly false alarms" to "mostly right" purely because of where it is used. That is why a vendor's accuracy figure means little without the setting in which it was measured, and why alert fatigue is a predictable consequence of deploying sensitive models in low-prevalence populations.

Calibration matters too: if a model says 30% risk, about 30% of such patients should have the outcome. A common summary is the Brier score, $\frac{1}{N}\sum_{i=1}^{N}(p_i - y_i)^2$, where $p_i$ is the predicted probability and $y_i \in \{0,1\}$ the outcome. Discrimination (AUROC) can survive a move to a new hospital while calibration and alert volume change sharply.

Code: internal versus external validation with subgroups

The sketch below trains a model at one synthetic "hospital", then evaluates it on a held-out split from the same site and on a second site where one feature is shifted (as a new lab assay or scanner might do). It reports AUROC with a bootstrap 95% confidence interval, the alert rate at a fixed threshold, and sensitivity in a subgroup where the signal is weaker.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score

rng = np.random.default_rng(0)

def make_site(n, shift, prev):
    y = rng.random(n) < prev
    X = rng.normal(size=(n, 5))
    X[:, 0] += 1.2 * y
    X[:, 1] += 0.8 * y + shift             # shift mimics a new scanner or assay
    group = rng.random(n) < 0.3            # a minority subgroup
    X[group, 0] -= 0.6 * y[group]          # weaker signal in that subgroup
    return X, y.astype(int), group

Xa, ya, ga = make_site(5000, shift=0.0, prev=0.10)   # development site
Xb, yb, gb = make_site(5000, shift=1.0, prev=0.10)   # external site
tr = np.arange(5000) < 3500                          # 70/30 split at site A
model = LogisticRegression(max_iter=1000).fit(Xa[tr], ya[tr])

def boot_auc(y, p, n_boot=1000):
    aucs = []
    for _ in range(n_boot):
        s = rng.choice(len(y), len(y), replace=True)
        if y[s].min() != y[s].max():
            aucs.append(roc_auc_score(y[s], p[s]))
    return np.percentile(aucs, [2.5, 97.5])

thr = np.quantile(model.predict_proba(Xa[tr])[:, 1], 0.85)  # flag top 15%
for name, X, y, g in [("internal", Xa[~tr], ya[~tr], ga[~tr]),
                      ("external", Xb, yb, gb)]:
    p = model.predict_proba(X)[:, 1]
    lo, hi = boot_auc(y, p)
    flag = p >= thr
    print(f"{name}: AUROC={roc_auc_score(y, p):.3f} ({lo:.3f}-{hi:.3f}) "
          f"alert rate={flag.mean():.2f}")
    for label, m in [("majority", ~g), ("subgroup", g)]:
        print(f"   {label} sensitivity={flag[m & (y == 1)].mean():.2f}")

On a typical run, AUROC is about 0.80 at both sites, yet the alert rate at the external site roughly doubles (from about 0.16 to about 0.29) because the shifted feature pushes everyone's scores up, and subgroup sensitivity trails the majority at both sites. A single headline AUROC would have hidden both problems: the doubled workload for clinicians and the gap for the subgroup.

Bias and equity

Sources of bias in clinical AI include unrepresentative training data (for example, dermatology images dominated by lighter skin), label bias (the Obermeyer cost-as-need problem), measurement bias (devices such as pulse oximeters that read less accurately on darker skin feed biased inputs), and access bias (patients who rarely see doctors have sparse records, so models know less about them). Mitigation starts with measuring performance by subgroup with confidence intervals, choosing labels that reflect the clinical goal, and monitoring after deployment. See Bias and Fairness.

AI in drug discovery and protein structure

AlphaFold

Predicting a protein's 3D shape from its amino-acid sequence was a 50-year grand challenge. At the CASP14 assessment in 2020, DeepMind's AlphaFold 2 produced predictions with accuracy competitive with experimental structures for a large share of targets, and the method was published in Nature in 2021 (Jumper et al.). DeepMind and EMBL's European Bioinformatics Institute then released the AlphaFold Protein Structure Database, which expanded in 2022 to over 200 million predicted structures, covering nearly every catalogued protein. AlphaFold 3 (Abramson et al., Nature, 2024) extended prediction to complexes of proteins with DNA, RNA, small-molecule ligands and ions. The 2024 Nobel Prize in Chemistry was awarded half to David Baker for computational protein design and half jointly to Demis Hassabis and John Jumper for protein structure prediction.

Know the limits. AlphaFold predicts a static structure, not the full range of motion a protein goes through; its own per-residue confidence score (pLDDT) is low for disordered regions; and a predicted structure is a hypothesis that still needs experimental confirmation for high-stakes uses such as drug design.

The drug pipeline

AI is used along the pipeline: target identification from genomic and literature data, virtual screening of huge compound libraries, generative design of molecules with desired properties, prediction of absorption, toxicity and other properties, and trial design and patient matching. Several AI-derived candidates have entered human trials. The honest evaluation standard is the same as for any drug: phase transition rates and approvals, not press releases about how fast a candidate was nominated. Discovery speed matters less if most failures happen later, in clinical trials, for reasons such as efficacy and safety that early models do not capture well.

AI for science more broadly

• Weather forecasting. Learned models trained on decades of reanalysis data now rival physics-based forecasts for many variables. DeepMind's GraphCast (Lam et al., Science, 2023) produced 10-day global forecasts in under a minute on a single accelerator and outperformed the European Centre for Medium-Range Weather Forecasts' (ECMWF) high-resolution deterministic forecast on about 90% of the targets it evaluated. GenCast (Price et al., Nature, 2024) extended this to probabilistic ensemble forecasts. ECMWF itself made its learned Artificial Intelligence Forecasting System (AIFS) operational on 25 February 2025, running alongside its physics-based model. These models still rely on physics-based data assimilation to produce their starting conditions.
• Materials discovery. DeepMind's GNoME (Merchant et al., Nature, 2023) used graph neural networks to predict about 2.2 million new crystal structures, of which about 380,000 were predicted to be stable. A companion paper described an autonomous lab synthesising new compounds; independent researchers later disputed whether some of those materials were genuinely new and correctly characterised. The episode is a useful case study: a computational "discovery" is a candidate until it is synthesised, characterised and independently confirmed.
• Other areas include fusion plasma control, mathematics (searching for counterexamples and constructions), and automated analysis of telescope and particle-detector data.

Common pitfalls

• Data leakage across patients. Splitting by image instead of by patient lets the same patient appear in training and test sets, inflating results. Split by patient, and for time-dependent problems, by time.
• Shortcut learning. Models learn hospital-specific artefacts (scanner text markers, portable versus fixed X-ray machines, the presence of a chest drain that signals treatment has already happened) instead of disease.
• Reporting accuracy on imbalanced data. With 1% prevalence, "always negative" is 99% accurate. Report sensitivity, specificity, PPV at a realistic prevalence, calibration and confidence intervals.
• Ignoring the workflow. A model that adds alerts without changing who acts on them, and how, rarely improves outcomes.
• No post-deployment monitoring. Case mix, coding practices and equipment change; performance silently decays.
• Treating a predicted structure or material as a result. Computational outputs are hypotheses until verified experimentally.

In practice

• Define the intended use precisely: which patients, which setting, which decision, which user. Regulation and validation both hang on this.
• Climb the evidence ladder: external validation across sites and subgroups before any deployment; a silent-mode prospective run (the model predicts but no one sees the output) before go-live.
• Measure in the units clinicians feel: alerts per shift, number needed to evaluate ($1/\text{PPV}$), minutes saved, errors found in audited notes.
• Plan change control: version models, document retraining triggers and validation steps, and align with PCCP-style processes where they apply.
• Keep accountability with clinicians for LLM-drafted documentation, and audit samples regularly for omissions and fabrications.
• For science applications, build an experimental confirmation step into every pipeline, and report the hit rate of computational predictions, not just their number.