Scientific AI Claim Audit Protocol

Scientific AI Claim Audit Protocol

A scientific AI claim should be rewritten before it is judged. Record the exact prediction target, intended use, input, output, data boundary, training cutoff, test set, comparator, metric, uncertainty, selection rule, and downstream decision.

Next, distinguish internal performance from external validity. A held-out benchmark can show performance on that benchmark. It does not automatically show performance on a different population, instrument, laboratory, disease, workflow, or regulatory use.

Check whether the comparator had the same information and operating constraints. AlphaFold 3's paper carefully distinguishes blind protein-ligand prediction from methods given privileged pocket information. A headline that removes that boundary changes the claim.

Inspect how outputs were selected. The paper reports top confidence-ranked samples from repeated seeds and diffusion samples under stated conditions. That is relevant to reproducibility, compute, and what a user must do to obtain comparable results.

Then look for uncertainty, failure modes, and evidence outside the originating team. Record what remains untested. If the intended use affects patients, laboratory decisions, regulated development, or significant capital, require qualified domain review and applicable validation.

The final statement should be narrow enough to survive source inspection. It should say what the evidence supports, the conditions under which it supports it, and what decision it cannot yet justify.