How to Build an AI Release Claim Ledger

How to Build an AI Release Claim Ledger

An AI release claim ledger is a maintained table that connects each public statement to the exact system, source, evaluation, conflicting evidence, confidence level, review owner, and refresh trigger behind it.

It turns launch coverage from a collection of links into an inspectable evidence record. When a benchmark changes, a model card is revised, or an endpoint behaves differently, the ledger shows which sentence needs to change.

flowchart LR
    A["Public claim"] --> B["System identity"]
    B --> C["Versioned support"]
    C --> D["Independent evaluation"]
    D --> E["Conflicts and gaps"]
    E --> F["Confidence language"]
    F --> G["Review and refresh"]
    G --> A

Make one row represent one claim

Do not put an entire launch into one row. Split the coverage into atomic claims that can be checked independently.

"Model X is the best open model" might become separate rows for benchmark performance, comparison set, artifact availability, license, reproducibility, and the meaning of open. A later correction to one row should not silently rewrite the others.

Preserve the exact public wording alongside a normalized testable statement. The exact wording protects historical accuracy. The normalized statement reveals what evidence would actually support it.

The ledger should link to public copy, but it should not become the public article. It is the control record that helps the article stay accurate.

Give the system a full identity

A claim about an AI system needs an identity stack.

Record the public artifact revision and hashes when weights are available. For an API, record provider, visible model identifier, timestamp, region, parameters, and response metadata. For a product, record wrapper version, system instructions when available, retrieval, tools, routing, retries, and postprocessing.

Mark every field by evidence class.

This prevents an endpoint name from quietly becoming a verified weight revision. [[Model Weights APIs and Wrappers Are Not the Same System]] provides the full identity method.

Link versioned support

Every material claim should point to the strongest available source and preserve the version reviewed.

A repository commit can establish what a public artifact contained on a date. A publisher report can establish what the publisher claimed and how it described its method. Evaluation code and raw output can make a result reconstructable. An independent test can add a separate measurement under its stated conditions.

Use the Model Cards for Model Reporting paper as a documentation baseline, then inspect the actual card, files, history, and linked records. [[How to Read a Model Card as an Evidence Record]] explains why the exact revision matters.

For web sources, preserve title, publisher, URL, publication date, access date, and a capture or archive route when allowed. If a page is access-controlled, record that limitation rather than pretending the source was independently preserved.

Store evaluation conditions beside the score

A benchmark score without its method should not receive a high-confidence row.

The ledger should capture dataset revision and split, prompt template, few-shot examples, inference settings, runtime, parser, scorer, exclusions, sample count, repeated runs, uncertainty, and raw-output location.

Stanford HELM shows the value of connecting scenarios, adaptations, metrics, and results. The ledger can use the same principle even when the evaluation comes from another framework.

Link the run manifest and item-level record rather than pasting every output into the ledger. The row should tell an editor whether the method is complete and where the evidence lives.

Preserve conflicts instead of averaging them away

When sources disagree, create a conflict record. Identify the systems and protocols being compared before declaring a contradiction.

Do not delete the earlier state after resolution. Append the correction, decision, reviewer, and date. An evidence system should show how confidence changed.

Translate the record into public language

Use a controlled language field so the article cannot outrun the ledger.

This is where editorial discipline becomes visible. The ledger does not ban strong conclusions. It requires the conclusion to earn its strength.

Assign owners and refresh triggers

Every row needs a review owner, next review date, and event-based refresh trigger.

Useful triggers include a repository revision, model-card edit, new technical report, evaluation release, endpoint version change, provider statement, correction, license change, or material challenge from a named source.

The NIST AI Risk Management Framework is useful here because it treats measurement and management as ongoing work. Release-day verification is not enough for an evergreen page.

Record the editorial reviewer, technical reviewer, legal reviewer when required, final approver, last verification time, and publication authority. A passed technical check does not authorize publication.

Use a compact reusable ledger

The working table can stay short if it links to deeper evidence.

The real ledger should replace the example with one row per atomic claim and stable links to source captures, run manifests, raw results, review decisions, and public destinations.

Reflection 70B shows why this matters. The historical record contains an early model-card claim, reports of differing public and private results, and a later card revision. A ledger would have separated those events, held the system identity as unresolved, and prevented technical uncertainty from becoming an unsupported conclusion about motive.

Use [[How to Evaluate an AI Model Release Claim]] to decide what the row means. Use [[How to Reproduce a Language Model Benchmark]] when an evaluation can be rerun. Keep the correction and chronology in [[What Venture Step Got Wrong About Reflection 70B]] and [[Reflection 70B Source Led Evidence Timeline]].

This method was developed with AI assistance from the immutable E038 transcript, the Model Cards paper, Stanford HELM, NIST material, versioned Reflection 70B records, and the linked claim-ledger framework. Dalton Anderson remains the author. Editorial, technical, current-source, and founder review are mandatory before publication. Publication is not authorized.