How to Read an LLM Training Data Disclosure

How to Read Training Data Disclosures Without Inventing the Dataset

Read a training-data disclosure by classifying exactly what it provides, preserving the model and version, and marking every missing field as undisclosed rather than filling it with a plausible estimate.

A pipeline description is not a corpus inventory.

flowchart LR
    A["Publisher statement"] --> B["Classify disclosure type"]
    B --> C["Bind model and version"]
    C --> D["Record exact scope"]
    D --> E["List supported inferences"]
    E --> F["List missing facts"]
    F --> G["Write bounded claim"]

Identify the artifact

Start with the model family, exact version, base or tuned form, release date, and source document.

A publisher may update a model card while older weights remain unchanged. A later family may use a different corpus.

Do not combine statements across versions unless the source explicitly connects them.

Classify the disclosure

Each row supports a different sentence.

Preserve the exact wording

Copy a short source excerpt into an internal ledger or make a faithful paraphrase with a citation.

Record the page, section, table, model, and date.

Then write what the statement does not say.

Meta's Llama 3.1 model card says the family used a new mix of publicly available online data, reports more than 15 trillion pretraining tokens, and gives a December 2023 knowledge cutoff.

Those fields do not identify every domain, document, license, author, language share, country, removal, or retained copy.

Do not reconstruct proprietary detail

A paper may describe web extraction, classifiers, deduplication, mixture experiments, code and math data, or multilingual preparation.

That does not allow a reader to reverse-engineer the exact corpus.

A filter description does not reveal every item that passed. A removal rule does not prove perfect removal. A category proportion does not identify the category's members.

Use "Meta reported" for publisher claims and "not disclosed" for absent detail.

Treat token counts carefully

Tokens depend on the tokenizer and pipeline stage.

A raw collection can shrink through parsing, filtering, and deduplication. A stored corpus can be sampled or repeated during training. A final reported training-token count may describe exposure rather than unique text.

Do not translate tokens directly into words or documents without a stated approximation and tokenizer.

The Llama 3 paper provides the project-specific method. It should control any more detailed Llama claim.

Meta's Llama 3.1 launch post adds first-party release framing. It does not replace the technical paper or identify the corpus.

Separate pretraining and post-training

Pretraining builds broad statistical behavior from large token sequences.

Post-training can add supervised examples, preferences, synthetic data, tool behavior, refusals, and safety tuning.

A statement about human-generated or synthetic fine-tuning data does not describe the pretraining corpus. A pretraining cutoff does not necessarily describe later system knowledge supplied through retrieval or tools.

Keep the stages separate in the ledger.

Ask about rights and governance without inventing answers

Training-data questions include copyright, contract, privacy, consent, sensitive information, child safety, security, jurisdiction, retention, deletion, provenance, and auditability.

A technical paper may not answer them.

Do not infer permission from public accessibility. Do not infer illegality from missing detail.

Record which governance evidence is present, which is outside the document, and which qualified reviewer owns the question.

Evaluate the disclosure itself

Assess specificity, versioning, provenance, reproducibility, exclusions, measurement, independent evidence, and update history.

A useful disclosure can remain incomplete. The editorial goal is not to reward or punish the publisher with a single transparency score. It is to tell the reader what the evidence supports.

Compare disclosures across releases only after normalizing the fields. A longer model card can disclose more prose while leaving the same material questions unanswered. Record additions, removals, changed definitions, and newly scoped claims rather than assigning a vague transparency improvement.

Write a defensible summary

Prefer: "Meta reported that Llama 3.1 used more than 15 trillion pretraining tokens from a new mixture of publicly available online data and described filtering and deduplication methods; it did not publish a document-level corpus inventory."

Avoid: "Meta trained Llama 3.1 on the entire internet" or a guessed list of sites.

Use [[How Foundation Model Pretraining Data Is Prepared]] for the pipeline. Use [[How to Build an AI Training Claim Ledger]] to record source and prohibited inference.

This evidence guide was developed with AI assistance from E029, Meta's paper and model card, and the linked disclosure framework. Dalton Anderson remains the author. Data-rights, privacy, technical, editorial, current-source, and founder review are mandatory before publication. Publication is not authorized.