How Foundation Model Pretraining Data Is Prepared

How Foundation Model Pretraining Data Is Prepared

Foundation-model pretraining data is prepared through a versioned pipeline that collects source material, extracts usable content, filters and deduplicates it, designs a mixture, tokenizes it, trains models, and evaluates both the data and resulting behavior.

A description of that pipeline does not reveal every document in the corpus.

flowchart LR
    A["Collect authorized source material"] --> B["Extract and normalize"]
    B --> C["Filter quality, language, and risk"]
    C --> D["Deduplicate"]
    D --> E["Design mixture and curriculum"]
    E --> F["Tokenize"]
    F --> G["Train and checkpoint"]
    G --> H["Evaluate and revise"]
    H --> C

Collection defines the possible corpus

Collection identifies source classes and acquires material through a crawler, licensed dataset, public repository, partner, internal source, synthetic process, or another authorized path.

Record the source category, acquisition method, time range, rights or authority, geography, language, access level, exclusions, and provenance.

A phrase such as publicly available online data does not identify each source. Public access and permission to train are also different questions.

Meta's Llama 3 paper and Llama 3.1 model card report a new mix of publicly available online data and more than 15 trillion pretraining tokens. They do not publish a document-level inventory.

Meta's Llama 3.1 launch record provides the separate first-party release summary. It should not be used to fill technical details that belong to the paper or model card.

Extraction turns sources into records

Webpages, PDFs, repositories, books, forums, code, and other sources have different structures.

An extractor decides which text, code, metadata, links, headings, or tables become records. It also handles encoding, scripts, menus, repeated templates, broken files, document boundaries, and language.

Extraction errors can remove valuable material or retain noise. Preserve parser versions, failure rates, samples, and source references where possible.

The data pipeline should remain traceable enough to investigate later issues.

Filtering makes value and risk judgments

Filters can target language, quality, spam, duplication, adult content, malware, personally identifying information, unsafe material, formatting, code quality, or other properties.

Some rules are deterministic. Others use classifiers or models.

Every filter has false positives and false negatives. Removing low-scoring text can erase dialects, minority languages, informal expertise, or unusual but valuable material.

Record the filter objective, version, threshold, test set, error analysis, removed proportion, and human review.

A paper saying a filter was used does not prove that every unwanted item was removed.

Deduplication changes representation and leakage

Deduplication can operate on URLs, exact documents, normalized documents, spans, lines, hashes, approximate signatures, or semantic similarity.

The choice affects repeated boilerplate, copied articles, code forks, quotations, templates, and legitimate recurring language.

Deduplication can reduce memorization and evaluation contamination while also removing useful variants.

Record normalization, unit of comparison, algorithm, threshold, bucket or index strategy, tie-breaking, and what was retained.

E029 described Meta's reported URL, document, and line-level work. The public explanation should remain tied to the paper rather than generalized as one standard pipeline.

Mixture design decides what the model sees often

After filtering and deduplication, teams select categories and sampling weights.

Web text, code, math, multilingual material, books, long documents, and synthetic examples can receive different emphasis.

The number of stored tokens is not necessarily the number of tokens presented during training. Sampling, repetition, curriculum, and annealing affect exposure.

Record category definitions, weights, schedule, changes, and evaluation used to select them.

Mixture proportions do not identify the underlying documents.

Tokenization changes the training units

A tokenizer converts text or code into token identifiers.

Vocabulary, normalization, byte handling, reserved tokens, and segmentation affect sequence length, multilingual efficiency, code, rare terms, and prompt format.

Record the exact tokenizer and version with the model artifact. Token counts cannot be compared cleanly across different tokenizers without qualification.

The Llama 3.1 repository identifies a TikToken-based tokenizer lineage. That does not make a token equivalent to a word.

Training is an iterative systems process

Pretraining presents token sequences to a model and updates parameters to reduce a defined objective.

The operational record includes model architecture, parameter count, optimizer, schedule, precision, batch, sequence length, parallelism, cluster, hardware, checkpoints, validation, failures, and restarts.

Infrastructure failures can corrupt or delay a run. Checkpoints and monitoring create recoverability.

Do not infer exact cost, power, emissions, failure rate, or economics unless the source reports them with a method.

Evaluation sends evidence back into the pipeline

Teams evaluate validation loss, downstream capabilities, language, code, math, long context, safety, memorization, contamination, bias, and other intended properties.

Results can lead to filter, mixture, tokenizer, training, or post-training changes.

Held-out evaluation also needs lineage. If test data appears in training, a score can overstate generalization.

No benchmark establishes every use case.

Keep disclosure separate from reconstruction

A strong public explanation says what the publisher reported, what artifact it applies to, and what remains missing.

Use [[How to Read Training Data Disclosures Without Inventing the Dataset]] to classify the evidence. Use [[How to Build an AI Training Claim Ledger]] before summarizing a proprietary pipeline.

This technical explainer was developed with AI assistance from E029, Meta's paper and model card, and the linked pipeline framework. Dalton Anderson remains the author. Technical, data-rights, privacy, safety, current-source, and founder review are mandatory before publication. Publication is not authorized.