Model Weights, APIs, and Wrappers Are Different

Model Weights, APIs, and Wrappers Are Not the Same System

Model weights, a hosted API, and a product wrapper are different evidence objects. They may be connected, but a result from one cannot automatically prove the behavior or identity of another.

The distinction matters whenever a public checkpoint performs differently from a private endpoint, a benchmark uses a product interface, or a provider keeps the request path private. Without a system map, people often compare labels instead of systems.

flowchart LR
    A["User request"] --> B["Product wrapper"]
    B --> C["Router and policy layer"]
    C --> D["Provider API"]
    D --> E["Inference runtime"]
    E --> F["Model weights"]
    F --> E
    E --> D
    D --> C
    C --> B
    B --> G["Displayed response"]

Weights are a parameter artifact

Model weights are numerical parameters associated with an architecture. A downloadable release normally also depends on configuration files, a tokenizer, special tokens, chat templates, adapters, and sometimes custom code.

Weights are not an application. They do not choose their own precision, hardware, inference engine, prompt, tools, safety policy, or user interface.

To identify public weights, preserve the repository, revision, file names, hashes, architecture configuration, tokenizer, license, and required code. A model name without a revision is a weak identifier because files and documentation can change behind the same page.

The current Reflection 70B repository is one public artifact. Its revision history shows why the repository and commit must be part of the identity record.

The runtime turns weights into outputs

An inference runtime loads the artifact and executes it. Precision, quantization, kernels, framework versions, hardware, sampling settings, batching, and memory constraints can affect output and performance.

Two evaluators can download the same files and still run materially different systems. One may use full precision while another uses a compressed quantization. One may render the publisher's chat template while another sends plain text. One may apply deterministic decoding while another samples.

That does not make comparison impossible. It means the runtime belongs in the method.

An API is a service boundary

An API accepts a request under a provider's contract. The visible model identifier may name a fixed snapshot, a moving alias, a family, or a product tier. The provider may apply routing, safety filters, caching, batching, or silent upgrades.

An evaluator can preserve request and response metadata, provider documentation, time, region, parameters, and observed behavior. Unless the provider exposes verifiable artifact identifiers, the evaluator may not be able to prove which exact weights served the request.

This is not unique to AI. Hosted services routinely hide implementation details. The mistake is treating an opaque service label as if it were a cryptographic identity.

A router can select a different path

Routing layers may choose a model by task, capacity, region, cost, latency, or policy. They can fail over when a preferred backend is unavailable. A cache can return an earlier response. A moderation layer can replace or suppress output.

If a benchmark traverses a router, record the visible route metadata and provider statements. Do not claim a hidden route unless there is direct evidence.

Observed similarity is not enough. Two systems can produce similar answers because they share training lineage, prompts, retrieval, or common benchmark knowledge. Observed difference is also not enough. Sampling and wrappers can make the same underlying model respond differently.

A wrapper is part of the tested product

A product wrapper can add a system prompt, memory, retrieval, web search, calculators, tools, multiple model calls, self-critique, retries, response ranking, and formatting.

Those additions may make the product more useful than bare weights. They also make the product a different object of evaluation.

If the claim concerns the product, evaluate the whole product and document the wrapper as far as possible. If the claim concerns a public checkpoint, a strong wrapper result cannot substitute for testing that checkpoint.

The distinction can be written simply: the product produced the observed response through its available interface; the underlying model identity was either verified or not independently verifiable.

Reflection 70B exposed the identity problem

The E038 record contains an attributed evaluator update preserved in Hugging Face discussion 58. It described weak results from an initial public release, stronger behavior from a private API, and an inability to independently verify the exact system behind that endpoint.

That record supports a system-identity problem. It does not, by itself, prove why the systems differed.

The careful next step is to preserve the public revision, request path, endpoint metadata, prompts, raw outputs, and evaluation method. Then compare the evidence layer by layer.

Draw the request path before comparing results

Start at the user's request and trace every known transformation to the displayed response.

Record the product version, wrapper instructions, tools, router, provider, visible model ID, timestamp, runtime details if known, artifact revision if known, and postprocessing. Mark each field as verified, publisher-reported, observed, inferred, or unknown.

Stanford HELM provides a structured way to think about scenarios, adaptations, metrics, and results. The same discipline should extend to system identity. A score is only interpretable when the tested object is defined.

Match the public sentence to the evidence

The wording should stop where verification stops.

This language prevents both false equivalence and premature accusation. It also makes the next evidence request obvious.

Use [[How to Reproduce a Language Model Benchmark]] when the artifact and method are testable. Use [[How to Evaluate an AI Model Release Claim]] when system identity is one part of a broader release claim. The historical sequence is preserved in [[Reflection 70B Source Led Evidence Timeline]].

This explainer was developed with AI assistance from the immutable E038 transcript, versioned Reflection 70B records, attributed evaluation discussion, Stanford HELM, and the linked identity framework. Dalton Anderson remains the author. Technical, current-source, and founder review are mandatory before publication. Publication is not authorized.