Prompt, RAG, Fine-Tune, or Distill an LLM?

Prompt, Retrieve, Fine-Tune, or Distill?

Choose the smallest customization method that fixes a measured failure on a frozen evaluation set. Start with prompting, add retrieval for changing external knowledge, consider fine-tuning for persistent task or behavior gaps with suitable labeled data, and consider distillation when a smaller model must reproduce defined behavior under valid rights and safety boundaries.

These methods can be combined, but each adds obligations.

flowchart TD
    A["Measure the failure"] --> B{"What is missing?"}
    B -->|Instruction, structure, examples| C["Prompt"]
    B -->|Changing external knowledge| D["Retrieve"]
    B -->|Stable behavior or task skill| E["Fine-tune"]
    B -->|Smaller model needs defined behavior| F["Distill"]
    C --> G["Run frozen evaluation"]
    D --> G
    E --> G
    F --> G
    G --> H{"Meets quality, risk, cost, and operations gates?"}

Begin with a failure record

Preserve the input, model, prompt, sources, tools, settings, output, expected properties, score, reviewer, and consequence.

Classify the problem. Is the instruction unclear? Is current knowledge absent? Is retrieval weak? Is the model failing a stable transformation? Is latency or cost too high? Is a safeguard blocking valid work?

Do not choose a technique because it is fashionable. Choose it because the evidence identifies the layer that failed.

Use prompting for instructions and structure

Prompting is the lowest-change starting point for task definition, context, examples, tone, constraints, and output structure.

Google Cloud's prompting strategy overview describes prompt engineering as iterative and test driven.

Prompting is attractive when the task can be explained, examples fit in context, requirements change often, and the model already has the necessary capability.

It does not add new factual knowledge to the weights. Long prompts increase tokens, latency, and maintenance. Instructions can conflict or be overridden by untrusted content.

Use retrieval for changing external knowledge

Retrieval augments the prompt with material selected from an external corpus.

Use it when answers depend on current policies, product documentation, customer records, research, or other sources that need provenance, access control, citation, refresh, and deletion.

Google Cloud's RAG documentation shows corpus ingestion, retrieval configuration, and generation with retrieved material. The product example does not establish that a particular corpus is authorized or that retrieval is accurate.

Evaluate retrieval and answer faithfulness separately. Treat retrieved text as untrusted input. Use [[How to Ground an Assistant in a Curated Knowledge Base]] for the full lifecycle.

Consider fine-tuning for stable behavior

Fine-tuning updates model parameters using examples. It can help with stable task behavior, domain syntax, classification, extraction, structured transformations, or response patterns when prompting is insufficient.

Google Cloud's current tuning overview recommends starting with prompting, measuring errors, and using high-quality labeled data that reflects production inputs.

Fine-tuning should not be used as a hidden document store for facts that change. It makes source-level correction and citation harder.

Review data rights, consent, privacy, security, label quality, representativeness, retention, deletion, and model-license terms before training.

Consider distillation for a defined compression goal

Distillation trains a student model to reproduce selected behavior from a teacher or teacher-generated data.

It can support a smaller, faster, or cheaper model for a bounded task. It can also reproduce teacher errors, bias, unsafe behavior, and unsupported claims.

Record the teacher identity, student identity, prompts, generated data, filters, labels, rights, license terms, safety rules, evaluation, and residual gaps.

Do not assume that access to teacher outputs permits every distillation use. The Llama 3.1 Community License is one example of terms that must be read for the exact intended activity.

Compare the operating burden

Prompting requires versioned instructions and regression tests.

Retrieval adds source governance, indexing, access, monitoring, refresh, and deletion.

Fine-tuning adds training data, compute, experiment tracking, artifact lineage, model evaluation, deployment, and rollback.

Distillation adds teacher and student lineage, generated-data governance, compression evaluation, and independent safety testing.

A combined system inherits the burden of every layer.

Run a matched evaluation

Use the same representative and boundary cases for each candidate method. Measure task quality, factual support, calibration, citations, safety, latency, cost, reviewer effort, and operations.

Freeze the model, prompt, corpus, adapter, or student artifact used for each result. Preserve failures, not only averages.

Select the simplest system that passes all required gates. If retrieval plus prompting solves the task, fine-tuning may add risk without enough value. If a stable structured transformation still fails, fine-tuning may be justified.

Reevaluate changes

Repeat the decision when the task, users, model, prompt, corpus, data, license, latency target, cost, threat, or consequence changes.

Use [[How to Build an AI Training Claim Ledger]] to keep public statements about prompting, retrieval, tuning, and distillation tied to evidence.

This guide was developed with AI assistance from E027, Google Cloud and official Llama documentation, and the linked customization framework. Dalton Anderson remains the author. Technical, data-rights, privacy, security, license, safety, domain, accessibility, source, and founder review are mandatory before publication or customization. Publication and model modification are not authorized.