Research Note

Model Customization Decision Framework

Start from a measured failure. Use prompting for instruction, structure, examples, and supplied context. Use retrieval when answers require changing external knowledge wi

Aug 4, 20261 min readBy Dalton Anderson
In this article

Model Customization Decision Framework

Start from a measured failure. Use prompting for instruction, structure, examples, and supplied context. Use retrieval when answers require changing external knowledge with provenance and access control. Consider fine-tuning when a stable task or behavior remains weak after prompt and system design and suitable labeled data exists. Consider distillation when a smaller model must reproduce defined behavior and the teacher, data, license, evaluation, and safety boundaries permit it.

Methods can be combined, but each adds data, evaluation, operations, security, privacy, and license obligations.

Choose the smallest intervention that passes a frozen evaluation set and operating constraints.

Sources

Follow the evidence.

  1. Introducing Llama 3.1ai.meta.com
  2. ai.meta.com: the llama 3 herd of modelsai.meta.com
  3. cloud.google.com: prompt design strategiescloud.google.com
  4. daltonanderson.ghost.io: unlocking llama 3 1 metas open source ai revolutiondaltonanderson.ghost.io
  5. docs.cloud.google.com: tune modelsdocs.cloud.google.com
  6. docs.cloud.google.com: rag quickstartdocs.cloud.google.com
  7. docs.nvidia.com: benchmarkingdocs.nvidia.com
  8. genai.owasp.org: owasp top 10 for llm applications 2025genai.owasp.org
  9. Meta Llama models repositorygithub.com
  10. github.com: LICENSEgithub.com
  11. github.com: MODEL CARDgithub.com
  12. github.com: USE POLICYgithub.com
  13. github.com: PurpleLlamagithub.com
  14. huggingface.co: modelshuggingface.co
  15. huggingface.co: model memory anatomyhuggingface.co
  16. open.spotify.com: 6DI2GolY5QTfflWBunE2xiopen.spotify.com
  17. opensource.org: open source ai definitionopensource.org
  18. owasp.org: www project top 10 for large language model applicationsowasp.org
  19. NIST AI Risk Management Frameworknist.gov
  20. NIST: Artificial Intelligence Risk Management Framework, Generative Artificial Intelligence Profilenist.gov
  21. youtu.be: gg2I5iO1x0oyoutu.be

From this episode

Two useful next steps.

Guide · 1 min

How to Read and Operationalize an AI Model License

Turn AI model license terms into artifact records, use boundaries, attribution, redistribution, acceptable-use controls, owners, evidence, and release gates.

Guide · 1 min

Prompt, RAG, Fine-Tune, or Distill an LLM?

Choose prompting, retrieval, fine-tuning, or distillation by measuring the failure, knowledge needs, labeled data, latency, rights, risk, and operations.

Return to the episode