How to Design Layered Controls for an Open LLM

How to Design Layered Controls for an Open Model System

Design layered controls for an open-model system by mapping each material threat to prevention, detection, recovery, evidence, and accountable owners across the whole application. A guard model can contribute to that architecture, but it cannot own identity, data access, tools, monitoring, or incident response.

Start with one use case, not a generic safe-AI checklist.

flowchart LR
    A["Users and identities"] --> B["Data and retrieval"]
    B --> C["Input and policy controls"]
    C --> D["Model and runtime"]
    D --> E["Tools and permissions"]
    E --> F["Output and human decision"]
    F --> G["Monitoring and incidents"]
    G --> H["Recovery and change control"]
    H --> A

Define the use case and consequence

Write down who the system serves, what job it performs, which data it receives, which sources it retrieves, which tools it can use, what output it produces, and what happens next.

The same model can support a low-consequence drafting tool or a system that changes customer records. Those systems do not share the same risk even if the weights are identical.

Name the decisions a person or downstream service may make from the output. Include foreseeable misuse and ordinary error.

NIST's AI Risk Management Framework organizes work around governing, mapping, measuring, and managing risk. The Generative AI Profile adapts that frame to generative systems. Neither document certifies a design. They provide a structure for decisions and evidence.

Draw the trust boundaries

Map users, administrators, model files, prompts, retrieval stores, uploaded files, connectors, tools, external APIs, databases, logs, reviewers, and output consumers.

Mark where data crosses between owners or privilege levels.

Untrusted text can arrive through a chat message, retrieved webpage, email, document, database field, tool result, or image. A model may treat all of it as context even when the application should treat part of it as data rather than instruction.

Identify which component makes each authorization decision. The model should not be the sole authority for access to money, customer data, production changes, private files, or external communication.

Build the threat-control matrix

For each material threat, record the actor, precondition, path, affected asset, plausible consequence, current control, evidence, gap, owner, and review trigger.

Then connect the threat to prevention, detection, and recovery.

The matrix is not proof. It is a map of claims that need testing.

Protect identity and permissions outside the model

Authenticate the user and service before the prompt reaches the model.

Determine access from an authoritative identity and policy system. Pass only the minimum context the model needs. Recheck authorization when a tool runs, not only when the conversation begins.

Use scoped, short-lived credentials where possible. Separate read from write. Separate drafting an action from executing it. Require explicit approval for consequential operations.

Do not assume that a system prompt such as "only use authorized tools" is an access-control mechanism.

Minimize and separate data

Classify data before connecting it to retrieval or context.

Define which users may retrieve which records and apply filtering before the results enter the model. Preserve provenance so an output can be traced to a source and version.

Treat retrieved content as untrusted for instruction purposes. Keep system rules, user instructions, data, and tool results in distinguishable channels when the platform supports it.

Avoid sending secrets, credentials, or unnecessary personal data to the model. Redact or tokenize when that meaningfully reduces exposure.

Plan deletion, retention, backup, legal hold, and offboarding. A vector index or model log can preserve information after a source record changes.

Place model safeguards at the right layer

Model alignment, content classifiers, prompt-attack detectors, and code-oriented filters can reduce specific risks.

Meta's Llama 3.1 model card says Llama models should be deployed as part of an overall system with additional safeguards. The Purple Llama repository contains current safeguard and evaluation artifacts.

For any such control, record the exact version, taxonomy, input, output, language, modality, threshold, placement, latency, failure behavior, false positives, false negatives, and bypasses.

Decide what happens when the control is unavailable or uncertain. Failing open and failing closed have different operational costs and risks.

Do not use a generic content policy as a substitute for a use-case policy.

Constrain tools and actions

Treat tool use as privileged execution.

Expose only the functions required for the job. Use typed schemas, value constraints, resource scopes, dry runs, idempotency where appropriate, rate limits, timeouts, and separate approval for high-impact actions.

Validate tool arguments outside the model. Authorize the action against the current user and resource. Verify the result before telling the user it succeeded.

Run code in an isolated environment with explicit filesystem, network, secret, process, and time limits.

OWASP's Top 10 for LLM applications includes risks around prompt injection, sensitive information, excessive agency, and other application failures. Use it as an input to the system threat model, not a replacement for it.

Validate outputs at the point of use

The required validation depends on what consumes the output.

Structured data should pass schema and business-rule checks. Code should pass review, testing, scanning, and sandboxing. Claims should carry source evidence where the use requires it. High-impact decisions need qualified human review and may be inappropriate for automation.

Do not ask one guard classifier to establish truth, authorization, policy compliance, and safe downstream use simultaneously.

Make uncertainty and incomplete evidence visible to the user or downstream process.

Monitor the system and prepare recovery

Log enough to reconstruct material behavior while minimizing sensitive data.

Record model and prompt versions, retrieval sources, policy and guard results, tools requested and executed, approvals, errors, and final outcomes where appropriate.

Define signals for abuse, drift, anomalous access, elevated refusals, guard disagreement, tool failure, quality regressions, and user-reported harm.

Every material control needs an owner and an incident path. The team should be able to revoke access, disable a tool, roll back a model or prompt, remove a retrieval source, correct a record, notify affected owners, and restore service.

Recovery should be rehearsed before the system is consequential.

Test control interaction

Controls can fail together.

A retrieval filter may pass untrusted instructions. A prompt detector may miss them. The model may request a tool. The authorization layer may accept an overbroad resource. The output filter may see only the final summary.

Use authorized red-team and integration tests that exercise the chain. Include control unavailability, malformed responses, stale policies, conflicting identities, partial tool failure, and rollback.

[[How AI Red Teaming Works]] provides the discovery-to-regression loop. [[Llama 3.1 Safety Tooling Record]] provides a versioned example of why named safeguards must remain distinct.

Assign residual risk

No layered design removes all risk.

Record what remains, why it remains, how it is monitored, who can accept it, when the decision expires, and what change reopens review.

Reevaluate when the model, quantized artifact, prompt, data, retrieval, tools, identities, user population, policy, threat, or consequence changes.

This risk guide was developed with AI assistance from E031, NIST, OWASP, Meta's official model and safeguard records, and the linked threat-control framework. Dalton Anderson remains the author. Security, privacy, legal, safety, technical, operations, accessibility, current-source, and founder review are mandatory before publication or implementation. Publication is not authorized.