How to Review an AI-Generated Pull Request

How to Review an AI-Generated Pull Request

Review an AI-generated pull request from the requirement and threat model inward. Restate the expected behavior before reading the agent's summary or diff, then verify scope, reproduce the change, challenge the tests, inspect data and security effects, review dependencies and operations, and preserve rollback.

A plausible diff and passing self-written tests can still implement the wrong thing.

Why the order matters

An agent usually returns a confident summary, a coherent implementation, and tests that fit its design. Reading those first can anchor the reviewer to the agent's interpretation.

Begin with the issue, user need, design decision, incident, or acceptance criteria that existed before the code. If that record is vague, the pull request is not ready for review. Clarify the requirement or split the task.

flowchart LR
    A["Independent requirement"] --> B["Threat and failure model"]
    B --> C["Scope and diff"]
    C --> D["Reproduce behavior"]
    D --> E["Challenge tests"]
    E --> F["Security and dependencies"]
    F --> G["Operations and rollback"]
    G --> H{"Merge, revise, split, or reject"}

The sequence prevents implementation detail from replacing product intent.

1. Restate the acceptance criteria

Write the expected inputs, outputs, invariants, errors, and non-goals in your own words.

For retry logic, define which failures are retryable, maximum attempts, timing, cancellation, idempotency, and what the caller sees when attempts end. "Add retries" is not reviewable.

Identify the evidence that would prove each criterion. Some behavior needs a unit test. Some needs an integration test, manual observation, migration rehearsal, security analysis, or domain approval.

Do not copy the agent's test names as the criteria.

2. Identify the threat and failure model

Ask what the change reads, writes, sends, deletes, exposes, retries, caches, or trusts.

Trace untrusted input, authentication, authorization, secret use, personal data, file paths, network calls, dependencies, logging, errors, and resource limits. Consider normal failure and adversarial use.

NIST's Secure Software Development Framework connects security requirements, protected development environments, code review, executable testing, provenance, and vulnerability response. The depth of review should follow business and security risk.

A documentation typo and an authentication change do not deserve the same gate.

3. Bound the change

Compare the changed files and behavior with the stated task.

Unrelated formatting, broad refactors, new abstractions, dependency upgrades, generated files, lockfile churn, configuration changes, and renamed interfaces increase review cost and risk.

Ask the agent to split unrelated work before review. Do not reward a large diff by skimming faster.

Google's current Jules review guide exposes the full diff, files changed, line counts, branch, and commit information. Use that summary to find scope, then inspect the repository state and actual diff.

4. Reproduce the important behavior

Check out the exact commit or pull-request head in a clean environment. Follow the repository's documented setup.

Run the build, relevant tests, linters, type checks, and static analysis. Exercise the main success path, important failure, boundary input, and one misuse case the agent did not mention.

Confirm that the command exits and reports failure correctly. A test process that never ran, silently skipped, or swallowed an error can still leave a green interface.

In E107, the local demonstration exposed a timeout that the agent's earlier build and test sequence did not settle. The production question was not merely whether increasing the number made the demo work.

5. Challenge the tests independently

Read the tests as claims about behavior.

Look for assertions that only repeat the implementation, mocks that bypass the risky boundary, snapshots that accept a large unexplained output, missing negative cases, broad exception handling, time-dependent flakiness, and tests that pass when the feature is removed.

Add or run at least one independent test for the most important criterion. When possible, use an existing regression suite or a reviewer-written test whose expected result was defined before the diff.

Passing checks are necessary evidence, not a verdict. GitHub's status-check documentation notes that a skipped job can report success in some workflow structures and that checks can be set by external systems or authorized users.

Confirm which workflow ran, against which commit, and what it actually tested.

6. Trace data, permissions, and secrets

Inspect every new source, sink, credential, environment variable, log, cache, queue, database field, and external call.

Verify least privilege. Confirm that sensitive values are not committed, printed, placed in error messages, added to fixtures, exposed to forked pull requests, or stored in generated artifacts.

For coding agents, inspect the execution environment as well as the code. Jules runs tasks in a cloud VM with internet access and warns users about repositories, dependencies, scripts, and secrets in its current FAQ.

An unexplained credential is a reason to stop and investigate, even when the feature works.

7. Review dependency and supply-chain effects

Examine lockfiles, package manifests, container bases, build actions, downloaded binaries, generated code, and installation scripts.

Ask whether the dependency is necessary, maintained, correctly pinned, compatible with the license and platform, and safe for the data it will receive.

GitHub's dependency review compares dependency changes between the base and pull-request head and can be required before merge. Use it as one input, not a substitute for architectural judgment.

Review action versions and build scripts because a small source diff can change what executes in CI.

8. Inspect maintainability

Check names, module boundaries, error handling, comments, public interfaces, duplication, dead code, configuration, and consistency with the repository.

The question is not whether the code looks clean in isolation. It is whether the team can understand, operate, change, and remove it later.

Ask the authoring agent to explain a non-obvious decision, then verify the explanation against the code. A fluent answer is not evidence unless the implementation supports it.

9. Verify operational behavior

Review timeouts, retries, cancellation, concurrency, resource use, logs, metrics, alerts, feature flags, migrations, deployment order, backward compatibility, and failure recovery.

Confirm that logs help diagnose the failure without leaking data. Confirm that a retry cannot duplicate a payment, message, claim, or record. Confirm that a partial deployment remains safe.

If the change alters a schema or API, test old and new components in the orders that can occur during rollout.

10. Preserve rollback and accountability

Decide how to disable or reverse the change before merge.

A feature flag, backward-compatible migration, revertable commit, documented repair, or safe deployment sequence may provide the recovery path. Some data changes are not truly reversible and need backups or compensating operations.

Record who reviewed the requirement, domain behavior, security surface, and code. AI assistance can be disclosed without pretending that the tool owns the result.

GitHub's protected-branch controls can require reviews, code owners, status checks, conversation resolution, and current approval after new commits.

Make one of four decisions

Merge when the requirement is clear, the change is bounded, the evidence is independent enough, the risk is understood, and recovery is credible.

Request revision when the task is right but the implementation needs focused correction. Split when the diff combines separable risks or exceeds comprehension. Reject when the change solves the wrong problem, creates unjustified risk, or cannot be understood well enough to own.

The reviewer is not grading the agent's effort. The reviewer is deciding whether the repository should accept the change.

Read the E107 episode story, What Happened When Jules Built a Go Backend, followed by [[What Is Google Jules|What Is Google Jules?]] and [[How to Run Parallel Coding Agents Without Losing Control]]. The publishing agent should activate the episode-story link only after its stable public route exists.

AI assisted with research organization and drafting. Dalton Anderson remains responsible for the analysis and publication decision.