Vibe Coding Benchmark Method Research Note

Vibe Coding Benchmark Method Research Note

A useful AI app-builder benchmark must answer a decision rather than manufacture a universal leaderboard. The decision might be which tool best supports a team's internal-tool workflow, which builder produces the strongest first prototype, or which platform leaves the cleanest path to independently operated software. Those are different tests.

The minimum protocol freezes the task, supplied context, account tier, model or mode, region, start state, time cap, spend cap, clarification policy, allowed interventions, and stop rule. It records every prompt, response, generated file, error, credit event, wait, manual change, test result, and final artifact. A one-shot run permits one initial builder instruction before the first output, but it must separately disclose all human work embodied in the supplied brief.

Software quality cannot be reduced to visual polish. NIST SP 800-218 supplies a secure-development vocabulary. OWASP ASVS supplies testable web-application security requirements. WCAG 2.2 supplies accessibility success criteria. Core Web Vitals supplies user-centered loading, responsiveness, and visual-stability measures. These sources do not create one combined benchmark score. They help define evidence that a reviewer can inspect.

Proposed result structure

The result should separate initial-build coverage, functional correctness, domain correctness, accessibility, security, performance, maintainability, operability, portability, cost, and evaluator confidence. A weighted total can be useful for one declared decision, but publishing only the total hides why the tools differ.

Functional tests should be derived from the frozen requirements. Each test needs a precondition, action, expected result, actual result, and artifact. A generated screen counts as implemented only when the required behavior can be exercised. Text that claims an API call exists does not prove the call was made.

Domain correctness needs an evaluator who understands the problem. In E098, a carrier logo, ACORD form label, or classification code could look credible while being operationally wrong. General software tests cannot replace that review.

Reruns and uncertainty

Generative systems vary. A single run shows that one outcome occurred, not how often it occurs. A credible comparison should use multiple fresh runs when the decision is important, preserve failures rather than silently retry them, and report how often each test passed.

Products also change without preserving a public experimental version. The report needs a test date and should avoid converting a dated observation into an evergreen recommendation.

E098 application

E098 held the general brief roughly constant but did not preserve enough account, model, cost, time, and rerun evidence for a reproducible benchmark. It remains valuable as an artifact-backed lab observation and as the source of a stronger future protocol.