Cheap Trust: Cost-Aware Verification for Agentic Software Development

As a project grows, the same pain appears again and again: things that were already fixed break again. Tests prevent regression debt, the expensive cycle in which every new feature quietly breaks an old one and the team repeatedly rediscovers the same lessons.

Agent-driven development multiplies that risk. Coding agents can inspect a repository, modify several subsystems, update tests, drive a browser, and prepare a pull request in minutes. Without strong regression protection, that speed produces compounding breakage rather than compounding productivity.

Protect yesterday’s working behavior from today’s optimization.

The bottleneck is increasingly verification. If every meaningful edit requires a full end-to-end suite, a frontier model to interpret its result, and a human to reconstruct what happened from raw logs, generation will outpace the system used to trust it.

The central question should therefore be:

What is the cheapest reliable way to know whether this change is safe enough to ship?

I call that target cheap trust: evidence strong enough for the risk at hand, but inexpensive enough to collect whenever software changes.

This article focuses on conventional software created or modified by coding agents. Testing an application that is itself an agent adds concerns such as tool trajectories, prompt injection, and permission boundaries, but the same verification economics still apply.

Verification is an economic system

Teams often reduce agent cost to token spend. Verification consumes more than tokens:

verification cost
≈ CI compute + model inference + wall-clock latency
  + human attention + expected cost of escaped defects

This is a conceptual model; the terms must either be normalized into a common measure or tracked separately. The objective is not to minimize one term in isolation. A five-second check that misses a payment regression is not cheap. A highly reliable three-hour suite that runs after every edit is not cheap either.

The useful target is decision-relevant signal per unit of verification cost.

Regression debt accumulates when a team fixes or learns something but does not encode what must remain true. A material or recurring failure should trigger a decision about what to preserve: a test, contract, invariant, artifact, rubric, or routing rule. When recurrence is plausible and consequential, a fix that does not improve future verification is incomplete.

What coding agents change about testing

Change volume can exceed verification capacity

Agents make implementation and revision much faster. That is valuable only when verification scales with the rate of change. Otherwise code becomes cheap to produce while confidence remains expensive to establish.

Plausible code still fails at boundaries

Generated code can look locally reasonable while breaking an API contract, changing database behavior, weakening an authorization rule, mishandling configuration, or disrupting a user workflow. The more subsystems an agent can change in one pass, the more important boundary evidence becomes.

Autonomous iteration needs reproducible evidence

When an agent can edit, test, inspect failures, and try again, “done” is not evidence. The loop should record what changed, which checks ran, why that scope was selected, what passed and failed, and why the result is sufficient for its assigned risk.

A layered architecture for cheap trust

The easiest way to make verification unaffordable is to put every requirement into one giant end-to-end evaluation. Different evidence has radically different costs.

Prefer a schema validator before a model evaluates quality. Prefer a targeted contract test before an end-to-end suite exercises the entire product. Prefer a screenshot diff that narrows a visual problem before anyone reads a long browser transcript.

change → risk classification → smallest sufficient scope
→ cheapest proven verifier → escalation when needed
→ durable regression protection

Diff → risk → scope

“Run everything” can be a reasonable release strategy and a poor inner-loop strategy. Inspect the diff, identify affected capabilities and boundaries, assign risk, and choose the smallest sufficient initial scope.

Mixed changes inherit the strongest applicable policy, while lower-risk portions can still use narrower checks. Classification should consider affected capabilities, contract changes, data sensitivity, reversibility, blast radius, trust boundaries, test-harness changes, and incident history, not file paths alone.

The router itself must be verified. Periodic broader runs can measure which failures targeted selection would have missed. That turns test selection into an auditable policy rather than an article of faith.

Separate the inner and outer loops

The inner loop should be narrow, fast, fail-fast, deterministic where possible, and repeatable many times per hour. The outer loop can include full unit and integration suites, cross-browser coverage, wider visual matrices, performance and security checks, migration rehearsal, and release readiness. The first keeps iteration affordable; the second protects against blind spots.

Turn judgment into checking

Model graders are useful, but they are tempting. It is easy to replace an unclear requirement with a prompt asking a powerful model, “Does this look correct?” That often hides the specification problem instead of solving it.

If a frontier model is required merely to decide whether a test passed, the test is often underspecified.

Before introducing a judge, ask whether the output can use a schema, the requirement can become an invariant, behavior can be asserted at an API or database boundary, a broad rubric can become narrow binary checks, or a prior human decision can become a regression case.

Recommendation quality, tone, prioritization, and UX judgment may remain qualitative. The goal is not to eliminate judgment. It is to shrink the fuzzy surface until expensive reasoning is used only where it adds information.

Evaluator routing: cheap by default, strong by exception

deterministic → calibrated low-cost model
→ high-capability model → human or domain review

Each qualitative test family should have a minimum sufficient verifier. A compact manifest can make the decision explicit:

test_family: checkout-copy-quality
min_verifier: low_cost_model
escalate_below_confidence: 0.85  # illustrative; calibrate per test family
reference_set: evals/checkout-copy-v2.jsonl

The label is evidence about one verifier on one slice of work. It is not a permanent certificate of intelligence.

Validate before delegation

Use a representative reference set containing normal, edge, adversarial, and previously failed cases. Compare the cheap verifier with a stronger reference process and measure false acceptance, false rejection, escalation frequency, performance by risk class, and confidence calibration where scores are used.

high-confidence pass → accept
high-confidence fail → reject
uncertain → defer to stronger verification

Revalidate after meaningful changes to the model, prompt, rubric, tools, product behavior, reference set, or production distribution.

Expensive verification should teach the system

A stronger verifier should return both a verdict and, when possible, a candidate way to reduce future ambiguity: clearer assertions, narrower context, added structure, or a test split. Escalation should create a cheaper path for the next occurrence rather than becoming a recurring premium toll.

Worked example: a checkout UI change

Suppose a coding agent moves a coupon field, updates validation, and changes a payment-summary component.

1. Classify. UI work is R3; the payment boundary follows R4 policy.
2. Run deterministic checks. Type checking, unit tests for validation, payment-summary contracts, and a total-calculation invariant.
3. Run targeted scenarios. Successful checkout, invalid and expired coupons, payment decline, and mobile checkout.
4. Preserve visual evidence. A mobile comparison shows the coupon error overlapping the submit button.
5. Classify the failure. A retry reproduces the same visual difference, increasing confidence that the problem is reproducible rather than a transient rendering artifact.
6. Escalate only the uncertain surface. A visual evaluator sees the expected, actual, and diff images plus the acceptance criterion.
7. Package the human decision. The reviewer receives one preview link and one question: “Is this placement acceptable at 390 pixels?”
8. Encode the decision. After rejection and repair, the agent adds a screenshot assertion for that exact state.

Prefer structured artifacts over raw context

Huge logs, DOM dumps, transcripts, and entire repositories are expensive and difficult to audit. Prefer machine-readable results, traces, screenshots and diffs, minimal error excerpts, exact commands, environment metadata, and compact summaries.

{
  "change_id": "checkout-coupon-layout",
  "risk": ["R3", "R4-payment-boundary"],
  "scope": ["checkout-unit", "payment-contract", "checkout-e2e"],
  "status": "FAIL",
  "failed": ["coupon-error-mobile"],
  "classification": "likely-regression",
  "classification_confidence": 0.82,
  "classifier": "visual-verifier-v3",
  "artifacts": ["trace.zip", "expected.png", "actual.png", "diff.png"],
  "next_action": "request_visual_review"
}

Evidence should carry provenance: commit, environment, dependency lockfile, browser project, configuration, and evaluator version. A result that cannot be reproduced is weaker than it appears.

Package human review

Human attention is often the most expensive resource in the loop because people are interruption-sensitive and uniquely valuable for questions of intent. Before requesting input, run cheap checks, narrow the uncertain surface, attach exact evidence, explain what has been ruled out, and ask one focused question.

When practical, establish acceptance criteria or reproduce the failure with a test before implementation begins. This makes it harder for the author to redefine success around whatever the change happens to produce.

Humans establish high-signal intent. Machines enforce it until product change or drift requires renewed judgment.

Security belongs in risk routing

Security review should be triggered by what changed, not by someone remembering a checklist. Changes involving authentication, authorization, secrets, data access, dependencies, input handling, infrastructure, permissions, destructive operations, or tenant boundaries should raise risk automatically.

Relevant checks can include positive and negative authorization tests, secret scanning, vulnerability checks, cross-tenant isolation, abuse cases, migration validation, infrastructure safety, and audit-log assertions. High-risk changes may justify a canary or gradual rollout even after the tests pass.

Use browsers to create regression tests, not replace them

Interactive browser-driving agents are excellent for exploration. Stable regression paths should become ordinary executable tests.

Explore rarely. Convert discoveries into tests. Run those tests often.

Playwright supports targeted files and titles, fail-fast execution, rerunning previous failures, traces, reports, and changed-test selection. Its documentation notes that --only-changed uses Git-based dependency analysis heuristically and can miss affected tests, so it is preliminary feedback rather than a replacement for the full gate.

npx playwright test --last-failed
npx playwright test tests/checkout.spec.ts
npx playwright test -g "checkout"
npx playwright test -x
npx playwright show-report
npx playwright show-trace path/to/trace.zip

For visual regression, use a small viewport matrix, controlled rendering conditions, expected/actual/diff images, and deliberate baseline updates. Do not pay a multimodal evaluator to rediscover obvious pixel differences on every run; escalate only when the meaning remains ambiguous.

Make verification cheaper over time

A mature system does not just test the product. It monitors whether testing remains economical and trustworthy. Track wall-clock time and CI compute by family, model cost by tier, human-review frequency, flake rate, false acceptance and rejection where measurable, escalation reasons, targeted-suite misses, and which checks actually catch regressions.

verify cheaply → escalate uncertainty → inspect why
→ encode the decision → recalibrate → lower future cost

Production monitoring closes the loop. Reduce escaped incidents to minimal reproducible cases, add them to the appropriate layer, and update routing when the original scope was insufficient. Feature flags, canaries, gradual rollouts, and rollback automation limit the cost of what pre-release verification fails to catch.

Implementation guide

The architecture does not require a large testing platform. It can begin with a small risk policy, stable test commands, a machine-readable run summary, and one escalation rule.

Read https://onr.github.io/blogs/cheap-trust-implementation-guide.html
before changing this repository's verification setup.

First inventory the existing tests and commands. Then propose the smallest
adaptation that adds risk routing, structured evidence, and explicit
escalation without duplicating working coverage. Stop after presenting the
proposal. Do not implement until the user approves it. The proposal must
identify affected files, validation commands, and rollback steps.

Related work

This architecture combines established engineering ideas: test pyramids, contracts, change-impact analysis, risk-based selection, visual regression, security gates, canaries, and human review.

• When to Trust the Cheap Check, a 2026 preprint on accept/reject/defer weak–strong verification policies.
• NIST AI RMF Core, covering repeatable evaluation, uncertainty, independent review, and production monitoring.
• Playwright CLI, CI guidance, and screenshot assertions.

Conclusion

AI agents make implementation cheaper. Sustainable productivity depends on making trust cheaper too.

That means contracts before judgment, targeted checks before broad suites, structured artifacts before raw context, calibrated cheap evaluators before expensive ones, and focused human decisions before open-ended review.

The strongest verification system is not the one that runs the most tests or uses the most capable evaluator every time. It is the one that spends attention where it changes the decision and turns expensive lessons into product memory.

That is cheap trust: repeatable evidence whose cost grows more slowly than the software it protects.