Implementing Cheap Trust

Read this guide completely before changing the repository:
https://onr.github.io/blogs/cheap-trust-implementation-guide.html

First inventory the existing tests, CI jobs, commands, risk-sensitive code,
and generated artifacts. Then propose the smallest compatible adaptation.

The proposal must identify:
1. existing coverage that will be reused;
2. missing risk and evidence policies;
3. files that would change;
4. validation commands;
5. rollback steps.

Do not duplicate working coverage. Do not replace the project's existing
test runner. Stop after presenting the proposal. Do not implement until the
user approves it.

This guide translates the ideas in Cheap Trust into a repository-level implementation. The examples are conventions, not requirements. Adapt them to the language, framework, risk profile, and CI system already in use.

Operating principles

1. Reuse before adding. Map existing tests and commands before creating new structure.
2. Route by risk. Select the initial scope from the affected capability, contract, data sensitivity, and blast radius.
3. Prefer deterministic evidence. Use types, schemas, contracts, and assertions before model judgment.
4. Separate instructions from evidence. Stable policy does not belong beside generated logs.
5. Escalate uncertainty. A cheap verifier may accept, reject, or defer. It must not invent confidence.
6. Keep humans focused. Human review receives a small evidence package and a specific decision.
7. Improve future verification. Material incidents and expensive decisions should sharpen tests, artifacts, or routing policy.

Inventory first

Before changing the repository, produce a short inventory:

• test frameworks and their configuration files;
• canonical local and CI commands;
• unit, integration, contract, end-to-end, security, and performance coverage;
• critical workflows and business invariants;
• authentication, authorization, payments, migrations, infrastructure, and tenant boundaries;
• current reports, traces, screenshots, logs, and their retention locations;
• known flaky tests and slow test families;
• release gates, canaries, feature flags, and rollback mechanisms.

If the project already has a working convention, extend it. A new folder is not an improvement if it creates a second source of truth.

One practical repository layout

The exact names are optional. The important boundary is between stable instructions, pending coordination, executable checks, evaluator assets, and generated evidence.

AGENTS.md (or the repository's agent entry point)

instructions/
  testing/
    INDEX.md
    risk-policy.yaml
    contract/
    e2e/
    security/
    review/

coordination/
  testing/
    proposals/
    specs/
    schedules/
    review-requests/

tests/
  unit/
  integration/
  contract/
  e2e/
  regression/

evals/
  datasets/
  rubrics/
  calibration/

artifacts/
  testing/
    runs/
    traces/
    screenshots/
    security/
    reports/

• instructions/ describes how verification works.
• coordination/ records pending work, proposals, and review requests.
• tests/ executes deterministic requirements.
• evals/ stores reference sets, rubrics, and calibration data.
• artifacts/ records what happened during a run.

The agent entry point should be a small router. It should direct the agent to the relevant policy instead of embedding the entire testing manual in every context window.

Encode a small risk policy

Start with a small taxonomy. Mixed changes inherit the strongest applicable policy.

version: 1
risk_classes:
  R0:
    examples: [docs, comments, copy]
    required: [static]
  R1:
    examples: [local_logic]
    required: [static, targeted_unit]
  R2:
    examples: [api, schema, dependency, integration]
    required: [static, unit, contract, integration]
  R3:
    examples: [ui, user_workflow]
    required: [targeted_e2e, visual_when_relevant]
  R4:
    examples: [auth, payments, migrations, infrastructure, permissions]
    required: [broad_suite, security, strong_review, rollout_plan]

mixed_change_policy: strongest_applicable
router_audit: scheduled_full_suite_comparison

File paths can be one signal, but they are not sufficient. The router should also consider capabilities, contract changes, sensitive data, reversibility, blast radius, and trust boundaries.

Attach verifier metadata to qualitative tests

Deterministic tests do not need a model tier. Qualitative test families should state the cheapest verifier that has been validated for that slice.

id: checkout-copy-quality
risk: R2
evaluator: qualitative
min_verifier: low_cost_model
escalate_below_confidence: 0.85
reference_set: evals/checkout-copy-v2.jsonl
revalidate_on:
  - model_change
  - rubric_change
  - product_change
  - production_drift

Validate the tier against representative normal, edge, adversarial, and previously failed cases. Measure false acceptance, false rejection, escalation frequency, and calibration. Overall agreement alone can hide consequential errors.

Expose canonical commands

Humans, coding agents, and CI should use the same stable entry points. Keep one broad command plus named layer commands. Adapt these examples to the project:

npm test                 # canonical broad suite
npm run test:fast        # inner-loop static and targeted checks
npm run test:unit
npm run test:contract
npm run test:integration
npm run test:e2e
npm run test:security
npm run test:report      # machine-readable summary

Every important interactive workflow should have a reproducible headless or command-line path. A release-critical check that only works through a person clicking a UI cannot be reliably scheduled or audited.

The coding-agent workflow

1. Read the repository agent entry point and the relevant testing policy.
2. Inspect the requested change and the existing test inventory.
3. Assign risk with a short rationale.
4. Select the smallest sufficient inner-loop scope.
5. State the exact commands before running them.
6. Run deterministic checks first.
7. Run targeted integration, scenario, or qualitative checks as required.
8. Write a structured run summary and preserve failure artifacts.
9. Pass, reject, or escalate according to policy.
10. Run the required outer-loop gate before merge or release.

The change author may run verification, but it should not redefine acceptance after seeing the output. For high-risk changes, final acceptance should come from established contracts, independent policy, or an appropriate reviewer.

Use a stable run artifact

{
  "change_id": "checkout-coupon-layout",
  "commit": "abc1234",
  "risk": ["R3", "R4-payment-boundary"],
  "scope": ["checkout-unit", "payment-contract", "checkout-e2e"],
  "commands": ["npm run test:fast", "npm run test:e2e -- checkout"],
  "status": "FAIL",
  "failed": ["coupon-error-mobile"],
  "classification": "likely-regression",
  "classification_confidence": 0.82,
  "classifier": "visual-verifier-v3",
  "artifacts": ["trace.zip", "expected.png", "actual.png", "diff.png"],
  "next_action": "request_visual_review"
}

Include enough provenance to reproduce the result: commit, environment, dependencies, browser project, configuration, test command, evaluator, and evaluator version.

Make escalation productive

A cheap verifier has three valid outcomes: accept, reject, or defer. Escalation triggers can include low confidence, conflicting evidence, high-risk changes, repeated nondeterminism, evaluator drift, or changes to the verification harness itself.

The stronger verifier should return:

1. the verdict and evidence;
2. the remaining uncertainty;
3. the next action;
4. a candidate way to make this decision cheaper next time.

That final item might be a clearer assertion, narrower context, new schema, split scenario, added fixture, better screenshot state, or routing-policy update.

Package human review

A human request should contain a preview or reproduction link, only the relevant artifacts, a short statement of what already passed, and one focused question. Do not ask a reviewer to reconstruct the run from raw logs.

Protect generated evidence

Traces, screenshots, logs, and model inputs can contain customer data, tokens, secrets, internal URLs, or sensitive business state.

• Exclude generated artifacts from Git unless they are intentionally reviewed fixtures.
• Store CI artifacts in an access-controlled system.
• Redact secrets and sensitive values before model or human review.
• Define retention periods by artifact type and risk.
• Record who or what produced each artifact.
• Do not send sensitive evidence to an evaluator that lacks the required data boundary.

Adopt the system incrementally

1. Week 1: inventory. Document commands, critical behaviors, test layers, and high-risk surfaces.
2. Week 2: routing. Add a small R0 to R4 policy and a structured run-summary schema.
3. Week 3: inner loop. Stabilize fast commands and affected-test selection.
4. Week 4: escalation. Add one calibrated qualitative evaluator or one focused human-review template.
5. Ongoing: audit. Compare targeted runs with broader gates, repair flaky tests, and convert incidents into regression memory.

Do not begin by building a large orchestration platform. Begin with one useful policy, one artifact, and one repeated workflow.

Definition of done

A repository has a useful first implementation when:

• one documented command runs the broad verification suite;
• fast layer-specific commands support the inner loop;
• risk-sensitive changes select stronger scope automatically or by explicit policy;
• non-trivial runs emit a machine-readable summary;
• failure artifacts are reproducible and protected;
• qualitative evaluators have reference sets and revalidation triggers;
• uncertainty has an escalation path;
• production incidents can become regression cases;
• the testing system has an owner and a maintenance cadence.