# AutoQA operating model

**Status:** Decision-ready operating model  
**Updated:** 2026-07-23

## Purpose

AutoQA checks whether human review decisions are supported by the project instructions and the evidence available for the work.

It exists because reviewer non-consensus is expensive and opaque - not because a model should become a casting vote. It exists under real pressure for throughput. That pressure is exactly when unmeasured automation becomes a weapon against data quality and workers.

AutoQA is not approved to reject work, change pay, restrict accounts, or make other consequential decisions until a specific lane meets release criteria and a separate decision says so.

## The problem

Human reviewers disagree for several reasons:

1. The instructions are unclear.
2. A reviewer made an error.
3. The decision requires expertise the reviewer did not have.
4. More than one interpretation is reasonable.

An agreement score does not distinguish among these cases. AutoQA is intended to identify why a review decision is supported, unsupported, or unresolved - and to route each cause differently.

**Consensus is not the objective function.** Explaining disagreement, fixing instructions, catching checkable error, and preserving legitimate splits is. Agreement may rise as a side effect. If it only rises because people imitate AutoQA, that is monoculture, not quality.

## What AutoQA evaluates

Two objects, kept separate:

- **The work:** Did the submitted work satisfy the project instructions?
- **The review:** Did the reviewer's label, rationale, cited evidence, criterion, and severity accurately describe that work?

A correct work product can receive an incorrect review. Collapsing them turns review error into fake work-quality error.

## The project contract

Each project needs an approved, versioned contract before evaluation:

- project instructions;
- criteria used to evaluate work;
- evidence AutoQA may use;
- examples of acceptable and unacceptable outcomes;
- how severity is assigned;
- how findings map to actions;
- who owns instruction changes and exceptions.

Models change. The contract remains the source of truth.

## How an evaluation works

1. **Validate the record.** Instructions, evidence, work, and review present and same project version.
2. **Identify relevant criteria.**
3. **Break the review into claims.**
4. **Find supporting evidence** for each material claim.
5. **Evaluate each claim:** supported, contradicted, cannot verify, or instructions insufficient.
6. **Evaluate work and review separately.**
7. **Apply project decision policy** (advisory until a lane graduates).
8. **Route unresolved cases** - only decision-changing uncertainty to a qualified human.
9. **Store the audit record** - inputs, versions, evidence links, findings, model config, human decisions, policy trace.

## Result types

- **Supported** - evidence supports the claim.
- **Contradicted** - evidence shows the claim is wrong.
- **Cannot verify** - record insufficient; not proof of error.
- **Instructions insufficient** - rules do not decide; route to owner; not worker fault.

## Decision rules

1. Use project instructions and permitted evidence - not model preference or reviewer consensus as ground truth.
2. Require evidence for material findings.
3. Keep quality and consequence separate.
4. Represent uncertainty directly; abstain when record, instructions, or measured capability is insufficient.
5. Treat model confidence as uncalibrated until measured.
6. Keep consequential decisions human until a specific lane meets release criteria.
7. Preserve replayability.
8. **Throughput, coverage, and automation-rate KPIs must not disable abstention or force pass/fail.**
9. **Do not use AutoQA solely as a casting vote to manufacture consensus on undecidable work.**

## Human responsibilities

People remain responsible for:

- approving the project contract;
- resolving unclear or conflicting instructions;
- creating and adjudicating benchmark data;
- reviewing consequential or unresolved cases (one decision-changing question, evidence first, hide provisional model answer for gold and consequence);
- handling appeals;
- approving any change in system authority;
- monitoring reliability after deployment - including blind audit of auto-accepts and model-human agreements.

When AutoQA escalates, the human receives criterion, relevant evidence, and unresolved question - not a steered essay toward the model's preferred conclusion.

## Measurement and release

1. Measure current human-review baseline and disagreement causes.
2. Build adjudicated development and held-out data for one project.
3. Evaluate AutoQA by criterion and case type in shadow mode.
4. Recommend: stop, narrow, remain assist-only, or approve a specific lane for further graduation work.

A lane is not eligible for autonomous use merely because it looks useful. It needs pre-defined accuracy, false-positive risk, stability, calibration, auditability, and monitoring on untouched test data.

Any approval applies only to tested project, contract version, criterion family, model configuration, and action. Expansion requires a separate decision.

## Known risks

- missed evidence in long records;
- weak omission and unsupported-praise detection;
- correlated errors among workers, reviewers, and models;
- prompt injection / manipulation;
- rubric and instruction drift;
- over-reliance and automation bias when humans see model scores;
- project and criterion transfer failure;
- unrepresentative benchmarks;
- privacy, worker-impact, and legal requirements;
- KPI override of abstention under volume pressure;
- monoculture if AutoQA becomes the agreement target.

Controls: held-out data, shadow deployment, human review, appeals, recurring audits that include agreement cases. Architecture alone does not solve these.

## What AutoQA does not do

- treat majority reviewer agreement as ground truth;
- score opinions as factual claims;
- convert missing evidence into automatic failure;
- penalize workers for unclear instructions;
- collapse work quality and review quality into one score;
- use model confidence without empirical calibration;
- change a project's quality standard through prompt wording;
- certify its own performance;
- authorize production consequences during the proposed pilot;
- manufacture consensus on undecidable cases to clear queues.

## Current decision

**Keep AutoQA running on live projects.** For **two weeks**, hold *new* consequential authority (reject / pay / account automation) and do not treat AutoQA as consensus ground truth, while the team returns with an **everywhere operating map** (advise / escalate / score by lane) and options to **tweak live**, **measure high-stakes lanes before automatic enforcement**, or **narrow what may auto-act**.

This does **not** pause or remove AutoQA from live work. It does **not** by itself authorize new punishment automation or pilot spend.

- [Write the decision](./decide.html)
- [Decision brief](./index.html)
- [Capability limits](./limits.html)


## Supporting detail

- [Detailed kernel](./KERNEL-DETAILED.md)
- [System design](./architecture.html)
- [Evidence ledger](./evidence.html)
- [Research review](./research.html)
- [Document library](./library.html)
- [Foundation package](./documents/view/foundation-readme-md.html)