AutoQAFoundation 0.1.0
Summary
Document library / Foundation specifications

Foundation Package - README

Package overview, core model, invariants, adoption sequence, and conformance levels.

MD191 lines7.9 KBSHA-256 4486f3fb62cf...foundationspecification
Download raw fileBack to library

AutoQA Foundation 0.1.0

A project-agnostic contract and benchmark package for evaluating human-produced AI-training data and the human annotations applied to it.

The package defines the stable domain boundary for an AutoQA system. It intentionally does not prescribe a model provider, orchestration framework, database, or Vercel implementation. Those components can change while the evaluation objects, proof obligations, decision policy, and benchmark remain versioned and auditable.

Core model

AutoQA evaluates two different objects:

  1. The attempt - whether the human-produced work satisfies the project's instructions.
  2. The annotation - whether the human reviewer's label, rationale, cited evidence, criterion attribution, and severity faithfully describe that attempt.

It does not reduce evaluation to a single score. The core unit is a criterion assertion:

Criterion C applies.
Attempt span A exhibits behavior B.
Evidence E supports, contradicts, or is insufficient for B.
B does or does not satisfy C.
The project decision policy assigns consequence S.

Non-negotiable invariants

A conforming implementation preserves these distinctions:

  • truth versus support by the permitted evidence;
  • source quality versus entailment;
  • relevance versus correctness;
  • factual accuracy versus inferential warrant;
  • positive proof versus absence of a detected defect;
  • attempt quality versus annotation fidelity;
  • criterion verdict versus severity;
  • severity versus operational action;
  • raw model confidence versus empirically calibrated confidence;
  • item validity versus marginal dataset value.

The engine may be project-agnostic. Its judgments may not be criterion-agnostic. Every production project must supply an approved, versioned evaluation contract.

Package contents

.
+-- README.md
+-- CONTRACT_SPEC.md
+-- BENCHMARK_PROTOCOL.md
+-- DECISION_POLICY.md
+-- CHANGELOG.md
+-- VERSION
+-- schemas/
| +-- common.schema.json
| +-- project-contract.schema.json
| +-- case-input.schema.json
| +-- evaluation-output.schema.json
| +-- benchmark-item.schema.json
| +-- human-resolution-packet.schema.json
+-- types/
| +-- autoqa.ts
| +-- tsconfig.json
+-- examples/
| +-- project-contract.example.json
| +-- case-input.example.json
| +-- evaluation-output.example.json
| +-- benchmark-item.example.json
| +-- human-resolution-packet.example.json
+-- templates/
| +-- criterion-authoring-template.csv
| +-- gold-set-template.csv
| +-- model-bakeoff-template.csv
+-- validation/
 +-- hash_utils.py
 +-- validate_examples.py
 +-- validate_package.py

Primary documents

  • CONTRACT_SPEC.md defines the normative semantics, lifecycle, invariants, and conformance requirements.
  • BENCHMARK_PROTOCOL.md defines how to build adjudicated gold data, calibrate evaluators, run ablations, set launch gates, and monitor drift.
  • DECISION_POLICY.md defines the recommended rule engine, action semantics, human-resolution trigger, and policy trace.

Machine-readable layer

  • Project contract: the approved interpretation of project instructions.
  • Case input: the task, evidence, attempt, optional reference, and optional human review.
  • Evaluation output: claims, expected elements, evidence relationships, findings, criterion assessments, annotation audit, decision trace, and feedback.
  • Benchmark item: adjudicated criterion truth, target-specific acceptable actions, ambiguity status, challenge attributes, and leakage controls.
  • Human-resolution packet: the one bounded, decision-changing question and its evidence context, with an optional response.

Runtime input and output validation should use the JSON Schemas. TypeScript interfaces are provided for application development but are not a runtime trust boundary.

Quick start

Validate the included examples:

python validation/validate_package.py

Type-check the TypeScript domain model:

cd types
tsc -p tsconfig.json

Recommended adoption sequence:

  1. Copy examples/project-contract.example.json and replace the synthetic instructions and criteria.
  2. Author each criterion in templates/criterion-authoring-template.csv before converting it to JSON.
  3. Run a contract review with the project owner, a domain expert, and a QA owner.
  4. Create representative case_input documents from historical production work.
  5. Independently adjudicate a calibration and test set using BENCHMARK_PROTOCOL.md.
  6. Compare evaluator configurations by criterion family rather than choosing one universal judge.
  7. Calibrate confidence and set project-specific action gates.
  8. Shadow the system before allowing autonomous acceptance or rejection.
  9. Preserve every model, prompt, source, rubric, and policy version required for replay.

Minimal evaluator pipeline

1. Validate and freeze inputs
2. Compile/load the approved project contract
3. Run deterministic integrity and compliance checks
4. Extract claims and expected elements
5. Map claims and requirements to evidence
6. Generate positive, negative, omission, and warrant observations
7. Challenge material observations
8. Adjudicate each criterion
9. Evaluate global qualities
10. Audit the human annotation independently
11. Apply the versioned decision policy
12. Ask at most one decision-changing human question when necessary
13. Generate and validate audience-specific feedback
14. Persist the complete audit record

Conformance levels

Level A - structured evaluation

The implementation validates inputs and outputs, uses an approved project contract, produces criterion-level assessments, and records evidence-linked findings.

Level B - calibrated selective evaluation

Level A plus empirical confidence calibration, risk-coverage reporting, explicit abstention/escalation, and locked test suites.

Level C - governed production AutoQA

Level B plus independent annotation auditing, one-interaction human resolution, shadow deployment, drift monitoring, replay, appeals, and restricted use of reviewer analytics.

Autonomous acceptance or rejection should not be enabled below Level B.

Example scenario

The included example contains a high-quality attempt that correctly concludes that the evidence does not establish a primary causal relationship. A human reviewer fails it for not choosing a forced yes/no answer. The project contract explicitly permits an inconclusive conclusion. The example evaluation therefore:

  • affirms the attempt's directness, grounding, warrant, and completeness;
  • accepts the attempt;
  • returns the annotation and composite case for rework;
  • marks the reviewer annotation as unsupported;
  • gives the attempter specific positive feedback;
  • gives the reviewer a minimal corrective explanation;
  • records a project-owner signal in case this misunderstanding recurs.

This demonstrates why attempt quality and annotation fidelity must be separate outputs.

Versioning rule

Three versions may change independently:

  • Schema version - shape and semantics of machine-readable documents.
  • Contract version - project instructions, criteria, anchors, or evidence policy.
  • Decision-policy version - operational mapping from criterion state to action.

A case must reference an exact contract version and content hash. An evaluation must record the exact case, contract, prompt, source snapshots, model runs, calibration profile, and decision-policy version used.

What 0.1.0 deliberately leaves open

  • model and provider selection;
  • retrieval architecture;
  • prompt templates;
  • long-context chunking strategy;
  • calibration algorithm;
  • reviewer-competence model;
  • dataset-level value model;
  • UI design;
  • persistence and job orchestration.

These should be decided empirically after the contract and benchmark are stable.