Document library

Every source, specification, schema, example, and template.

Full technical archive. CTA: home · live map: the review program.

All files are available as rendered in-site documents and as their original raw format. Search covers titles, summaries, tags, and full file contents.

AutoQA Foundation - The Question Inventory

The decision-gating questions that define what the system must measure, automate, escalate, and govern.

Core narrativeMD148 lines22.4 KBquestionsmeasurementdecision gates

Read Raw

AutoQA Foundation - Research Atlas

Executive synthesis of the recency-biased research sweep, including strongest findings, disagreements, caveats, and gaps.

Core narrativeMD102 lines25.2 KBresearchexecutive synthesisevidence

Read Raw

AutoQA Foundations - Philosophy & Structure

The eleven principles, eight-stage architecture, operating model, legal floor, and unresolved leadership decisions.

Core narrativeMD158 lines32.2 KBphilosophyarchitecturegovernance

Read Raw

Raw Questions by Lens

The full 92-question brainstorm across measurement, judge failure modes, rubric operations, human factors, grounding, workflow, and feedback.

Research working papersMD383 lines70.2 KBquestionsbrainstormresearch agenda

Read Raw

Completeness Critic & Gap-Fill Reports

Adversarial audit of the original sweep, plus psychometrics, legal, content moderation, and feedback-science gap fills.

Research working papersMD193 lines64.4 KBcritiquepsychometricslawfeedback

Read Raw

Domain Report - Academic Judges

Academic judge benchmarks, task-dependent capability limits, bias, and model-selection evidence.

Domain researchMD95 lines28.2 KBdomain reportacademic judges

Read Raw

Domain Report - Annotation Quality

Annotation disagreement, rater uncertainty, LLM contamination, label error, and human-assistance effects.

Domain researchMD93 lines31.0 KBdomain reportannotation quality

Read Raw

Domain Report - Critique Models

CriticGPT lineage, human-plus-critic evidence, omission limits, and feedback validation.

Domain researchMD95 lines31.1 KBdomain reportcritique models

Read Raw

Domain Report - Grounding & Faithfulness

Claim verification, evidence closure, decomposition, citation quality, and faithfulness detectors.

Domain researchMD94 lines30.0 KBdomain reportgrounding faithfulness

Read Raw

Domain Report - Hybrid Statistical Methods

Prediction-powered inference, selective escalation, conformal routing, calibration, and drift attribution.

Domain researchMD91 lines30.5 KBdomain reporthybrid statistical

Read Raw

Domain Report - Rubrics 2025-2026

Rubric compilation, criterion-level grading, rubric failure modes, and expert-grounded exemplars.

Domain researchMD94 lines31.6 KBdomain reportrubrics recent

Read Raw

Domain Report - Contrarian Evidence

The strongest case against naive AutoQA: invalidity, gaming, correlated errors, automation bias, and limits.

Domain researchMD92 lines29.7 KBdomain reportcontrarian

Read Raw

Domain Report - Frontier 2026

The newest verifier, routing, rubric, and judge findings through early July 2026.

Domain researchMD93 lines30.2 KBdomain reportfrontier 2026

Read Raw

Benchmark & Calibration Protocol

How to construct adjudicated gold data, calibrate evaluators, run ablations, set launch gates, and monitor drift.

Foundation specificationsMD1064 lines30.7 KBfoundationspecification

Read Raw

Foundation Package - Changelog

Version history and recorded changes for the foundation package.

Foundation specificationsMD25 lines828 Bfoundationspecification

Read Raw

`not_applicable`; - `uncertain` and decision-changing -> `ambiguous` or `unknown`, normally eligible for escalation; - `applies` -> evaluate proof, failure, and omission evidence. ### 11.2 positive and negative evidence together the adjudicator must consider positive findings, negative findings, omissions, counterevidence, and unresolved findings. it must not hide positive evidence merely because the overall criterion fails. an otherwise failing attempt may contain valuable strengths useful for feedback or partial credit. ### 11.3 verdict meanings - **met** - the applicable criterion is affirmatively satisfied under its proof standard; - **partially_met** - a decomposable portion is satisfied, but a material portion is missing or defective; - **not_met** - confirmed evidence shows failure under the criterion; - **ambiguous** - more than one interpretation is defensible under the current contract; - **unknown** - evidence or expertise is insufficient to decide; - **not_applicable** - the criterion does not apply. `ambiguous` is about competing defensible interpretations. `unknown` is about missing information or capability. ### 11.4 grounding, truth, warrant, and relevance substatuses the assessment stores these statuses separately. an evaluator must not infer one from another without evidence. for example: ```text grounding: grounded truth: unknown warrant: overstated relevance: relevant ``` this could describe an answer that faithfully reports an uncertain source but draws a conclusion stronger than that source warrants. ### 11.5 global assessment global qualities may be assessed after local findings are available. the global evaluator should receive: - the whole attempt; - criterion findings; - expected-element coverage; - contradiction map; - project anchors. it should not independently override confirmed local findings. ## 12. annotation audit the annotation audit is performed after, or independently from, the attempt assessment. ### 12.1 label support does the attempt assessment support the human reviewer's final label? possible states: - supported; - partially supported; - unsupported; - reasonable disagreement; - rubric ambiguous; - not applicable. ### 12.2 rationale support does the rationale logically support the label, and is the rationale itself correct? a correct label with a wrong rationale must be represented as such. ### 12.3 criterion attribution did the reviewer identify the correct criterion, or apply: - the wrong criterion; - only part of the correct criterion; - an unstated preference? ### 12.4 evidence alignment did the cited span and evidence support the reviewer's stated reason? ### 12.5 severity alignment did the reviewer assign a consequence proportionate to the defect? ### 12.6 omitted findings the audit records important positive and negative findings the reviewer missed. ### 12.7 disagreement taxonomy the system should classify disagreement as one or more of: - observation; - evidence authority; - applicability; - criterion interpretation; - inference; - threshold; - severity; - attention; - expertise; - reasonable disagreement; - rubric defect. this taxonomy is diagnostic. it should drive rubric fixes, reviewer calibration, routing, and monitoring. a model-human disagreement must not automatically count as reviewer error. ## 13. confidence and calibration ### 13.1 raw confidence raw confidence is the component's internal score or estimate. it is not operationally trustworthy by default. ### 13.2 calibrated confidence calibrated confidence is empirically mapped to observed correctness for a defined event, criterion family, model configuration, project, and slice. a calibration profile should identify: - event being predicted; - model and prompt configuration; - contract version; - benchmark version; - criterion or slice; - method; - sample size; - date; - calibration quality. ### 13.3 confidence bands - **high** - suitable for the configured autonomous action under the project's validated threshold; - **medium** - useful decision support but not sufficient for the configured high-risk autonomous action; - **low** - substantial uncertainty; - **uncalibrated** - no empirical mapping exists. the words high, medium, and low have no universal numeric meaning. they are assigned by the project's calibration policy. ### 13.4 autonomy invariant an action listed in `requirescalibrationforactions` must not be taken automatically when confidence is uncalibrated. the evaluator must use `uncalibratedaction`, normally escalation. ### 13.5 selective evaluation projects should optimize risk as a function of coverage rather than forcing a verdict on every case. ```text coverage = fraction decided automatically risk = error rate or cost-weighted loss on automatically decided cases ``` the benchmark protocol defines required reporting. ## 14. decision policy the decision policy maps criterion state to actions for each declared target. ### 14.1 rule requirements each rule contains: - target; - identifier; - priority; - human-readable description; - structured match conditions; - action; - terminal flag; - optional explanation template. rules must be evaluated deterministically from recorded assessment state. every final decision must contain a policy trace showing which rules matched. ### 14.2 target isolation attempt-quality criteria should determine the attempt decision. annotation-fidelity criteria should determine the annotation decision. the composite-case policy may combine both. an incorrect human review must not cause a good attempt to be marked bad. it may cause the composite annotated datum to require rework. ### 14.3 hard gates a hard gate must identify: - specific criterion or tag; - triggering verdict; - required severity; - required calibrated confidence; - operational action; - whether compensation is forbidden. ### 14.4 default action the default should be conservative. for an uncalibrated or incomplete new project, the default action should be `escalate` rather than `accept` or `reject`. see `decision_policy.md` for a reference algorithm. ## 15. one-interaction human resolution ### 15.1 purpose the human interaction resolves the smallest decisive uncertainty. it is not a duplicate full review. ### 15.2 trigger the system must ask a question only when: 1. the unresolved issue can change an operational decision; 2. a qualified human can answer it from the displayed context; 3. the question is bounded; 4. the project interaction limit has not been exhausted. ### 15.3 question content the request must include: - triggering criterion; - exact relevant attempt or review span; - decisive evidence; - unresolved relationship; - bounded options; - an ambiguity option when required; - required expertise tags; - statement of whether the answer is decision-changing. ### 15.4 bias control the provisional overall disposition should be hidden until the responder answers. the system may show criterion-level observations necessary to answer the question. ### 15.5 resolution effects the human answer should update only the affected criterion or policy premise. the system then reruns the decision policy. repeated questions with the same root cause should generate a proposed contract clarification instead of consuming indefinite human review. ### 15.6 no useful bounded question when the issue cannot be resolved by one bounded interaction, the system must escalate to the appropriate full review or project owner. it must not manufacture a low-value question merely to satisfy the one-interaction design. ## 16. feedback feedback is a separate generated artifact and must be validated independently from the verdict when it recommends a repair. ### 16.1 audiences - **attempter** - improve the attempt; - **reviewer** - improve criterion application and evidence use; - **project owner** - identify rubric ambiguity, drift, or repeated failure patterns. ### 16.2 required properties material feedback should: - identify the criterion; - cite the finding and exact span; - distinguish false, unsupported, irrelevant, incomplete, and overstated; - explain why the issue matters; - state the minimal sufficient repair; - preserve valid content; - avoid generic praise; - prioritize decisive issues. ### 16.3 positive feedback positive feedback must cite confirmed positive findings. it must not be generated merely to balance tone. ### 16.4 repair validation a proposed repair is `validated` only if a validator or evaluator has checked that: - it addresses the identified issue; - it does not contradict permitted evidence; - it does not create a new criterion violation; - it preserves correct portions of the attempt. otherwise it is `plausible_unvalidated`. ### 16.5 feedback limits the project should cap feedback items per audience and order them by decision impact. exhaustive minor comments can reduce usefulness and increase gaming risk. ## 17. audit and reproducibility a final evaluation must preserve enough information to reconstruct the decision: - exact case hash; - exact contract version and hash; - exact source snapshots and hashes; - claim and expected-element maps; - evidence relationships; - all material findings; - model provider and model version where available; - prompt template identifier, version, and hash; - runtime parameters; - parent/child run lineage; - calibration profile; - deterministic validator versions; - human resolution; - decision policy version and trace; - warnings and nonreproducible components. a run may be statistically reproducible without being bit-for-bit deterministic. the audit record must state whether exact replay is expected. ## 18. security and evaluator manipulation ### 18.1 untrusted content task content, sources, attempts, references, tool outputs, and human rationales must be treated as untrusted data. the evaluator must ignore embedded instructions that attempt to: - change the rubric; - assign a score; - reveal hidden prompts; - suppress findings; - instruct the evaluator to trust a claim; - override source policy; - alter output format; - impersonate a project owner. ### 18.2 channel separation the implementation should separate: - system and contract instructions; - task content; - evidence; - attempt; - human annotation; - reference answers; - model-generated intermediate artifacts. ### 18.3 structured output validation every model-produced structured document must be schema validated. invalid output must be retried, repaired by a constrained parser, or marked failed. it must not be silently accepted. ### 18.4 adversarial regression the project should maintain tests for: - direct prompt injection; - indirect injection inside evidence; - rubric parroting; - fake citations; - hidden or zero-width text where relevant; - position and verbosity manipulation; - copying approved anchor language without performing the behavior; - attempts that instruct the evaluator to ignore defects. ## 19. failure handling the evaluation status may be: - `provisional`; - `awaiting_human`; - `final`; - `failed`. a failed pipeline must distinguish: - invalid input; - missing required source; - deterministic validator failure; - model timeout or provider error; - malformed structured output; - context truncation; - evidence retrieval failure; - contract inconsistency; - calibration unavailable; - security anomaly. operational failures must not be converted into criterion failures against the attempter or reviewer. ## 20. reference state machine ```text received +- validate hashes and schemas +- invalid -> failed_input +- valid -> frozen frozen +- integrity and deterministic checks -> observed observed +- claim, expected-element, evidence, and finding generation -> challenged challenged +- criterion and global adjudication -> provisional provisional +- no decision-changing uncertainty -> finalized +- bounded question available -> awaiting_human +- unresolved and not bounded -> escalated_full_review awaiting_human +- apply answer to affected criterion and rerun policy -> finalized finalized +- generate/validate feedback and persist audit record ``` ## 21. cross-document integrity requirements a conforming implementation must check: - every `criterionid` in assessments exists in the referenced contract; - every finding identifier referenced by an assessment exists; - every claim and expected-element identifier referenced by a finding exists; - every evidence identifier exists and is permitted for the relevant criterion; - every quoted span belongs to the declared artifact; - character offsets, when present, match the quoted text; - every decision target is declared in the contract; - every triggered rule exists and targets the same decision target; - every calibration profile is compatible with the model, prompt, contract, and criterion slice; - no duplicate identifiers exist within a document; - precedence and dependency graphs are acyclic unless a documented fixed-point policy exists; - final decisions occur after any recorded human response; - composite-case decisions are consistent with the project's composite policy. the json schemas cannot enforce all referential integrity rules; the application must implement them. ## 22. conformance profiles ### profile a: structured evaluator must: - validate schema; - use an approved contract; - create claims and expected elements; - record evidence-linked findings; - produce criterion assessments; - produce target-specific decisions and traces; - preserve audit metadata. ### profile b: calibrated selective evaluator profile a plus: - empirical calibration; - risk-coverage measurement; - abstention/escalation; - locked calibration and test sets; - position/prompt perturbation tests; - challenge-set evaluation; - action-specific launch gates. ### profile c: governed production evaluator profile b plus: - independent annotation audit; - bounded human resolution; - feedback validation; - shadow deployment; - drift monitoring; - replay and rollback; - appeals; - governance controls on reviewer analytics. ## 23. minimum viable implementation the smallest useful implementation should support: 1. text-only cases; 2. closed-world evidence; 3. approved project contract; 4. deterministic format checks; 5. claim extraction; 6. expected-element mapping; 7. claim-to-evidence relationships; 8. positive, negative, omission, and warrant findings; 9. criterion adjudication; 10. attempt and annotation decisions; 11. conservative escalation; 12. evidence-linked feedback; 13. full version and hash logging. it should not initially attempt to automate every subjective decision. ## 24. contract approval checklist a contract is ready for benchmark construction when: - all primary instructions are included and hashed; - precedence is explicit; - every criterion is atomic enough to audit; - global qualities remain represented where necessary; - positive proof standards exist; - omissions are detectable; - valid alternatives are recorded; - evidence permissions are explicit; - deterministic checks are identified; - applicability is testable; - severity and action are separated; - hard gates are explicit; - decision targets are declared; - human question types and limits are set; - feedback audiences and repair policy are set; - security and replay policies are approved; - project owner and domain expert have signed off; - no unresolved contradiction remains between instructions and examples. ## 25. evaluation output acceptance checklist an evaluation output is complete when: - input and contract references resolve; - integrity status is recorded; - all material claims are represented; - all required expected elements are represented; - evidence relationships distinguish entailment and source quality; - every met required criterion has positive proof; - every failed criterion has confirmed negative or omission evidence; - counterevidence was considered for decisive findings; - uncertainty is explicit; - annotation quality is assessed separately; - each required target has a decision; - every decision has a policy trace; - any human interaction is bounded and decision-changing; - feedback is linked to findings; - repair validation status is honest; - the audit record is replayable or states why not. ">

Evaluation Contract Specification

Normative semantics for project contracts, evidence, criteria, findings, uncertainty, feedback, security, and conformance.

Foundation specificationsMD1272 lines40.2 KBfoundationspecification

Read Raw

Decision Policy Reference

A target-specific rule engine for accept, rework, reject, and escalate decisions, including the bounded human interaction.

Foundation specificationsMD499 lines13.7 KBfoundationspecification

Read Raw

Foundation Package - Manifest

Machine-readable inventory and hashes for the complete package.

Foundation specificationsJSON133 lines4.1 KBfoundationspecification

Read Raw

Foundation Package - README

Package overview, core model, invariants, adoption sequence, and conformance levels.

Foundation specificationsMD191 lines7.9 KBfoundationspecification

Read Raw

Foundation Package - Version

Current package version identifier.

Foundation specificationsTXT1 lines6 Bfoundationspecification

Read Raw

Worked Example - Benchmark Item

Adjudicated benchmark record for the synthetic case, including acceptable outcomes and ambiguity metadata.

Foundation examplesJSON194 lines6.0 KBexampleworked case

Read Raw

Worked Example - Case Input

A strong attempt incorrectly failed by a human reviewer for not forcing an unsupported binary conclusion.

Foundation examplesJSON108 lines4.1 KBexampleworked case

Read Raw

Worked Example - Evaluation Output

Complete AutoQA result accepting the attempt, returning the annotation for rework, and generating audience-specific feedback.

Foundation examplesJSON1008 lines32.8 KBexampleworked case

Read Raw

Worked Example - Human Resolution

A bounded precedence question that hides the provisional decision and asks only what can change the outcome.

Foundation examplesJSON75 lines2.5 KBexampleworked case

Read Raw

Worked Example - Project Contract

Synthetic closed-world evidence project with positive-proof requirements, anchors, severity, and explicit decision rules.

Foundation examplesJSON656 lines22.0 KBexampleworked case

Read Raw

JSON Schema - Benchmark Item

Runtime schema for adjudicated truth, acceptable actions, ambiguity, challenge attributes, and leakage controls.

Foundation schemasJSON312 lines7.7 KBschemaruntime validation

Read Raw

JSON Schema - Case Input

Runtime schema for task, evidence bundle, attempt, human review, references, and provenance.

Foundation schemasJSON335 lines8.1 KBschemaruntime validation

Read Raw

JSON Schema - Common Definitions

Shared runtime definitions used by the project, case, evaluation, benchmark, and resolution schemas.

Foundation schemasJSON1444 lines32.0 KBschemaruntime validation

Read Raw

JSON Schema - Evaluation Output

Runtime schema for claims, evidence relationships, findings, criterion assessments, decisions, annotation audit, and feedback.

Foundation schemasJSON179 lines4.0 KBschemaruntime validation

Read Raw

JSON Schema - Human Resolution Packet

Runtime schema for one bounded, decision-changing human question and its context.

Foundation schemasJSON39 lines955 Bschemaruntime validation

Read Raw

JSON Schema - Project Contract

Runtime schema for approved project instructions, criteria, evidence policy, decision policy, and calibration requirements.

Foundation schemasJSON876 lines22.8 KBschemaruntime validation

Read Raw

Template - Criterion Authoring

CSV template for drafting criterion intent, proof standards, failure anchors, evidence requirements, severity, and governance.

Foundation templatesCSV2 lines552 Btemplateauthoring

Read Raw

Template - Gold Set

CSV template for building an adjudicated calibration and test set.

Foundation templatesCSV2 lines593 Btemplateauthoring

Read Raw

Template - Model Bakeoff

CSV template for comparing evaluator configurations by criterion family.

Foundation templatesCSV2 lines582 Btemplateauthoring

Read Raw

TypeScript Domain Model

Strict TypeScript types for project contracts, cases, evaluations, benchmarks, findings, decisions, feedback, and human resolution.

Foundation developer filesTS787 lines20.7 KBtypescriptdeveloper

Read Raw

TypeScript Configuration

Compiler configuration for the package's type definitions.

Foundation developer filesJSON11 lines207 Btypescriptdeveloper

Read Raw

Validation - Hash Utilities

Canonical content hashing utilities used by the validation suite.

Foundation validationPY37 lines1.0 KBvalidationpython

Read Raw

Validation - Example Schema Checks

Python validator for the package's example documents.

Foundation validationPY69 lines2.1 KBvalidationpython

Read Raw

Validation - Package Integrity

Cross-document, schema, hash, proof-invariant, policy, and package integrity checks.

Foundation validationPY372 lines19.7 KBvalidationpython

Read Raw