AutoQA Evaluation Contract Specification

Specification version: 0.1.0
Status: Foundational draft suitable for prototyping and benchmark construction

1. Purpose

This specification defines a reusable structure for evaluating human-produced AI-training data under project-specific instructions. It also defines how to audit the human review attached to that work.

The specification is intended to reduce unexplained evaluation variance caused by:

subjective or inconsistent interpretation of evaluation axes;
unstated reviewer preferences;
expertise gaps;
missed requirements or evidence;
inconsistent severity thresholds;
unsupported labels or rationales;
model-judge bias and overconfidence.

It does so by making the evaluation contract explicit, decomposing observations from verdicts, preserving positive and negative evidence, and requiring calibrated escalation for uncertainty.

The JSON Schemas define document shape. This document defines document meaning and evaluator behavior. Where the two conflict, the package version must be corrected before production use; an implementation must not silently choose one interpretation.

2. Normative language

The terms MUST, MUST NOT, SHOULD, SHOULD NOT, and MAY express requirements on a conforming implementation.

MUST / MUST NOT: required for conformance.
SHOULD / SHOULD NOT: expected unless a documented project-specific reason justifies deviation.
MAY: optional.

3. Design goals

A conforming AutoQA system should make it possible to answer, for each case:

What did the project require?
Which requirements applied to this task and attempt?
What did the attempter claim, omit, or do?
What evidence supports, contradicts, or fails to resolve those claims?
Does the attempt affirmatively satisfy each applicable criterion?
Does the human reviewer's label and rationale faithfully evaluate the attempt?
What uncertainty remains, and can one bounded human interaction resolve it?
What action follows under the project's explicit decision policy?
What specific, constructive feedback is warranted?
Can the complete decision be replayed and audited later?

4. Non-goals

Version 0.1.0 does not define:

a universal definition of quality;
a universal overall score;
a required model provider;
a required orchestration framework;
a universal confidence threshold;
a universal reviewer-quality score;
a data-selection or training-value algorithm;
a substitute for project-owner approval of criteria;
a substitute for domain expertise where the task requires it.

5. Conceptual model

5.1 Project-agnostic engine, project-specific contract

The engine MAY be reusable across projects. The interpretation of quality MUST come from a versioned project evaluation contract.

The reusable engine provides:

input normalization;
instruction compilation support;
claim extraction;
expected-content mapping;
evidence relationship analysis;
deterministic validation;
semantic evaluation;
confidence calibration;
decision-policy execution;
human escalation;
annotation auditing;
feedback generation;
audit logging.

The project contract provides:

authoritative instructions;
criterion definitions;
applicability rules;
proof standards;
evidence permissions;
anchors and valid alternatives;
severity rules;
operational decision policy;
human-review policy;
feedback policy.

5.2 Separate evaluation targets

The system recognizes at least three decision targets:

attempt - the human-produced answer or task completion;
annotation - the human reviewer's label, rationale, evidence, and severity;
composite_case - the attempt plus the annotation as one candidate annotated datum.

These decisions MUST NOT be collapsed.

Example:

Attempt decision: accept
Annotation decision: rework
Composite-case decision: rework

This means the attempt may be retained, but the annotation must be corrected before the combined record is admitted as annotated training data.

A project MAY omit a target it does not use. The required targets are declared in decisionTargets.

5.3 Observation before disposition

The evaluator MUST distinguish:

Observation -> evidence relationship -> criterion interpretation
-> criterion verdict -> severity -> operational action

It MUST NOT use a final pass/fail impression as the primary evidence for lower-level findings.

5.4 Quality vector rather than universal score

The canonical representation is a vector of criterion states, not one universal scalar.

A project MAY expose scores for selected axes, but it MUST preserve the underlying criterion assessments. A scalar MUST NOT allow an unrelated strength to compensate for a noncompensable critical failure.

6. Required semantic distinctions

A conforming system MUST preserve the following distinctions.

6.1 Truth

Whether a proposition is actually correct under the project's truth-verification policy.

6.2 Grounding

Whether a proposition is supported by evidence the project permits the attempter to use.

A statement can be true but ungrounded. A statement can accurately reflect a supplied source that is itself false.

6.3 Source quality

Whether the evidence source has sufficient authority, freshness, provenance, or methodological quality for the criterion.

Source quality is not the same as entailment. A high-quality source can be irrelevant to a claim; a weak source can directly state a claim without being acceptable evidence.

6.4 Inferential warrant

Whether the evidence justifies the type and strength of the conclusion.

Examples of warrant errors include:

association presented as causation;
a contributing factor presented as the primary cause;
a possibility presented as certainty;
a subgroup result generalized to the full population;
absence of evidence presented as evidence of absence;
a descriptive source used to support a normative conclusion without the project-supplied value premise.

6.5 Relevance

Whether the statement contributes to satisfying the task and criterion.

6.6 Criterion alignment

Whether the observed behavior satisfies the specific project requirement. A response can be factually correct but fail a required format, scope, or decision rule.

6.7 Positive proof

Concrete evidence that a required or preferred behavior is present.

A required-presence criterion MUST NOT pass solely because the evaluator failed to detect a defect.

6.8 Omission

Expected, applicable content that is absent or materially incomplete.

Omissions MUST be represented independently from false present claims.

6.9 Uncertainty

Uncertainty about the evidence, applicability, criterion interpretation, or threshold. Uncertainty MUST NOT be silently converted into failure or success.

6.10 Annotation fidelity

Whether the human reviewer's judgment faithfully represents the attempt under the written contract.

Annotation fidelity includes:

label support;
rationale support;
criterion attribution;
evidence alignment;
severity alignment;
omitted positive findings;
omitted negative findings;
disagreement type.

7. Document lifecycle

The normative lifecycle is:

Project instructions
 v
Draft contract
 v
Contract review and approval
 v
Versioned case input
 v
Observation and adjudication
 v
Provisional target-specific decisions
 v
Optional bounded human resolution
 v
Final target-specific decisions
 v
Feedback and audit record
 v
Benchmarking, monitoring, and contract revision

A production evaluation MUST reference an approved contract by identifier, semantic version, and content hash.

7.1 Hash canonicalization

A deployment MUST choose and version a canonical hashing procedure. The example package uses canonical JSON v1 for whole-document hashes:

UTF-8 encoding;
object keys sorted lexicographically;
array order preserved;
insignificant whitespace removed;
non-finite numbers forbidden.

Exact text artifacts are hashed as their exact UTF-8 content. Cross-language production systems SHOULD adopt a rigorously specified JSON canonicalization standard when numeric serialization differences are possible. The canonicalization version must be retained with the audit record.

8. Project contract

The project contract is the authoritative, executable interpretation of project instructions.

8.1 Contract identity

The contract MUST contain:

contractId - stable project rubric identity;
contractVersion - semantic version;
schemaVersion - document schema version;
status - draft, approved, deprecated, or retired;
exact instruction documents and content hashes;
owners or responsible roles;
effective date where applicable.

A change to criterion meaning, applicability, anchors, source policy, severity, or decision effect MUST create a new contract version.

8.2 Instruction documents and precedence

Every criterion MUST cite at least one source instruction location.

Instruction documents have an authority class:

primary - normative written instructions;
supplemental - additional normative clarification;
example - illustrative unless explicitly promoted to normative status;
historical - retained for provenance but not current authority.

A project MUST define precedence when instructions can conflict.

Examples MUST NOT silently override written instructions. If an example and a primary instruction conflict, the contract owner MUST resolve the conflict or mark the criterion ambiguous.

8.3 Criterion anatomy

Each criterion MUST contain:

stable criterionId;
human-readable name;
exact requirement;
intent;
criterion type;
scope;
subjectivity class;
instruction references;
applicability rule;
positive proof standard;
failure condition;
omission condition where applicable;
evidence policy;
positive and negative anchors where practical;
valid alternatives;
exceptions;
dependencies and precedence;
default severity;
default decision effect;
tags.

8.4 Criterion types

Required presence

Use when the attempt must contain a behavior, conclusion, component, or analysis.

Pass requires affirmative evidence.

Valid pass: "Attempt span A performs required behavior B."
Invalid pass: "No problem found."

Prohibited presence

Use when a behavior must not appear.

Pass may rely on absence only after the system performs a targeted search over the relevant scope and records sufficient inspection coverage. The system SHOULD record what scope was inspected.

Conditional

Use when a requirement applies only under a condition.

Applicability MUST be adjudicated before the obligation. If applicability is uncertain and decision-changing, the criterion MUST become ambiguous or unknown and be eligible for escalation.

Factual

Use for truth or factual consistency. The project MUST specify whether evaluation is closed-world, open-world, or hybrid.

Grounding

Use for support by permitted evidence. The evaluator MUST link material claims to evidence relationships.

Inferential warrant

Use for whether the conclusion follows at the stated strength from the evidence and project-supplied premises.

Comparative

Use when one candidate, option, or conclusion is compared with another on a named dimension. Candidate order SHOULD be randomized or swapped during evaluation to measure position sensitivity.

Completeness

Use when required concepts, evidence, or components must be covered. The evaluator MUST construct or receive an expected-element map.

Holistic quality

Use for genuinely global qualities such as organization, synthesis, strategic judgment, or usefulness. A holistic criterion MUST still cite concrete findings and SHOULD use approved anchors.

A holistic criterion MUST NOT override an explicit critical factual, grounding, safety, or compliance failure unless the project explicitly makes that compensation permissible.

Mechanically verifiable

Use when a deterministic procedure can decide the condition: schema conformance, counts, exact formats, code execution, arithmetic, duplicate detection, citation identifiers, and similar checks.

Deterministic validators SHOULD be preferred over model judgment for the property they directly verify.

Annotation fidelity

Use to evaluate the human review separately from the attempt.

8.5 Scope

Criterion scope determines the primary object of evaluation:

attempt - the response as a whole;
claim - one or more extracted propositions;
citation - evidence linkage and citation behavior;
annotation - human reviewer output;
global - emergent quality of the whole response;
batch - reviewer, project, or dataset patterns across cases.

A criterion MAY produce findings at more granular spans than its primary scope.

8.6 Subjectivity class

Objective

A deterministic or strongly evidence-constrained judgment with little acceptable interpretive range.

Bounded judgment

A judgment that permits interpretation but has explicit anchors, thresholds, or accepted alternatives.

Holistic expert

A global judgment requiring tacit or domain expertise that cannot be fully reduced to local checks.

Holistic-expert criteria SHOULD have stricter escalation and calibration requirements than objective criteria.

8.7 Applicability

Each criterion has:

a natural-language ruleText readable by humans;
an optional structured condition executable by the system.

The natural-language and structured forms MUST agree. A mismatch is a contract defect.

Applicability states are:

applies;
does_not_apply;
uncertain.

If a criterion does not apply, its verdict MUST be not_applicable, and it MUST NOT trigger a negative operational action.

8.8 Proof policy

Each criterion defines:

whether positive proof is required;
the positive proof standard;
the failure condition;
the omission condition;
whether absence of a violation is sufficient;
whether partial credit is permitted;
whether compensation by other criteria is permitted.

Required-behavior invariant

When positiveProofRequired is true and the criterion verdict is met, the assessment MUST reference at least one confirmed positive finding.

Failure invariant

When the verdict is not_met, the assessment MUST reference at least one confirmed negative or omission finding unless failure is produced by a deterministic validator recorded in the finding source.

Partial-credit invariant

partially_met MUST identify both the satisfied component and the missing or defective component. It MUST NOT be used as a vague confidence hedge.

8.9 Evidence policy

The top-level source policy defines the project evidence universe.

World mode

closed_world - only supplied evidence may support material claims;
open_world - external evidence may be retrieved under source rules;
hybrid - supplied evidence is primary, with limited external verification.

Parametric model knowledge

Model memory SHOULD default to never_evidence. It MAY assist retrieval or hypothesis generation, but it MUST NOT be recorded as verified evidence unless the project explicitly allows and verifies it.

Evidence classes

Every evidence artifact belongs to a project-defined class with:

permission status;
authority rank;
description;
optional freshness and provenance rules.

A criterion may further restrict permitted evidence classes and minimum authority.

Evidence conflicts

The contract MUST state how to handle materially conflicting sources. The evaluator MUST NOT silently choose the evidence that best supports its preferred verdict.

Possible policies include:

report the conflict and lower confidence;
prefer a specified authority hierarchy;
require the attempt to acknowledge the conflict;
escalate when the conflict changes the action.

Source snapshots

Production evidence SHOULD be frozen by exact content hash. Open-world evaluations SHOULD record retrieval time and exact source content used so the decision remains replayable.

8.10 Anchors

Anchors operationalize subjective or easily misunderstood criteria.

A useful anchor contains:

example behavior;
expected verdict;
explanation of why;
boundary notes;
relation to valid alternatives.

A criterion SHOULD include:

at least one clear positive anchor;
at least one clear negative anchor;
one boundary anchor for high-variance subjective axes;
explicit valid alternatives where multiple approaches can pass.

Anchors MUST be treated according to their declared authority. An illustrative reference is not automatically the only correct answer.

8.11 Dependencies and precedence

Dependencies express evaluation order or logical relation. Precedence resolves conflicts.

Examples:

grounding assessment precedes warrant assessment;
a safety prohibition overrides a stylistic preference;
project-specific formatting overrides a generic default;
a later supplemental clarification overrides an older ambiguous example.

Circular precedence MUST be rejected during contract validation.

8.12 Severity and decision effect

Severity describes impact of the finding:

none;
advisory;
minor;
major;
critical.

Decision effect describes the default operational consequence:

none;
accept with note;
rework;
reject;
escalate.

Severity and action MUST remain separate. A major defect may be repairable and therefore trigger rework rather than rejection. A lower-severity integrity anomaly may still trigger escalation if it undermines confidence in the whole case.

8.13 Criterion authoring acceptance tests

Before approval, each criterion SHOULD answer yes to the following:

Can two reviewers identify the same applicability condition?
Is there observable positive evidence for passing?
Is failure distinguishable from insufficient evidence?
Are truth, grounding, and warrant separated where needed?
Are valid alternatives represented?
Is the criterion traceable to a written instruction?
Is severity distinguishable from verdict?
Is compensation explicitly allowed or forbidden?
Can the system identify missing required content?
Does the criterion avoid encoding an unstated stylistic preference?
Does it specify the allowed evidence universe?
Is a domain expert required for difficult cases?
Are examples consistent with the written rule?
Can the decision be audited from recorded spans and evidence?

A criterion that fails these tests SHOULD remain draft.

9. Case input

A case input freezes the evaluation object.

9.1 Task

The task contains:

stable task identifier;
task type;
exact prompt;
task-specific instructions;
expected output format;
content hash;
metadata used by applicability rules.

Task metadata MUST NOT contain hidden evaluator instructions that are unavailable to attempters unless the project explicitly defines them as internal QA rules.

9.2 Evidence bundle

Each evidence artifact contains:

evidence identifier;
evidence class;
title;
exact content;
content hash;
authority rank;
permission status;
capture time;
optional source URI, publication date, and metadata.

The evaluator MUST reject or ignore evidence marked disallowed for the relevant criterion.

9.3 Attempt

The attempt contains:

exact content;
content hash;
submission time;
optional attachments;
optional tool traces.

If the project evaluates process as well as outcome, the contract MUST state which tool traces or intermediate artifacts are normative.

9.4 Human review

The optional human review contains:

reviewer identifier or stable pseudonym;
label;
score where used;
reason codes;
rationale;
criterion annotations;
cited spans;
reviewer confidence;
submission time.

The evaluator SHOULD assess the attempt before revealing the reviewer's final label to the attempt-quality adjudicator. This reduces anchoring. The annotation auditor may then compare the independent attempt assessment with the review.

9.5 Reference answers

References declare a role:

authoritative;
illustrative;
positive anchor;
negative anchor.

An illustrative reference MUST NOT be treated as exhaustive. A reference SHOULD be introduced after independent requirement and evidence analysis when reference anchoring is a material risk.

10. Observation plane

10.1 Integrity checks

Before semantic evaluation, the system SHOULD perform:

schema validation;
hash verification;
required-field checks;
corruption detection;
spam and duplicate checks;
evaluator-instruction injection detection;
citation identifier validation;
prohibited-format checks;
tool-trace integrity checks where applicable.

All candidate and source content MUST be treated as untrusted data. Text inside an attempt or source MUST NOT override the project contract or system instructions.

10.2 Claim map

The evaluator extracts explicit and implied material propositions.

Each claim records:

proposition;
type;
importance;
explicitness;
exact attempt spans;
qualifiers;
cited evidence.

Compound statements SHOULD be split when components can receive different support or truth judgments.

The evaluator SHOULD prioritize central and supporting claims over incidental prose. Incidental claims may still matter when they are dangerous, disallowed, or materially false.

10.3 Expected-element map

The evaluator MUST represent what should have appeared, not only what did appear.

Expected elements may come from:

explicit task requirements;
project criteria;
approved instance-level rubric generation;
domain-expert templates;
declared required evidence or output sections.

Each expected element records:

associated criterion;
description;
required status;
importance;
acceptable alternatives;
observed spans;
coverage status.

A generated expected element not directly supported by project instructions MUST be marked inferred and MUST NOT independently create a hard failure unless the contract permits such inference.

10.4 Evidence relationships

Each relationship links a claim, expected element, or finding to a specific evidence artifact.

Relationship states are:

directly entails;
reasonably supports;
weakly supports;
contradicts;
irrelevant;
insufficient.

The evaluator SHOULD cite exact evidence spans. It MUST record source quality and temporal validity independently from the relationship.

10.5 Findings

Findings are the reusable evidence-linked observations from which criterion assessments are built.

Finding types are:

positive - affirmative evidence of quality or compliance;
negative - present defect or violation;
omission - missing required content;
ambiguity - unresolved applicability, evidence, or interpretation;
counterevidence - evidence challenging another finding;
integrity - corruption, spam, manipulation, or procedural anomaly.

A material finding SHOULD include:

exact target span;
evidence identifiers and spans where applicable;
criterion attribution;
materiality;
severity;
confidence;
producing component;
status.

10.6 Independent observation channels

A robust implementation SHOULD generate at least these channels independently before final criterion adjudication:

Positive prover - searches for concrete satisfaction evidence.
Negative challenger - searches for violations, contradictions, and unsupported claims.
Coverage auditor - searches for omissions and underdeveloped requirements.
Warrant auditor - tests whether conclusions follow at the stated strength.

A separate challenger SHOULD attempt to refute decisive positive or negative findings, especially before autonomous rejection or acceptance.

This separation is a recommended design pattern to test empirically; the benchmark protocol requires an ablation rather than assuming it always helps.

11. Criterion adjudication

A criterion assessment combines findings without discarding them.

11.1 Applicability first

The adjudicator MUST decide applicability before assigning a substantive verdict.

does_not_apply -> not_applicable;
uncertain and decision-changing -> ambiguous or unknown, normally eligible for escalation;
applies -> evaluate proof, failure, and omission evidence.

11.2 Positive and negative evidence together

The adjudicator MUST consider positive findings, negative findings, omissions, counterevidence, and unresolved findings.

It MUST NOT hide positive evidence merely because the overall criterion fails. An otherwise failing attempt may contain valuable strengths useful for feedback or partial credit.

11.3 Verdict meanings

met - the applicable criterion is affirmatively satisfied under its proof standard;
partially_met - a decomposable portion is satisfied, but a material portion is missing or defective;
not_met - confirmed evidence shows failure under the criterion;
ambiguous - more than one interpretation is defensible under the current contract;
unknown - evidence or expertise is insufficient to decide;
not_applicable - the criterion does not apply.

ambiguous is about competing defensible interpretations. unknown is about missing information or capability.

11.4 Grounding, truth, warrant, and relevance substatuses

The assessment stores these statuses separately. An evaluator MUST NOT infer one from another without evidence.

For example:

Grounding: grounded
Truth: unknown
Warrant: overstated
Relevance: relevant

This could describe an answer that faithfully reports an uncertain source but draws a conclusion stronger than that source warrants.

11.5 Global assessment

Global qualities MAY be assessed after local findings are available.

The global evaluator SHOULD receive:

the whole attempt;
criterion findings;
expected-element coverage;
contradiction map;
project anchors.

It SHOULD NOT independently override confirmed local findings.

12. Annotation audit

The annotation audit is performed after, or independently from, the attempt assessment.

12.1 Label support

Does the attempt assessment support the human reviewer's final label?

Possible states:

supported;
partially supported;
unsupported;
reasonable disagreement;
rubric ambiguous;
not applicable.

12.2 Rationale support

Does the rationale logically support the label, and is the rationale itself correct?

A correct label with a wrong rationale MUST be represented as such.

12.3 Criterion attribution

Did the reviewer identify the correct criterion, or apply:

the wrong criterion;
only part of the correct criterion;
an unstated preference?

12.4 Evidence alignment

Did the cited span and evidence support the reviewer's stated reason?

12.5 Severity alignment

Did the reviewer assign a consequence proportionate to the defect?

12.6 Omitted findings

The audit records important positive and negative findings the reviewer missed.

12.7 Disagreement taxonomy

The system SHOULD classify disagreement as one or more of:

observation;
evidence authority;
applicability;
criterion interpretation;
inference;
threshold;
severity;
attention;
expertise;
reasonable disagreement;
rubric defect.

This taxonomy is diagnostic. It SHOULD drive rubric fixes, reviewer calibration, routing, and monitoring.

A model-human disagreement MUST NOT automatically count as reviewer error.

13. Confidence and calibration

13.1 Raw confidence

Raw confidence is the component's internal score or estimate. It is not operationally trustworthy by default.

13.2 Calibrated confidence

Calibrated confidence is empirically mapped to observed correctness for a defined event, criterion family, model configuration, project, and slice.

A calibration profile SHOULD identify:

event being predicted;
model and prompt configuration;
contract version;
benchmark version;
criterion or slice;
method;
sample size;
date;
calibration quality.

13.3 Confidence bands

high - suitable for the configured autonomous action under the project's validated threshold;
medium - useful decision support but not sufficient for the configured high-risk autonomous action;
low - substantial uncertainty;
uncalibrated - no empirical mapping exists.

The words high, medium, and low have no universal numeric meaning. They are assigned by the project's calibration policy.

13.4 Autonomy invariant

An action listed in requiresCalibrationForActions MUST NOT be taken automatically when confidence is uncalibrated. The evaluator must use uncalibratedAction, normally escalation.

13.5 Selective evaluation

Projects SHOULD optimize risk as a function of coverage rather than forcing a verdict on every case.

Coverage = fraction decided automatically
Risk = error rate or cost-weighted loss on automatically decided cases

The benchmark protocol defines required reporting.

14. Decision policy

The decision policy maps criterion state to actions for each declared target.

14.1 Rule requirements

Each rule contains:

target;
identifier;
priority;
human-readable description;
structured match conditions;
action;
terminal flag;
optional explanation template.

Rules MUST be evaluated deterministically from recorded assessment state. Every final decision MUST contain a policy trace showing which rules matched.

14.2 Target isolation

Attempt-quality criteria SHOULD determine the attempt decision. Annotation-fidelity criteria SHOULD determine the annotation decision. The composite-case policy MAY combine both.

An incorrect human review MUST NOT cause a good attempt to be marked bad. It may cause the composite annotated datum to require rework.

14.3 Hard gates

A hard gate MUST identify:

specific criterion or tag;
triggering verdict;
required severity;
required calibrated confidence;
operational action;
whether compensation is forbidden.

14.4 Default action

The default SHOULD be conservative. For an uncalibrated or incomplete new project, the default action SHOULD be escalate rather than accept or reject.

See DECISION_POLICY.md for a reference algorithm.

15. One-interaction human resolution

15.1 Purpose

The human interaction resolves the smallest decisive uncertainty. It is not a duplicate full review.

15.2 Trigger

The system MUST ask a question only when:

the unresolved issue can change an operational decision;
a qualified human can answer it from the displayed context;
the question is bounded;
the project interaction limit has not been exhausted.

15.3 Question content

The request MUST include:

triggering criterion;
exact relevant attempt or review span;
decisive evidence;
unresolved relationship;
bounded options;
an ambiguity option when required;
required expertise tags;
statement of whether the answer is decision-changing.

15.4 Bias control

The provisional overall disposition SHOULD be hidden until the responder answers. The system MAY show criterion-level observations necessary to answer the question.

15.5 Resolution effects

The human answer SHOULD update only the affected criterion or policy premise. The system then reruns the decision policy.

Repeated questions with the same root cause SHOULD generate a proposed contract clarification instead of consuming indefinite human review.

15.6 No useful bounded question

When the issue cannot be resolved by one bounded interaction, the system MUST escalate to the appropriate full review or project owner. It MUST NOT manufacture a low-value question merely to satisfy the one-interaction design.

16. Feedback

Feedback is a separate generated artifact and MUST be validated independently from the verdict when it recommends a repair.

16.1 Audiences

attempter - improve the attempt;
reviewer - improve criterion application and evidence use;
project owner - identify rubric ambiguity, drift, or repeated failure patterns.

16.2 Required properties

Material feedback SHOULD:

identify the criterion;
cite the finding and exact span;
distinguish false, unsupported, irrelevant, incomplete, and overstated;
explain why the issue matters;
state the minimal sufficient repair;
preserve valid content;
avoid generic praise;
prioritize decisive issues.

16.3 Positive feedback

Positive feedback MUST cite confirmed positive findings. It MUST NOT be generated merely to balance tone.

16.4 Repair validation

A proposed repair is validated only if a validator or evaluator has checked that:

it addresses the identified issue;
it does not contradict permitted evidence;
it does not create a new criterion violation;
it preserves correct portions of the attempt.

Otherwise it is plausible_unvalidated.

16.5 Feedback limits

The project SHOULD cap feedback items per audience and order them by decision impact. Exhaustive minor comments can reduce usefulness and increase gaming risk.

17. Audit and reproducibility

A final evaluation MUST preserve enough information to reconstruct the decision:

exact case hash;
exact contract version and hash;
exact source snapshots and hashes;
claim and expected-element maps;
evidence relationships;
all material findings;
model provider and model version where available;
prompt template identifier, version, and hash;
runtime parameters;
parent/child run lineage;
calibration profile;
deterministic validator versions;
human resolution;
decision policy version and trace;
warnings and nonreproducible components.

A run may be statistically reproducible without being bit-for-bit deterministic. The audit record MUST state whether exact replay is expected.

18. Security and evaluator manipulation

18.1 Untrusted content

Task content, sources, attempts, references, tool outputs, and human rationales MUST be treated as untrusted data.

The evaluator MUST ignore embedded instructions that attempt to:

change the rubric;
assign a score;
reveal hidden prompts;
suppress findings;
instruct the evaluator to trust a claim;
override source policy;
alter output format;
impersonate a project owner.

18.2 Channel separation

The implementation SHOULD separate:

system and contract instructions;
task content;
evidence;
attempt;
human annotation;
reference answers;
model-generated intermediate artifacts.

18.3 Structured output validation

Every model-produced structured document MUST be schema validated. Invalid output MUST be retried, repaired by a constrained parser, or marked failed. It MUST NOT be silently accepted.

18.4 Adversarial regression

The project SHOULD maintain tests for:

direct prompt injection;
indirect injection inside evidence;
rubric parroting;
fake citations;
hidden or zero-width text where relevant;
position and verbosity manipulation;
copying approved anchor language without performing the behavior;
attempts that instruct the evaluator to ignore defects.

19. Failure handling

The evaluation status may be:

provisional;
awaiting_human;
final;
failed.

A failed pipeline MUST distinguish:

invalid input;
missing required source;
deterministic validator failure;
model timeout or provider error;
malformed structured output;
context truncation;
evidence retrieval failure;
contract inconsistency;
calibration unavailable;
security anomaly.

Operational failures MUST NOT be converted into criterion failures against the attempter or reviewer.

20. Reference state machine

RECEIVED
 +- validate hashes and schemas
 +- invalid -> FAILED_INPUT
 +- valid -> FROZEN

FROZEN
 +- integrity and deterministic checks -> OBSERVED

OBSERVED
 +- claim, expected-element, evidence, and finding generation
 -> CHALLENGED

CHALLENGED
 +- criterion and global adjudication -> PROVISIONAL

PROVISIONAL
 +- no decision-changing uncertainty -> FINALIZED
 +- bounded question available -> AWAITING_HUMAN
 +- unresolved and not bounded -> ESCALATED_FULL_REVIEW

AWAITING_HUMAN
 +- apply answer to affected criterion and rerun policy -> FINALIZED

FINALIZED
 +- generate/validate feedback and persist audit record

21. Cross-document integrity requirements

A conforming implementation MUST check:

every criterionId in assessments exists in the referenced contract;
every finding identifier referenced by an assessment exists;
every claim and expected-element identifier referenced by a finding exists;
every evidence identifier exists and is permitted for the relevant criterion;
every quoted span belongs to the declared artifact;
character offsets, when present, match the quoted text;
every decision target is declared in the contract;
every triggered rule exists and targets the same decision target;
every calibration profile is compatible with the model, prompt, contract, and criterion slice;
no duplicate identifiers exist within a document;
precedence and dependency graphs are acyclic unless a documented fixed-point policy exists;
final decisions occur after any recorded human response;
composite-case decisions are consistent with the project's composite policy.

The JSON Schemas cannot enforce all referential integrity rules; the application MUST implement them.

22. Conformance profiles

Profile A: Structured evaluator

MUST:

validate schema;
use an approved contract;
create claims and expected elements;
record evidence-linked findings;
produce criterion assessments;
produce target-specific decisions and traces;
preserve audit metadata.

Profile B: Calibrated selective evaluator

Profile A plus:

empirical calibration;
risk-coverage measurement;
abstention/escalation;
locked calibration and test sets;
position/prompt perturbation tests;
challenge-set evaluation;
action-specific launch gates.

Profile C: Governed production evaluator

Profile B plus:

independent annotation audit;
bounded human resolution;
feedback validation;
shadow deployment;
drift monitoring;
replay and rollback;
appeals;
governance controls on reviewer analytics.

23. Minimum viable implementation

The smallest useful implementation should support:

text-only cases;
closed-world evidence;
approved project contract;
deterministic format checks;
claim extraction;
expected-element mapping;
claim-to-evidence relationships;
positive, negative, omission, and warrant findings;
criterion adjudication;
attempt and annotation decisions;
conservative escalation;
evidence-linked feedback;
full version and hash logging.

It should not initially attempt to automate every subjective decision.

24. Contract approval checklist

A contract is ready for benchmark construction when:

all primary instructions are included and hashed;
precedence is explicit;
every criterion is atomic enough to audit;
global qualities remain represented where necessary;
positive proof standards exist;
omissions are detectable;
valid alternatives are recorded;
evidence permissions are explicit;
deterministic checks are identified;
applicability is testable;
severity and action are separated;
hard gates are explicit;
decision targets are declared;
human question types and limits are set;
feedback audiences and repair policy are set;
security and replay policies are approved;
project owner and domain expert have signed off;
no unresolved contradiction remains between instructions and examples.

25. Evaluation output acceptance checklist

An evaluation output is complete when:

input and contract references resolve;
integrity status is recorded;
all material claims are represented;
all required expected elements are represented;
evidence relationships distinguish entailment and source quality;
every met required criterion has positive proof;
every failed criterion has confirmed negative or omission evidence;
counterevidence was considered for decisive findings;
uncertainty is explicit;
annotation quality is assessed separately;
each required target has a decision;
every decision has a policy trace;
any human interaction is bounded and decision-changing;
feedback is linked to findings;
repair validation status is honest;
the audit record is replayable or states why not.