AutoQA Decision Policy Reference

Reference version: 0.1.0

1. Purpose

The decision policy converts evidence-backed criterion assessments into operational actions. It is deliberately separate from semantic evaluation so that:

• project owners can change operational thresholds without rerunning all semantic analysis;
• evidence and criterion verdicts remain inspectable;
• severity does not silently become action;
• attempt, annotation, and composite-case decisions remain independent;
• every action has a deterministic policy trace.

2. Action semantics

Accept

The target is ready for its intended use. All hard requirements are affirmatively satisfied at the required calibrated confidence, and no unresolved major issue remains.

Accept with notes

The target is usable, but one or more advisory or minor findings should be retained. Notes must not hide a major unresolved defect.

Rework

The target is not ready, but the defect is specific and repairable. Feedback should identify the minimal sufficient correction.

Reject

The target has a disqualifying defect under an explicit project rule. Rejection should require strong evidence and calibrated confidence unless the failure is fully deterministic.

Escalate

The system should not decide autonomously. Reasons include:

• uncertain applicability;
• ambiguous instruction interpretation;
• insufficient evidence;
• required domain expertise;
• conflicting high-quality evidence;
• uncalibrated semantic judgment;
• unsupported input type or context range;
• security or integrity anomaly;
• disagreement near a high-cost threshold.

3. Target semantics

Attempt target

Uses attempt-quality criteria. An annotation error must not lower the attempt verdict.

Annotation target

Uses annotation-fidelity criteria. A good attempt can have a bad annotation, and vice versa.

Composite-case target

Represents whether the attempt-plus-annotation record is ready as annotated data. Its policy may require both targets to be acceptable.

Typical mapping:

The exact mapping belongs in the project contract.

4. Inputs

The policy engine consumes only recorded state:

• criterion assessments;
• criterion metadata and tags;
• confirmed findings;
• unresolved findings;
• calibrated confidence;
• annotation disagreement types;
• deterministic validator outcomes;
• human-resolution response where present;
• policy rules and priorities.

It must not independently reinterpret the raw attempt.

5. Rule match semantics

Each rule targets one of:

attempt
annotation
composite_case
batch

A rule may select assessments by:

• criterion identifiers;
• criterion tags;
• verdicts;
• severities;
• default decision effects;
• minimum calibrated confidence;
• presence of unresolved findings;
• annotation disagreement type.

5.1 Multiple selectors

Selectors within one rule use logical AND unless the contract explicitly defines otherwise.

Example:

criterionTags: [hard-quality]
verdicts: [not_met]
severities: [critical]
minimumCalibratedConfidence: 0.95

This matches a criterion only when all four conditions are satisfied.

5.2 Multiple criteria

A project must declare whether a rule over multiple criteria means:

• any matching criterion;
• all listed criteria;
• at least N matching criteria.

Version 0.1.0 recommends the following default:

• negative and escalation rules use any;
• clean-accept rules use all.

The human-readable rule description must make this explicit until a later schema version adds a formal quantifier.

5.3 Rule priority

Rules are evaluated from lowest numeric priority to highest. A terminal matched rule ends evaluation for that target. Nonterminal matches remain in the policy trace and may be superseded by a later terminal rule according to project logic.

5.4 Tie-break order

If multiple nonterminal actions remain, use the contract's tieBreakOrder. A conservative default is:

reject > escalate > rework > accept_with_notes > accept

This is not universally appropriate. For repairable work, a project may prefer escalation or rework over rejection.

6. Required preconditions

Before applying semantic action rules, the policy engine must check:

1. input and contract hashes resolve;
2. required assessment targets exist;
3. no cross-document referential error exists;
4. decisive findings are confirmed;
5. the calibration profile is compatible;
6. required deterministic checks completed;
7. any human response is valid for its request;
8. no unresolved security anomaly invalidates the run.

A failed precondition normally produces escalation or a system failure, not a negative judgment against the attempter.

7. Reference algorithm

function decideTarget(
 target: DecisionTarget,
 contract: ProjectContract,
 evaluation: EvaluationState,
): Decision {
 const applicableRules = contract.decisionPolicy.rules
 .filter((rule) => rule.target === target)
 .sort((a, b) => a.priority - b.priority);

 const trace: PolicyTraceEntry[] = [];
 const matchedActions: OperationalAction[] = [];
 const triggeredRuleIds: string[] = [];

 assertReferentialIntegrity(contract, evaluation);
 assertCalibrationCompatibility(contract, evaluation, target);

 for (const rule of applicableRules) {
 const match = evaluateRuleMatch(rule.match, contract, evaluation, target);

 trace.push({
 ruleId: rule.ruleId,
 matched: match.matched,
 inputs: match.traceInputs,
 result: match.explanation,
 });

 if (!match.matched) continue;

 matchedActions.push(rule.action);
 triggeredRuleIds.push(rule.ruleId);

 if (rule.terminal) {
 return buildDecision(target, rule.action, triggeredRuleIds, trace, evaluation);
 }
 }

 const action = matchedActions.length
 ? resolveByTieBreak(matchedActions, contract.decisionPolicy.tieBreakOrder)
 : contract.decisionPolicy.defaultAction;

 return buildDecision(target, action, triggeredRuleIds, trace, evaluation);
}

The production implementation must also enforce the safeguards below.

8. Safeguards

8.1 Positive-proof safeguard

An attempt or annotation cannot be automatically accepted merely because no failure rule matched.

For every required-presence, completeness, grounding, or positive-quality criterion needed for acceptance:

• applicability must be applies;
• verdict must be met;
• at least one confirmed positive finding must be referenced;
• required expected elements must be complete;
• calibrated confidence must meet the action gate.

8.2 Unresolved-finding safeguard

A decision-changing unresolved finding blocks automatic acceptance or rejection unless the policy explicitly routes it to a qualified human resolution.

8.3 Calibration safeguard

If the chosen action is listed in requiresCalibrationForActions, every decisive semantic assessment must have a compatible calibrated score or band.

Otherwise use uncalibratedAction.

A deterministic validator may be exempt when its correctness has been separately verified and the contract explicitly permits autonomous action from it.

8.4 Counterevidence safeguard

Before autonomous reject or accept, decisive findings should have undergone a counterevidence challenge. A confirmed counterfinding that materially weakens the decision should lower confidence or trigger escalation.

8.5 Noncompensation safeguard

A criterion with allowCompensation = false cannot be offset by strengths on other criteria.

8.6 Annotation isolation safeguard

Annotation-fidelity failures cannot change the attempt target unless the project defines the annotation as part of the attempt itself. They may change the annotation and composite targets.

8.7 Ambiguity safeguard

A criterion marked ambiguous must not be treated as not_met. A criterion marked unknown must not be treated as met.

8.8 System-error safeguard

Provider failures, truncation, malformed output, unsupported media, or retrieval failure must never be counted as attempter defects.

9. Human-resolution policy

9.1 Candidate unresolved issues

The engine identifies unresolved findings that are:

• decision-changing;
• answerable by one qualified human;
• expressible as a bounded choice;
• within the project's allowed question types.

9.2 Select one question

When multiple issues are eligible, prioritize by expected reduction in decision loss:

expected value = loss before answer
 - expected loss after answer
 - interaction cost

Practical ranking factors:

• severity of affected criterion;
• probability the answer changes the target action;
• confidence that the human can resolve it;
• human effort;
• recurrence value for contract improvement.

9.3 Apply response

A response must map to an explicit update such as:

• applicability becomes applies or not applicable;
• one instruction gains precedence;
• one evidence class becomes authoritative;
• a specified inference is permitted or forbidden;
• severity becomes major or minor;
• an alternative is accepted;
• the contract is declared ambiguous.

The policy engine reruns only after the update is recorded in the evaluation state and audit trail.

9.4 Project ambiguity response

When the responder chooses project instruction is ambiguous:

• final target action should normally be escalate;
• the case should not count as reviewer error;
• a contract issue should be created;
• recurring affected cases should be grouped for re-adjudication after clarification.

10. Example policy

10.1 Attempt rules

- target: attempt
 ruleId: reject-critical-grounding
 priority: 1
 match:
 criterionTags: [hard-quality]
 verdicts: [not_met]
 severities: [critical]
 minimumCalibratedConfidence: 0.95
 action: reject
 terminal: true

- target: attempt
 ruleId: rework-major
 priority: 2
 match:
 verdicts: [not_met]
 severities: [major]
 minimumCalibratedConfidence: 0.85
 action: rework
 terminal: false

- target: attempt
 ruleId: escalate-unresolved
 priority: 3
 match:
 hasUnresolvedFindings: true
 action: escalate
 terminal: true

- target: attempt
 ruleId: accept-clean
 priority: 10
 match:
 criterionTags: [required-for-attempt-acceptance]
 verdicts: [met]
 minimumCalibratedConfidence: 0.90
 hasUnresolvedFindings: false
 action: accept
 terminal: true

10.2 Annotation rules

- target: annotation
 ruleId: rework-unfaithful-review
 priority: 1
 match:
 criterionTags: [annotation-fidelity]
 verdicts: [not_met]
 severities: [major, critical]
 minimumCalibratedConfidence: 0.90
 action: rework
 terminal: true

- target: annotation
 ruleId: accept-faithful-review
 priority: 10
 match:
 criterionTags: [annotation-fidelity]
 verdicts: [met]
 minimumCalibratedConfidence: 0.90
 hasUnresolvedFindings: false
 action: accept
 terminal: true

10.3 Composite rules

- target: composite_case
 ruleId: escalate-any-target-uncertain
 priority: 1
 match:
 hasUnresolvedFindings: true
 action: escalate
 terminal: true

- target: composite_case
 ruleId: rework-bad-annotation
 priority: 2
 match:
 criterionIds: [annotation-fidelity]
 verdicts: [not_met]
 severities: [major, critical]
 action: rework
 terminal: true

- target: composite_case
 ruleId: accept-both-clean
 priority: 10
 match:
 criterionTags: [required-for-composite-acceptance]
 verdicts: [met]
 minimumCalibratedConfidence: 0.90
 hasUnresolvedFindings: false
 action: accept
 terminal: true

11. Policy trace requirements

Every decision records:

• target;
• chosen action;
• policy version;
• triggered rules;
• decisive criteria;
• unresolved findings;
• confidence;
• full rule trace;
• human-readable rationale;
• decision time.

A useful policy trace must allow an auditor to answer:

• Which rule was evaluated?
• What criterion state was supplied?
• Did it match?
• Why or why not?
• Which rule determined the action?
• What would have changed the action?

12. Policy testing

The decision engine should have deterministic unit tests for:

• each rule independently;
• tie-break behavior;
• no-rule default;
• uncalibrated confidence;
• not-applicable criteria;
• ambiguous and unknown criteria;
• positive-proof failure;
• missing expected elements;
• counterevidence;
• noncompensable failures;
• annotation isolation;
• composite mapping;
• human response updates;
• invalid rule identifiers;
• contract version mismatch;
• calibration profile mismatch;
• system error versus attempter failure.

Golden policy tests should be run whenever the contract, policy engine, or schema changes.

13. Policy-version changes

Create a new policy version when changing:

• action thresholds;
• rule priority;
• target mapping;
• hard gates;
• tie-break order;
• confidence requirements;
• default action;
• human escalation conditions;
• composite-case admission policy.

A policy-only change does not necessarily require rerunning claim and evidence extraction. It does require replaying the stored criterion state through the new policy and validating the operational effect on a locked set.

14. Initial default

For a new project without sufficient calibration:

Deterministic clean pass/failure: project-defined automatic action
Semantic clear finding: reviewer assistance
Semantic uncalibrated decision: escalate
Subjective disagreement: escalate or advisory
Reviewer performance consequence: adjudicated samples only

The default action should remain escalate until benchmark gates justify selective autonomy.