# AutoQA Foundation - The Question Inventory

**Date:** 2026-07-14
**What this is:** The decision-gating questions that must be answered to build a reliable, useful AutoQA for human-annotated AI training data. Generated by a 7-lens brainstorm (measurement, judge failure modes, rubric ops, human factors, grounding, hybrid workflow, feedback/product), 92 raw questions merged into 9 themes. Themes are ordered by how foundational they are - earlier themes gate later ones.
**Companions:** `01_RESEARCH_ATLAS.md` (what the literature answers), `02_FOUNDATIONS.md` (the resulting philosophy + structure), `atlas/raw_questions.md` (full 92-question inventory with per-question rationale).

Questions marked [A] now have a research-backed answer or strong default (see the atlas); unmarked questions require in-house measurement or a product-owner decision.

---

## 1. Ground truth, validation targets & meta-evaluation

What the AutoQA is measured against, and whether that target is even measurable. Gates every downstream accuracy claim, threshold, and routing rule.

- [A] What is the ground-truth construct - "would an expert consensus panel pass this item?" or "does this item satisfy the written project instructions?" - and when they diverge, which wins? (Instruction-satisfaction lets the AutoQA be MORE valid than the humans it's checked against; consensus-prediction caps validity at human reliability.)
- What is the measured inter-reviewer reliability floor (Krippendorff's alpha on a common item set), and is it high enough for human agreement to serve as a validation target at all? *(Never measured for our population - the variance-attribution pilot is not optional.)*
- [A] What is the minimum viable gold set per project given low failure base rates - and does the arithmetic force pooled/hierarchical validation across projects? *(Answered: PPI-family closed-form sizing; see atlas hybrid-statistical.)*
- [A] What agreement statistic do we standardize on given extreme pass rates - and do we ban raw percent-agreement and single-number "accuracy vs humans" from all reporting?
- What is the empirical false-agreement rate - judge and attempter agreeing but both wrong per expert ground truth - and does that mandate a standing independent expert-audit channel? *(Unmeasured anywhere; requires seeded known-flaw items.)*
- Is reviewer variance actually caused by criteria vagueness, or would perfect rubrics leave most variance intact (expertise/attention)? Does a cheap variance-attribution pilot run before the build order is set? *(Psychometrics gap-fill says: substantial rater-intrinsic residual survives perfect rubrics - but the split must still be measured per population.)*
- [A] What is the explicit ground-truth hierarchy - what outranks what (seeded gold > criterion author > human reviewer > AutoQA?) - and which streams may feed judge calibration?

## 2. The objective-subjective boundary & verdict ontology

Where verifiable fact ends and legitimate judgment begins, and what output types the system may emit. This boundary IS the system's scope.

- [A] Which failure types are objectively verifiable vs irreducibly judgment-laden - and should the AutoQA emit categorically different output types (verdict vs flag-for-human vs confidence-weighted opinion) per class?
- Can we build a reliable axis-triage classifier that sorts a project's evaluation axes into "compile to binary checks" / "anchor with exemplars" / "route to human" buckets before go-live? *(Mechanism exists - per-criterion conformal set width - but needs in-house validation.)*
- [A] Where is the line between a verifiable claim and a legitimate opinion, and what does the system do with opinion-class claims?
- [A] Should the verdict taxonomy include "instructions underdetermine this case" - triggered by measured disagreement, feeding an instruction-gap channel to project owners, exempt from attempter penalty?
- For genuinely contested calls, verdict pluralism ("defensible under reading A, not B") or forced resolution - and can the system distinguish contested-by-nature from contested-because-underspecified?
- [A] Is inter-reviewer disagreement the disease, or partly legitimate signal - and is the target agreement-with-truth or consistency-of-application? Would an AutoQA that enforces consistency launder one arbitrary interpretation into false objectivity?
- If two independently-configured AutoQA instances judge the same items, do they disagree at rates comparable to the human reviewers - indicating the variance lives in criterion ambiguity, not people?

## 3. Grounding machinery: claim decomposition, evidence scope & entailment

The core novel requirement - verifying the attempter's POSITIVE statements against a defined evidence record.

- [A] At what claim-decomposition granularity does judge agreement stop improving, and does over-decomposition degrade it via checklist myopia? *(Answered: granularity must be adaptive - atomic decomposition measurably degrades strong verifiers.)*
- What is the aggregation rule from claim-level verdicts to item-level pass/fail - are severity/weight taxonomies mandatory per project, or does the system emit claim-level findings plus a recommended verdict?
- [A] What is the evidence closure - attempter-cited spans only, full item context, or open world - and how do we stop the judge from "verifying" against its own priors? Must groundedness verdicts quote span-level evidence to be admissible?
- [A] What entailment standard must an attempter statement meet - strict entailment (fails nearly all human prose) or reasonable-expert support (reintroduces subjectivity) - and is the output 3-way (entailed/contradicted/unsupported) or graded?
- [A] What claim ontology does verification need - grounding claims, entailment claims, calibration claims (stated severity matches evidence strength) - with different machinery and different reliability ceilings per type?
- Can the judge distinguish warranted from empty praise at usable precision/recall per praise type on a seeded set? *(Zero published evidence anywhere - this is a from-scratch in-house measurement.)*
- [A] Do universal/absence claims ("no errors present," "fully addresses the prompt") require independent re-attempt - and if the judge's seeded-defect recall isn't materially above the attempter population's, are such verdicts labeled "not-contradicted" rather than "verified"?
- Is omission detection tractable project-agnostically, or is it a customization-layer plugin requiring a per-project defect taxonomy? *(AbsenceBench says: structurally weakest judge lane; convert completeness criteria to enumerable checklists.)*
- [A] When cited evidence supports the conclusion but uncited in-scope evidence undermines it - is cherry-picking a grounding failure or an omission, and does verification read cited-only or full-scope?
- Should attempters be required to attach explicit evidence pointers to every load-bearing claim - converting open verification into closed entailment - and what does piloting show about citation-padding?
- What is the base-rate split between attempter claims that are actually false vs merely uncited-but-true - and at what judge precision does flagging "unsupported" beat spending the human touch elsewhere?
- Contrarian: is grounding verification (including re-finding defects to check "clean" verdicts) actually less capability-demanding than doing the annotation - and if not, what is the genuinely verification-easier subset?

## 4. Rubric compilation & the per-project customization contract

How project instructions become executable judge configuration - the load-bearing claim behind "project-agnostic."

- [A] What is the minimal artifact set an instruction set must compile into - checkable claims, positive/negative exemplars, counter-exemplars, edge-case adjudications, forbidden-inference rules?
- [A] Can one compilation procedure hit target fidelity across instruction-set styles, or is ingestion a structured authoring workshop with tooling rather than an automated pipeline? *(Answered: instruction-only auto-compilation leaves a ~22-28pt fidelity gap; expert grounding artifacts are structurally required.)*
- What exactly is the per-project customization surface as a closed config schema - proven falsifiably by onboarding a new project through config alone, with zero prompt-engineering of the core judge?
- How many expert-adjudicated exemplars per criterion before judge-human agreement plateaus (5 vs 50 vs 500) - and does the answer quietly reintroduce the expensive aligned-expert dependency? *(One datapoint: 5-shot -> 80% on one dataset; exemplar QUALITY - hard contrastive near-miss pairs - appears to dominate count.)*
- Who is the authority when the compiled rubric and the project owner's case-by-case intuitions diverge - rubric-as-law or rubric-as-guidance - and does an amendment retroactively re-score past verdicts?
- [A] Does constructive feedback require compile-time authoring ("what good looks like," repair templates, doctrine constraints), or can it be generated at verdict time from the claim-evidence trace alone?

## 5. Judge reliability, bias & competence limits

The judge's own failure modes measured against the human baseline it augments - if the judge's variance or bias exceeds the humans', we've automated the problem.

- [A] What is the run-to-run verdict flip rate on identical items - noise to vote away (k-sampling) or signal that the criterion is underspecified (rubric rewriting)? *(Answered: both, by design - vote AND route high-entropy items to the contested class.)*
- [A] Does the judge exhibit agreement-default/sycophancy bias on positive claims - passing plausible-but-ungrounded claims at a higher rate than it fails grounded ones? *(FACTS: positive-class ~85 F1 vs negative-class ~46 F1 - yes, quantified, large.)*
- [A] Does the judge favor verbose, hedged, rubric-vocabulary-mimicking writeups over terse-but-correct ones - and do we need canonicalization/style-stripping before judgment?
- [A] When the attempter critiques output from the judge's own model family, does self-preference suppress agreement with valid criticisms - forcing cross-family judge routing? *(Preference leakage: up to 28.7pp, prompt-level debiasing fails.)*
- [A] How sensitive are verdicts to semantically-neutral prompt perturbations - and is a perturbation-robustness harness a mandatory ship gate per project config?
- For each axis type, what is the judge-vs-adjudicated-human agreement ceiling - and on which axes does the judge fall below the human inter-rater baseline, forcing lanes? *(Must be assembled per project; published anchors exist only for grounding.)*
- What is the judge's blind-spot profile on expertise-heavy content - can it detect its own incompetence and abstain, and are some projects AutoQA-ineligible?
- [A] Does a multi-model ensemble buy error reduction, or do frontier judges share correlated blind spots - such that cross-family disagreement should be an escalation signal rather than a vote?
- Is judge-judge agreement on grounding verdicts actually higher than the human reviewer-reviewer agreement we're fixing - as a falsifiable go/no-go gate before deploying at all?

## 6. Human-budget allocation, routing & arbitration

With at most one human touch per item, routing IS the system.

- [A] Is one-human-interaction-per-item even the right target - would zero per-item touch plus randomized deep blind audits outperform one shallow, automation-biased touch? *(Fork resolved by evidence + law: see FOUNDATIONS Section 5 - consequential adverse verdicts require a meaningful human decision for EU attempters; measurement is best served by sampled audits.)*
- [A] What is the single interaction spent on - blind re-judgment, verdict verification, targeted micro-adjudication of contested claims, or the attempter's contest? *(Evidence-only assisted adjudication of the routed slice wins; showing the AI verdict causes measured over-reliance.)*
- [A] Where in the verdict distribution does the touch buy most reliability - and must routing be closed-loop on observed overturn rates?
- [A] Should scarce human minutes go to per-item adjudication at all, versus upstream compounding work (rubric disambiguation, exemplar-generating edge-case adjudications that write back into config)?
- [A] Does escalation route the whole item or the specific contested claim/criterion?
- Are escalation queues organized by criterion (specialist reviewers) or by item (generalist) - and does specialist routing measurably close the expertise-gap root cause?
- [A] Does the escalated human judge blind or anchored - and at what point does a falling overturn rate signal rubber-stamping?
- [A] Is the escalation threshold set to a fixed human-capacity budget or a target residual error rate? *(Both exist as dial profiles: error-target mode outputs the budget; pace-bound mode outputs the reported residual error - FOUNDATIONS P11.)*
- [A] When contractual pace makes the standard threshold unaffordable, what may the quality/leniency dial legitimately move (aggregation cutoffs, escalation rate, verification depth, enforcement disposition) - and what may it never touch (judge perception, integrity verdicts, seeded gold/audit channels, the legal floor)?
- [A] Is the dial one-sided by construction - dispositions loosen while measurement persists, so an expedited batch can be re-filtered to any strictness later from recorded scores (attempter-disposition vs dataset-admission consumers)?
- [A] How do the Goodhart/divergence sensors condition on the active dial setting, so a leniency period is not misread as attempter improvement (or a strictness change as attempter decay)?
- Who has authority to change a project's active operating profile, is every change logged, prospective-only, and announced to attempters - and is the stamped residual-error estimate disclosed to the data buyer? *(product-owner decision - FOUNDATIONS Section 8.6.)*
- Which confidence proxy (stated confidence, k-sample agreement, paraphrase flip rate, ensemble disagreement, conformal set width, rationale-entailment score) actually predicts human overturn at routable AUROC? *(Never benchmarked head-to-head - in-house measurement.)*
- [A] What is the asymmetric, cost-weighted decision rule for auto-pass / auto-fail / escalate - is auto-FAIL held to a stricter bar than auto-PASS, and who bears each error's cost?
- [A] When human and AutoQA persistently disagree, what arbitrates - and how do we prevent the override stream from becoming unvalidated calibration data?
- [A] What statistical model fuses sparse human labels with dense AI judgments? *(PPI++/EIF for rates and dashboards; TPR/FPR-corrected tests for threshold certification.)*
- What is the cold-start protocol for a zero-label project, and what quantitative graduation criterion flips it into confidence-routed mode?

## 7. Constructive feedback: generation, grounding & efficacy

The feedback channel held to its own standards - and falsifiable by attempter behavior change.

- [A] Is the judge's feedback held reflexively to the same grounding standard it enforces - every criticism citing entailing evidence, passing its own verification gate before delivery - and what false-accusation precision SLO keeps attempter trust?
- [A] Must every feedback statement be traceable to a specific clause in the project instructions - and what happens to real problems that map to no written criterion?
- [A] What is the minimal feedback unit attempters actually act on - e.g., the 4-tuple (quoted claim, evidence span, criterion clause, concrete fix)? *(Feedback science: high-information task-referenced feedback ~ doubles effect vs bare verdicts; person/rank-referenced feedback backfires.)*
- [A] What is the explainability contract for a failing verdict - verbatim evidence spans, checkable reasoning chain, fail-closed if the bar can't be met? *(Also a legal requirement for EU attempters: plain-language explanation + human review on request.)*
- Does feedback need a different confidence bar than the verdict - can the system ship coaching on auto-passed items, and what is the harm model when a feedback claim is itself wrong?
- [A] Does feedback measurably change attempter behavior - what longitudinal metric do we commit to, and after how long with no movement do we conclude decoration and reallocate? *(Priors now exist: median modest, concentrated in low performers, ~1/4 act on it; instrument per-attempter pre/post error-rate deltas and expect backfire in a minority.)*

## 8. Adversarial dynamics, incentives & consequences

The system operates on paid workers who adapt to it. The moment verdicts carry consequences, the system is adversarial by construction.

- Does feedback leak the judge's decision boundary - after N cycles, do attempter scores rise on the AutoQA while staying flat on held-out human-graded golden items? *(The single most important standing sensor; no published precedent - must build.)*
- [A] Which shortcut heuristics does the judge use as pass/fail proxies, can pay-motivated attempters reverse-engineer them blind, and is the defense designed in from day one? *(Universal adversarial phrases transfer to unseen judges - filtering rationales is insufficient alone.)*
- [A] Can the judge distinguish genuine grounding from citation theater - if we perturb or swap the cited evidence, does the verdict flip?
- [A] How do we prevent feedback-induced monoculture - annotations converging on judge-pleasing patterns that destroy the diversity the training data exists to capture - and should feedback ever include stylistic guidance? *(Default: no stylistic guidance; population-level diversity drift metric from day one.)*
- [A] For wholesale LLM-generated submissions, is the defense content-based detection or economics-based gating? *(Resolved: economics/provenance-based; content detection is beaten - agent passes 99.8% of checks at $0.05/task.)*
- Do we care about LLM-assistance provenance or only content validity - is a correct, well-grounded, heavily-LLM-assisted annotation a pass, a fail, or per-project policy? *(product-owner/per-project decision; process telemetry has legal constraints - see FOUNDATIONS Section 10.)*
- [A] Are AutoQA verdicts advisory or enforcement-grade - and if consequences attach, what precision floor, audit trail, and contest rights does each verdict need? *(Enforcement-grade adverse verdicts trigger the legal floor: human decision-maker, plain-language explanation, appeal with SLA.)*
- [A] What appeal mechanism absorbs disputes without re-importing full human review load - and what appeal-overturn band signals a healthy vs broken judge?

## 9. Drift, monitoring & lifecycle integrity

Keeping the certificate valid after launch.

- [A] How do we detect drift in three distinct moving parts - attempter adaptation, instruction revisions, judge-model updates - with a different sensor and response for each? *(Frozen human-labeled anchor set + anytime-valid attribution separates "judge moved" from "population moved.")*
- What falsifiable signal triggers a rubric revision (per-axis disagreement trend, override cluster density, appeal rate) at what threshold? *(No published drift-rate parameters; instrument and learn.)*
- [A] Are judge model versions pinned per project with golden-set regression gates on upgrade?
- [A] Once a project runs trusted-standalone, what randomized audit rate and drift statistic triggers automatic de-graduation - and is audit sampling unpredictable enough that attempters cannot write for the judge?
- [A] Does the AutoQA prefilter shift the human reviewer's base rates enough to cause induced criterion drift - and does queue design mandate seeded known-verdict items and easy-case dilution? *(Content-moderation practice: yes, both are standard; seeded items must be indistinguishable.)*
- [A] What keeps the calibration workforce competent once the AutoQA writes verdicts - does reviewer skill decay under reconcile-only work, and is a standing blind-judgment quota mandatory? *(Psychometrics: rater calibration decays within a single session; rolling-window reviewer-effect estimates, not one-time certification.)*

---

## Cross-cutting tensions

Places where two legitimate goals pull in opposite directions. These are managed, not solved; FOUNDATIONS states the default posture for each.

1. **Standardization vs legitimate disagreement.** Enforcing consistent verdicts kills the reviewer-variance problem but risks laundering one arbitrary reading of a contested criterion into false objectivity; admitting ambiguity classes preserves legitimate disagreement but can become an excuse bucket.
2. **Decomposition vs holistic validity.** Atomic claim-level checks are tractable, auditable, falsifiable - and strip context (checklist myopia), re-importing subjectivity through the aggregation rule.
3. **Feedback usefulness vs Goodhart resistance.** The more specific and actionable the feedback, the more it publishes a gradient toward the judge's decision boundary that paid attempters will descend.
4. **Human as ground truth vs human as noise source.** The system exists because human reviewers are unreliable, yet escalation, overrides, appeals, and calibration all quietly treat a single human's judgment as truth. Every human touch chooses between efficient anchored review (contaminated labels) and expensive blind independence (usable calibration signal).
5. **Per-item accuracy vs compounding upstream investment.** The touch spent adjudicating this item maximizes per-item quality; the same minutes spent on rubric disambiguation improve all future items - but leave today's items unconfirmed.
6. **Grounding strictness vs false-flag rate.** Strict entailment fails nearly all human prose (which compresses and generalizes); loose "reasonable support" reintroduces the subjectivity the system was built to remove.
7. **Project-agnostic foundation vs per-project fidelity.** A closed config surface keeps "agnostic" falsifiable - but compilation fidelity, judge domain competence, and defect taxonomies constantly pressure the boundary.
8. **Enforcement weight vs error tolerance and trust.** Consequential verdicts create real deterrence and collapse acceptable error rates (and trigger legal duties); advisory verdicts preserve trust and tolerate error but lose incentive force.
9. **Contractual pace vs quality floor** *(added 2026-07-14 from the product owner's throughput requirement)*. Delivery obligations sometimes demand a looser operating point than the standard threshold; refusing any dial forces the loosening to happen off the books, while a dial that quietly loosens *measurement* (not just disposition) destroys the system's evidentiary value. Managed by pricing leniency: dispositions move, measurement doesn't, and every batch ships stamped with its profile and estimated residual error (FOUNDATIONS P11).