The Question Inventory - AutoQA Foundation

Date: 2026-07-14 What this is: The decision-gating questions that must be answered to build a reliable, useful AutoQA for human-annotated AI training data. Generated by a 7-lens brainstorm (measurement, judge failure modes, rubric ops, human factors, grounding, hybrid workflow, feedback/product), 92 raw questions merged into 9 themes. Themes are ordered by how foundational they are - earlier themes gate later ones. Companions: 01_RESEARCH_ATLAS.md (what the literature answers), 02_FOUNDATIONS.md (the resulting philosophy + structure), atlas/raw_questions.md (full 92-question inventory with per-question rationale).

Questions marked [A] now have a research-backed answer or strong default (see the atlas); unmarked questions require in-house measurement or a product-owner decision.

1. Ground truth, validation targets & meta-evaluation

What the AutoQA is measured against, and whether that target is even measurable. Gates every downstream accuracy claim, threshold, and routing rule.

[A] What is the ground-truth construct - "would an expert consensus panel pass this item?" or "does this item satisfy the written project instructions?" - and when they diverge, which wins? (Instruction-satisfaction lets the AutoQA be MORE valid than the humans it's checked against; consensus-prediction caps validity at human reliability.)
What is the measured inter-reviewer reliability floor (Krippendorff's alpha on a common item set), and is it high enough for human agreement to serve as a validation target at all? (Never measured for our population - the variance-attribution pilot is not optional.)
[A] What is the minimum viable gold set per project given low failure base rates - and does the arithmetic force pooled/hierarchical validation across projects? (Answered: PPI-family closed-form sizing; see atlas hybrid-statistical.)
[A] What agreement statistic do we standardize on given extreme pass rates - and do we ban raw percent-agreement and single-number "accuracy vs humans" from all reporting?
What is the empirical false-agreement rate - judge and attempter agreeing but both wrong per expert ground truth - and does that mandate a standing independent expert-audit channel? (Unmeasured anywhere; requires seeded known-flaw items.)
Is reviewer variance actually caused by criteria vagueness, or would perfect rubrics leave most variance intact (expertise/attention)? Does a cheap variance-attribution pilot run before the build order is set? (Psychometrics gap-fill says: substantial rater-intrinsic residual survives perfect rubrics - but the split must still be measured per population.)
[A] What is the explicit ground-truth hierarchy - what outranks what (seeded gold > criterion author > human reviewer > AutoQA?) - and which streams may feed judge calibration?

2. The objective-subjective boundary & verdict ontology

Where verifiable fact ends and legitimate judgment begins, and what output types the system may emit. This boundary IS the system's scope.

[A] Which failure types are objectively verifiable vs irreducibly judgment-laden - and should the AutoQA emit categorically different output types (verdict vs flag-for-human vs confidence-weighted opinion) per class?
Can we build a reliable axis-triage classifier that sorts a project's evaluation axes into "compile to binary checks" / "anchor with exemplars" / "route to human" buckets before go-live? (Mechanism exists - per-criterion conformal set width - but needs in-house validation.)
[A] Where is the line between a verifiable claim and a legitimate opinion, and what does the system do with opinion-class claims?
[A] Should the verdict taxonomy include "instructions underdetermine this case" - triggered by measured disagreement, feeding an instruction-gap channel to project owners, exempt from attempter penalty?
For genuinely contested calls, verdict pluralism ("defensible under reading A, not B") or forced resolution - and can the system distinguish contested-by-nature from contested-because-underspecified?
[A] Is inter-reviewer disagreement the disease, or partly legitimate signal - and is the target agreement-with-truth or consistency-of-application? Would an AutoQA that enforces consistency launder one arbitrary interpretation into false objectivity?
If two independently-configured AutoQA instances judge the same items, do they disagree at rates comparable to the human reviewers - indicating the variance lives in criterion ambiguity, not people?

3. Grounding machinery: claim decomposition, evidence scope & entailment

The core novel requirement - verifying the attempter's POSITIVE statements against a defined evidence record.

[A] At what claim-decomposition granularity does judge agreement stop improving, and does over-decomposition degrade it via checklist myopia? (Answered: granularity must be adaptive - atomic decomposition measurably degrades strong verifiers.)
What is the aggregation rule from claim-level verdicts to item-level pass/fail - are severity/weight taxonomies mandatory per project, or does the system emit claim-level findings plus a recommended verdict?
[A] What is the evidence closure - attempter-cited spans only, full item context, or open world - and how do we stop the judge from "verifying" against its own priors? Must groundedness verdicts quote span-level evidence to be admissible?
[A] What entailment standard must an attempter statement meet - strict entailment (fails nearly all human prose) or reasonable-expert support (reintroduces subjectivity) - and is the output 3-way (entailed/contradicted/unsupported) or graded?
[A] What claim ontology does verification need - grounding claims, entailment claims, calibration claims (stated severity matches evidence strength) - with different machinery and different reliability ceilings per type?
Can the judge distinguish warranted from empty praise at usable precision/recall per praise type on a seeded set? (Zero published evidence anywhere - this is a from-scratch in-house measurement.)
[A] Do universal/absence claims ("no errors present," "fully addresses the prompt") require independent re-attempt - and if the judge's seeded-defect recall isn't materially above the attempter population's, are such verdicts labeled "not-contradicted" rather than "verified"?
Is omission detection tractable project-agnostically, or is it a customization-layer plugin requiring a per-project defect taxonomy? (AbsenceBench says: structurally weakest judge lane; convert completeness criteria to enumerable checklists.)
[A] When cited evidence supports the conclusion but uncited in-scope evidence undermines it - is cherry-picking a grounding failure or an omission, and does verification read cited-only or full-scope?
Should attempters be required to attach explicit evidence pointers to every load-bearing claim - converting open verification into closed entailment - and what does piloting show about citation-padding?
What is the base-rate split between attempter claims that are actually false vs merely uncited-but-true - and at what judge precision does flagging "unsupported" beat spending the human touch elsewhere?
Contrarian: is grounding verification (including re-finding defects to check "clean" verdicts) actually less capability-demanding than doing the annotation - and if not, what is the genuinely verification-easier subset?

4. Rubric compilation & the per-project customization contract

How project instructions become executable judge configuration - the load-bearing claim behind "project-agnostic."

[A] What is the minimal artifact set an instruction set must compile into - checkable claims, positive/negative exemplars, counter-exemplars, edge-case adjudications, forbidden-inference rules?
[A] Can one compilation procedure hit target fidelity across instruction-set styles, or is ingestion a structured authoring workshop with tooling rather than an automated pipeline? (Answered: instruction-only auto-compilation leaves a ~22-28pt fidelity gap; expert grounding artifacts are structurally required.)
What exactly is the per-project customization surface as a closed config schema - proven falsifiably by onboarding a new project through config alone, with zero prompt-engineering of the core judge?
How many expert-adjudicated exemplars per criterion before judge-human agreement plateaus (5 vs 50 vs 500) - and does the answer quietly reintroduce the expensive aligned-expert dependency? (One datapoint: 5-shot -> 80% on one dataset; exemplar QUALITY - hard contrastive near-miss pairs - appears to dominate count.)
Who is the authority when the compiled rubric and the project owner's case-by-case intuitions diverge - rubric-as-law or rubric-as-guidance - and does an amendment retroactively re-score past verdicts?
[A] Does constructive feedback require compile-time authoring ("what good looks like," repair templates, doctrine constraints), or can it be generated at verdict time from the claim-evidence trace alone?

5. Judge reliability, bias & competence limits

The judge's own failure modes measured against the human baseline it augments - if the judge's variance or bias exceeds the humans', we've automated the problem.

[A] What is the run-to-run verdict flip rate on identical items - noise to vote away (k-sampling) or signal that the criterion is underspecified (rubric rewriting)? (Answered: both, by design - vote AND route high-entropy items to the contested class.)
[A] Does the judge exhibit agreement-default/sycophancy bias on positive claims - passing plausible-but-ungrounded claims at a higher rate than it fails grounded ones? (FACTS: positive-class ~85 F1 vs negative-class ~46 F1 - yes, quantified, large.)
[A] Does the judge favor verbose, hedged, rubric-vocabulary-mimicking writeups over terse-but-correct ones - and do we need canonicalization/style-stripping before judgment?
[A] When the attempter critiques output from the judge's own model family, does self-preference suppress agreement with valid criticisms - forcing cross-family judge routing? (Preference leakage: up to 28.7pp, prompt-level debiasing fails.)
[A] How sensitive are verdicts to semantically-neutral prompt perturbations - and is a perturbation-robustness harness a mandatory ship gate per project config?
For each axis type, what is the judge-vs-adjudicated-human agreement ceiling - and on which axes does the judge fall below the human inter-rater baseline, forcing lanes? (Must be assembled per project; published anchors exist only for grounding.)
What is the judge's blind-spot profile on expertise-heavy content - can it detect its own incompetence and abstain, and are some projects AutoQA-ineligible?
[A] Does a multi-model ensemble buy error reduction, or do frontier judges share correlated blind spots - such that cross-family disagreement should be an escalation signal rather than a vote?
Is judge-judge agreement on grounding verdicts actually higher than the human reviewer-reviewer agreement we're fixing - as a falsifiable go/no-go gate before deploying at all?

6. Human-budget allocation, routing & arbitration

With at most one human touch per item, routing IS the system.

[A] Is one-human-interaction-per-item even the right target - would zero per-item touch plus randomized deep blind audits outperform one shallow, automation-biased touch? (Fork resolved by evidence + law: see FOUNDATIONS Section 5 - consequential adverse verdicts require a meaningful human decision for EU attempters; measurement is best served by sampled audits.)
[A] What is the single interaction spent on - blind re-judgment, verdict verification, targeted micro-adjudication of contested claims, or the attempter's contest? (Evidence-only assisted adjudication of the routed slice wins; showing the AI verdict causes measured over-reliance.)
[A] Where in the verdict distribution does the touch buy most reliability - and must routing be closed-loop on observed overturn rates?
[A] Should scarce human minutes go to per-item adjudication at all, versus upstream compounding work (rubric disambiguation, exemplar-generating edge-case adjudications that write back into config)?
[A] Does escalation route the whole item or the specific contested claim/criterion?
Are escalation queues organized by criterion (specialist reviewers) or by item (generalist) - and does specialist routing measurably close the expertise-gap root cause?
[A] Does the escalated human judge blind or anchored - and at what point does a falling overturn rate signal rubber-stamping?
[A] Is the escalation threshold set to a fixed human-capacity budget or a target residual error rate? (Both exist as dial profiles: error-target mode outputs the budget; pace-bound mode outputs the reported residual error - FOUNDATIONS P11.)
[A] When contractual pace makes the standard threshold unaffordable, what may the quality/leniency dial legitimately move (aggregation cutoffs, escalation rate, verification depth, enforcement disposition) - and what may it never touch (judge perception, integrity verdicts, seeded gold/audit channels, the legal floor)?
[A] Is the dial one-sided by construction - dispositions loosen while measurement persists, so an expedited batch can be re-filtered to any strictness later from recorded scores (attempter-disposition vs dataset-admission consumers)?
[A] How do the Goodhart/divergence sensors condition on the active dial setting, so a leniency period is not misread as attempter improvement (or a strictness change as attempter decay)?
Who has authority to change a project's active operating profile, is every change logged, prospective-only, and announced to attempters - and is the stamped residual-error estimate disclosed to the data buyer? (product-owner decision - FOUNDATIONS Section 8.6.)
Which confidence proxy (stated confidence, k-sample agreement, paraphrase flip rate, ensemble disagreement, conformal set width, rationale-entailment score) actually predicts human overturn at routable AUROC? (Never benchmarked head-to-head - in-house measurement.)
[A] What is the asymmetric, cost-weighted decision rule for auto-pass / auto-fail / escalate - is auto-FAIL held to a stricter bar than auto-PASS, and who bears each error's cost?
[A] When human and AutoQA persistently disagree, what arbitrates - and how do we prevent the override stream from becoming unvalidated calibration data?
[A] What statistical model fuses sparse human labels with dense AI judgments? (PPI++/EIF for rates and dashboards; TPR/FPR-corrected tests for threshold certification.)
What is the cold-start protocol for a zero-label project, and what quantitative graduation criterion flips it into confidence-routed mode?

7. Constructive feedback: generation, grounding & efficacy

The feedback channel held to its own standards - and falsifiable by attempter behavior change.

[A] Is the judge's feedback held reflexively to the same grounding standard it enforces - every criticism citing entailing evidence, passing its own verification gate before delivery - and what false-accusation precision SLO keeps attempter trust?
[A] Must every feedback statement be traceable to a specific clause in the project instructions - and what happens to real problems that map to no written criterion?
[A] What is the minimal feedback unit attempters actually act on - e.g., the 4-tuple (quoted claim, evidence span, criterion clause, concrete fix)? (Feedback science: high-information task-referenced feedback ~ doubles effect vs bare verdicts; person/rank-referenced feedback backfires.)
[A] What is the explainability contract for a failing verdict - verbatim evidence spans, checkable reasoning chain, fail-closed if the bar can't be met? (Also a legal requirement for EU attempters: plain-language explanation + human review on request.)
Does feedback need a different confidence bar than the verdict - can the system ship coaching on auto-passed items, and what is the harm model when a feedback claim is itself wrong?
[A] Does feedback measurably change attempter behavior - what longitudinal metric do we commit to, and after how long with no movement do we conclude decoration and reallocate? (Priors now exist: median modest, concentrated in low performers, ~1/4 act on it; instrument per-attempter pre/post error-rate deltas and expect backfire in a minority.)

8. Adversarial dynamics, incentives & consequences

The system operates on paid workers who adapt to it. The moment verdicts carry consequences, the system is adversarial by construction.

Does feedback leak the judge's decision boundary - after N cycles, do attempter scores rise on the AutoQA while staying flat on held-out human-graded golden items? (The single most important standing sensor; no published precedent - must build.)
[A] Which shortcut heuristics does the judge use as pass/fail proxies, can pay-motivated attempters reverse-engineer them blind, and is the defense designed in from day one? (Universal adversarial phrases transfer to unseen judges - filtering rationales is insufficient alone.)
[A] Can the judge distinguish genuine grounding from citation theater - if we perturb or swap the cited evidence, does the verdict flip?
[A] How do we prevent feedback-induced monoculture - annotations converging on judge-pleasing patterns that destroy the diversity the training data exists to capture - and should feedback ever include stylistic guidance? (Default: no stylistic guidance; population-level diversity drift metric from day one.)
[A] For wholesale LLM-generated submissions, is the defense content-based detection or economics-based gating? (Resolved: economics/provenance-based; content detection is beaten - agent passes 99.8% of checks at $0.05/task.)
Do we care about LLM-assistance provenance or only content validity - is a correct, well-grounded, heavily-LLM-assisted annotation a pass, a fail, or per-project policy? (product-owner/per-project decision; process telemetry has legal constraints - see FOUNDATIONS Section 10.)
[A] Are AutoQA verdicts advisory or enforcement-grade - and if consequences attach, what precision floor, audit trail, and contest rights does each verdict need? (Enforcement-grade adverse verdicts trigger the legal floor: human decision-maker, plain-language explanation, appeal with SLA.)
[A] What appeal mechanism absorbs disputes without re-importing full human review load - and what appeal-overturn band signals a healthy vs broken judge?

9. Drift, monitoring & lifecycle integrity

Keeping the certificate valid after launch.

[A] How do we detect drift in three distinct moving parts - attempter adaptation, instruction revisions, judge-model updates - with a different sensor and response for each? (Frozen human-labeled anchor set + anytime-valid attribution separates "judge moved" from "population moved.")
What falsifiable signal triggers a rubric revision (per-axis disagreement trend, override cluster density, appeal rate) at what threshold? (No published drift-rate parameters; instrument and learn.)
[A] Are judge model versions pinned per project with golden-set regression gates on upgrade?
[A] Once a project runs trusted-standalone, what randomized audit rate and drift statistic triggers automatic de-graduation - and is audit sampling unpredictable enough that attempters cannot write for the judge?
[A] Does the AutoQA prefilter shift the human reviewer's base rates enough to cause induced criterion drift - and does queue design mandate seeded known-verdict items and easy-case dilution? (Content-moderation practice: yes, both are standard; seeded items must be indistinguishable.)
[A] What keeps the calibration workforce competent once the AutoQA writes verdicts - does reviewer skill decay under reconcile-only work, and is a standing blind-judgment quota mandatory? (Psychometrics: rater calibration decays within a single session; rolling-window reviewer-effect estimates, not one-time certification.)

Cross-cutting tensions

Places where two legitimate goals pull in opposite directions. These are managed, not solved; FOUNDATIONS states the default posture for each.

Standardization vs legitimate disagreement. Enforcing consistent verdicts kills the reviewer-variance problem but risks laundering one arbitrary reading of a contested criterion into false objectivity; admitting ambiguity classes preserves legitimate disagreement but can become an excuse bucket.
Decomposition vs holistic validity. Atomic claim-level checks are tractable, auditable, falsifiable - and strip context (checklist myopia), re-importing subjectivity through the aggregation rule.
Feedback usefulness vs Goodhart resistance. The more specific and actionable the feedback, the more it publishes a gradient toward the judge's decision boundary that paid attempters will descend.
Human as ground truth vs human as noise source. The system exists because human reviewers are unreliable, yet escalation, overrides, appeals, and calibration all quietly treat a single human's judgment as truth. Every human touch chooses between efficient anchored review (contaminated labels) and expensive blind independence (usable calibration signal).
Per-item accuracy vs compounding upstream investment. The touch spent adjudicating this item maximizes per-item quality; the same minutes spent on rubric disambiguation improve all future items - but leave today's items unconfirmed.
Grounding strictness vs false-flag rate. Strict entailment fails nearly all human prose (which compresses and generalizes); loose "reasonable support" reintroduces the subjectivity the system was built to remove.
Project-agnostic foundation vs per-project fidelity. A closed config surface keeps "agnostic" falsifiable - but compilation fidelity, judge domain competence, and defect taxonomies constantly pressure the boundary.
Enforcement weight vs error tolerance and trust. Consequential verdicts create real deterrence and collapse acceptable error rates (and trigger legal duties); advisory verdicts preserve trust and tolerate error but lose incentive force.
Contractual pace vs quality floor (added 2026-07-14 from the product owner's throughput requirement). Delivery obligations sometimes demand a looser operating point than the standard threshold; refusing any dial forces the loosening to happen off the books, while a dial that quietly loosens measurement (not just disposition) destroys the system's evidentiary value. Managed by pricing leniency: dispositions move, measurement doesn't, and every batch ships stamped with its profile and estimated residual error (FOUNDATIONS P11).