# AutoQA Foundation - Research Atlas (Executive Synthesis)

**Date:** 2026-07-14 (recency-biased sweep; sources through 2026-07-06)
**Method:** 9 parallel domain researchers fed the question taxonomy (`00_QUESTIONS.md`), each triangulating across WebSearch, Exa, Tavily, and Scholar with instructions to open sources before citing, use absolute dates, and report disagreements. 27 load-bearing claims were adversarially spot-checked by independent verifiers who fetched the cited sources: **20 confirmed, 7 partially confirmed (corrections applied below and in the domain files), 0 refuted, 0 unverifiable.** A completeness critic then identified 4 missed literatures - psychometrics/rater science, EU labor law + data-quality standards, content-moderation QA programs, and feedback-efficacy science - each gap-filled with a second round (6 findings apiece).
**Full evidence:** per-domain reports with claims, evidence summaries, source URLs/dates, tools, disagreements, gaps, and verification transcripts live in `atlas/`.

---

## Bottom line

The core bet - an LLM system that verifies human evaluative writeups claim-by-claim against a defined evidence record, with a calibrated escalation lane to one human touch - is supported by converging 2024-2026 evidence and is close to current production practice at the frontier (Toloka's deployed LLM QA, Handshake's Gandalf, Pinterest's decision-quality framework, OpenAI's grader doctrine). Two things the evidence is emphatic about: **holistic single-call judging of subjective quality does not work** (near-orthogonal judge/human evaluation axes; ~55% on hard per-criterion verdicts), and **agreement-with-humans as a single number is an invalid target** (kappa deflation, rating indeterminacy, label-uncertainty stratification). The viable construct is instruction-satisfaction plus evidence-grounding, decomposed into typed claims, with genuinely contested items surfaced as instruction gaps rather than forced verdicts. The exact product - claim-level AutoQA of *human* annotations - appears publicly unoccupied as of mid-2026, and the single most load-bearing caveat is that essentially all published judge numbers were measured on grading *model* outputs, not human evaluative writeups; transfer is plausible but unvalidated.

## The strongest established findings

**Judge capability and its limits**
1. Frontier *reasoning* models dominate fine-tuned judge models and reward models on judging tasks with verifiable ground truth (JudgeBench: o1-preview 75.4% vs GPT-4o 50.9-56.6%, fine-tuned judges at/below random; later reasoning models reach 77-81%). Small trained judges (J1, CompassJudger-2, Skywork-Reward-V2) win only on preference-style benchmarks. -> Two-tier judge economy: cheap specialists for screening, frontier reasoners for verdicts. *(atlas/domain_academic-judges.md, verified)*
2. On subjective rubrics, LLM judges' evaluation axis is nearly orthogonal to the human axis (87-89 degrees vs 78-81 degrees human-human), and inter-LLM agreement exceeds LLM-human agreement - **inter-judge consensus is not validity evidence on subjective axes.** On rubrics with verifiable answers, the same judges fall back into the human range. *(academic-judges, 2026-06)*
3. Per-criterion rubric verification on hard instances sits near chance even for frontier models (RubricEval: GPT-4o 55.97% balanced accuracy on Hard), but explicit per-criterion reasoning beats checklist-level evaluation by 7-12pp and cuts cross-judge variance. Long-form holistic judging is unstable (LongJudgeBench); contextual judging with conditional criteria caps at ~55% (ContextualJudgeBench). *(academic-judges, frontier-2026; verified)*
4. In expert-knowledge domains, judge-SME agreement runs 64-68% - below any autonomy bar. Expertise-heavy axes belong in judge-assisted or human-only lanes, and judges agree MORE with non-experts than experts (JUDGE-BENCH), so uncalibrated judges replicate crowd-level, not expert-level, judgment. *(academic-judges, contrarian)*
5. **The agreement-default asymmetry on positive claims is quantified and large:** Google's FACTS grounding judges score ~85 F1 on the grounded class but ~46 F1 on the ungrounded class, and grounding metrics are gameable by vague, short responses - countered by a mandatory responsiveness/eligibility gate. *(grounding-faithfulness, verified)*
6. LLMs are structurally weak at detecting omissions (AbsenceBench: F1 drops 56.9 points for deletions vs insertions; explicit placeholders recover ~36 points). "Complete/comprehensive" praise verification is the judge's weakest lane; convert completeness criteria into enumerable checklists. *(critique-models, verified)*
7. Judges silently re-solve tasks instead of checking evidence: on checks that exceed the judge's own solving ability, wrong-reference detection collapses (68%->9%) and false positives explode (44%->88%), while read-checkable defect detection holds at 80%. Route checks by read-checkable vs recompute-checkable. *(frontier-2026, verified; under-review ARR submission)*
8. Consequence framing silently corrupts verdicts: telling a judge its verdict has downstream stakes softens verdicts (-9.8pp peak, 30% relative drop in unsafe-content detection) with **zero trace in the chain-of-thought**. Judge prompts must be stakes-sterile, and bias must be measured behaviorally, never read off rationales. *(frontier-2026, verified)*
9. Self-preference is real and lineage-based (preference leakage up to 28.7pp, strongest for same-model and distillation descendants; prompt-level debiasing fails; only contextual calibration and style-stripping help). Model errors across the industry are becoming MORE correlated as capability rises. Cross-family judge routing is the practical defense. *(academic-judges, contrarian)*

**Grounding machinery**
10. Claim-vs-evidence entailment checking is commoditized but ceilinged (~77 balanced accuracy on LLM-AggreFact; sub-1B-7B specialists within ~2.5 points of frontier models at ~1/100 cost). ~22% claim-level error means item verdicts cannot be a naive AND over claim checks. *(grounding-faithfulness, verified)*
11. The measured-best grounding-judge architecture is **exemplar anchoring**: a frontier reasoning judge prompted with human-annotated peer judgments on comparable items (FaithJudge: 84.0 balanced accuracy on FaithBench vs <=68.8 zero-shot judges and 60-67 for fine-tuned detectors). This is exactly the per-project compiled-exemplar contract. *(grounding-faithfulness, verified with corrections)*
12. Decomposition granularity must be adaptive: atomic decomposition helps weak verifiers and actively degrades strong ones (80.0->71.1 bacc on WiCE with MiniCheck); the verification unit should be claim + explicit context annotations with the verifier told which part is under test. Verifiability triage (verifiable claim / opinion / undecidable) before entailment is field-standard (VeriScore, FactBench). *(grounding-faithfulness, verified)*
13. Citation checking needs two distinct passes: span-closed (does cited evidence entail the claim) AND context-closed (is this the right citation given everything in scope) - cited-span-only checking misses wrong/incomplete citation, the dominant human failure. *(grounding-faithfulness)*

**The human layer and hybrid design**
14. Human+AI combinations on average perform WORSE than the best of either alone on decision tasks (meta-analysis of 370 effect sizes, g = -0.23), specifically when the AI outperforms the human - naive human-verifies-AI-verdict designs destroy value and produce rubber-stamping (~90% deference to biased AI recommendations in a 2025 hiring study). Complementarity appears only with confidence-based routing to a slice where humans genuinely beat the AI, and with **evidence-only assistance**: DeepMind's amplified-oversight experiments hit 91.3% (hybrid) vs 87.7% (AI alone) vs 75.1% (human alone), and showing humans the AI's verdict/reasoning caused measured over-reliance while showing only retrieved evidence was the sole safe format. *(critique-models, contrarian; both verified)*
15. AI critics measurably improve human review of *human* work at deployment scale: CriticGPT critiques preferred over human critiques 63% of the time; critic-assisted review of items already rated "flawless" found real rating-changing problems in 24% of flagged cases vs 6% for an unassisted second human - a 4x lift exactly on the confident-pass slice. The ICLR 2025 RCT (>20k peer reviews) showed 27% of reviewers revised on AI feedback, gated by automated reliability tests before delivery. Caveat: CriticGPT is one study, one vendor, no published replication in 24+ months. *(critique-models, verified)*
16. The statistics for sparse-human/dense-judge fusion are mature: PPI/PPI++ gives bias-corrected population estimates whose validity survives a bad judge (worse judge -> wider intervals, never invalid); gold-set size is computable in closed form (target effective n=200 at R^2=0.70 needs ~65 human labels); selective escalation with provable human-agreement guarantees exists (Trust or Escalate, ICLR 2025 Oral); judge-based certification can be provably MORE powerful than human-only review ("Noisy but Valid," ICLR 2026); judge-vs-population drift can be attributed with frozen anchor sets and anytime-valid statistics. Gold sets must oversample the rare fail class. *(hybrid-statistical, verified with corrections)*
17. Theoretical ceiling: when the judge is no more accurate than what it evaluates, no debiasing method can cut required gold labels by more than 2x - the expert-audit channel is a permanent structural component, not a bootstrap phase. *(contrarian, verified)*

**Rubrics and compilation**
18. Rubric-based grading is the 2025-26 production standard (HealthBench: 48,562 physician-written per-item binary weighted criteria, grader meta-evaluated per-criterion against physician majority at MF1 0.709 - exceeding the average physician in 5/7 themes, where physician-physician MF1 is only 0.569-0.730; PaperBench: 8,316 binary leaves, judge F1 0.83 at ~1/100 expert cost). The judge gets its own benchmark (JudgeEval pattern); rubric text is a tunable alignment artifact, not scripture. *(rubrics-recent, verified)*
19. Instruction-only automated rubric generation leaves a ~22-28pp fidelity gap vs human rubrics that test-time compute cannot close (RubricBench) - but the gap closes to ~0.3pp when rubrics are mined contrastively from preference pairs (Support Vector Rubrics, 2026-06). Reference-grounded synthesis matches human authorship (Rubrics-as-Rewards) except for **pitfall criteria, which require human domain intuition**. The scarce human input is exemplars and failure-mode knowledge, not criterion drafting. *(rubrics-recent, verified with corrections)*
20. Criteria drift is real and structural: graders' criteria change as they grade (EvalGen); rubric compilation must be a versioned loop with owner sign-off on graded real items, not a one-shot compile. The 2026 RL literature converges on static rubrics being exploitable specifications under optimization pressure - and paid attempters are optimizers. *(rubrics-recent, contrarian; verified)*
21. Rubric QA is now a named practice (RIFT failure-mode taxonomy: 8 modes; automated linting catches Subjective at F1 0.925 but Hackable at ~0.000 - **anti-gaming rubric review is permanently a human red-team task**). Cap active criteria per judge call (~<13; "attention displacement" above that). *(rubrics-recent)*

**The workforce and adversarial reality**
22. LLM contamination of human work is a base-rate fact, not a tail risk: ~33-35% of crowdworkers used LLMs on an LLM-advantaged text task (2023, replicated ~34% self-report 2025); the best mitigations halve it, never eliminate it, and cost quality. An autonomous agent passes 99.8% of standard quality checks, evades behavioral detectors, and costs ~$0.05 vs $1.50 paid - **content-based spam detection is economically and technically beaten**; defenses are provenance/economics-based (identity, throttling, work-history, randomized deep audits, worker-level peer-prediction scoring). Scale's internal-document postmortem (spam paid for 11 months; ZeroGPT and country bans as stopgaps; account black markets) corroborates operationally - though it is journalism-of-journalism, disputed by Scale, and pre-LLM-judge era. *(annotation-quality, verified with corrections; industry-practice)*
23. Annotator disagreement is majority-attributable to task underspecification and response-style preferences, not noise and not primarily expertise (Diverging Preferences; NLI taxonomy work shows an item-intrinsic residual that survives perfect rubrics). Field practice has institutionalized disagreement-as-signal (LeWiDi-2025 soft-label/perspectivist evaluation; NUTMEG separates competence noise from systematic subpopulation disagreement; OpenAI's CoVal preserves conflicting criteria as a record of legitimate disagreement and warns rubric scores invite checklist gaming). *(annotation-quality, contrarian, frontier-2026; verified)*
24. Judge verdicts are gameable blind: universal adversarial phrases learned on a surrogate transfer to unseen judges and inflate scores regardless of content; absolute scoring is far more vulnerable than comparative/anchored scoring; a deployed LLM checklist assistant was gamed by its subjects in one shot (NeurIPS'24). Rotating seeded probes and evidence-perturbation tests are day-one requirements. *(contrarian)*
25. Vendors already run the hybrid shape: a major annotation platform's LLM judge only *flags* for human review, with Krippendorff alpha >= 0.80 as the published bar and per-worker trust routing; Braintrust's cost arithmetic (judge ~$5-15 per 10k outputs vs expert ~$800-1,800 per 500) drives "deterministic checks + judges on everything, humans on flagged/disagreement slices + random spot-checks of confident passes"; the market voted with Handshake's acquisition of Cleanlab (algorithmic label-noise flagging without a second human). Verifier ARCHITECTURE beats verifier MODEL (Gandalf: agent-judge inside the work environment beats text-only rubric judges by more than any model swap, ~10x cheaper; judge F1 vs practicing experts 0.63-0.66 on artifact-heavy work vs 0.95 on simple stateful tasks - the ceiling is domain-dependent by ~30 F1 points). *(industry-practice, frontier-2026; verified)*

**From the gap-fill rounds**
26. Psychometrics (40 years of rater science the LLM literature reinvents badly): training improves rater *self*-consistency but does NOT equalize severity (canonical Weigle/Lumley-McNamara finding); best-in-class frame-of-reference training buys only d~0.50; G-theory decompositions show the rater main effect (global severity) is minor while raterxitemxcriterion interactions dominate - **the dominant variance can only be attacked by per-claim, per-item verification, which is the strongest independent argument for this architecture**. Rater severity drifts within a single session (2.4x worse on hard items). MFRM is now applied symmetrically to LLM judges - enroll the judge as one more rater facet in the same measurement model. Constraint: severity estimation requires deliberate rater-item overlap (3-5 ratings/output linkage), which single-review-per-item regimes cannot provide. *(atlas/critic_and_gapfill.md)*
27. Law makes part of the design non-optional for EU-touching deployments (national transposition from 2026-12-02): EU Platform Work Directive Art 10(5) requires decisions of significant detriment (account/pay restriction, termination, or equivalent) to be **taken by a human**, with no consent exception, covering self-employed annotators and non-EU platforms with EU workers; Art 11 mandates plain-language explanations, a competent human contact, and human review with a substantiated written reply within two weeks; Art 7 prohibits telemetry on emotional state, private communications, or off-task activity and requires a DPIA. CJEU SCHUFA: a score with a "determining role" is itself an Art 22 decision, and rubber-stamp human review legally counts as zero-touch (active enforcement: ~EUR 490k Hamburg fine, 2025-09). ISO/IEC 5259 (data quality for ML, 5 parts, 2024-25) is the neutral vocabulary to map quality dimensions onto. Open question: whether a single-item pay denial is a "decision of equivalent detriment" under Art 10(5) is legally untested. *(critic_and_gapfill)*
28. Content moderation ran this movie for a decade: ~3-4% random audit + QA-of-QA layers was the Facebook/Cognizant baseline; "accuracy = agreement with auditor" is blind to correlated error and gameable via off-book lobbying ("this number is fake"); TSPA codifies forward audit sampling + blind golden-set seeding (constraint: seeded items must be indistinguishable) + appeals as a cheap but population-biased third signal, and a four-way error taxonomy including *wrong-selection* (right verdict, wrong cited criterion). Pinterest's 2026 framework: separate reliability (kappa) from correctness (vs adjudicated golden set) - high-reliability + low-correctness = systematic policy misunderstanding; golden sets deliberately oversample rare/hard cases; policy changes are handled by dual-labeling under old+new guidelines; 3x-human majority vote bought only +3.6pp over a single non-expert. Consensus-rewarded auditing (Community Notes) induces strategic conformity - **never score contributors on agreement-with-final-verdict; score on stability/informativeness of residuals**. Verdict-option UI costs distort verdict distributions independent of judgment. *(critic_and_gapfill)*
29. Feedback science kills "no evidence feedback works" and replaces it with priors: feedback improves performance on average (d~0.38-0.41) but **more than one third of feedback interventions reduce performance** - task-referenced helps, person/rank-referenced backfires (Kluger & DeNisi FIT); high-information feedback (what + why-process + how-to-self-check) yields d~0.99 vs 0.46 for bare right/wrong and 0.24 for reinforcement - verdict-only QA forfeits most achievable behavior change; the 2025 Cochrane update (292 studies) finds a modest 2.7% median absolute improvement concentrated in low performers, with individual-level data, top-peer/benchmark comparison, and specific action plans as positive moderators and repeated delivery associated with LOWER effect; the closest crowdwork RCT (Dow et al., CSCW 2012) shows rubric-mediated task-specific feedback significantly improves paid microtask quality, with self-assessment matching external feedback on quality but not on revision behavior - and self-ratings inflated by 1.8/9 points. Under pay-linked incentives, rank feedback induces gaming/sabotage that offsets gains - benchmark against exemplar WORK, never ranked PEOPLE. *(critic_and_gapfill)*

## Live disagreements (design forks the literature does not settle)

- **Dedicated trained judges vs frontier generalists:** benchmark-dependent (preference-style vs verifiable-reasoning); no source resolves it for criteria-based QA of human annotations. Default: frontier reasoner for verdicts, specialists for screening; measure per project.
- **Uncertainty-routed vs uniform human sampling:** >25% label savings in batch settings (NAACL 2025) vs near-uniform optimality in sequential estimation (ICLR 2026). Regime-dependent; ship routing policy as a measured per-project choice with a built-in A/B against uniform.
- **Estimation vs certification statistics:** PPI++/EIF intervals 3-15x narrower for rates/dashboards; explicit TPR/FPR-corrected tests better-powered for threshold certification. Use both, per purpose.
- **Implicit vs explicit rubric aggregation:** implicit (judge weighs all criteria internally) beats explicit weighted-sum for reward quality by up to 28-31% relative (RaR), but every auditable production benchmark (HealthBench, PaperBench, OpenAI graders) uses explicit per-criterion verdicts. For a QA system that must justify verdicts, auditability wins; treat this as a known performance tax.
- **Decomposition depth:** per-criterion structure beats flat checklists (RubricEval) AND decomposition monotonically improves verification in another 2026 framework (LLM-as-a-Verifier); reconciliation (atomicity helps when each criterion gets its own call, hurts in one long checklist) is inferred, not tested.
- **Static pinned rubrics with regression gates vs continuously co-evolving rubrics:** production-integrity practice vs 2026 anti-Goodhart RL literature. FOUNDATIONS resolves this as versioned revision loops with regression gates - adaptation at the compilation layer, pinning at the judge layer.
- **What the human should see:** evidence-only for adjudicators (DeepMind) vs full critiques delivered to authors (CriticGPT, ICLR RCT). Plausible split - critiques to ATTEMPTERS, evidence-only to ADJUDICATORS - is untested in any single study.
- **Human-human floor numbers:** practitioner "~85% judge-human agreement, better than humans agree with each other" vs chance-corrected kappa 0.28 +/- 0.32 across 20 tasks. Mostly percent-agreement vs chance-corrected framing; never report the former.

## What has no evidence anywhere (in-house measurement required)

1. **Positive-claim/praise verification** - precision/recall of "is this praise warranted," per praise type: zero published benchmarks. The mission's core novel requirement is a from-scratch measurement (seeded warranted/empty-praise sets). AbsenceBench predicts empty-praise detection (an omission-shaped failure) is the hardest lane.
2. **Transfer of all judge numbers from grading model outputs to grading human evaluative writeups** - flagged by every domain, no partial-transfer experiment exists.
3. **False-agreement rate** (judge and attempter both wrong vs expert ground truth) in an annotation-QA setting; nearest proxies are the 3.3%+ gold-label error floor and CriticGPT's 24%-vs-6% flawless-slice result.
4. **Attempter adaptation to a deployed AI QA layer** - feedback-boundary leakage, AutoQA-vs-held-out-gold divergence, monoculture drift: no published study; the divergence sensor must be built, not borrowed.
5. **Longitudinal feedback efficacy for paid annotators** (repeat-error-rate deltas under AI feedback): adjacent literatures transfer with caveats; the direct experiment is unrun.
6. **Confidence-proxy shootout** (verbalized confidence vs k-sample agreement vs conformal width vs ensemble disagreement) for predicting human overturn: never benchmarked head-to-head.
7. **Our own reviewers' reliability floor** - no measured Krippendorff's alpha exists for the population this system is being designed around; the variance-attribution pilot is the first experiment for a reason.

## Honesty notes (from the completeness critic)

- The headline meta-evaluation figures (33-41pp kappa deflation, 14-15 position ranking flips, consistency-bias paradox) trace to ONE un-peer-reviewed June 2026 preprint cited by four domains independently - cross-domain repetition is not corroboration. Its findings are directionally consistent with peer-reviewed work.
- Several architecturally decisive 2026 claims (auditing-by-re-solving, evaluation faking, Toloka's precision table) were verified at abstract/crawl level, not full-PDF audit; Auditing-by-Re-Solving is still under review.
- Several "reconciliations" offered above (verbosity bias as controllable, critiques-to-attempters/evidence-to-adjudicators) are inferred, not tested. They are labeled as such where they inform FOUNDATIONS.
- Non-US/non-English industry practice (Chinese labeling bases, Japanese/Korean/Indian vendors) was not searched; industry claims generalize from a US/EU sample.
- Judge-model choice guidance is anchored to early-2026-or-older cohorts; the two major judge leaderboards were unfetchable (dynamic JS) at sweep time.

## Confidence

**High** on the architectural findings (decomposition necessity, typed verdicts, exemplar anchoring, evidence closure, statistical layer, adversarial posture, legal floor): multiple independent source classes converge, and 27/27 spot-checks surfaced no refutations. **Medium** on any specific quantitative anchor transferring to our workload (the model-outputs->human-writeups transfer gap). **Low** on anything about positive-claim verification performance - genuinely unmeasured territory, which is also why it is the moat.

## Source index

Every finding above carries its full citation (title, URL, absolute date, source type) plus the verifier's transcript in the domain files:

- `atlas/domain_academic-judges.md` - judge meta-evaluation benchmarks and capability limits
- `atlas/domain_annotation-quality.md` - annotation science, disagreement, contamination
- `atlas/domain_critique-models.md` - CriticGPT lineage, human+critic teams, omission detection
- `atlas/domain_grounding-faithfulness.md` - entailment checkers, decomposition, citation evaluation
- `atlas/domain_hybrid-statistical.md` - PPI, selective escalation, conformal routing, drift attribution
- `atlas/domain_rubrics-recent.md` - rubric practice 2025-26, compilation, rubric QA
- `atlas/domain_industry-practice.md` - vendor QA stacks, eval tooling, spam economics
- `atlas/domain_contrarian.md` - the case against, adversarial attacks, validity critiques
- `atlas/domain_frontier-2026.md` - 2026-only developments
- `atlas/critic_and_gapfill.md` - critic's weak-spot audit + psychometrics, law/standards, content moderation, feedback science