AutoQAFoundation 0.1.0
Summary
Document library / Core narrative

AutoQA Foundation - Research Atlas

Executive synthesis of the recency-biased research sweep, including strongest findings, disagreements, caveats, and gaps.

MD102 lines25.2 KBSHA-256 d2cdfd0acbbb...researchexecutive synthesisevidence
Download raw fileBack to library

Date: 2026-07-14 (recency-biased sweep; sources through 2026-07-06) Method: 9 parallel domain researchers fed the question taxonomy (00_QUESTIONS.md), each triangulating across WebSearch, Exa, Tavily, and Scholar with instructions to open sources before citing, use absolute dates, and report disagreements. 27 load-bearing claims were adversarially spot-checked by independent verifiers who fetched the cited sources: 20 confirmed, 7 partially confirmed (corrections applied below and in the domain files), 0 refuted, 0 unverifiable. A completeness critic then identified 4 missed literatures - psychometrics/rater science, EU labor law + data-quality standards, content-moderation QA programs, and feedback-efficacy science - each gap-filled with a second round (6 findings apiece). Full evidence: per-domain reports with claims, evidence summaries, source URLs/dates, tools, disagreements, gaps, and verification transcripts live in atlas/.

Bottom line

The core bet - an LLM system that verifies human evaluative writeups claim-by-claim against a defined evidence record, with a calibrated escalation lane to one human touch - is supported by converging 2024-2026 evidence and is close to current production practice at the frontier (Toloka's deployed LLM QA, Handshake's Gandalf, Pinterest's decision-quality framework, OpenAI's grader doctrine). Two things the evidence is emphatic about: holistic single-call judging of subjective quality does not work (near-orthogonal judge/human evaluation axes; ~55% on hard per-criterion verdicts), and agreement-with-humans as a single number is an invalid target (kappa deflation, rating indeterminacy, label-uncertainty stratification). The viable construct is instruction-satisfaction plus evidence-grounding, decomposed into typed claims, with genuinely contested items surfaced as instruction gaps rather than forced verdicts. The exact product - claim-level AutoQA of human annotations - appears publicly unoccupied as of mid-2026, and the single most load-bearing caveat is that essentially all published judge numbers were measured on grading model outputs, not human evaluative writeups; transfer is plausible but unvalidated.

The strongest established findings

Judge capability and its limits

  1. Frontier reasoning models dominate fine-tuned judge models and reward models on judging tasks with verifiable ground truth (JudgeBench: o1-preview 75.4% vs GPT-4o 50.9-56.6%, fine-tuned judges at/below random; later reasoning models reach 77-81%). Small trained judges (J1, CompassJudger-2, Skywork-Reward-V2) win only on preference-style benchmarks. -> Two-tier judge economy: cheap specialists for screening, frontier reasoners for verdicts. (atlas/domain_academic-judges.md, verified)
  2. On subjective rubrics, LLM judges' evaluation axis is nearly orthogonal to the human axis (87-89 degrees vs 78-81 degrees human-human), and inter-LLM agreement exceeds LLM-human agreement - inter-judge consensus is not validity evidence on subjective axes. On rubrics with verifiable answers, the same judges fall back into the human range. (academic-judges, 2026-06)
  3. Per-criterion rubric verification on hard instances sits near chance even for frontier models (RubricEval: GPT-4o 55.97% balanced accuracy on Hard), but explicit per-criterion reasoning beats checklist-level evaluation by 7-12pp and cuts cross-judge variance. Long-form holistic judging is unstable (LongJudgeBench); contextual judging with conditional criteria caps at ~55% (ContextualJudgeBench). (academic-judges, frontier-2026; verified)
  4. In expert-knowledge domains, judge-SME agreement runs 64-68% - below any autonomy bar. Expertise-heavy axes belong in judge-assisted or human-only lanes, and judges agree MORE with non-experts than experts (JUDGE-BENCH), so uncalibrated judges replicate crowd-level, not expert-level, judgment. (academic-judges, contrarian)
  5. The agreement-default asymmetry on positive claims is quantified and large: Google's FACTS grounding judges score ~85 F1 on the grounded class but ~46 F1 on the ungrounded class, and grounding metrics are gameable by vague, short responses - countered by a mandatory responsiveness/eligibility gate. (grounding-faithfulness, verified)
  6. LLMs are structurally weak at detecting omissions (AbsenceBench: F1 drops 56.9 points for deletions vs insertions; explicit placeholders recover ~36 points). "Complete/comprehensive" praise verification is the judge's weakest lane; convert completeness criteria into enumerable checklists. (critique-models, verified)
  7. Judges silently re-solve tasks instead of checking evidence: on checks that exceed the judge's own solving ability, wrong-reference detection collapses (68%->9%) and false positives explode (44%->88%), while read-checkable defect detection holds at 80%. Route checks by read-checkable vs recompute-checkable. (frontier-2026, verified; under-review ARR submission)
  8. Consequence framing silently corrupts verdicts: telling a judge its verdict has downstream stakes softens verdicts (-9.8pp peak, 30% relative drop in unsafe-content detection) with zero trace in the chain-of-thought. Judge prompts must be stakes-sterile, and bias must be measured behaviorally, never read off rationales. (frontier-2026, verified)
  9. Self-preference is real and lineage-based (preference leakage up to 28.7pp, strongest for same-model and distillation descendants; prompt-level debiasing fails; only contextual calibration and style-stripping help). Model errors across the industry are becoming MORE correlated as capability rises. Cross-family judge routing is the practical defense. (academic-judges, contrarian)

Grounding machinery 10. Claim-vs-evidence entailment checking is commoditized but ceilinged (~77 balanced accuracy on LLM-AggreFact; sub-1B-7B specialists within ~2.5 points of frontier models at ~1/100 cost). ~22% claim-level error means item verdicts cannot be a naive AND over claim checks. (grounding-faithfulness, verified) 11. The measured-best grounding-judge architecture is exemplar anchoring: a frontier reasoning judge prompted with human-annotated peer judgments on comparable items (FaithJudge: 84.0 balanced accuracy on FaithBench vs <=68.8 zero-shot judges and 60-67 for fine-tuned detectors). This is exactly the per-project compiled-exemplar contract. (grounding-faithfulness, verified with corrections) 12. Decomposition granularity must be adaptive: atomic decomposition helps weak verifiers and actively degrades strong ones (80.0->71.1 bacc on WiCE with MiniCheck); the verification unit should be claim + explicit context annotations with the verifier told which part is under test. Verifiability triage (verifiable claim / opinion / undecidable) before entailment is field-standard (VeriScore, FactBench). (grounding-faithfulness, verified) 13. Citation checking needs two distinct passes: span-closed (does cited evidence entail the claim) AND context-closed (is this the right citation given everything in scope) - cited-span-only checking misses wrong/incomplete citation, the dominant human failure. (grounding-faithfulness)

The human layer and hybrid design 14. Human+AI combinations on average perform WORSE than the best of either alone on decision tasks (meta-analysis of 370 effect sizes, g = -0.23), specifically when the AI outperforms the human - naive human-verifies-AI-verdict designs destroy value and produce rubber-stamping (~90% deference to biased AI recommendations in a 2025 hiring study). Complementarity appears only with confidence-based routing to a slice where humans genuinely beat the AI, and with evidence-only assistance: DeepMind's amplified-oversight experiments hit 91.3% (hybrid) vs 87.7% (AI alone) vs 75.1% (human alone), and showing humans the AI's verdict/reasoning caused measured over-reliance while showing only retrieved evidence was the sole safe format. (critique-models, contrarian; both verified) 15. AI critics measurably improve human review of human work at deployment scale: CriticGPT critiques preferred over human critiques 63% of the time; critic-assisted review of items already rated "flawless" found real rating-changing problems in 24% of flagged cases vs 6% for an unassisted second human - a 4x lift exactly on the confident-pass slice. The ICLR 2025 RCT (>20k peer reviews) showed 27% of reviewers revised on AI feedback, gated by automated reliability tests before delivery. Caveat: CriticGPT is one study, one vendor, no published replication in 24+ months. (critique-models, verified) 16. The statistics for sparse-human/dense-judge fusion are mature: PPI/PPI++ gives bias-corrected population estimates whose validity survives a bad judge (worse judge -> wider intervals, never invalid); gold-set size is computable in closed form (target effective n=200 at R^2=0.70 needs ~65 human labels); selective escalation with provable human-agreement guarantees exists (Trust or Escalate, ICLR 2025 Oral); judge-based certification can be provably MORE powerful than human-only review ("Noisy but Valid," ICLR 2026); judge-vs-population drift can be attributed with frozen anchor sets and anytime-valid statistics. Gold sets must oversample the rare fail class. (hybrid-statistical, verified with corrections) 17. Theoretical ceiling: when the judge is no more accurate than what it evaluates, no debiasing method can cut required gold labels by more than 2x - the expert-audit channel is a permanent structural component, not a bootstrap phase. (contrarian, verified)

Rubrics and compilation 18. Rubric-based grading is the 2025-26 production standard (HealthBench: 48,562 physician-written per-item binary weighted criteria, grader meta-evaluated per-criterion against physician majority at MF1 0.709 - exceeding the average physician in 5/7 themes, where physician-physician MF1 is only 0.569-0.730; PaperBench: 8,316 binary leaves, judge F1 0.83 at ~1/100 expert cost). The judge gets its own benchmark (JudgeEval pattern); rubric text is a tunable alignment artifact, not scripture. (rubrics-recent, verified) 19. Instruction-only automated rubric generation leaves a ~22-28pp fidelity gap vs human rubrics that test-time compute cannot close (RubricBench) - but the gap closes to ~0.3pp when rubrics are mined contrastively from preference pairs (Support Vector Rubrics, 2026-06). Reference-grounded synthesis matches human authorship (Rubrics-as-Rewards) except for pitfall criteria, which require human domain intuition. The scarce human input is exemplars and failure-mode knowledge, not criterion drafting. (rubrics-recent, verified with corrections) 20. Criteria drift is real and structural: graders' criteria change as they grade (EvalGen); rubric compilation must be a versioned loop with owner sign-off on graded real items, not a one-shot compile. The 2026 RL literature converges on static rubrics being exploitable specifications under optimization pressure - and paid attempters are optimizers. (rubrics-recent, contrarian; verified) 21. Rubric QA is now a named practice (RIFT failure-mode taxonomy: 8 modes; automated linting catches Subjective at F1 0.925 but Hackable at ~0.000 - anti-gaming rubric review is permanently a human red-team task). Cap active criteria per judge call (~<13; "attention displacement" above that). (rubrics-recent)

The workforce and adversarial reality 22. LLM contamination of human work is a base-rate fact, not a tail risk: ~33-35% of crowdworkers used LLMs on an LLM-advantaged text task (2023, replicated ~34% self-report 2025); the best mitigations halve it, never eliminate it, and cost quality. An autonomous agent passes 99.8% of standard quality checks, evades behavioral detectors, and costs ~$0.05 vs $1.50 paid - content-based spam detection is economically and technically beaten; defenses are provenance/economics-based (identity, throttling, work-history, randomized deep audits, worker-level peer-prediction scoring). Scale's internal-document postmortem (spam paid for 11 months; ZeroGPT and country bans as stopgaps; account black markets) corroborates operationally - though it is journalism-of-journalism, disputed by Scale, and pre-LLM-judge era. (annotation-quality, verified with corrections; industry-practice) 23. Annotator disagreement is majority-attributable to task underspecification and response-style preferences, not noise and not primarily expertise (Diverging Preferences; NLI taxonomy work shows an item-intrinsic residual that survives perfect rubrics). Field practice has institutionalized disagreement-as-signal (LeWiDi-2025 soft-label/perspectivist evaluation; NUTMEG separates competence noise from systematic subpopulation disagreement; OpenAI's CoVal preserves conflicting criteria as a record of legitimate disagreement and warns rubric scores invite checklist gaming). (annotation-quality, contrarian, frontier-2026; verified) 24. Judge verdicts are gameable blind: universal adversarial phrases learned on a surrogate transfer to unseen judges and inflate scores regardless of content; absolute scoring is far more vulnerable than comparative/anchored scoring; a deployed LLM checklist assistant was gamed by its subjects in one shot (NeurIPS'24). Rotating seeded probes and evidence-perturbation tests are day-one requirements. (contrarian) 25. Vendors already run the hybrid shape: a major annotation platform's LLM judge only flags for human review, with Krippendorff alpha >= 0.80 as the published bar and per-worker trust routing; Braintrust's cost arithmetic (judge ~$5-15 per 10k outputs vs expert ~$800-1,800 per 500) drives "deterministic checks + judges on everything, humans on flagged/disagreement slices + random spot-checks of confident passes"; the market voted with Handshake's acquisition of Cleanlab (algorithmic label-noise flagging without a second human). Verifier ARCHITECTURE beats verifier MODEL (Gandalf: agent-judge inside the work environment beats text-only rubric judges by more than any model swap, ~10x cheaper; judge F1 vs practicing experts 0.63-0.66 on artifact-heavy work vs 0.95 on simple stateful tasks - the ceiling is domain-dependent by ~30 F1 points). (industry-practice, frontier-2026; verified)

From the gap-fill rounds 26. Psychometrics (40 years of rater science the LLM literature reinvents badly): training improves rater self-consistency but does NOT equalize severity (canonical Weigle/Lumley-McNamara finding); best-in-class frame-of-reference training buys only d~0.50; G-theory decompositions show the rater main effect (global severity) is minor while raterxitemxcriterion interactions dominate - the dominant variance can only be attacked by per-claim, per-item verification, which is the strongest independent argument for this architecture. Rater severity drifts within a single session (2.4x worse on hard items). MFRM is now applied symmetrically to LLM judges - enroll the judge as one more rater facet in the same measurement model. Constraint: severity estimation requires deliberate rater-item overlap (3-5 ratings/output linkage), which single-review-per-item regimes cannot provide. (atlas/critic_and_gapfill.md) 27. Law makes part of the design non-optional for EU-touching deployments (national transposition from 2026-12-02): EU Platform Work Directive Art 10(5) requires decisions of significant detriment (account/pay restriction, termination, or equivalent) to be taken by a human, with no consent exception, covering self-employed annotators and non-EU platforms with EU workers; Art 11 mandates plain-language explanations, a competent human contact, and human review with a substantiated written reply within two weeks; Art 7 prohibits telemetry on emotional state, private communications, or off-task activity and requires a DPIA. CJEU SCHUFA: a score with a "determining role" is itself an Art 22 decision, and rubber-stamp human review legally counts as zero-touch (active enforcement: ~EUR 490k Hamburg fine, 2025-09). ISO/IEC 5259 (data quality for ML, 5 parts, 2024-25) is the neutral vocabulary to map quality dimensions onto. Open question: whether a single-item pay denial is a "decision of equivalent detriment" under Art 10(5) is legally untested. (critic_and_gapfill) 28. Content moderation ran this movie for a decade: ~3-4% random audit + QA-of-QA layers was the Facebook/Cognizant baseline; "accuracy = agreement with auditor" is blind to correlated error and gameable via off-book lobbying ("this number is fake"); TSPA codifies forward audit sampling + blind golden-set seeding (constraint: seeded items must be indistinguishable) + appeals as a cheap but population-biased third signal, and a four-way error taxonomy including wrong-selection (right verdict, wrong cited criterion). Pinterest's 2026 framework: separate reliability (kappa) from correctness (vs adjudicated golden set) - high-reliability + low-correctness = systematic policy misunderstanding; golden sets deliberately oversample rare/hard cases; policy changes are handled by dual-labeling under old+new guidelines; 3x-human majority vote bought only +3.6pp over a single non-expert. Consensus-rewarded auditing (Community Notes) induces strategic conformity - never score contributors on agreement-with-final-verdict; score on stability/informativeness of residuals. Verdict-option UI costs distort verdict distributions independent of judgment. (critic_and_gapfill) 29. Feedback science kills "no evidence feedback works" and replaces it with priors: feedback improves performance on average (d~0.38-0.41) but more than one third of feedback interventions reduce performance - task-referenced helps, person/rank-referenced backfires (Kluger & DeNisi FIT); high-information feedback (what + why-process + how-to-self-check) yields d~0.99 vs 0.46 for bare right/wrong and 0.24 for reinforcement - verdict-only QA forfeits most achievable behavior change; the 2025 Cochrane update (292 studies) finds a modest 2.7% median absolute improvement concentrated in low performers, with individual-level data, top-peer/benchmark comparison, and specific action plans as positive moderators and repeated delivery associated with LOWER effect; the closest crowdwork RCT (Dow et al., CSCW 2012) shows rubric-mediated task-specific feedback significantly improves paid microtask quality, with self-assessment matching external feedback on quality but not on revision behavior - and self-ratings inflated by 1.8/9 points. Under pay-linked incentives, rank feedback induces gaming/sabotage that offsets gains - benchmark against exemplar WORK, never ranked PEOPLE. (critic_and_gapfill)

Live disagreements (design forks the literature does not settle)

  • Dedicated trained judges vs frontier generalists: benchmark-dependent (preference-style vs verifiable-reasoning); no source resolves it for criteria-based QA of human annotations. Default: frontier reasoner for verdicts, specialists for screening; measure per project.
  • Uncertainty-routed vs uniform human sampling: >25% label savings in batch settings (NAACL 2025) vs near-uniform optimality in sequential estimation (ICLR 2026). Regime-dependent; ship routing policy as a measured per-project choice with a built-in A/B against uniform.
  • Estimation vs certification statistics: PPI++/EIF intervals 3-15x narrower for rates/dashboards; explicit TPR/FPR-corrected tests better-powered for threshold certification. Use both, per purpose.
  • Implicit vs explicit rubric aggregation: implicit (judge weighs all criteria internally) beats explicit weighted-sum for reward quality by up to 28-31% relative (RaR), but every auditable production benchmark (HealthBench, PaperBench, OpenAI graders) uses explicit per-criterion verdicts. For a QA system that must justify verdicts, auditability wins; treat this as a known performance tax.
  • Decomposition depth: per-criterion structure beats flat checklists (RubricEval) AND decomposition monotonically improves verification in another 2026 framework (LLM-as-a-Verifier); reconciliation (atomicity helps when each criterion gets its own call, hurts in one long checklist) is inferred, not tested.
  • Static pinned rubrics with regression gates vs continuously co-evolving rubrics: production-integrity practice vs 2026 anti-Goodhart RL literature. FOUNDATIONS resolves this as versioned revision loops with regression gates - adaptation at the compilation layer, pinning at the judge layer.
  • What the human should see: evidence-only for adjudicators (DeepMind) vs full critiques delivered to authors (CriticGPT, ICLR RCT). Plausible split - critiques to ATTEMPTERS, evidence-only to ADJUDICATORS - is untested in any single study.
  • Human-human floor numbers: practitioner "~85% judge-human agreement, better than humans agree with each other" vs chance-corrected kappa 0.28 +/- 0.32 across 20 tasks. Mostly percent-agreement vs chance-corrected framing; never report the former.

What has no evidence anywhere (in-house measurement required)

  1. Positive-claim/praise verification - precision/recall of "is this praise warranted," per praise type: zero published benchmarks. The mission's core novel requirement is a from-scratch measurement (seeded warranted/empty-praise sets). AbsenceBench predicts empty-praise detection (an omission-shaped failure) is the hardest lane.
  2. Transfer of all judge numbers from grading model outputs to grading human evaluative writeups - flagged by every domain, no partial-transfer experiment exists.
  3. False-agreement rate (judge and attempter both wrong vs expert ground truth) in an annotation-QA setting; nearest proxies are the 3.3%+ gold-label error floor and CriticGPT's 24%-vs-6% flawless-slice result.
  4. Attempter adaptation to a deployed AI QA layer - feedback-boundary leakage, AutoQA-vs-held-out-gold divergence, monoculture drift: no published study; the divergence sensor must be built, not borrowed.
  5. Longitudinal feedback efficacy for paid annotators (repeat-error-rate deltas under AI feedback): adjacent literatures transfer with caveats; the direct experiment is unrun.
  6. Confidence-proxy shootout (verbalized confidence vs k-sample agreement vs conformal width vs ensemble disagreement) for predicting human overturn: never benchmarked head-to-head.
  7. Our own reviewers' reliability floor - no measured Krippendorff's alpha exists for the population this system is being designed around; the variance-attribution pilot is the first experiment for a reason.

Honesty notes (from the completeness critic)

  • The headline meta-evaluation figures (33-41pp kappa deflation, 14-15 position ranking flips, consistency-bias paradox) trace to ONE un-peer-reviewed June 2026 preprint cited by four domains independently - cross-domain repetition is not corroboration. Its findings are directionally consistent with peer-reviewed work.
  • Several architecturally decisive 2026 claims (auditing-by-re-solving, evaluation faking, Toloka's precision table) were verified at abstract/crawl level, not full-PDF audit; Auditing-by-Re-Solving is still under review.
  • Several "reconciliations" offered above (verbosity bias as controllable, critiques-to-attempters/evidence-to-adjudicators) are inferred, not tested. They are labeled as such where they inform FOUNDATIONS.
  • Non-US/non-English industry practice (Chinese labeling bases, Japanese/Korean/Indian vendors) was not searched; industry claims generalize from a US/EU sample.
  • Judge-model choice guidance is anchored to early-2026-or-older cohorts; the two major judge leaderboards were unfetchable (dynamic JS) at sweep time.

Confidence

High on the architectural findings (decomposition necessity, typed verdicts, exemplar anchoring, evidence closure, statistical layer, adversarial posture, legal floor): multiple independent source classes converge, and 27/27 spot-checks surfaced no refutations. Medium on any specific quantitative anchor transferring to our workload (the model-outputs->human-writeups transfer gap). Low on anything about positive-claim verification performance - genuinely unmeasured territory, which is also why it is the moat.

Source index

Every finding above carries its full citation (title, URL, absolute date, source type) plus the verifier's transcript in the domain files:

  • atlas/domain_academic-judges.md - judge meta-evaluation benchmarks and capability limits
  • atlas/domain_annotation-quality.md - annotation science, disagreement, contamination
  • atlas/domain_critique-models.md - CriticGPT lineage, human+critic teams, omission detection
  • atlas/domain_grounding-faithfulness.md - entailment checkers, decomposition, citation evaluation
  • atlas/domain_hybrid-statistical.md - PPI, selective escalation, conformal routing, drift attribution
  • atlas/domain_rubrics-recent.md - rubric practice 2025-26, compilation, rubric QA
  • atlas/domain_industry-practice.md - vendor QA stacks, eval tooling, spam economics
  • atlas/domain_contrarian.md - the case against, adversarial attacks, validity critiques
  • atlas/domain_frontier-2026.md - 2026-only developments
  • atlas/critic_and_gapfill.md - critic's weak-spot audit + psychometrics, law/standards, content moderation, feedback science