# Domain: rubrics-recent

## OpenAI HealthBench meta-evaluates its LLM grader per-criterion against physician majority grades using Macro F1 (met/not-met, class-balanced), and reports the grader as a percentile of individual physicians: GPT-4.1 grader MF1 = 0.709, exceeding the average physician in 5 of 7 themes, while physician-vs-physician MF1 is only 0.569-0.730 with wide individual spread. [LOAD-BEARING]
EVIDENCE: Full paper text (Section 8.1, Tables 5-6) confirmed: 60,896 meta-examples over 34 physician-consensus criteria; 'typical physician' baseline computed by scoring each physician against the others exactly as the model is scored; grader model choice matters (GPT-4.1 0.709 > o4-mini 0.692 > o3 0.681 > GPT-4.1-nano 0.580); prompt AND individual criterion phrasings were tuned 'so that their intent was unmistakable to the grader'. Overall benchmark std across 16 full runs is ~0.002.
SOURCE: HealthBench: Evaluating Large Language Models Towards Improved Human Health (OpenAI) | https://arxiv.org/html/2505.08775v1 | 2025-05-13 | primary
IMPLICATION: The 2025-26 standard validation design for a rubric judge: per-criterion binary meta-eval against adjudicated expert-consensus labels with a chance-robust class-balanced statistic, reporting the judge as a percentile of individual human raters - not raw accuracy vs. a single reviewer. The human agreement ceiling is empirically low (MF1 ~0.6-0.7 even among physicians on physician-written criteria), so 'exceeds median human' is an attainable and meaningful bar, and consensus-adjudicated labels (not single-reviewer labels) must be the target. Also: rubric criterion text is a tunable judge-alignment artifact, not fixed scripture.

## HealthBench's production rubric format is per-item (conversation-specific) criteria written by 262 physicians (48,562 criteria), each an independently-judged binary met/not-met check with a nonzero weight from -10 to +10 (negative criteria encode pitfalls), aggregated as a weighted sum normalized by max attainable positive points.
EVIDENCE: Confirmed in fetched full text: 'Each rubric criterion has an associated nonzero point value between -10 and 10, with negative points used for criteria that are undesirable'; grader judges each criterion independently; item score can go negative; the vast majority of criteria were written specifically for one example.
SOURCE: HealthBench (rubric structure, Section 3) | https://arxiv.org/html/2505.08775v1 | 2025-05-13 | primary
IMPLICATION: Reference design for the AutoQA verdict aggregation question: per-item compiled criteria (not one generic project rubric), explicit negative-weighted pitfall criteria as first-class citizens, independent binary judging per criterion, and a transparent weighted-sum aggregation - i.e., claim-level findings with an arithmetic, auditable roll-up rather than a holistic judge verdict.

## a large data vendor's Rubrics-as-Rewards found that LLM-synthesized rubrics grounded in expert reference answers performed on par with human-authored rubrics as reward signals (0.359 vs 0.348), that prompt-specific rubrics substantially beat generic ones, and that the one component synthetic generation could NOT produce well was Pitfall criteria - anticipating failure modes 'often demands human intuition and domain expertise'. [LOAD-BEARING]
EVIDENCE: Full paper (arXiv 2507.17746) confirmed: four design principles (grounded in expert guidance; comprehensive coverage; semantic weighting Essential 1.0 / Important 0.7 / Optional 0.3 / Pitfall 0.8 penalty; self-contained per-criterion evaluability); rubrics generated by o3-mini/GPT-4o from reference answers, ~7-20 criteria per prompt; implicit aggregation (judge sees all criteria+weights and emits one score) consistently beat explicit per-criterion binary + weighted-sum, up to 28% relative on HealthBench-1k; rubrics helped small judges most and reduced cross-judge variance.
SOURCE: Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains (a large data vendor) | https://arxiv.org/html/2507.17746v1 | 2025-07-23 | primary
IMPLICATION: Rubric compilation can be automated IF anchored to expert reference material - the scarce human input to budget for is reference exemplars and failure-mode/pitfall knowledge, not criterion drafting labor. Caveat for AutoQA: RaR's implicit-aggregation win optimizes reward quality, not auditability; an AutoQA that must justify verdicts may still prefer explicit per-criterion verdicts (HealthBench/PaperBench style) and treat RaR as evidence that fixed hand-crafted weights are not sacred.

## RubricBench (Mar 2026) measures a stable ~26-27 point preference-accuracy gap between self-generated and human-annotated rubrics across all judge backbones (e.g., DeepSeek-v3.2 57.8% vs 84.9%; Gemini-3-Flash 58.0% vs 85.3%), and scaling test-time compute (more sampled rubrics, refinement) does not close it - 'rubric formation', not judge reasoning, is the bottleneck. [LOAD-BEARING]
EVIDENCE: Full HTML (arXiv 2603.01562) confirmed: 1,147 adversarial pairwise comparisons where surface cues mislead (rejected responses longer/better-formatted/more confident); expert rubrics are 2-10 atomic binary checks derived only from the instruction; LLM-generated rubrics over-produce unnecessary (17.9% vs 10.1%) and overly rigid (12.8% vs 7.7%) rules; checklists of 13+ items add noise ('attention displacement'); even with human rubrics accuracy plateaus ~85% because judges treat must-have constraints as soft signals; safety domain worst for self-generated rubrics (~25-30% vs >90% human).
SOURCE: RubricBench: Aligning Model-Generated Rubrics with Human Standards | https://arxiv.org/html/2603.01562v1 | 2026-03-02 | academic
IMPLICATION: Directly bounds the per-project customization contract: instruction-only automated rubric compilation (no expert exemplars/references) leaves a large fidelity gap that more compute cannot buy back - an expert-grounding step (reference answers, adjudicated exemplars, or human rubric sign-off) is structurally required, not optional. Also: cap compiled checklist length (~<13 active criteria per judge call), and give hard constraints enforcement semantics (gating) rather than listing them as soft criteria.

## EvalGen ('Who Validates the Validators', UIST 2024) established criteria drift: evaluation criteria cannot be fully specified a priori because grading outputs itself changes the criteria - 'users need criteria to grade outputs, but grading outputs helps users define criteria' - and some criteria are dependent on the specific outputs observed. [LOAD-BEARING]
EVIDENCE: Fetched abstract/paper confirmed the catch-22 formulation, the finding that criteria appear dependent on observed outputs rather than definable in advance, and the conclusion that judge-alignment is inherently iterative and subjective; EvalGen operationalizes this by generating candidate assertions/judge prompts and selecting those best aligned with accumulated human grades.
SOURCE: Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences | https://arxiv.org/abs/2404.12272 | 2024-10-13 | academic
IMPLICATION: The rubric compilation pipeline must be a loop, not a compile step: initial rubric -> human grades a sample -> criteria are revised -> re-validate. Foundational consequence for the AutoQA: budget a per-project calibration phase where criteria are expected to change, version every rubric, and treat early judge-human disagreement partly as rubric-specification signal (rewrite) rather than judge noise (vote harder) - the two-builds fork in design question 5 has an evidence-backed default.

## RIFT (Snorkel AI, Apr 2026) provides the first rubric failure-mode taxonomy - 8 modes in 3 categories (Reliability: Subjective, Non-Atomic, Ungrounded; Content Validity: Misaligned/Rigid, Missing Criteria; Consequential Validity: Hackable, Low Signal, Redundant) - and shows failure-mode density correlates with judge-human misalignment (r=0.162, p=0.0021); synthetic rubrics skew Subjective (86.7% vs 52.6% for human-crafted) while human rubrics skew Misaligned/Rigid (63.2% vs 20.0%); automated LLM detection reaches F1 0.925 for Subjective but ~0.000 for Hackable.
EVIDENCE: Full HTML (arXiv 2604.01375v2) confirmed all numbers: 85 rubrics / 255 expert annotations across 5 sources (human: AdvancedIF, ResearchRubrics; synthetic: WildChecklists, OpenRubrics, AutoRubrics); taxonomy built via grounded theory, mean Cohen's kappa 0.64; diagnostics include LLM-as-judge classifiers (GPT-5.2, Gemini 3 Pro), inter-rater reliability across labeler models, alignment to a strong reference, and reward-variance probes.
SOURCE: RIFT: A RubrIc Failure Mode Taxonomy and Automated Diagnostics (Snorkel AI) | https://arxiv.org/html/2604.01375v2 | 2026-04-20 | academic
IMPLICATION: Rubric QA/linting is now a named, tooled 2026 practice: run compiled rubrics through a RIFT-style lint before go-live. Two structural lessons: (1) human and LLM rubric authors fail in complementary ways, motivating co-authoring rather than either alone; (2) hackability/gameability is the one failure mode automated diagnostics cannot detect - the anti-gaming review of the AutoQA's compiled rubrics must be a human red-team task, permanently.

## Support Vector Rubrics (Jun 2026) identifies the mechanism behind the LLM-vs-human rubric gap - 'self-generated rubrics describe good responses, whereas effective criteria must discriminate between close candidates' - and closes the RubricBench gap from 24.1 to 0.3 points by mining contrastive rubrics from preference pairs (support-pair selection plus adversarial probing of hard negatives).
EVIDENCE: Fetched abstract (arXiv 2606.08077, Sun et al.) confirmed the objective-mismatch framing, the max-margin method (rubric bank mined from preference data, prompt-conditioned selector, iterative refinement on hard negatives), transfer of the rubric bank across judges without retraining, and competitiveness with dedicated reward models on RewardBench 1/2 and RM-Bench.
SOURCE: Support Vector Rubrics: Closing the Gap Between Self-Generated and Human Rubrics | https://arxiv.org/abs/2606.08077 | 2026-06-06 | academic
IMPLICATION: The highest-value compilation artifacts are contrastive: borderline pass/fail adjudication pairs, not descriptions of quality. For the AutoQA's exemplar question (5 vs 50 vs 500), this says invest in hard-negative/near-miss pairs per criterion and derive discriminative criteria from them - descriptive 'what good looks like' statements alone are the documented failure pattern.

## PaperBench (OpenAI, Apr 2025) demonstrates the extreme end of decomposition: 8,316 individually-graded binary leaf criteria across 20 papers, sibling-relative manual weights, rubrics co-developed with each paper's original authors over multiple weeks per rubric; an o3-mini judge then grades leaves at F1 = 0.83 vs expert human judgments for ~$66/paper versus ~12 hours of expert grading time.
EVIDENCE: Full paper (arXiv 2504.01848) confirmed: each leaf is pass/fail; parent scores are weighted averages propagating to a root score; weights reflect importance relative to siblings, not difficulty; JudgeEval is a separate benchmark for judges (o1 scored 0.84 but ~$830/paper - o3-mini chosen as cost-effective); human expert grading estimated at 'tens of hours' per paper.
SOURCE: PaperBench: Evaluating AI's Ability to Replicate AI Research (OpenAI) | https://arxiv.org/html/2504.01848v2 | 2025-04-02 | primary
IMPLICATION: Atomic binary decomposition of genuinely expert judgments is feasible and auto-gradable at usable-but-imperfect agreement (F1 0.83), with ~100x cost reduction over expert grading - but rubric authoring cost is weeks of joint expert time per complex artifact. For AutoQA: fine decomposition pays at grading time and the judge gets its own meta-benchmark (JudgeEval pattern), but the compilation budget, not the judging budget, is the binding constraint per project.

## OpenAI's official grader documentation prescribes that every model grader be validated on its own eval built from expert-graded answers (grader score ordering must reproduce expert ranking), and names the canonical reward-hacking detection signature: the graded model scores well on the model-grader eval while doing poorly on expert human evaluations.
EVIDENCE: Fetched live doc (developers.openai.com/api/docs/guides/graders, accessed 2026-07-14): grader taxonomy = string_check, text_similarity, score_model (LLM judge with numeric range), python (sandboxed), multigrader (formula-combined sub-graders, RFT only); guidance: iterate grader prompts, use few-shot examples of good/fair/poor answers, prefer smooth over binary scores, add edge cases to the grader eval over time, 'guard against reward hacking'. Page carries a deprecation note for graders within the evals/fine-tuning workflows they support.
SOURCE: OpenAI API documentation: Graders | https://developers.openai.com/api/docs/guides/graders | 2026-07-14 | primary
IMPLICATION: Lab-published operational doctrine matching the AutoQA's needs: (1) mixed grader stacks - deterministic checks composed with LLM judges via explicit formulas - are the production pattern for the objective/subjective split; (2) a standing expert-audit channel is the designed-in hack detector (grader-vs-expert divergence), i.e., the independent expert-audit sampling channel is a permanent component in the reference architecture, not a bootstrap phase.

## A cluster of 2026 papers (Rubric-ARM Feb 2026; EvoRubrics, ARBOR, AMARIS Jun 2026) converges on the position that static rubrics are an exploitable reward specification under optimization pressure and must be adapted/co-evolved during training, with Rubric-ARM jointly optimizing a rubric generator and judge via alternating RL from preference feedback.
EVIDENCE: Rubric-ARM abstract opened and confirmed (prior approaches 'rely on static rubrics or disjoint training pipelines, which limits their adaptability'; alternating optimization with variance-reduction analysis; SOTA on judge benchmarks and downstream policy alignment). EvoRubrics/ARBOR/AMARIS confirmed at title/abstract level via search only - trend triangulated across 5+ 2026 arXiv titles, not deep-read.
SOURCE: Rubric-ARM: Alternating Reinforcement Learning for Rubric-Based Reward Modeling (plus 2026 co-evolution cluster) | https://arxiv.org/abs/2602.01511 | 2026-02-02 | academic
IMPLICATION: Attempters paid per accepted item exert the same optimization pressure as an RL policy: the 2026 consensus is that a frozen compiled rubric will be Goodharted, so rubric revision must be a designed, continuous loop (with regression gates), not an exceptional maintenance event. This directly supports treating criteria drift response and anti-gaming rubric rotation as foundational architecture.

## Autorubric (Feb 2026, Rao & Callison-Burch) consolidates scattered rubric-evaluation techniques into one open-source framework with opinionated defaults - binary/ordinal/nominal criterion types, judge ensembles, few-shot calibration, bias mitigations, and psychometric reliability metrics - reporting 87% binary accuracy with moderate-to-substantial kappa on CHARM-100, and showing per-criterion explanations double as improvement signals (peer-review agent 0.47 -> 0.85, beating a 0.82 expert-curated baseline).
EVIDENCE: Fetched abstract (arXiv 2603.00077, v2 2026-04-03, 52 pages) confirmed the unification claim ('scattered across papers with inconsistent terminology and partial implementations'), the three validation benchmarks (RiceChem 80% with 5-shot calibration; ResearcherBench 931 criteria cross-judge agreement; CHARM-100), and RL-reward use (+0.039 AdvancedIF, Wilcoxon p=0.032, positive IFEval transfer).
SOURCE: Autorubric: Unifying Rubric-based LLM Evaluation | https://arxiv.org/abs/2603.00077 | 2026-02-13 | academic
IMPLICATION: By early 2026 rubric-based judging has a consolidated reference implementation whose defaults (ensembles, few-shot calibration on human-graded examples, psychometric reliability reporting per criterion type) can be adopted wholesale rather than reinvented; and the same per-criterion explanation trace that justifies a verdict is demonstrated to work as constructive improvement feedback - verdict machinery and feedback generation can share one artifact.

## LangChain's Align Evals (LangSmith, Jul 2025) productized judge validation as a standard workflow: humans grade a representative golden set per criterion, an 'alignment score' measures judge-vs-human match, unaligned cases are surfaced for prompt iteration, and each judge-prompt version is compared against a saved baseline alignment score.
EVIDENCE: Fetched launch blog confirmed the four-step workflow (select criteria; select representative good+bad examples; human-grade expected scores; iterate evaluator prompt against alignment score), the stated problem ('Our evaluation scores don't match what we'd expect a human on our team to say'), inspiration from Eugene Yan's AlignEval, and roadmap items (alignment tracking over time, automatic prompt optimization).
SOURCE: Introducing Align Evals: Streamlining LLM Application Evaluation (LangChain) | https://www.langchain.com/blog/introducing-align-evals | 2025-07-29 | practitioner
IMPLICATION: The industry-default minimum-viable gold set practice: small per-criterion human-graded golden sets with versioned alignment baselines, refreshed as unaligned cases surface - evidence that per-criterion (not per-item-holistic) alignment scoring with regression baselines is the practical validation loop the AutoQA should ship with from day one.

## Tools & artifacts
- openai/simple-evals healthbench_meta_eval.py: Reference implementation of HealthBench's grader meta-evaluation (grader vs physician grades, Macro F1 per criterion) | https://github.com/openai/simple-evals/blob/main/healthbench_meta_eval.py | Directly reusable pattern for validating the AutoQA judge against expert-adjudicated labels per criterion - the meta-eval harness the foundation needs, already open-sourced
- Autorubric: Open-source framework (Rao & Callison-Burch, Feb 2026) unifying rubric-based LLM evaluation: binary/ordinal/nominal criteria, judge ensembles, few-shot calibration, bias mitigations, psychometric reliability metrics | https://arxiv.org/abs/2603.00077 | Closest existing embodiment of the AutoQA's judging core; its opinionated defaults are a checklist of what a 2026-credible rubric judge must include
- RIFT taxonomy + diagnostics: Snorkel AI's 8-mode rubric failure taxonomy with automated LLM lint classifiers and reliability probes (Apr 2026) | https://arxiv.org/html/2604.01375v2 | Ready-made rubric-QA lint stage for the compilation pipeline; also documents which rubric defects (Hackable) automated linting cannot catch
- RubricBench: 1,147 adversarial pairwise comparisons with expert atomic rubrics for measuring rubric-generation and rubric-execution quality (Mar 2026) | https://arxiv.org/html/2603.01562v1 | Off-the-shelf stress test for any automated rubric-compilation procedure the AutoQA adopts; its surface-bias-controlled pairs also probe judge style bias (design question 5)
- LangSmith Align Evals: Productized judge-alignment workflow: human-graded golden set, per-criterion alignment score, versioned baselines (Jul 2025) | https://www.langchain.com/blog/introducing-align-evals | Working example of the minimum-viable judge-validation loop and UI pattern for surfacing judge-human disagreements for iteration
- OpenAI Graders API + documentation: Composable grader stack (string_check, text_similarity, score_model, python, multigrader with formula aggregation) with published calibration and anti-reward-hacking guidance | https://developers.openai.com/api/docs/guides/graders | Lab-published architecture for mixing deterministic checks with LLM judges under one aggregation formula - the objective/subjective lane split in API form (note: page carries a deprecation notice)
- PaperBench rubrics + SimpleJudge (openai/preparedness): 20 author-co-developed hierarchical rubrics (8,316 binary leaves) plus an LLM judge and JudgeEval, a benchmark for the judge itself | https://arxiv.org/html/2504.01848v2 | The 'benchmark your judge separately' (JudgeEval) pattern and the sibling-relative weighting scheme are directly transferable to AutoQA verdict aggregation

## Disagreements
- LLM-generated vs expert rubrics: Scale's RaR (Jul 2025) reports reference-grounded synthetic rubrics match human-authored ones (0.359 vs 0.348 as RL rewards), while RubricBench (Mar 2026) measures a stable ~26-27pt preference-accuracy gap that test-time compute cannot close, and SVR (Jun 2026) claims the gap closes (24.1 -> 0.3) only by mining discriminative rubrics from preference pairs. The likely reconciling variable is grounding: RaR's generator saw expert reference answers, RubricBench's saw only the instruction - but no paper tests all conditions head-to-head, so 'can compilation skip expert grounding' remains unsettled; the safe design assumes it cannot.
- Aggregation rule: RaR found implicit aggregation (judge internally weighs all criteria into one score) consistently beats explicit per-criterion binary + weighted-sum for reward quality (up to 28% relative), yet the flagship production benchmarks (HealthBench, PaperBench) and OpenAI's grader doctrine use explicit independent per-criterion judgments with arithmetic roll-ups for auditability. Performance and auditability pull in opposite directions; no source resolves the tradeoff for a QA setting where verdicts must be explained.
- Decomposition granularity: PaperBench succeeds with 8,316 atomic binary leaves (judge F1 0.83, each leaf graded separately), while RubricBench finds checklists of 13+ items degrade judgment via 'attention displacement', and RIFT lists both Non-Atomic AND Redundant as failure modes. The apparent resolution - atomicity helps when each criterion gets its own judge call, hurts when one call processes a long checklist - is inferred, not directly tested anywhere.
- Static vs dynamic rubrics: production eval practice (HealthBench, PaperBench, OpenAI graders) pins rubrics and regression-tests judges against them, treating rubric stability as integrity; the 2026 RL literature (Rubric-ARM, EvoRubrics, ARBOR, AMARIS) argues static rubrics are inherently exploitable specifications that must co-evolve with the optimizing population. For an AutoQA facing adaptive paid attempters, these prescriptions conflict: pin-and-regression-gate vs continuously-adapt.
- Judge-model reasoning strength: HealthBench found the non-reasoning GPT-4.1 outperformed reasoning models o3/o4-mini as a rubric grader (MF1 0.709 vs 0.681/0.692), while PaperBench selected reasoning model o3-mini (high effort) and found o1 best (F1 0.84). Whether rubric grading benefits from reasoning-class judges appears task-dependent and both labs partially attribute results to prompt-tuning conducted with the winning model.

## Gaps
- No published 2025-2026 study applies rubric-based LLM autograding to QA of HUMAN annotators' work product (annotations/critiques/ratings). All quantitative evidence in this domain grades model outputs or RL rollouts; transfer to human-attempter QA - including how paid humans adapt adversarially versus how policies reward-hack - is an assumption, not a finding.
- No major training-data vendor (Scale, Surge, Mercor) has published a complete rubric-writing methodology guide. The closest public artifacts are RaR's four design principles, OpenAI's grader-doc guidance, HealthBench's physician criteria process, and RIFT's failure taxonomy; Appen markets 'rubric design' services but publishes no method detail. Vendor-internal rubric-writing guides likely exist but are not public.
- Criteria drift has no 2026 quantitative follow-up: EvalGen's finding is qualitative (small-n user study, 2024). No source measures drift rates, drift half-life, or drift-triggered revision thresholds in a production rubric pipeline - the exact parameters design question 9 needs.
- The 2026 rubric co-evolution cluster (EvoRubrics, ARBOR, AMARIS) was verified only at abstract/title level; their specific numbers were not confirmed by opening full texts. The trend claim rests on Rubric-ARM (opened) plus five-plus corroborating 2026 titles.
- OpenAI's graders documentation page carries a deprecation notice for the graders/evals workflows; what replaces them in OpenAI's mid-2026 stack was not tracked down, so the grader taxonomy finding may describe a sunsetting API surface (the design doctrine it encodes remains valid evidence of practice).
- No source directly measures the minimum number of adjudicated exemplars per criterion before judge-human agreement plateaus (the 5 vs 50 vs 500 question); Autorubric's '5-shot calibration -> 80%' on one chemistry dataset is the only datapoint found, and SVR implies exemplar QUALITY (hard contrastive pairs) dominates count.

## Verifications
- CLAIM: OpenAI HealthBench meta-evaluates its LLM grader per-criterion against physician majority grades using Macro F1 (met/not-met, class-balanced), and reports the grader as a percentile of individual physicians: GPT-4.1 grader MF1 = 0.709, exceeding the average physician in 5 of 7 themes, while physician-vs-physician MF1 is only 0.569-0.730 with wide individual spread.
  VERDICT: confirmed | Fetched https://arxiv.org/html/2505.08775v1 (marked arXiv:2505.08775v1 [cs.CL] 13 May 2025 - date correct). All claim elements verified against Section 8.1 / Tables 5-7: (1) meta-evaluation uses macro F1 on binary met/not-met per consensus criterion, explicitly chosen for class imbalance (random baseline MF1=0.50); (2) GPT-4.1 grader MF1 = 0.709, best of graders tested (o4-mini 0.692, o3 0.681, GPT-4.1 mini 0.661, GPT-4.1 nano 0.580); (3) over 60,896 meta-examples across the 34 physician-consensus criteria (avg 1,791/criterion); (4) 'typical physician' baseline computed by scoring each physician against the others the same way as the model, reported as a percentile - GPT-4.1 exceeds the average physician in 5 of 7 themes (below only expertise-tailored communication 0.610 vs 0.618 and health data tasks 0.683 vs 0.730), upper half in 6/7, above 33rd percentile in all; (5) theme-level weighted-average physician-vs-physician MF1 ranges 0.569 (response depth) to 0.730 (health data tasks), with paper noting physician-physician and model-physician agreement both span ~55-75% and wide individual spread; (6) grading prompt AND individual criterion phrasings were tuned so intent was unmistakable to the grader (paper also cautions GPT-4.1's top rank may be partly explained by its use during prompt tuning); (7) std across 16 full runs ~0.002 (Table 7: 0.0016-0.0029 by model). Supersession check: no arXiv v2 exists; 2026 activity is HealthBench Professional (OpenAI, April 2026) and third-party replications (e.g., open-source graders: Kimi-K2 0.693, Qwen3-235B 0.681 vs GPT-4.1 0.709) - these extend, not contradict, the original meta-eval numbers. Only pedantic nuance: 0.569-0.730 are theme-level weighted averages of physician MF1, not the full range across individual physicians (individual spread is wider) - the claim's own 'wide individual spread' wording already captures this correctly.
- CLAIM: a large data vendor's Rubrics-as-Rewards found that LLM-synthesized rubrics grounded in expert reference answers performed on par with human-authored rubrics as reward signals (0.359 vs 0.348), that prompt-specific rubrics substantially beat generic ones, and that the one component synthetic generation could NOT produce well was Pitfall criteria - anticipating failure modes 'often demands human intuition and domain expertise'.
  VERDICT: confirmed | All core assertions verified against the source (arXiv 2507.17746, a large data vendor). (1) Table 3: RaR-Implicit-Synthetic 0.359 vs RaR-Implicit-Human 0.348 on HealthBench - synthetic rubrics grounded in expert reference answers on par with human-authored; ungrounded synthetic dropped to 0.320, confirming the grounding caveat. (2) Prompt-specific vs generic confirmed: "predefined generic rubrics substantially underperform compared to prompt-specific ones" (generic ~0.29 vs ~0.37 prompt-specific in ablations; Predefined 0.2472 vs RaR-Implicit 0.3194 on HealthBench-1k). (3) Pitfall claim confirmed verbatim: paper states synthetically generating effective pitfall criteria "is inherently difficult" and anticipating failure modes "often demands human intuition and domain expertise." Evidence-summary details also check out: four desiderata (expert-grounded, comprehensive, semantic weighting Essential 1.0 / Important 0.7 / Optional 0.3 / Pitfall 0.8 penalty, self-contained), o3-mini/GPT-4o generation, 7-20 criteria per prompt, implicit aggregation beating explicit weighted-sum with up to 28% relative gain over Simple-Likert on HealthBench-1k (v1 figure; the current v2 abstract cites up to 31%), smaller judges benefiting most, and reduced cross-judge score variance (0.250-0.279 vs 0.220-0.254). Date correct: v1 submitted 2025-07-23. Minor currency notes, not refutations: (a) a v2 exists (2025-10-03) and the paper was accepted at ICLR 2026 - citing the v1 HTML is fine but the 28% headline became 31% in the current version; (b) substantial 2026 follow-on literature (Open Rubric System arXiv 2602.14069, ARBOR 2606.03239, Scale's own "Chasing the Tail" 2509.21500 and Enterprise RaR blog) critiques RaR's static pointwise-weighted rubrics as having a discriminability ceiling and gaming vulnerability, but none contradicts the specific findings claimed. Sources: https://arxiv.org/abs/2507.17746, https://arxiv.org/html/2507.17746v1, https://openreview.net/pdf?id=c1bTcrDmt4, https://scale.com/blog/enterprise-rar
- CLAIM: RubricBench (Mar 2026) measures a stable ~26-27 point preference-accuracy gap between self-generated and human-annotated rubrics across all judge backbones (e.g., DeepSeek-v3.2 57.8% vs 84.9%; Gemini-3-Flash 58.0% vs 85.3%), and scaling test-time compute (more sampled rubrics, refinement) does not close it - 'rubric formation', not judge reasoning, is the bottleneck.
  VERDICT: partially_confirmed | Source check (arXiv 2603.01562v1, fetched): the RubricBench paper exists, is dated 2026-03-02 (v2 2026-03-03), and confirms nearly every specific: 1,147 adversarial pairwise comparisons; DeepSeek-v3.2 57.8% self-generated vs 84.9% human (vanilla 38.8%); Gemini-3-Flash 58.0% vs 85.3% (vanilla 56.4%); test-time scaling fails (Rub@4->Rub@32 flat or declining, refinement depth declining, while scaling HUMAN rubrics helps 75.4->85.3); unnecessary-rule 17.9% vs 10.1% and rigid-rule 12.8% vs 7.7%; 13+-item checklists / 'Attention Displacement' with >70% hallucination rates; ~85% human-rubric plateau; safety ~25-30% self-generated vs >90% human. Two corrections: (1) the gap across ALL backbones is ~22-28 points, not a tight '~26-27' - 26-27 fits only the two cited models; (2) the 'rubric formation is the bottleneck' framing is the paper's interpretation and, more importantly, has been operationally SUPERSEDED in part: 'Support Vector Rubrics' (arXiv 2606.08077, June 2026, PKU/USTC) closes the gap on RubricBench to 0.3 points (82.8 vs 83.1 human-oracle, vs 59.0 self-generated with same GPT-OSS-120B judge) via max-margin contrastive rubric-bank learning over preference data - so the gap is stable under naive self-generation and test-time compute, but not under trained discriminative rubric construction. Safety remains the residual weak domain even for SVR (83.8 vs 92.5). Claim is accurate as a description of the March paper; the 'does not close' generalization no longer holds as of June 2026.
  CORRECTED: RubricBench (arXiv 2603.01562, Mar 2026) measures a stable ~22-28 point preference-accuracy gap between self-generated and human-annotated rubrics across judge backbones (DeepSeek-v3.2 57.8% vs 84.9%; Gemini-3-Flash 58.0% vs 85.3%), and scaling test-time compute (more sampled rubrics, refinement depth) does not close it - the paper attributes the bottleneck to rubric formation, not judge reasoning. However, follow-up work (Support Vector Rubrics, arXiv 2606.08077, Jun 2026) closes the gap to ~0.3 points on RubricBench via max-margin rubric-bank learning from preference data, showing the gap is specific to naive self-generation rather than fundamental.