← Field Notes
JUN 17 · Paper · via arXiv Case DrilldownComparison View

21 judge models agreed with themselves and still carried bias

A score you cannot open is a score you cannot trust. The judge's reasoning, and the cases where humans overruled it, belong on the screen beside the number.

Read the source ↗ attached to Case Drilldown →
watching

watching → emerging → established

A candidate behavior, not yet a published pattern. Early signal — worth tracking, not yet worth standardizing on.

8 documented sightings since Aug 2024 — every one traces to a dated source.

See the evidence →

Machine summary of the source

A study of 21 judge models over about 541,000 judgments found that a judge can be very consistent with itself and still carry bias. Test-retest agreement above 0.95 sat beside position bias above 0.10 in two production judges. Agreement with humans dropped by 33 to 41 points once chance agreement was removed. The authors argue that reliability and validity are different tests, and that a consistent judge is not for that reason a correct one.

The summary above is generated; the note at the top is the editorial judgment. Primary source ↗