← Field Notes
JUN 13 · Clipped · via Google Cloud blog Comparison ViewEval Set Authoring

Google writes the rubric per example and checks each rule pass or fail

A rubric per example hides a cost: a standard that shifts with each case is hard for a team to share or defend. The pass and fail examples beside each rule are what let a team hold the rubric steady across cases.

Read the source ↗ attached to Comparison View →
watching

watching → emerging → established

A candidate behavior, not yet a published pattern. Early signal — worth tracking, not yet worth standardizing on.

10 documented sightings since Jul 2024 — every one traces to a dated source.

See the evidence →

Machine summary of the source

Vertex AI generates adaptive rubrics for each example, then applies them as pass or fail checks with a reason for each rule. The autorater can be benchmarked against human-rated examples. The console adds a comparison candidate to a run for a side-by-side view with win and tie rates. Google's own line: trust is built through transparency and control.

The summary above is generated; the note at the top is the editorial judgment. Primary source ↗