Annotations & review

Human labels are the ground truth for Neens: a reviewer looks at a session and records a pass or fail verdict with a critique. The Review page queues the sessions most worth a human look, the Annotations tab is the browsable history of every label, and the Alignment view measures how well your judges agree with those humans — closing the loop that keeps automated scoring honest.

At a glance

The Review queue

The Queue tab is a prioritized worklist, not a raw session list. Each candidate session is ranked by five signals — Impact, Uncertainty, Novelty, Disagreement, and Redundancy — so reviewer time goes where a human verdict changes the most: high-impact sessions, scores sitting near their pass/fail threshold, traffic unlike anything labeled before, and places where judges contradict each other, while clusters that already have several labels are de-prioritized.

Configure the queue

The queue configuration is saved per agent (open the config drawer on the Queue tab):

Label a session

The Annotations tab

The Annotations tab on the Review page browses the same label store: every label with its target, verdict, critique, failure-mode/cluster tag, gold flag, timestamp, and who wrote it — each label permanently carries its reviewer’s name, and superseded labels remain inspectable.

It also hosts the Annotator leaderboard: per reviewer, label volume, pass/fail split, gold labels authored, and — where a reviewer has labeled targets that also have expert truth — their agreement rate and Cohen’s κ against it. Volume gets recognition; agreement tells you whose labels to trust.

Judge ↔ human alignment

Once you have expert labels, the Alignment tab on the Judges page measures each judge against them. A judge’s most recent verdict per session is paired with the expert truth for that session, treating failure as the positive class:

Convergence chart

Each recompute persists a measurement, so the chart plots every judge as a line over time — toggle between κ, Precision, and Recall, and narrow the window with the time-range picker (Today / 24h / 7d / 30d / All / Custom). A rising line means your judge edits are converging on human judgment; a flat low line means the rubric still doesn’t match what your experts consider a failure.

Alignment is recomputed automatically every night at 3:00 UTC, and on demand with the refresh action. With no gold/expert labels yet, it degrades gracefully — you’ll see an empty trend with the reason, never fabricated numbers.

Disagreement drill-down

Select a judge to see exactly where it diverged from the experts, split into:

• Missed failures — the judge passed a session an expert failed (the costly kind).
• False alarms — the judge failed a session an expert passed.

Each row shows the judge’s score and reasoning next to the expert’s critique. If the human was wrong (or the case is genuinely ambiguous), click Relabel to write a corrected gold label in place — then hit Recompute alignment to refresh the metrics immediately.

Closing the loop: iterate the judge

Alignment turns judge editing from guesswork into a measured cycle:

1. Measure — check κ/precision/recall for the judge on the Alignment tab.
2. Diagnose — read the disagreements: is the judge missing a failure type your experts catch, or nitpicking things they don’t care about?
3. Edit — update the judge’s instructions or criteria; saving publishes a new immutable version (score history from older versions is preserved). See Judges.
4. Re-run — run the new version over labeled sessions and recompute alignment.
5. Repeat until the trend converges — then trust the judge at scale, including as a gate in pre-prod evaluations.

Troubleshooting

See also the FAQ.