Review

The Review page is where people label agent runs pass or fail. Neens builds you a queue of the items where a human verdict is worth the most, shows you what happened, and records your verdict as ground truth — the reference your judges are measured against, the source of your regression sets, and the confirmation that a failure mode is real.

At a glance

What your verdicts feed

Every verdict you record lands in the same shared ground-truth store the rest of Neens reads. Nothing needs to be exported or synced.

Traces or sessions

Use the Traces / Sessions switch at the top of the page to choose what you are judging. Each has its own queue, so switching never loses your place in the other.

Sessions only appear if your agent sends a conversation id with its traces — see Traces & sessions → Sessions.

The workspace

The Queue tab has three columns.

1. Queue list (left). Every item in your queue, in order, with its age, agent and a reason chip — Errored, Judge on the fence, Judges disagree, New pattern, Low score or Random check. A dot shows each item’s state (to do, pass, fail, skipped). Filter between To do and Done. A progress bar above shows how many you have reviewed, the pass / fail / skipped split, and an estimate of the time left.
2. What happened (centre). For a trace: what the user asked, what the agent answered, and the steps it took (LLM calls, tool calls) with durations; errors are shown in place. Open the Spans tab to inspect a step’s inputs and outputs. For a session: the transcript of the whole conversation, and the list of traces it is made of. When Neens knows what to check, a Look for hint sits on top (for example “the refund for the duplicate charge was never issued”), taken from the judge that flagged the item or from the error.
3. Verdict panel (right).
   • Pass or Fail.
   • Skip when you can’t tell; Not sure: flag for discussion to mark it for a teammate.
   • Failure mode — if it failed, pick the failure mode. Neens pre-selects a Suggested one when the item belongs to a known failure mode.
   • What went wrong? — an optional critique. Keep it to one sentence a judge author can act on.
   • Mark as a gold example — tick this when you are confident and the case is a good yardstick; gold labels are used to check judge accuracy.
   • Why you’re seeing this — the item’s rank in the queue and the signals that put it there (see below).

Recording a verdict moves you to the next to-do item. Changing your mind later is safe: relabelling the same item replaces your earlier label but keeps the old one in the history.

Build a queue

Open the queue builder from the controls at the top of the Queue tab (the source, time window and size chips, or More filters).

How queues behave

• Your queue is yours. Each person has their own queue per agent and per grain (one for traces, one for sessions). Teammates’ queues can overlap, so keep Skip items I or a teammate already labelled on to avoid labelling the same item twice.
• Order is stable. A queue stays in the same order across reloads and devices until you regenerate it, so you can stop and pick up where you left off.
• Regenerate keeps your labels. Regenerate rebuilds the queue with the current settings. Items you already labelled stay, in order; only unlabelled items are replaced with fresh candidates.
• Labels made elsewhere show up. If someone labels an item from the Traces page, it shows as done in your queue too.
• No queue yet? The first time you open Review, Neens builds one from the agent’s saved default (or Smart mix, last 7 days, 25 items).

Why you’re seeing this

Every item in a Smart mix queue is ranked by five signals. The verdict panel shows the item’s rank (“Ranked #3 of 25”) and each signal rated High, Medium, Low or None relative to the other items in the queue, with the evidence behind it. How items are ranked opens the full breakdown.

The priority score combines them:

priority = 0.35 × impact + 0.25 × uncertainty + 0.25 × novelty + 0.15 × disagreement − 0.20 × redundancy

Each signal is between 0 and 1, so a run that errored badly (impact 0.95), with a judge near its line (uncertainty 0.55), that judges disagree about (disagreement 1), in a familiar failure mode (novelty 0.05), that nobody has labelled yet (redundancy 0) scores 0.33 + 0.14 + 0.01 + 0.15 − 0 ≈ 0.63. Items below about 0.30 usually teach judges little.

Already covered is the only signal that lowers priority: once a failure mode has several labels, more of the same teaches little, so the queue moves on to something new.

Keyboard shortcuts

Keyboard shortcuts work on the Queue tab whenever you are not typing in a text box.

When the queue is complete

When the last item is done, Neens shows a summary of what your labels changed:

• Reviewed — the count, the pass / fail / skipped split and how long it took.
• Judge disagreements found — verdicts that contradicted a judge. Each was added to the regression set automatically.
• Judge agreement — each affected judge’s agreement with people, before and after your labels, and how many labels it is now based on (for example “Resolution quality” agreement 71% → 76%, now based on 41 labels (was 16)).
• Failure modes confirmed — the failure modes you tagged, with counts.

Then pick what’s next: Build another queue, Check judge alignment, Review the skipped items, or review the next batch when more candidates still match your filters (“142 traces still match this queue’s filters, and 117 of them are unlabelled”).

How-tos

Confirm a failure mode before fixing it

A failure mode is a hypothesis until a person has looked at it. Confirm it before you build a regression set or start a fix.

Calibrate a judge with a random sample

Ranked queues are deliberately full of hard cases, which makes them great for teaching judges and bad for measuring them. To measure a judge’s real accuracy, label a random sample.

Review whole conversations for a support agent

For a support agent, the question that matters is often whether the customer’s problem got solved, not whether each step looked right.

Review via the API

Everything the workspace does is available over the API, relative to your Neens host. Calls use an API key from Settings → API keys, sent as a bearer token. Every route is also available without the /api prefix.

Build (or regenerate) a queue. The body is the same as the builder’s settings; only the fields you want to change are needed.

curl -X POST https://your-neens/api/review/queues \
  -H "Authorization: Bearer $NEENS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "grain": "trace",
    "source": "failure_mode",
    "window": "7d",
    "size": 25,
    "filters": { "failureModeId": "fm_wrong_resolution" },
    "skipLabelled": true,
    "exploreShare": 0.2
  }'

The response is the queue plus its items, in order. Each item carries the targetType and targetId to label, its status, its rank and priority, and the plain-language reasons:

{
  "queue": { "id": "rq_8f2c…", "grain": "trace", "source": "failure_mode", "size": 25,
             "candidateCount": 142, "mix": { "errored": 14, "borderline": 6, "conflict": 3, "novel": 2 } },
  "items": [
    { "position": 0, "targetType": "session", "targetId": "29c14f8f…", "status": "todo",
      "rank": 1, "priority": 0.75,
      "primaryReason": { "kind": "errored", "label": "Errored" },
      "reasons": [ { "signal": "impact", "level": "high", "title": "It failed",
                     "detail": "tool.issue_refund returned an error; 18.4k tokens" } ] }
  ],
  "progress": { "total": 25, "todo": 25, "pass": 0, "fail": 0, "skipped": 0, "flagged": 0 }
}

Record a verdict. Pass the queue_id so the item is marked done in the queue:

curl -X POST https://your-neens/api/review \
  -H "Authorization: Bearer $NEENS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "queue_id": "rq_8f2c…",
    "target_type": "session",
    "target_id": "29c14f8f…",
    "verdict": "fail",
    "critique": "Offered a credit instead of refunding the duplicate charge.",
    "failure_mode_id": "fm_wrong_resolution",
    "is_gold": false
  }'

Troubleshooting

See also Annotations & alignment and the FAQ.