Judges

A judge is a scorer that grades each trace or session against a rubric and produces a quality signal between 0 and 1 — plus a written reason you can audit. Judges are how raw traces become scores: enable one, run it, and every graded trace gets a number you can filter, chart, and gate on.

At a glance

Judge types

Each judge targets either a trace (one span tree) or a session (a conversation rollup).

Neens also ships a built-in judge library — Faithfulness, Answer Relevancy, Hallucination, Toxicity, Bias, PII Leakage, Correctness, and more — visible in every agent under Available judges. Library judges are visible but score nothing until you enable them. Two scorers are provisioned per agent out of the box:

• Primary Score — a composite that blends Faithfulness, Answer Relevancy, and Coherence into one headline quality number and scores a sample of incoming traces continuously (see Continuous evaluation).
• Issue Classification — a classifier that tags failing sessions with the failure mode they exhibit, from your taxonomy. It runs on the failure set only, not on healthy traffic.

Create a judge

[{"lc": 1, "type": "constructor",
  "id": ["langchain", "schema", "messages", "HumanMessage"],
  "kwargs": {"content": "Can I return a sale dress?"}}]

Evidence:
INPUT (the user's request / task):
Can I return a sale dress?

OUTPUT (the agent's response to evaluate):
Sale items are final sale.

Output types

By default an LLM Prompt judge returns a Score 0–1 and Neens asks the model for a score float. But many rubrics are naturally a numeric range or a set of labels — so the Output type control lets a judge score on its own native scale while Neens keeps a normalized 0–1 score under the hood for dashboards, composites, clustering, and pass-rate. The judge’s native value is captured alongside for display, and you define what counts as a pass in native terms.

The stored score is always the normalized 0–1 value, so every downstream chart and composite keeps working unchanged; the native score/label is what you see on run drill-downs and the score catalogue. A judge with no output type set behaves exactly like the classic 0–1 path.

Composite judges

A composite judge combines other judges. Pick a method:

• Weighted sum — assign a weight to each component; Normalize to 1.0 rebalances weights over the components that are actually present per target.
• Expression — arithmetic over component scores (e.g. 0.6*answer_completeness + 0.4*bias), referencing each component by its variable name. Supported operations: + - * / ** % and min / max / abs / round. An invalid expression fails the run up front rather than degrading silently.

Set a Missing policy — Skip missing (a target missing a component score is skipped) or Fail on missing — and a Threshold for the composite’s pass/fail label.

When a composite runs, each component’s most recent score is reused if one exists; an LLM-prompt component with no score yet is scored inline once (its own score row is written too, so it is cached for next time). Primary Score works exactly this way, which is why its component metrics appear in the catalogue without their own enabled scorers.

Lifecycle stage

Every judge carries a lifecycle stage — a curation signal that says how far along the prompt is, without changing anything about how the judge scores:

Change a judge’s stage from the Judges page. The move is one-directional in spirit — experimental → finalized → archived — but you can set any stage at any time (e.g. re-open an archived judge back to experimental).

The change is recorded in the activity/audit trail and takes the same permission as configuring a judge, so a viewer can’t silently re-stage someone else’s scorer. Via the API, PATCH /judges/{id}/lifecycle with {"stage": "experimental" | "finalized" | "archived"} (any other value is a 422); a scoped caller can only re-stage a judge visible to its agent.

Enable it as a scorer

Creating (or having) a judge scores nothing by itself. Click Configure to enable it as a scorer in the current agent, where you choose:

• Version to run.
• Apply for — Manual (run on demand) or Continuous (score new traces). See Continuous evaluation.
• What traces to score — everything eligible, a fixed count, or a percentage sample. A live estimate shows how many traces are Eligible and how many it Will score.
• Target source — Filter (traces matched by status/agent/start-date filters) or Dataset (score only the members of a dataset; sampling still applies).
• Scoring connections (pool) — which LLM connection(s) score the traces. Pick several to spread load across providers and dodge rate limits; leave empty to use the agent’s default connection.
• Success threshold — an optional per-scorer minimum success rate for a run to count as fully completed (blank → the platform default, see Run outcomes).

Enabled scorers appear in the Enabled scorers table with two independent controls: Run now (a one-off run over existing traces) and Pause / Resume (continuous) or Disable / Enable (manual) for automatic scoring. Remove deletes the scorer; removing a platform-default judge sticks — it won’t silently re-enable itself.

You can also ask the Assistant to do this — “turn that judge on and score the last week” — or drive it from an agent over the MCP server. Neither needs a version id: omit it and the judge’s latest version is deployed, and an eval can be started from the judge alone when it has a single enabled deployment.

Run it

# Preview how many traces a filter would score
curl -X POST /eval-targets/estimate \
  -H 'Content-Type: application/json' \
  -d '{"filter": {"sampling": {"mode": "percentage", "percentage": 10}}}'
# → {"eligible": 1240, "sampled": 124}
 
# Start a run for an enabled scorer (deployment)
curl -X POST /eval-runs \
  -H 'Content-Type: application/json' \
  -d '{"deployment_id": "dep-…"}'
# → {"id": "run-…", "status": "queued", "total": 124, …}
 
# Cap THIS run at the 200 most-recent eligible traces (the deployment's filter is untouched)
curl -X POST /eval-runs \
  -H 'Content-Type: application/json' \
  -d '{"deployment_id": "dep-…", "sample_size": 200}'
# → {"id": "run-…", "status": "queued", "total": 200, …}  (total is what was actually queued)
 
# Poll the run: status, per-target tasks, scores + reasons
curl /eval-runs/run-…?limit=200
 
# Stop it
curl -X POST /eval-runs/run-…/cancel

Cancel or recover a run

A scorer run moves through queued → running to a terminal state — completed, completed-with-failures, failed, cancelled, or skipped (see Run outcomes for the grading bands and the Activity feed for the full status list). Two things are worth knowing before you launch one.

Enabling a scorer doesn’t run it. Turning a scorer on only makes it eligible: a manual scorer scores nothing until you click Run now, and a continuous scorer scores only new traces as they arrive — never your existing history. To score the traces you already have, use Run now. (The Judges page says the same inline: “Enabling a scorer doesn’t run it automatically. Click Run now…”)

Cancel a running run

Click Cancel on a live Run now run — or POST /eval-runs/{id}/cancel. Cancelling takes effect promptly: any scoring call already in flight finishes, but the run’s remaining queued targets are dropped rather than drained, so scoring stops within seconds instead of grinding through work you no longer want.

The cancelled run then reads Cancelled in both places it appears — its scorer’s run history on the Judges page and the run’s row on the Activity page — so the two never disagree about whether it is still going. Whatever scored before you stopped it keeps its scores, but a cancelled run’s partial counts are never graded into a pass/fail verdict.

If a run is interrupted

If the processing behind a run is interrupted while it is still in flight, the run is automatically moved to a terminal state — it reads Cancelled rather than being left showing Running forever. So a scorer never gets wedged: once the stuck run is closed out, simply Run now again to re-score cleanly. This is the same self-healing you see for clustering runs on the Activity feed (a stalled clustering run is marked failed there).

How it works

When a run starts, Neens resolves the scorer’s judge version and target set (the filter or dataset, sampled as configured), creates one task per target, and scores them:

1. Evidence assembly — for each target, the selected evidence blocks are read from the trace, parsed, and rendered into the prompt. The live preview, the Test run button and a real run all call the same assembly and the same renderer, so they cannot drift: what the preview shows is byte-for-byte what the model receives.
2. Scoring — the prompt is sent to a connection from the run’s pool; targets round-robin across pool members. The model must return strict JSON — {"score": …, "reason": …}, or {"label": …, "reason": …} for a categorical or binary output type — which is parsed defensively and normalized to a 0–1 score.
3. Persistence — each verdict is written as a score row (metric = the judge, source llm_judge, or composite for composites) and appears immediately in the score catalogue, on run drill-downs, and in dashboards.

Judge calls within a run execute sequentially — one LLM call at a time per run.

Transient failures are retried. A timeout, 429, or 5xx from the LLM provider is retried with exponential backoff — up to 3 retries per target by default, with delays growing from 0.5s up to a 30s cap. Terminal failures (an unparseable verdict, a config or auth error) are never retried; that target is marked failed with its error and the run moves on.

Run outcomes

A finished run is graded by its success rate — the fraction of attempted targets that scored successfully (skipped and cancelled targets don’t count against it):

The green cutoff can be overridden per scorer (Success threshold in Configure, or per run in the Run dialog).

Filter a run’s scores, compare versions, and export

A judge’s run history (open it by clicking a scorer row) is also where you audit and compare ad-hoc batch runs — the manual Run now runs, each tied to one judge version. These flows read each run’s own per-target snapshot, so a later run overwriting the shared score row never corrupts an older run’s numbers.

• Scope the Scores page to one run. From a specific ad-hoc run you can view exactly the scores that run produced, rather than the metric’s current live scores. API: GET /scores?runId=<runId> (returns the run’s snapshot — this is meant for ad-hoc runs, not a continuous scorer that overwrites each trace’s row every time it re-scores).
• Compare two versions. Pick two runs of the same judge — typically runs of two different versions — to see, per target, which sessions improved, regressed, or stayed unchanged, plus each side’s score and reason and the average delta. API: GET /judges/{id}/compare?runA=<runId>&runB=<runId>; a run from another judge or agent is a 404.
• Export a run with input and output. Download a run’s (or several runs’) scores as CSV or JSON, including each target’s input and the agent’s output alongside the score, label, threshold, and reason — handy for spreadsheets, offline review, or sharing a regression. API: GET /eval-runs/export?runs=<id>,<id>&format=csv|json.

Judge ↔ human alignment

An LLM judge is only useful if you can trust it. The Alignment tab measures each judge’s agreement (precision, recall, Cohen’s κ) against human gold labels and lets you drill into disagreements. See Annotations & review for the full label-review-align loop, and Pre-prod evaluations for running your trusted judges against a candidate release.

The Run Targets and Comparison tabs

Two more tabs on this page go one level deeper on what a scorer runs on and which scorer model to trust:

• Run Targets — the library of reusable populations (a golden dataset, a live sample, a filtered sweep) that scorers run against, plus each target’s Used by list. Attaching a scorer to a Run Target is done from that scorer’s own Run Targets tab. Full guide: Run Targets.
• Comparison — holds one judge’s rubric fixed and scores the same window of traces with two or more candidate scorer LLMs, side by side, so you can see whether a cheaper model agrees closely enough to switch. Full guide, with a worked example: Cost Optimization → Compare scorer models side by side.

Troubleshooting