Scores

A score is one quality verdict about one trace or session — a number between 0 and 1, plus the reasoning behind it. Scores are what judges produce and what everything downstream consumes: filters, dashboards, failure clustering, and release gates. The Scores page is the catalogue where every metric your agent produces lives, with per-user favorites and hiding so you see the metrics you care about first.

At a glance

Anatomy of a score

Every score row carries:

The last two are what make quality sliceable per model; they surface through the metrics catalogue (as the Model, Agent, and Model attribution dimensions) rather than as columns on the Scores list — see Model comparison.

A score type — one card in the catalogue — is identified by the triple (metric_key, source, target_type). The same metric key can appear more than once if different sources or target types produce it.

Where scores come from

Browse the catalogue

The Scores page opens on a card grid — one card per score type — over a selectable time window (default 30d), showing scores from Finalized judges only by default (see Lifecycle filter). Every card shows how many targets were scored, the pass rate, the fail count, a sparkline, and pills for its source and target type.

What the card leads with depends on what the scorer emits. A numeric scorer shows its average as a donut gauge and its delta; a boolean or categorical scorer shows a label distribution — the same donut shape, cut into one arc per label with a named legend beside it — because a mean of low / medium / high answers nothing. See Boolean and categorical scorers.

Click a card to drill into that metric:

• Summary tiles — Pass rate, Scored, Fails for the window, led by Average for a numeric scorer or Most common (the dominant label and its share) for a boolean or categorical one.
• Donut — the average gauge, or the full label distribution: the same donut, with a legend listing every label’s share and its count.
• Chart — the average score over time for a numeric scorer, or the share of each label over time for a boolean or categorical one.
• Score rows table — every individual verdict, each score shown as a donut gauge (with its native label where the judge emits one), with a column picker, sortable headers, and filters for Result (pass/fail), Score range, and Source. Click a row to open the score modal: the score against its threshold, the judge’s Rationale, the Raw output, and a View trace button that jumps to the graded trace.

Reading the donut and scale

Every score is stored normalized to 0–1, but the donut center shows it on a friendlier display scale — 0–100 by default — so 0.75 reads as 75. The ring fills in proportion to the underlying 0–1 score and is colored by the metric’s threshold bands: green at or above the green cutoff, yellow between the two cutoffs, and red below the yellow cutoff. When a judge carries a native readout (e.g. 7.5 / 10 or high), the center shows that value verbatim while the ring still fills on the normalized score.

Boolean and categorical scorers

Not every scorer produces a number. A guard answers yes or no; a classifier picks one label from a set. For those, an average is not a weaker summary — it is a meaningless one, so the Scores page renders a label distribution instead: a donut cut into one arc per label, with a legend naming each one.

Percentages are always computed server-side over the whole selected window, not over the page of rows in the table below — so changing the time range changes the shares, and paging through the rows does not.

Reading the label donut

The label donut is the same object as the numeric gauge next to it in the grid, so a mixed catalogue reads as one board rather than two:

• The centre is the denominator. It carries the number of scores in the selected window — the count every percentage beside it divides by. 40% over 5 scores and 40% over 5,000 are very different evidence, and the legend alone can’t tell you which you’re looking at.
• The legend is the chart. Each row is a swatch, the label, and its share; the drill-down adds the raw count. Labels render in the judge’s declared order (worst → best), not by popularity, so a severity scale always reads the same way from window to window.
• Colour means nothing beyond “different slice”. A boolean’s two verdicts are deliberately not painted green and red: which verdict is the good one is only knowable from a declared output spec, and colouring a guess would be a quality claim the data doesn’t support. Pass/fail lives in the Pass rate tile, which comes from your own passing / pass_label rule. Every slice is named in text, so nothing is carried by colour alone.
• A rare label still shows. A slice worth a fraction of a percent keeps a hairline arc rather than vanishing, and the legend carries its real number.

Other and No label

Two slices are generated rather than emitted by the judge, and both exist so that nothing is silently dropped:

A card and everything under it always describe the same scores: whatever a card counts, its breakdown covers and its drill-down lists. No scores in this window yet. therefore means exactly that — the selected window holds no scores for that scorer — and never appears beside a card that is counting some.

Declare the output type on the judge

The page reads the type from the judge’s output spec — the same output_spec that lets a judge honour its own rubric. Set it when you create or edit the judge; you don’t configure anything on the Scores page.

A boolean judge. pass_label and fail_label are the literal words the judge returns, and they define which verdict counts as a pass:

{
  "criteria": "Does the answer expose personally identifiable information the user had not already shared?",
  "required_data": ["input", "output"],
  "output_spec": {
    "type": "binary",
    "pass_label": "false",
    "fail_label": "true"
  }
}

The Scores card then draws a two-arc donut whose legend reads False 88% and True 12%, with the window’s scored count in the centre. Neens localizes the generic verdict words (true/false, yes/no, pass/fail); a judge that names its own verdicts — safe/unsafe — has them rendered verbatim, which is usually the more readable choice for a safety guard.

A categorical judge. labels is an ordered list, worst to best, and that order is the order the legend and the breakdown render in — so a severity scale reads top to bottom in your own order rather than by popularity:

{
  "criteria": "Rate the business risk this answer creates for the company.",
  "required_data": ["input", "output", "context"],
  "output_spec": {
    "type": "categorical",
    "labels": ["none", "low", "medium", "high"],
    "higher_is_better": false,
    "passing": ["none", "low"]
  }
}

The card draws four arcs and a legend reading none 37%, low 30%, medium 18%, high 15% — in that order, because that is the order you declared, not the order the window happened to produce them in. Four labels is exactly what a card renders, so nothing folds into Other here. In a window where high never occurred, the card drops to three arcs and the drill-down still lists high at 0%.

Many labels: a worked example

A wide label set — a triage judge over your agent’s failure taxonomy — is folded on the card. Declare the vocabulary the same way:

{
  "criteria": "Which failure class best explains why this trace went wrong? Choose exactly one.",
  "required_data": ["input", "output", "messages"],
  "output_spec": {
    "type": "categorical",
    "labels": [
      "tool_selection_error", "tool_execution_error", "missing_context",
      "hallucinated_fact", "instruction_ignored", "unsafe_output",
      "format_violation", "retry_loop", "no_failure"
    ],
    "higher_is_better": false,
    "passing": ["no_failure"]
  }
}

Over a week of 1,240 classified traces you get:

1. The card draws five arcs — the four largest labels (tool_selection_error, missing_context, hallucinated_fact, no_failure) plus Other, whose tooltip reads Grouped remainder — 4 further labels. 1,240 sits in the centre.
2. Click the card. The drill-down redraws the same donut with every label listed individually, each with its share and its count, and unsafe_output still listed at 0% because you declared it and this week produced none. A remainder past 24 labels would again fold into Other.
3. The chart underneath switches from the numeric progression to share over time — one stacked 0–100% band per label, using the same colour each label has in the donut above it. It answers the categorical question (“is the mix shifting?”) rather than the numeric one, and a bucket where nothing was scored is drawn as a gap in the bands, not as a row of zeroes — “nothing happened this hour” and “everything dropped to zero” are different findings.
4. The rows table below lists the individual verdicts. Its paging never moves the percentages above it: those are a server-side aggregate over the whole window.

The classifier is categorical too

The built-in Issue Classification judge writes the detected failure-mode name as its label, so its card is a distribution over your taxonomy — which failure mode is actually most common, as a share of everything classified in the window.

You do not declare an output spec for it, and you do not need to. A classification judge picks a label from your taxonomy and has no number to average, so it is categorical by construction — its taxonomy is the vocabulary, and that taxonomy changes whenever you edit it, which is why the labels are not frozen into a spec. The number it does store is the classifier’s confidence in the label it chose, and a mean of that answers nothing about which failure modes you have. If you do give a classification judge an explicit output_spec — a fixed triage vocabulary, say — that declaration wins, exactly like any other judge’s.

When the type can’t be declared

Neens works out a score type in this order, and tells you which it used:

1. Declared — the producing judge’s current output_spec. This is the normal case, and it is what you should rely on. A judge with no spec is declaring the default: a plain 0–1 number.
2. Recorded — the output type stamped on each score row when it was written. This answers for rows whose judge was since deleted or renamed.
3. Classifier — source = classifier rows are categorical by construction.
4. Inferred — a last resort, read from the labels the rows actually carry.

A classification judge is the one exception to step 1: leaving its spec unset declares nothing, because it has no number to fall back to, so it is answered at step 2 or step 3 instead. Every other kind of judge — and any classification judge that does declare a spec — is answered at step 1.

Lifecycle filter (Finalized / Experimental / All)

Judges carry a lifecycle stage — experimental, finalized, or archived — and the Scores page uses it to keep the catalogue focused. A Finalized / Experimental / All control switches which scores the whole page (cards, drill-downs, and sparklines) shows:

This is the point of lifecycle staging: an ad-hoc experimental judge doesn’t flood the main view until you promote it to Finalized on the Judges page.

Scope to a single run

For an ad-hoc batch eval run, you can view exactly the scores that one run produced — its captured snapshot — instead of the metric’s current live scores. Open it from a judge’s run history on the Judges page, or via GET /scores?runId=<evalRunId>. This is meant for bounded ad-hoc runs; a continuous scorer overwrites each trace’s score row as it re-grades, so a single run isn’t a meaningful boundary for it. The run history is also where you compare two versions of a judge and export a run with each target’s input and output.

Favorites and hiding

Each catalogue card has a star (favorite) and an eye (hide) toggle. These are personal view preferences — stored per user, per agent — and are saved via PUT /scores/prefs keyed on the (metric_key, source, target_type) triple:

• Favorites float into a Favorites section at the top of the page.
• Hidden metrics collapse into a Hidden (n) reveal at the bottom. A hidden metric can’t be favorited until you unhide it.

Hiding vs. disabling the judge

Hiding is view-only: if an enabled judge is still producing new scores for the metric you just hid, Neens shows a banner naming that judge and offering Disable judge as an explicit second step. Disabling stops new scores from being produced (for everyone); existing scores are kept. You can always re-enable the judge from the Judges page.

Score display settings

The display scale and color bands are configured per agent under Settings → Score display. The card shows a live preview of three sample scores rendered with your saved settings, so you can see the effect before applying it. Editing is admin-only; everyone else sees it read-only. It is presentation only — it never changes how a score is stored or how pass/fail is graded.

Thresholds always stay on the underlying 0–1 scale; a score at or above the green line reads green, between the two lines yellow, and below the yellow line red. Neens keeps the bands non-inverted (the yellow cutoff can’t sit above the green one).

Primary Score

Primary Score (primary_score) is the headline quality number every new agent gets out of the box — a composite that blends Faithfulness, Answer Relevancy, and Coherence into one session-level score. It plays two special roles:

• Continuous evaluation. It ships as the agent’s one enabled continuous scorer, sampling newly ingested traces automatically. Its component metrics appear in the catalogue as a byproduct of each composite run — they don’t need their own enabled scorers. See Continuous evaluation.
• Failure clustering. The default selection criterion for failure clustering is a low Primary Score — which is how silent failures (bad answers with no hard error) end up in the failure set.

Where scores surface

• Trace detail — from any score’s modal, View trace opens the graded trace; classifier scores also surface there as the trace’s issue chip.
• Traces page — the Score filter selects traces carrying a score for a metric, optionally Pass/Fail against its threshold (URL params score_metric, score_status, score_label). See Traces & sessions.
• Dashboards & metrics — the score-grain catalogue measures eval_pass_rate, avg_score, and score_count, sliceable by judge, score_source, score_label, target_type, score_bucket, and — because every score records the model and agent behind the answer it graded — by model, agent, and model_source. See the Metrics catalogue and Model comparison.
• Judge run drill-downs — each eval run lists its per-target scores, reasons, and errors on the Judges page.
• Pre-prod comparisons — candidate-vs-baseline verdicts in pre-prod evaluations are built from preprod-sourced scores.

Troubleshooting