Run Targets

A Run Target is a named, reusable population for your scorers to run on — a golden dataset, a live sample of production traffic, or a filtered sweep of history. It separates what a judge measures (the rubric) from what it measures it on (the traffic), so one scorer definition can grade several populations at once and every scorer can share the same populations.

At a glance

Why Run Targets exist

A scorer answers one question — is this answer faithful? is this reply coherent? — but you rarely want that question answered on just one slice of traffic. Take Primary Score, the built-in composite of Faithfulness, Answer Relevancy, and Coherence. You want it on:

• your Golden Q&A dataset, so a regression can’t sneak past the benchmark you trust;
• a live sample of what support-bot is actually saying customers today;
• last night’s failures, swept nightly, to watch a known problem area.

That’s one rubric over three populations. Before Run Targets, “what to score” was baked into each scorer’s deployment, so answering that meant three near-duplicate scorers to keep in sync. A Run Target lifts the population out into its own object: define Primary Score once, then attach it to all three targets. Results come back split per target, so you can see at a glance that the same scorer reads 0.95 on the golden set, 0.82 on live traffic, and 0.61 on the error sweep — three honest numbers instead of one blurred average.

The three kinds

Every Run Target is one of three kinds. The kind decides how its population is resolved and which trigger fits it.

Create a Run Target

Attach a scorer

You attach a Run Target from a scorer, not the other way round: open a judge, go to its Run Targets tab, and click Attach Run Target. Each attachment is one binding of this scorer to one Run Target with its own run-time settings.

Attach one scorer to several targets

Repeat Attach Run Target for each population. Attaching Primary Score to Golden Q&A, Live production, and Nightly support-bot errors gives it three bindings — one rubric, three populations — each with its own trigger, cutoff, and connection. The scorer’s Run Targets tab lists all three, each with its latest metric and a sparkline.

Grain must match

Read results split per target

Because a scorer can run on several populations, its results are reported per Run Target rather than pooled. On the scorer’s Run Targets tab, each attached target carries its own scored count, average, pass rate, and trend — so the same rubric reads independently on each:

That spread is the point: a benchmark near the top, live traffic a step below, and the error sweep lowest of all. A single blended number would hide exactly the signal you’re watching for — a golden set holding steady while live quality slips, or an error sweep that stops improving after a fix.

A target with no runs yet reads as blank (an em dash), never a misleading 0 — an unscored population and a zero-scoring one are different things.

Which target a run evaluated

Because one scorer can be attached to several Run Targets, its run history and the Judges list both label each row with the Run Target it evaluated, so two runs of the same scorer against different populations never read as duplicates:

• On the scorer’s run history (click a scorer to open its runs), the Target column names the Run Target each eval run scored — e.g. a Live production run and a Nightly support-bot errors run sit side by side, each tagged with its own target.
• On the Judges list, each enabled scorer row shows the Run Target its deployment is bound to, so the same scorer attached to two targets appears as two clearly labelled rows.

A run that wasn’t tied to a named Run Target — an ad-hoc Run now over an inline filter — shows a neutral em dash in the Target column rather than an invented label.

Reuse: editing a target updates every bound scorer

A Run Target is shared state. Editing it changes the population for every scorer bound to it — that reuse is the feature. Widen the Nightly support-bot errors filter to cover a second agent, or bump Live production from 10% to 25%, and every attached scorer picks up the change on its next run. There’s no per-scorer copy to update, and no drift between scorers that are meant to watch the same slice.

How it works

• Resolution at run time. When a scorer runs a binding, Neens resolves the Run Target’s population then — a Dataset target reads its pinned version (or the live members, if Follow latest), a Live sample samples the just-arrived traces, and a Filtered batch runs the filter over history — applies the target’s sampling, and scores exactly that set. Editing the target changes what the next run resolves.
• Settings split by owner. The population (filter, sampling, dataset, version) lives on the Run Target and is shared; the run-time settings (trigger, pass cutoff, model connection) live on each attachment and are private to that scorer. The two never fight — a run reads the population from the target and the run-time settings from its own binding.
• Nothing is retroactive. A Live sample only scores traffic arriving after it’s attached and running, exactly like continuous evaluation; use a Filtered batch (or Run now) to score history.
• Every LLM-scored population needs a connection. Like every judge, a Run Target run uses the agent’s configured LLM connection (Settings → Connections). With none configured the run finishes Skipped rather than borrowing another agent’s credentials — see continuous evaluation.

Troubleshooting

Related

• Judges — define, version, and enable the scorers you attach to Run Targets.
• Datasets — the versioned collections a Dataset target points at.
• Continuous evaluation — how a Live sample scores new traffic.
• Scores — the catalogue every Run Target’s verdicts land in.
• Pre-prod evaluations — replay a golden dataset version against a candidate release before you ship.