Continuous evaluation

Continuous evaluation scores your traffic as it arrives: every time new traces are ingested, Neens samples them and runs your continuous judges against the sample — no manual runs, no schedules to manage. It’s how an agent accumulates a live quality signal (and a failure set for clustering) without anyone pressing a button.

At a glance

How it works

After a batch of traces is ingested and persisted, Neens checks the agent’s enabled scorers whose trigger is Continuous (score new traces) and, for each one:

1. Samples the just-arrived traces at the scorer’s effective sampling rate (a coin flip per trace at the configured percentage).
2. Applies the daily budget. Each continuous scorer has its own daily cap (max_scored_per_day); once a scorer has dispatched its cap for the day, further traces are skipped until the next UTC day. Budgets are per scorer, so one continuous judge can never starve another’s.
3. Creates a small eval run scoped to exactly the sampled traces and queues it for the eval worker — the same scoring path, retries, and run outcomes as a manual run. These runs are attributed as Automated in run history and the Activity feed.
4. Writes scores. Verdicts land in the score catalogue with source llm_judge (or composite for composites). A composite like Primary Score inline-scores any component that has no score yet, so its component metrics fill in as byproducts.

Continuous evaluation is best-effort by design: it runs after ingestion completes and can never block, slow down, or fail a trace write. If scoring hiccups, the traces are still safely stored.

Primary Score, cold start, and steady state

Every new agent ships with one continuous scorer enabled: the Primary Score composite (Faithfulness + Answer Relevancy + Coherence). Its defaults balance signal against cost:

The cold-start boost graduates automatically once the threshold is crossed — if your scoring rate drops from “everything” to “a sample” after the first couple hundred traces, that’s expected, not a failure.

What continuous evaluation is not

• It is not retroactive. A continuous scorer only scores traces that arrive after it’s enabled (or resumed). To score existing traces, use Run now on the scorer — a one-off run over eligible historical traffic.
• It is not the Issue Classification path. The built-in Issue Classification judge also shows up as a continuous scorer, but it’s dispatched by a background sweep (every 30 minutes) over the agent’s failure set only — not on ingest, and never over healthy traffic. See Issues & failure modes.

Enable, pause, and resume

Sampling and cost

Every trace a continuous scorer grades costs at least one LLM call on the agent’s configured LLM connection — and a composite costs one call per component that doesn’t already have a score (up to three for Primary Score). Your levers:

• Sampling percentage (per scorer, in Configure → What traces to score) — the fraction of arriving traces that get scored.
• Daily cap (per scorer) — the hard ceiling regardless of traffic volume. Resets at midnight UTC.
• Scoring connections (pool) — spread continuous load across several connections to dodge provider rate limits.

If the agent has no LLM connection, continuous runs still fire, but they score nothing: the run finishes Skipped and every target records the reason no_llm_connection. Neens never borrows another agent’s credentials, and ingestion is unaffected. Configure one under Settings → Connections (LLM connections).

When a run is Skipped

A run finishes Skipped when it reached the end having scored nothing — no target succeeded and none failed either. It is deliberately neither green nor red: skipping can be perfectly legitimate (there is genuinely nothing to score yet) or it can mean something is misconfigured. The reason tells you which.

Every skipped target carries a reason code plus a plain sentence, visible on the Activity feed row and in its Details drawer. A run can carry more than one code; the drawer lists each with how many targets it accounts for.

Skip reason codes

Are skipped targets retried?

Transient causes are; permanent ones are not. A background sweep (every 30 minutes) looks for terminal runs holding targets skipped for a retryable reason and opens a new run scoped to exactly those targets — never a re-drive of the finished run, so the original’s history and counts stay intact. Both runs appear in the Activity feed, and the drawer links them (“Retried as …” on the original, “Retry 1 of 2 — re-attempt of …” on the new one).

Every retry is bounded twice:

• Attempts. The retry chain is capped, so a target is attempted at most three times, ever.
• Rate. The wait before the first retry is 10 minutes and doubles each attempt (10 → 20 → 40 minutes), so a provider that is already struggling isn’t hammered. At most 20 runs per agent are re-dispatched per sweep.

A permanent reason is never retried — it would produce the identical outcome every time and cost real money doing it. Neither is an unknown skip: Neens cannot argue the condition will clear, so it surfaces it loudly instead of looping on it.

Debugging a skipped run, concretely

3× no_llm_connection   not retryable
No LLM connection is configured for this agent, so the judge could not run.
Add an LLM connection in Settings → Connections, then re-run the scorer.

No retry: none of these reasons clear on their own.

Troubleshooting

Related: Judges · Scores · Pre-prod evaluations · Metrics catalogue