Activity

Activity is the live feed of background work Neens runs over your data: scoring runs (judges evaluating traces), enrichment runs, and clustering runs. Use it to confirm an evaluation finished, watch a live run’s progress, or dig into why a run failed.

At a glance

What appears in the feed

Each row shows the run’s Name, Type, Status, Target (what kind of item it processes), a Progress bar showing how the run’s targets split across succeeded / failed / running (see Reading the Progress bar), Triggered by, Started, and Duration. Columns are configurable with the Columns button and sortable by clicking headers; the feed lists the most recent runs (up to 100).

Focusing the feed

• A summary row counts Total, Active, Running, Queued, Historic, and Failed runs across the agent.
• Toggle All / Live / History to show everything, only in-flight work (queued + running), or only finished runs.
• The Filters button narrows by Type (Clustering, Scores, Enrichments), Status (Queued, Running, Completed, Degraded, Failed, Skipped), and Started (a date range on when the run began).

Run statuses

Runs move through queued → running to a terminal state:

A scoring run is graded by its success rate — completed targets over targets that actually ran. It’s not realistic to expect 100% against external LLM providers, so by default a run is completed at ≥ 90% success, failed below 10%, and completed_with_failures in between (a judge can override its own green cutoff). Individual target failures retry automatically on transient errors — up to 3 attempts by default — before counting against the run.

Why a run was skipped

A skipped run scored nothing, and Neens records why — per target and rolled up on the run. The Activity row’s error summary and the Details drawer both show a machine reason code (no_llm_connection, insufficient_components, target_not_found, no_taxonomy, …), the sentence it means, whether it will be retried, and what to do about it. If Neens cannot classify the cause it says unknown out loud rather than showing a blank.

Skips caused by a transient condition — the LLM connection could not be resolved on that pass, a component judge’s call kept erroring, the trace hadn’t finished landing — are re-dispatched automatically as a new run over exactly those targets, bounded to a couple of attempts (2 by default) with a doubling backoff. Permanent causes are never retried; they need a change from you. The drawer links the two runs in both directions, so “was this retried?” is answerable without reading a log.

The full code table, and a step-by-step for debugging one, live in Continuous evaluation → When a run is Skipped.

Reading the Progress bar

The Progress cell is a distribution, not a completion meter. It shows what happened to every target the run was given, colour-coded, on a single bar:

Segments are drawn left to right in that order, so the green tip is always in the same place and two rows are comparable at a glance. Beside the bar is a text readout — 120/165 · 45 failed. The failure clause appears only when something actually failed; a clean run reads 165/165 with nothing after it. Colour is never the only signal: the readout and the status pill both state the outcome in words.

Worked example — a finished scoring run

A judge ran against 165 sessions. 120 were scored; 45 errored against the provider.

• The bar is about three-quarters green, then a red block, and no grey — 120 + 45 = 165, so every target was attempted.
• The readout reads 120/165 · 45 failed. The leading number is the succeeded count, not “how far along” the run is. That distinction is the point: a finished run at 120/165 used to look like a bar frozen short of the end, with nothing on screen saying that the missing 45 had failed.
• Click Details and the drawer leads with 73% success rate — 120 of 165 tasks succeeded, 45 failed — over a thicker copy of the same bar, with a legend giving each category’s count and share: Succeeded 120 · 73%, Failed 45 · 27% (and a Not run row whenever targets were never attempted).
• 73% is 120 ÷ (120 + 45). With the default cutoffs that sits between 10% and 90%, so the run is graded completed_with_failures.

Worked example — a run still in flight

The same judge over 900 targets, checked mid-run: 600 succeeded, 30 failed, 24 currently executing. All four categories are visible at once — green 600, red 30, aqua 24, and 246 targets of exposed grey track that the run has not reached yet. The readout reads 600/900 · 30 failed.

While the run is live the grey shrinks as work is dispatched, and the aqua segment moves along in front of the green. The success rate so far is 600 ÷ 630 = 95%, but no verdict exists yet — the pill stays running until the run reaches a terminal state.

Skipped targets don’t drag the rate down

A run’s success rate is measured over attempted targets only:

success rate = succeeded ÷ (succeeded + failed)

Targets that were skipped never ran, so they are not in the denominator — they appear as Not run grey. If the run above had 185 targets with 120 succeeded, 45 failed and 20 skipped, the success rate would still be 73%, and the extra 20 would show as grey. This is exactly the rule the status pill is graded on, so the bar and the pill can never disagree about what counted.

An unknown is not a zero

Some runs report no failure count at all:

• a run recorded before the distribution bar existed — that history only ever kept a completed/total pair;
• a run kind with no per-target failure notion — a clustering run is a single refit over the failure set, not a fan-out of independently failing tasks.

Those rows render the plain single-fill bar, with no colour split and no failure clause.

The pill is the verdict; the bar is the breakdown

The Status pill is not derived from these counts. It is the server’s graded verdict, using the success-rate bands in Run statuses; the bar is the evidence behind that verdict. So the two can look like they disagree, and shouldn’t surprise you:

• A run graded completed can still show a thin red segment — the green cutoff is 90% by default, not 100%.
• A skipped run shows an all-grey bar: nothing was attempted, so there is nothing to colour. The skip reason says whether that was legitimate.
• A cancelled run’s bar shows how far it got before it was stopped; those partial counts are never graded into a verdict. Cancelling takes effect promptly — the run stops scoring within seconds rather than draining its queued work — and it reads Cancelled here and on the scorer’s run history on the Judges page, so the two pages never disagree.

What to do when you see red

1. Expand the row. A scoring run’s inline metrics include Failure rate and Top errors — the real error messages, grouped and counted. That usually names the cause outright (provider rate limit, timeout, revoked credential).
2. Open Details. The per-target breakdown lists every failing target with its error, duration and a retried N× marker where a target needed several attempts. Transient errors were already retried automatically — up to 3 attempts by default — before counting as failed, so a red segment is a persistent failure, not a blip.
3. For grey, look for a skip reason. A skipped target carries a typed reason code, the sentence it means, whether it will be retried, and the remedy — see Why a run was skipped. Grey that is simply “not reached yet” needs nothing from you.
4. Then act on the cause, not the run. Re-running a judge against a dead LLM connection reproduces the same red bar; fix the connection, quota or judge configuration first.

Who triggered a run

The Triggered by column separates human-initiated from automated work:

• A person’s name with avatar initials — someone kicked the run off manually (e.g. Run now on a judge or enrichment). Attribution is stamped when the run is created and survives later display-name changes.
• A Scheduled chip — the Neens scheduler started it (e.g. nightly clustering).
• An Automated chip — the platform triggered it in response to data, such as continuous on-ingest scoring or on-ingest enrichment.
• An API key chip — a machine credential (a data-plane API key) triggered it.
• A System chip or — — attribution is unknown (including runs recorded before attribution existed).

Run details

Scoring runs

Click a Scores row to expand its pipeline metrics inline:

• Throughput (targets/second) and ETA while running.
• Failure rate and Top errors — the actual error messages, grouped and counted.
• Workers / Concurrency — how many judge calls run in parallel.
• Per-connection (req/min) — the request rate against the LLM connection backing the run.

The Details button opens the drawer. It leads with the run’s success rate as a large percentage, a full-width copy of the distribution bar, and a legend giving each category’s count and share, then lists the per-target breakdown: each target’s status, score, pass/fail label, duration, any error, and a retried N× marker when a target needed multiple attempts. A chip row totals the targets by status (completed / failed / skipped / running / queued / cancelled). A skipped target additionally shows its reason code, the sentence it means, whether it will be retried, and the remedy — see Why a run was skipped.

Clustering runs

Clustering rows have a Details action summarizing the run — Failure traces analyzed, Duration, Started, and a note explaining the outcome — with a link into Failure clustering to explore the resulting clusters.

A clustering run’s Progress bar has no colour split: the run is one refit over the whole failure set, so there is no per-target success/failure to break down. Its verdict lives in the status (completed, degraded, failed) — see An unknown is not a zero.

Reference

Troubleshooting

• A run sits in queued — no worker is consuming its queue yet (on a distributed deployment) or earlier work is still draining. It starts as soon as capacity frees up.
• A scoring run is completed_with_failures — its Progress bar carries a red segment sized to the failures. Expand the row and check Top errors, then open Details for the failing targets. Provider rate limits and timeouts are the usual culprits; failed targets were already retried automatically. Full walkthrough: What to do when you see red.
• A finished run’s bar isn’t full, and nothing is red — the grey remainder is Not run: targets that were skipped or never reached. Open Details and read the skip reason.
• A scoring run is skipped — it scored nothing. Open Details: the header names the reason(s). no_llm_connection means add a connection; insufficient_components means the composite couldn’t gather enough component scores; target_not_found and llm_connection_error are transient and already scheduled for retry.
• An enrichment or judge you expected isn’t producing runs — continuous work only triggers on new traces, and the judge/enrichment must be enabled. See Scores and Enrichments.