Failure clustering

Failure clustering is how Neens turns a pile of failing traces into a short, named list of failure modes. It embeds each failing session’s behavior — what the agent did, where it errored, which quality checks it failed — and groups similar sessions together, so a thousand bad traces become a dozen patterns you can actually triage. The results power the Failure Modes page (see Issues & failure modes for the taxonomy built on top).

At a glance

Reading the Failure Modes tab

The Failure Modes tab groups clusters into buckets (Timeouts, Tool failures, and so on — see Issues & failure modes) and, within each bucket, sorts cards by classified trace count descending — the highest-impact failure mode in the bucket is always first. Use Expand all / Collapse all at the top of the tab to open or close every bucket group at once, or a group’s own header to toggle just that one.

A summary strip above the groups gives you the shape of the page at a glance: total modes, affected traces (the sum of each mode’s trace count in the window), how many buckets are represented, how many modes are eroding a business KPI (see Business KPIs), and a breakdown of fix status across every mode — how many already have an applied fix, how many have one proposed and in flight, and how many have no fix yet.

The time window

The same time-range picker used on the Issues & Taxonomy tab (Today, 24h, 7d, 30d, All time, or custom) also scopes the Failure Modes tab: it re-derives each mode’s displayed trace count, trend sparkline, and the summary’s affected-traces total to the window you pick. It does not re-cluster — the clusters themselves (their membership, labels, and root causes) come from the last completed analysis run regardless of the window; the picker only changes which of that run’s members are counted as “in window” for display. Leave it at the default to see the same window the last analysis used, or narrow it to see what’s happening right now within an existing clustering.

What counts as a failure

Clustering runs over the agent’s failure set, and you control what’s in it. The Analyzing: chip at the top of the Failure Modes page shows the current criteria in plain language; click Configure clustering on the same page to change them and every other knob (see Clustering configurations below).

In Score mode you also pick which score is consulted:

The Score threshold (0–1) defaults to 0.5: a session scoring below it counts as a failure even if it finished without an error.

The failure set is also windowed and capped so runs stay fast at volume: by default the last 30 days, at most 5,000 sessions per run (sampled with day-and-agent stratification when the set is larger — the page then labels counts as a sample-based estimate).

When clustering runs

• Nightly — a full clustering pass runs at 2:00 AM (server timezone) for every agent.
• First-run readiness sweep — every 30 minutes, Neens checks agents that have never been analyzed and triggers their first run as soon as they cross the readiness threshold (30 failure traces by default, shown as a progress meter on the page). A low-traffic agent older than a week gets its first run at 15 failures so the feature is never invisible.
• On demand — click Run analysis now (or Re-analyze once results exist), or call POST /clusters/run. A manual run uses exactly the same per-agent settings as the nightly job, so the button and the scheduler always produce the same clustering of the same data.

Only one run per agent executes at a time; a crashed run stops blocking after an hour. The run returns an honest outcome — completed, skipped_insufficient_data (with the count it found vs. the minimum it needs), queued (the analysis was handed to a background worker), or failed — and the page shows Last analyzed freshness from the most recent run.

How it works

Labeling and root cause — the LLM connection

Cluster labeling, cluster-level root-cause hypotheses, and draft remediations all use the agent’s default LLM connection, configured in Settings → LLM connections (Anthropic, Bedrock, or any OpenAI-compatible endpoint). There is no platform-wide key.

Labels are pushed to be specific: if the model returns something generic (“Agent Error”), Neens retries with an explicit correction, then falls back to a deterministic label derived from the dominant failing tool or failed metric (for example get_order_status tool failure or Low answer relevancy score).

Online assignment, novelty, and “more like this”

Between full runs, Neens classifies new failing sessions against the fitted clusters without re-clustering:

• Intraday assignment — every 30 minutes, recently arrived failure sessions (the last hour, up to 500) are matched to the nearest cluster, so fresh failures land in existing modes and novel ones raise insights the same day, not the next morning.
• On demand — POST /clusters/assign with a list of session ids returns a status per session: assigned (joins a known cluster), novel (far from everything — a candidate new failure mode, which also fires a novelty insight), unassigned (in between), or no_model (no clustering run has completed yet).
• Similar sessions — GET /sessions/{id}/similar returns the sessions semantically closest to a given one (nearest-neighbor over the same embeddings), useful for “show me more traces like this failure”.

Exemplars, status, and the taxonomy

• Exemplars — GET /clusters/{id}/exemplars returns a cluster’s most representative members (closest to its center; 5 by default, up to 50). The UI uses these when you promote a cluster: Confirm candidate requires reviewing at least 3 exemplar traces first, so the taxonomy stays grounded in real evidence.
• Status — clusters are active (current) or resolved (superseded by a later run or no longer recurring). Pickers and the Failure Modes page show active clusters only.
• Failure modes — clusters are the machine-discovered evidence behind the human-curated taxonomy. Confirming a cluster creates a failure mode linked to it, and after each run Neens auto-proposes cluster-to-mode links wherever a clear majority of a cluster’s members were classified into one mode. See Issues & failure modes, and Remediations for drafting a fix from a cluster.

From a cluster to action

Every cluster card carries the same fix path as an Issue, plus one action of its own:

• View traces — opens the sessions behind the cluster in the selected window.
• Generate fix → View fix — the same stateful action as on an Issue card: Generate fix drafts a typed remediation grounded in the cluster’s evidence when none exists yet; once one does, the same spot becomes View fix. See Remediations.
• Promote to issue — link the cluster into your curated taxonomy as evidence for a named issue, instead of Create eval (that action lives on Issue cards, once a mode already has evidence behind it).

Promote a cluster to an issue

Keeping failure modes clean

Over time an agent can end up with near-duplicate failure modes — two cards with the same name, or one broad mode that should really be several. Neens keeps the list tidy in two ways: it dedupes duplicate names automatically on every run, and it lets you merge and rename modes by hand from a card.

Automatic dedup of duplicate names

Cluster names are auto-generated (from the dominant failing tool or metric), so two unrelated modes can land on the same label — e.g. two different clusters both named Low faithfulness score. When that happens, the dedup pass resolves the collision one of two ways:

• Disambiguate — if the two modes are genuinely distinct, Neens keeps them separate and appends a distinguishing suffix so each name is unique, e.g. Low faithfulness score (checkout) and Low faithfulness score (search).
• Coalesce — if the two modes are effectively the same failure, Neens folds them into one so you don’t triage the same pattern twice.

This runs on every clustering pass and is controlled by the Dedup duplicate labels toggle in the clustering config (Advanced section), on by default. Turn it off if you’d rather see the raw, un-disambiguated names. A name you set yourself (see Rename below) is never touched by this pass.

Merge a failure mode

When two cards describe the same problem, merge one into the other from its card.

The traces from the mode you started on move into the target, and the now-empty source mode is marked resolved (kept as history, not deleted). Behind the card this calls POST /clusters/{id}/merge against the target, with the source mode’s id. If a bucket has no other modes, the dialog says so and there’s nothing to merge into.

Rename a failure mode (sticky names)

Auto-generated names are a starting point — rename any mode to something your team recognizes.

A renamed mode is pinned: the name you set sticks across every future analysis run, so the nightly pass and every Re-analyze keep your wording instead of reverting to the auto-generated label. Pinned modes show a small pin marker next to their name (hover it for “This name was set manually — future analysis runs won’t rename it.”). Renaming a mode also exempts it from automatic dedup. Behind the card this is PATCH /clusters/{id} with the new label.

Clustering configurations

Everything above — what counts as a failure, which score is consulted, how fine the clusters are, how much data a run reads — is captured in a clustering configuration. Configs work like judges: an agent keeps a collection of named configs, but exactly one is active, and only the active config drives analysis. Open the collection from Configure clustering on the Failure Modes page.

Out of the box, there’s nothing to set up. Every agent starts with one active config named Default that mirrors the sane defaults in the reference table below. You only create more configs when you want to try a different way of grouping failures.

The knobs

Every config carries the same set of tunables, grouped in the editor:

Selection criteria — what goes into the failure set:

• What gets clustered (selectionMode) — Score (default), All traces, or Errors only. See What counts as a failure for the full behavior of each.
• Score metric and Score threshold (0–1) — in Score mode, which score is consulted (Primary Score (recommended), Any score, or a specific metric) and the cutoff below which a session counts as a failure (default 0.5).

Advanced — how the run behaves:

• Min cluster size — the granularity lever, and the one worth tuning first. It’s the fewest sessions that can form a cluster; smaller = more, finer failure modes, larger = fewer, broader ones. Default 3, floor 2. (Clustering is density-based, so there’s no “number of clusters” to set — you steer granularity with this instead.)
• Min traces to run (minFailureSessions) — how many failing traces must exist before the first run starts (default 30). Lower it for a low-volume agent so analysis begins sooner.
• Sampling %, Window (days), and Max traces per run — down-sample the eligible set, set the rolling look-back window, and cap how many sessions a single run reads (defaults 100, 30, 5000). Together they keep runs fast at volume.
• Dedup duplicate labels — automatically disambiguate or merge failure modes that get the same auto-generated name (default on). See Automatic dedup of duplicate names.
• Enable this config for scheduled clustering — turn scheduled and manual runs off without deleting the config.

As you edit, a live estimate shows how many traces match the current criteria in the window — e.g. “142 eligible traces · meets the 30-trace minimum” — so you can see the effect before you save. It’s a quick count, not a trial run, so it returns instantly.

Lifecycle: Experimental, Finalized, Archived

Each config carries a lifecycle stage — a colored badge of Experimental (amber), Finalized (green), or Archived (gray). It’s a curation label only: it tells your team how settled a config is and controls which configs show by default. It never changes what runs — only activating a config does that.

• A config you create starts Experimental. The seeded Default config is Finalized.
• The config list defaults to the Finalized view (plus the active config, which is always shown); switch the filter to All to see experimental and archived configs too.
• Move a config between stages any time with Change lifecycle stage — there’s no fixed order and no config is ever frozen. Archive a config you’ve stopped using to tuck it out of the default view without losing it.

Activate a config

Activate is the switch that matters: the active config — and only the active config — drives the nightly run, the readiness sweep, and every Run analysis now. Activation is independent of lifecycle stage; you can run an Experimental config while you’re still tuning it.

Reference: clustering configs

Manage the collection per agent via the config routes; the active config’s knobs are also readable through the compatibility route GET /clusters/settings.

Each config’s tunable knobs and their defaults:

Troubleshooting