Remediations

A remediation is a typed, tracked fix proposal for one recurring failure — drafted from the real evidence in your failing traces (system prompts, tool schemas, tool-call errors), not from generic advice. The detail reads as the fix loop it actually is: Root cause → Fix → Proof → Ship. You simulate a remediation against real failing sessions before you ship it, apply it, and then measure whether the failure actually went away.

Remediations are the action half of the Neens Fix pillar: Issues and failure modes tell you what keeps going wrong; a remediation is the concrete change that stops it.

At a glance

Generate a fix

curl -X POST "https://<your-neens-host>/remediations/generate" \
  -H "Authorization: Bearer $NEENS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"failureModeId": "fm-abc123"}'

How generation works

Neens reads the failure’s real trace evidence — captured system prompts, tool schemas, tool-call arguments and errors, HTTP status codes, span kinds and statuses, model parameters, and representative failing input/output examples — then:

1. Derives the root cause from the evidence, not the label. Neens reads the dominant error signature across the failing sessions — the HTTP status codes in the errors, the error class names, the span kinds and their pass/fail status — and writes a likely cause grounded in what it found, e.g. “≈94% of sampled sessions fail with HTTP 503 ‘connection pool exhausted’ from the ticket service.” The likely cause is never a restatement of the cluster label.
2. Classifies where the failure lives — its failure locus: is it your agent’s reasoning, a tool contract, a downstream or upstream service, a genuine quality problem, or a control working exactly as intended? This is decided from the evidence (status codes, error classes, guardrail messages), so a real 503 becomes an infra advisory and a correctly-firing refund limit becomes working-as-intended — organically, not from a keyword in the label.
3. Routes to a locus-appropriate fix type (see Remediation types) and drafts a grounded artifact with the agent’s LLM connection: for an agent-fixable failure the artifact’s before quotes the real captured text (the actual system prompt, tool schema, or error) and after is the proposed replacement, with a unified line diff. For a service failure the artifact is an advisory that names the owning component and the recommended resilience fix instead of a prompt edit.
4. Self-critiques the draft in a second LLM pass — checking that before is a genuine quote, that the proof eval is specific enough to catch this exact failure, and that the fix type and target are consistent — and keeps the improved version.
5. Attaches a proof eval — an LLM rubric defining what “fixed” means for this failure. It is reused as the default judge when you simulate.
6. Applies the grounding gate — a proposal with no concrete change (or too little evidence to classify) is held as needs grounding and kept out of the actionable backlog.
7. Computes a priority (see below), ranked by fixability — an advisory or a working-as-intended item ranks below a genuinely agent-fixable change.

Where the failure lives (the failure locus)

Before it proposes anything, Neens decides where the failure actually lives — because the honest fix for “your agent chose the wrong tool” is nothing like the honest fix for “the ticket service was down.” That classification is the failure locus, and it’s read from the evidence, not the cluster label:

From the locus, Neens sets an actionability — how you’re meant to act on the fix — and it drives both the backlog and the sort order:

Remediation types

Every remediation carries a type — what kind of change the fix is — so it can carry a concrete artifact (or a concrete recommendation) rather than free-text advice. The agent-fixable types stay as before; the grounded flow adds four types for the service and control-plane loci:

Worked examples

Priority

Every remediation carries a RICE-style priority score (0–100) so you can work the highest-leverage fixes first. In the detail it folds into a single priority chip (its band); the full breakdown lives in the chip’s tooltip. It combines:

The score is bucketed into a band: critical (≥ 60), high (≥ 35), medium (≥ 15), else low. Sort the table by Priority (or newest-first), and filter by work state, status, type, or label. Because fixability is a factor, a genuinely fixable prompt change out-ranks a higher-volume infra advisory — the backlog is sorted by what you can fix, not just by how loud a failure is.

Needs grounding: held out of the backlog

A remediation is only useful if it carries a concrete change. Neens will not put a fix in your actionable backlog just because it exists — an empty or templated proposal is worse than none, because it looks like work that isn’t there. A proposal is held as needs grounding (and kept out of the actionable/open backlog) when any of the following is true:

• it’s an agent-fixable type but the artifact has no concrete change — before equals after, or the change is the generic filler “clarify the agent’s instructions…” that the deterministic fallback emits when it has nothing specific to say;
• there’s no usable error signature and no confidence to stand on;
• the locus is incoherent (the grouped sessions aren’t one failure) or unknown (not enough evidence to classify) — here the honest next step is to re-cluster or gather more evidence, not to ship a guess.

A needs-grounding item isn’t deleted — it’s parked. In the UI it lives in a distinct Needs grounding section, separate from the actionable backlog, so a real fix is never buried under placeholders.

One root cause, one remediation (deduplication)

The same underlying failure often fragments across several agents or clusters — three different agents all timing out against the same ticket service look like three problems but are one. Neens collapses them: each remediation carries a root-cause signature (a normalized locus:component:error key, e.g. downstream_unavailable:ticket_service:503_pool_exhausted), and when a new proposal matches the signature of an open remediation already in the agent, Neens merges it instead of inserting a duplicate — it adds the new cluster’s reach to the existing row, records the additional affected agent/cluster in the evidence, and recomputes priority against the combined impact.

The result is one remediation that says “this affects N agents” with the total reach behind it — so a widespread root cause rises in the backlog on its true blast radius, and you fix it once rather than triaging the same thing three times.

Applying a fix requires a bound proof

You can’t move an actionable remediation to applied or verified on a hunch. Neens requires a bound proof first — evidence that the fix actually works:

• a completed verification run (the pass^k result from an eval-verified PR) bound to the remediation, or
• a proof-eval gate — the failure mode’s dataset + judge (an eval gate) — that a fix can be run against.

Without one, the transition is refused with a clear message telling you what’s missing (an HTTP 409, mirrored in the UI as a disabled Apply / Verify control with an explanatory tooltip) — never a silent success and never a crash. The fastest way to bind a proof is to simulate the fix and then let the eval-verified PR flow open a verified pull request; both leave a bound proof on the remediation.

The remediation detail: the fix loop

Click any row in the list to open that remediation’s own page, /remediations/{id} — and the detail is laid out as the four-stage loop the fix moves through: Root cause → Fix → Proof → Ship. A persistent header sits on top; the four stages sit below it.

Each stage toggles independently (it’s not an accordion), so you can have several open at once. The live stage opens by default and a completed stage collapses to a one-line receipt. Use Expand all / Collapse all (just above the stages) to open or fold every section — including the trailing Affected sessions disclosure — in one click.

The header (always visible)

The header never collapses, so the three things you most need are on screen no matter which stages you have open:

• Title and a one-line root cause — what’s being fixed, in a sentence.
• A proof-state pill (the proof status) and a work-state chip (your team’s triage lane) — the two independent tracks, side by side.
• Four stat tiles — Sessions (how many failing sessions ground the fix), Confidence (earned by simulation), Sim Δ (the before→after pass-rate lift from the latest conclusive simulation), and Efficacy (the post-deploy verdict). A tile with nothing to report yet reads as a muted ”—” — never 0, never a green number (see Reading “unknown” honestly).
• One primary action — the single most useful next step (below), a View diff button (shown unless View diff is already the primary action, so it never appears twice), and an overflow ⋯ menu (move to another work state, archive/unarchive, delete).

The primary action is whatever the loop is waiting on:

While a simulation or a fix run is actually in progress, the header shows that it’s running rather than offering an action to start another.

Root cause

The first stage is always complete — it’s why the remediation exists. It shows the failure summary, the evidence-derived likely cause (the dominant error signature across the failing sessions, e.g. “≈94% fail with HTTP 503 ‘connection pool exhausted’ from the ticket service” — not a restatement of the cluster label), a failure-locus badge and an actionability badge (see Where the failure lives), a failure-category chip, and the evidence it’s grounded in (the failing sessions and cluster it was drafted from). For an advisory, this stage also names the owning component — the service responsible — so you can route it. Once you’ve read it, collapse it to its one-line receipt: e.g. “Grounded in 14 hallucination sessions.” An ungrounded receipt (a fix with no attached failing sessions) is itself a signal — the fix is advice, not evidence, until it’s linked to a failure.

Fix

This stage holds the grounded fix artifact: the target (what to change), the change instruction and rationale, and the concrete change itself. For a prompt_change you see the prompt before → after; for other types you see the raw unified diff — one representation, never both.

When the fix engine is enabled for your agent, its controls live in this stage too — launch an eval-verified PR, and the stage’s receipt reflects the result (“PR opened, fixes 12 of 14 evals”, a draft that needs a human, or a failed run to read).

Proof

Before shipping, simulate the fix: Neens replays it against a sample of the real failing sessions and shows a before → after pass-rate comparison, so you apply fixes that actually help and skip the ones that don’t. This stage shows the proof-eval rubric, the simulation plan, and the results; its Configure controls open in a slide-over drawer so the stage stays readable.

Use Simulate fix (POST /remediations/items/{id}/simulate). On production deployments the simulation runs in the background on a worker — the stage shows a pending banner and polls for the result, so you can keep working. Simulation requires an LLM connection; without one the request is rejected with a clear 400 rather than failing mid-run.

Editable criteria

Open the Configure drawer to steer exactly how the simulation runs; a live plan preview shows what would run before you commit:

The resolved choices — including the exact session ids — are pinned to the run, so the background worker replays exactly the cohort you previewed even if new failures arrive in between.

Two strategies

Auto picks record-and-replay when the fix is a system-prompt change and the model connection supports tool calling; otherwise it falls back to answer-rewrite. If record-and-replay turns out to be impossible at run time (e.g. the model rejects tool calling), the run degrades to answer-rewrite automatically and the result says so. The results always show which strategy actually ran.

Reading the results

Results show pass-rate before and after, plus a per-trace breakdown: which traces the fix fixed, which it broke, and which were unchanged. A delta-keyed next step appears — if the fix improved things you get an Apply call to action; if it’s flat or regressed you’re prompted to adjust the criteria (with a warning when any trace got worse). When it’s done, the stage collapses to a one-line receipt — e.g. “Passed: 21% → 76%.”

A completed simulation updates the remediation’s confidence — it is earned from the replay: the after pass-rate dominates (weight 0.6), blended with the before→after improvement (weight 0.25). The basis is stored with the fix so you can see where the number came from.

Ship

The last stage answers did shipping it actually work? — one “After merge” card that folds together the deploy/proof status, the PR / commit / verification links, and the efficacy verdict.

Once a fix is applied, Neens measures whether the failure actually shrank in live traffic. GET /remediations/items/{id}/efficacy compares:

• Baseline volume — how many sessions the failure’s cluster had when the fix was proposed (snapshotted at generation time), against
• Current volume — how many sessions that cluster has now,

and returns the reduction, the reduction percentage, and a verdict: improved (volume fell), flat (no change), regressed (volume grew), or unknown (not enough data to compare). The card sits next to the What changed list of deploy events recorded in the 7 days before the fix was proposed, so you can correlate the failure (and the fix) with real changes — see What changed.

Use the verdict to move the proof status forward: mark the fix verified when the volume drop holds, or regressed if the failure returns. To make sure it stays fixed, turn the failure mode into a standing eval gate.

Reading “unknown” honestly

Everywhere in the detail, a value that hasn’t been earned yet is shown as exactly that — and never dressed up as a success:

• A stat tile with no result reads a muted ”—”, not 0.
• The Proof stage reads Not run until you simulate, and Inconclusive when the replay couldn’t be trusted — neither is a pass.
• The Ship stage reads unknown when there isn’t enough post-deploy data to compare — not “no improvement.”

Each of these is a receipt of what hasn’t happened yet, so you can tell “this fix is proven” apart from “we haven’t checked.” An unknown is never a green tick and never a zero — it’s your cue to run the next step (simulate, apply, or wait for more traffic).

Affected sessions

At the bottom of the detail, an Affected sessions disclosure shows the count of failing sessions grounding the remediation. Expand it to browse the actual sessions; it loads only when you open it, so a remediation with a large cluster stays fast to view.

Triage: status vs. work state

Each remediation has two independent tracks — keep them straight, they answer different questions.

Proof status — is this fix proven and shipped?

Work state — where is this in my team’s queue? A simple triage lane, orthogonal to proof status:

A fix can be applied (proof) while still in_progress (work), or proposed while parked in todo. Both are updated with PATCH /remediations/items/{id} (status, workState, labels) — in the UI, from the header’s overflow ⋯ menu — and an unknown value is rejected with 422. Delete is a permanent hard-delete that also removes the fix’s simulation history.

Moving a remediation to accepted is normally a human decision, and by default it stays one. An agent can hand part of that over with an autonomy level; a human still merges every resulting PR.

The Remediations list

Fix → Remediations is a dense, sortable table — one row per remediation — so you can scan and triage many fixes at once:

Click a sortable column header (every column except Work state) to sort by it. Search and the filters (sort, work state, status, type, label, and Include archived) sit right beside the rows, and label chips filter with one click.

By default the table shows the actionable backlog — grounded, agent-fixable fixes. Service Advisories and Needs grounding items live in their own views so they don’t crowd the work you can actually ship; switch to them (or add include_advisories=true / include_ungrounded=true to the API call) when you want to route advisories to a service owner or review what’s being held. You can also filter by actionability and failure locus directly.

Open a remediation. Click any row to go straight to that remediation’s full page — the fix-loop detail at its own shareable URL, /remediations/{id}, the link to paste into a ticket or hand to a teammate. A thin coloured rail on the left of each row marks its proof status and the Confidence column shows a small meter, so the table scans at a glance.

• Press J / K to move the highlighted row down / up without reaching for the mouse.
• Press Enter to open the highlighted remediation.

Overview. The failure-volume trend and the summary stats are tucked into a collapsed Overview disclosure at the top, so the table and filters get the space by default. Expand it when you want the fleet-level picture; it remembers your choice.

Related

• Issues and failure modes — where the failures come from
• Fix bundles — export a remediation as a coding-agent-ready pack
• Eval-verified PR — let Neens open a proven PR for the fix
• Eval gates — keep a fixed failure from silently coming back
• What changed — correlate regressions with deploys
• Judges — the evaluators simulations use to score before/after
• Insights — regression and anomaly detection across your fleet