Fix outcomes

Opening an eval-verified PR proves a fix works before it ships. Fix outcomes closes the loop after it ships: when a Neens-opened fix PR is merged, Neens records the deploy, watches the fixed failure in real production traffic for a window, and reports whether the failure actually went down — per agent, per merged fix.

This is the difference between a dashboard number and a business outcome. “Failures shown” is a chart. “Failure volume is down 60% across 8 merged fixes, with a median cluster-to-merge time of 2 days” is a result you can take to a stakeholder — the public close-out of the Neens failure → fix → proof loop.

At a glance

How the loop closes

Tell Neens a fix merged

Neens needs to know when a PR merges. Pick whichever path fits how you ship.

https://<neens-host>/api/webhooks/github

curl -X POST https://your-neens/api/remediations/items/rem_123/record-merge \
  -H "Authorization: Bearer nk_live_…" \
  -H "Content-Type: application/json" \
  -d '{
    "prUrl": "https://github.com/acme/agent/pull/42",
    "commitSha": "9f3c1ab",
    "mergedAt": "2026-07-10T00:00:00Z"
  }'

Read the Outcomes report

Open Fix → Outcomes for a per-agent rollup of every merged fix and what it did in production. The same data is served by GET /remediations/efficacy-report.

The KPIs

The headline tiles answer “did shipping fixes make the product better?”:

• Fixes shipped / verified / regressed — how many merged fixes have a verdict, and how many held up in production vs. came back.
• Failure-volume reduction % — total volume of the fixed failure modes before vs after their deploys, as one headline percentage across all merged fixes.
• Median MTTR — the median time from a failure first being seen (its cluster’s first-seen timestamp) to the fix merging. This is your cluster-to-merge time: how fast the loop actually turns.

Before vs after, per fix

Each merged fix shows its failure volume before the deploy next to its volume after, and the per-fix reduction. A fix that drove its failure mode from 40 occurrences to 6 reads as an 85% drop and a verified verdict; a fix whose volume didn’t move (or whose gate broke) reads as regressed.

What verified / regressed / inconclusive mean

• Verified — production volume of the failure mode dropped and the proof-eval gate still holds. The fix did what it claimed.
• Regressed — volume didn’t drop, or a proof gate that used to pass now fails. The failure is still happening (or came back) despite the merge — worth reopening.
• Inconclusive — too little traffic to call it. When the before-window volume is too low to form a baseline, there is no reduction to measure against, so Neens won’t manufacture a verdict from noise. (An after-window spike is a regression, not an inconclusive.)

{
  "window": { "days": 7 },
  "summary": {
    "fixesShipped": 8,
    "fixesVerified": 6,
    "fixesRegressed": 1,
    "fixesPending": 1,
    "fixesInconclusive": 0,
    "totalVolumeBefore": 240,
    "totalVolumeAfter": 96,
    "volumeReductionPct": 60.0,
    "mttrMedianSeconds": 172800,
    "mttrMedianDays": 2.0,
    "mttrCount": 7
  },
  "closeouts": [
    {
      "id": "co-…",
      "remediationId": "rem-…",
      "remediationTitle": "Guardrail over-blocks refunds",
      "status": "verified",
      "prUrl": "https://github.com/acme/agent/pull/42",
      "mergedAt": "2026-07-10T00:00:00Z",
      "beforeVolume": 40,
      "afterVolume": 6,
      "reductionPct": 85.0,
      "volumeStatus": "improved",
      "gatesHeld": true,
      "mttrDays": 1.5,
      "windowDays": 7
    }
  ],
  "trend": [
    { "period": "2026-07", "fixesVerified": 6, "volumeBefore": 240, "volumeAfter": 96, "reductionPct": 60.0 }
  ]
}

Did the KPI move?

Volume answers “is the failure happening less?” It does not answer “did the number we funded this fix to move actually move?” — and those two come apart all the time: a failure can get rarer while the business metric it was hurting barely budges, or a fix can quietly lift a KPI well beyond the one cluster it targeted. So beside the volume leg, each close-out now carries a KPI leg. For every active business KPI, Neens measures the KPI over the window before the deploy and over the equal window after it, and reports the before→after move.

It measures each KPI at two scopes, so you can tell a local win from a fleet-wide one:

• Cluster — the KPI computed over just the failure cluster this fix targeted. “Did containment recover for the cases this fix was about?”
• Agent — the KPI over the whole agent, unfiltered. “Did the fleet number move, or only this corner of it?”

A fix can read improved at the cluster scope and flat at the agent scope — that is a real, common, honest result (the fix worked where it was aimed, and that corner is a small slice of all traffic), not a contradiction.

The four statuses

Every KPI-delta row lands in one of four statuses, and each renders differently on purpose.

A worked example

The remediation “Guardrail over-blocks refunds” was funded because an over-eager guardrail was escalating refund cases a human then had to pick up. After it merged, its close-out shows:

• Volume — the failure mode fell from 40 occurrences to 6 (an 85% drop): the failure is happening far less.
• KPI, cluster scope — containment rate on that cluster’s cases rose 62% → 81%, a distinguishable +19-point move: the cases this fix was about are now handled without a human.
• KPI, agent scope — agent-wide containment moved 87% → 88%, reported flat: the fleet number barely shifted, because this one cluster is a small share of all traffic. Honest, and not oversold.

That is what the KPI leg adds: not just “the failure is rarer”, but “and here is the business number it was costing you, measurably recovered for the cases it touched.”

{
  "id": "co-9f2a1c",
  "remediationId": "rem-4471",
  "remediationTitle": "Guardrail over-blocks refunds",
  "status": "verified",
  "beforeVolume": 40,
  "afterVolume": 6,
  "reductionPct": 85.0,
  "volumeStatus": "improved",
  "kpiDeltas": [
    {
      "kpiId": "kpi-7c1a", "kpiLabel": "Containment rate", "measureKey": "containment_rate",
      "scope": "cluster", "kind": "rate",
      "before": 0.62, "after": 0.81, "delta": 0.19, "deltaPct": 30.65,
      "ciLow": 0.04, "ciHigh": 0.34, "distinguishable": true,
      "status": "improved", "unknownReason": null,
      "beforeN": 210, "afterN": 240, "targetStatus": "met"
    },
    {
      "kpiId": "kpi-7c1a", "kpiLabel": "Containment rate", "measureKey": "containment_rate",
      "scope": "project", "kind": "rate",
      "before": 0.87, "after": 0.88, "delta": 0.01, "deltaPct": 1.15,
      "ciLow": -0.02, "ciHigh": 0.04, "distinguishable": false,
      "status": "flat", "unknownReason": null,
      "beforeN": 704, "afterN": 731, "targetStatus": "met"
    },
    {
      "kpiId": "kpi-3d81", "kpiLabel": "Cost per case", "measureKey": "cost_per_case",
      "scope": "cluster", "kind": "cost",
      "before": 0.51, "after": 0.42, "delta": -0.09, "deltaPct": -17.65,
      "ciLow": null, "ciHigh": null, "distinguishable": null,
      "status": "improved", "unknownReason": null,
      "beforeN": 210, "afterN": 240, "targetStatus": "met"
    },
    {
      "kpiId": "kpi-55c0", "kpiLabel": "Deflection rate", "measureKey": "custom:deflection-rate",
      "scope": "cluster", "kind": "rate",
      "before": 0.44, "after": 0.51, "delta": null, "deltaPct": null,
      "ciLow": null, "ciHigh": null, "distinguishable": null,
      "status": "unknown", "unknownReason": "definition_changed",
      "beforeN": 96, "afterN": 120, "targetStatus": "unknown"
    }
  ]
}

The same moves also feed the KPI’s own page: every business KPI lists the fixes that moved it, so you can start from a number that recovered and see which merged fixes recovered it.

How the verdict is decided

For each merged fix, once the watch window has elapsed:

1. Measure production volume of the fixed failure mode in the window before the deploy and in the equal window after it.
2. Re-check the proof gate — does the remediation’s failure-derived eval gate still pass?
3. Decide:
   • volume dropped and the gate holds → verified;
   • volume did not drop, or a gate broke → regressed;
   • before-window volume too low to form a baseline → inconclusive.

The verdict is written back onto the remediation, so its lifecycle (applied → verified / regressed) reflects what really happened in production — not just that a PR merged.

Troubleshooting

• A merge didn’t start a watch. Neens couldn’t correlate the PR to a remediation. Confirm the PR URL matches the one Neens stamped when it opened the fix, or record the merge explicitly with POST /remediations/items/{id}/record-merge.
• Webhook deliveries are ignored. The X-Hub-Signature-256 didn’t verify — the secret on the GitHub side doesn’t match the one Neens was configured with.
• A fix stays inconclusive. Not enough of the failure mode’s traffic landed in the window. Wait for more production traffic to accumulate before the verdict can be called.