The eval-verified PR

The fix engine closes the Neens failure→fix loop into a single, trustworthy action: from a confirmed remediation, Neens proposes a code/prompt change, applies it on a branch, verifies it with pre-prod evals, and opens a pull request — but only when the fix actually holds. It never merges: a human always reviews and merges the PR.

This is the difference between “here’s a suggested fix” and “here’s a fix we proved works.” Every PR the fix engine opens carries its proof: the pass^k eval results, the held regression set, and the human-aligned judges that verified it.

At a glance

The loop

1. Propose — a driver turns the remediation’s fix bundle into a patch. The default artifact driver replays the remediation’s own typed before/after fix deterministically (no LLM, no repo access). A webhook driver hands the bundle to your coding agent; a cli driver runs a headless coding-agent CLI in a clone.
2. Apply — the patch is applied on a new branch through a VCS driver (a real GitHub App, or a local git remote for self-hosting/demos). The PR is not opened yet.
3. Verify (pass^k) — Neens runs the candidate through k independent pre-prod eval runs against the failure’s own failure-derived evals and your accumulated regression set.
4. Gate — the fix passes only when every run is green (zero regressions) and a human-aligned judge that the fix did not optimize against verified it.
5. Open the PR — only a passing candidate opens a pull request; Neens stamps the PR URL and the verification run back onto the remediation. A failing candidate retries up to a bounded budget, then lands a draft with the failure report instead of a clean PR.

At no point does Neens merge, hold your repository credentials beyond the VCS installation you configured, or modify a running system.

And to run this loop unattended — Neens starting the verified run itself, or accepting an eligible remediation and starting it with no human in between — set a per-agent autonomy level. The human merge gate stays exactly where it is at every level.

Start a fix run

curl -X POST https://your-neens/api/fix-engine/runs \
  -H "Authorization: Bearer nk_live_…" \
  -H "Content-Type: application/json" \
  -d '{
    "remediationId": "rem_…",
    "vcsInstallationId": "vcs_…",
    "versionLabel": "fix/order-status-grounding"
  }'

Configure a VCS installation

The fix engine opens PRs through a VCS installation you register once under Fix → Remediations (or via POST /fix-engine/vcs-installations, admin-only). The credential is encrypted at rest and never returned — reads show only whether one is present.

pass^k — why one green run isn’t enough

A single eval run can pass by luck: a non-deterministic agent, a flaky tool, or a judge that happened to grade generously. pass^k requires the candidate fix to pass k independent verification runs — all green, zero regressions — before a PR is opened. k ≥ 3 is what defeats a flaky green: a fix that only passes sometimes will fail at least one of the k runs and never opens a PR.

The default is 3. Raise it for higher-stakes changes; the run reports greens/k and refuses to open a PR until every run is green.

Anti-gaming — who is allowed to verify

A fix that is graded by the same signal it was tuned against isn’t verified — it’s overfit. Neens enforces two non-negotiable rules when it picks the judges that VERIFY a fix:

1. Separation from the authoring signal. Any judge the fix optimized against is excluded from verification. The verifier must be a different judge than the one the change was aimed at.
2. Human anchoring. A verifying judge must be human-aligned — its agreement with expert labels (Cohen’s κ) must clear a floor (default 0.4). A judge with no measurement, or one below the floor, cannot anchor a verification. See Annotations & review for how judge↔expert alignment is measured.

If pass^k is green but no human-anchored judge is available to verify, Neens lands a draft PR (lower confidence) rather than a clean one — an unanchored verification is never presented as trusted.

This is also why a fix that the Neens prompt optimizer wrote needs a second human-aligned judge: the optimizer records the judges it optimized the prompt against, and those judges are excluded from verifying it. If they are the only ones available, the run lands a draft by design.

The living regression set

Every fix run verifies against two things: the failure’s own failure-derived evals and your accumulated regression set — a single, growing golden dataset of everything the agent must keep getting right. A candidate that fixes the target failure but breaks a past correction fails the gate.

The regression set compounds: when a human corrects a judge during review, that corrected example is captured into the set. So every human correction makes the gate stronger, and each new fix must hold the entire accumulated history — not just its own narrow test.

Two ways to verify: preview URL vs. CI gate

The verification step reuses the Neens pre-prod evaluation engine, so you can run it in whichever lane fits your deploy:

• Preview URL — Neens calls your agent. Point the run at a preview deployment’s endpoint URL (or an existing agent connection) at a version label. Neens calls it per golden prompt, captures the traces, and scores them — no test harness on your side. This is the one-command “verify this candidate” lane.
• Customer-CI gate. Run the neens eval run command the fix bundle pre-generates in your CI, with your gate policy. The pull request only merges if the gate passes. Neens provides the dataset, the judges, and the gate-as-code; your pipeline runs it against your build.

Both lanes verify against the same failure-derived evals plus the regression set, and both are inspectable in Neens — every verification eval is a real run you can open and read.

Read the eval-verified PR

The PR the fix engine opens is written to be reviewed. Its body leads with the proof, because that’s what a human merges on:

• A one-line proof headline — e.g. “Fixes 12 of 14 new evals, regresses 0 of 210.”
• pass^k — how many verification runs were green out of k.
• Root cause — the diagnosed reason the agent was failing, and where to look.
• The change — the fix summary and the target it touches.
• Verified by — the human-κ-anchored judges that verified it (with their κ), and an explicit note that they are distinct from any signal the fix optimized against.
• Regression set — how many accumulated regression checks the fix held.

On your side you review the diff and the proof, then merge (Neens never does) and move the remediation to Applied in Neens to record that the fix shipped.

Reachability & self-hosting

The fix engine’s outbound calls — a self-hosted GitHub Enterprise API, a preview-deploy endpoint, or a webhook coding-agent — are SSRF-gated: a host that resolves to a private, loopback, or link-local address is refused unless it is allowlisted. Public hosts (like github.com) are never gated. To reach an internal or local target, ask your operator to allowlist its host or CIDR.

Troubleshooting

• The run is rejected with “no proof-gate dataset.” The remediation has no failure-derived eval gate to verify against. Generate one for its failure mode first — see Eval gates.
• pass^k never turns green. The fix isn’t reliably fixing the failure. Read the per-run results in the fix run detail; a candidate that’s green sometimes but not always is exactly what pass^k is meant to catch.
• The run lands a draft, not a clean PR. pass^k passed but no human-anchored judge was available to verify. Label more gold items so a judge’s alignment is measured (see Annotations & review).
• A verification call is blocked. The target host resolves private/loopback and isn’t allowlisted — ask your operator to allowlist its host.