Eval gates
An eval gate is a standing evaluation built from one of your own confirmed failures and used as a release guard. This is the evals-from-failures flywheel: instead of writing evals from scratch, you turn a real recurring failure into a regression test — so once you fix it, it can’t silently come back.
One action materializes the whole chain from a failure mode: a dataset of its real evidence sessions, a judge that scores the failure as absent, and the gate binding them together with a tracked baseline.
At a glance
| Where | Fix → Eval Gates in the sidebar; created from an Issue card’s Generate eval in Diagnose |
| Key API | POST /flywheel/failure-modes/{id}/generate-eval, GET /flywheel/gates, GET /flywheel/gates/{id}, PATCH /flywheel/gates/{id}, POST /flywheel/gates/{id}/run |
| Needs | A failure mode with evidence (classified sessions, a linked cluster, or exemplars). An LLM connection to actually run the drafted judge |
| Scope | Agent-scoped |
Create a gate
Confirm a failure mode
Gates come from failure modes in Issues and failure modes — a named, defined pattern of failure with real evidence sessions behind it.
Generate the eval
On the mode’s Issue card use Generate eval, or call
POST /flywheel/failure-modes/{id}/generate-eval. In one step Neens:
- builds a dataset from the mode’s evidence — its exemplar sessions, every session the classifier tagged with the mode, and the members of any linked failure cluster (de-duplicated);
- drafts an LLM judge whose rubric scores the failure as absent: a high score means the agent did not exhibit it (the mode’s definition is folded into the criteria);
- parks a human-gated deployment for that judge — created disabled, so nothing runs until an expert has reviewed the draft and enabled it;
- registers the gate binding mode ↔ judge ↔ dataset ↔ deployment, active with an empty baseline.
The response returns all the created ids (gateId, judgeId, datasetId, deploymentId,
datasetItemCount).
Review and enable the judge
The drafted judge is a starting point, not gospel. Review its rubric on the Judges page and enable its deployment when you’re satisfied — until then the gate’s detail shows the deployment as disabled.
Run it
Use Run gate to fire the first eval run. The first completed run seeds the baseline pass rate; every later run updates the latest pass rate and is compared against that baseline.
A gate needs evidence. If the failure mode has no classified sessions, linked cluster, or
exemplars yet, generation is refused with 422 — Neens won’t create a gate bound to an empty
dataset that would pass forever. Classify sessions into the mode or link a cluster first.
What a gate tracks
Each gate row carries:
| Field | Meaning |
|---|---|
name | Derived from the failure mode (FM: <mode name>) |
status | active or paused |
baselinePassRate | Seeded by the first completed run; the reference every later run is compared to |
lastPassRate | Pass rate of the most recent completed run |
lastRunId / lastRunAt | The most recent run and when it fired |
failureModeId / judgeId / datasetId / deploymentId | The provenance chain the gate binds together |
A run’s pass rate is computed from its scored items: an item passes when its score meets the judge’s threshold (0.7 when the judge doesn’t set one). Since the judge scores the failure as absent, a drop in pass rate means the failure is coming back.
The Eval Gates page lists every gate with its source failure mode, status, baseline, and latest pass rate; you can filter by status or failure mode, search by name, and sort any column. Per-row actions: Run gate, and Pause / Activate.
The gate detail drawer
Click a gate to open its detail drawer (GET /flywheel/gates/{id}), which explains the gate on
one screen:
- Provenance — the What this gate checks section shows the source failure mode (name, definition, severity), the drafted judge (name, type), the dataset (name, item count), and the deployment’s status and trigger — so you always know why this gate exists and whether its judge has been enabled yet.
- Baseline vs. last — the baseline pass rate side by side with the most recent run’s pass rate, plus a green/red delta badge. This is the at-a-glance regression signal.
- Recent runs — the last 10 eval runs, newest first, each with its status, progress, computed pass rate, and who (or what) triggered it.
From the drawer you can Run gate, Pause/Activate, and re-baseline: PATCH the gate
with a new baselinePassRate when you’ve intentionally changed the bar (for example after a big
fix landed and you want future runs compared to the new normal).
Using gates to guard releases
A practical rhythm:
- After shipping a fix — run the gate for that failure to confirm the fix holds against the
original failing evidence. A remediation reaching
verifiedand its gate staying green are the two halves of “this failure is handled” (see Remediations). - As part of release checks — run your active gates when a new agent version is about to ship. A pass-rate drop below baseline shows up as a failing gate instead of a silent production incident.
- When a gate drops — check What changed: recording deploy events lets you correlate the drop with the prompt/model/tool change that landed just before it, turning a failing gate into “this deploy did it” rather than a mystery.
Pause gates you’re not ready to enforce; they keep their history and can be re-activated any time.
Gates vs. pre-prod evals
Both guard releases, at different granularities:
- An eval gate guards one known failure — a focused regression test over that failure’s own evidence dataset, with a per-gate baseline.
- A pre-prod evaluation scores a whole candidate version against a golden dataset and surfaces regressions versus a baseline run or your production window.
Use gates for “this specific bug must never return,” and pre-prod evals for “this release, as a whole, is not worse.” Both are powered by the same judges machinery, so a judge you refine in one place improves the other.
API reference
| Endpoint | Purpose |
|---|---|
POST /flywheel/failure-modes/{id}/generate-eval | Materialize dataset + judge + disabled deployment + gate from a failure mode (404 unknown mode, 422 no evidence) |
GET /flywheel/gates | List gates; filters status, failureModeId, name; sortable by name, failure mode, status, baseline, last pass rate, last run, created |
GET /flywheel/gates/{id} | Rich detail: provenance, deployment status, and the 10 most recent runs with pass rates |
PATCH /flywheel/gates/{id} | Update status (active | paused) and/or baselinePassRate |
POST /flywheel/gates/{id}/run | Fire an eval run against the gate’s dataset; the first completed run seeds the baseline |
Related
- Issues and failure modes — the confirmed failures gates are built from
- Remediations — fix the failure the gate guards
- Judges — review and tune the drafted judge
- Pre-prod evaluations — whole-version release gating
- What changed — correlate a gate drop with a deploy
- Insights — automatic regression detection across metrics