Sweep decisions

A model sweep produces evidence: N models × k runs over one frozen golden dataset. The comparison turns that evidence into the answer — “for the refund agent the cheap model clears your 90% bar at a fraction of the cost; for the escalation agent it does not.”

That second half is the point. “A cheaper model still hits the number” is almost never true of a whole workspace; it is true of one agent and false of another, and a decision made on the workspace average quietly ships the wrong model to the work that matters most.

At a glance

The whole thing in one worked example

make demo-stack seeds a real sweep you can open and follow along with: Support agent — model selection, six arms over 30 frozen golden items spanning two agents — a refund-agent (22 items: lookups, policy answers, multi-step resolutions) and an escalation-agent (8 items: fraud, chargebacks, safety, over-limit money — cases the agent must not settle alone), each arm run k = 3 times.

Set the quality bar

A bar you did not choose is a bar you should not trust, so the comparison reports where its bar came from — bar.source on the wire, and a line under the verdict in the UI:

Pinning {"min_pass_rate": 0.9} on the sweep’s gate at launch is the durable option: everyone who opens the comparison later reads it against the same number, with nothing to configure and no query parameter to remember. That is what the seeded demo does.

The baseline arm

Savings, regressions and new passes are all measured against one arm — the declared incumbent, which is the arm at position 0 (the first one you listed when you created the sweep). Override it with ?baselineArm=<armId> or the Baseline arm control.

Neens never picks the best-performing arm as the baseline. A baseline chosen after seeing the results is not a baseline, it is a flattering comparison.

How to read the honesty signals

Every one of these exists because the alternative is a number that reads like a fact and isn’t.

Sample size and confidence interval

Every rate carries its n and a 95% Wilson confidence interval, and an arm that this sample cannot separate from the winner is annotated “not distinguishable at n=…” rather than silently ranked above or below it.

94% and 91% over 40 items are not different. At those sample sizes the intervals overlap heavily, and the ordering you see is sampling noise wearing a ranking’s clothes.

Two bounds are reported, and they answer different questions:

In the worked example the winner clears on the point estimate and not on the lower bound — which is why the verdict says so out loud. The fix is more golden items, not a rounder number.

pass^k, and why 2 of 3 is not a pass

Two independent rules, both of which have to hold:

• Per arm: the arm must be green on all k of its runs. An arm at 2/3 reports passed: false and is excluded from the ranking — see pass^k.
• Per item: an item counts as passed only when every scored repeat of it passed. An item that passed 2 of 3 identical attempts has not passed; it is counted in flakyItems instead.

Averaging the flaky items into the rate would re-introduce exactly the lucky green that k repeats exist to catch. In the worked example previous-gen carries 2 flaky items — the same prompts, a different answer on one repeat.

Unpriced arms — the one to read twice

A partly-priced arm is reported the same way: pricedCases versus cases tells you how much of the arm the figure covers, and unpriced sessions are excluded from the mean rather than counted as zero.

Unattributed agents

The per-agent split groups items by the agent name on the captured trace. Items whose sessions carry no agent name group under a row labelled Unattributed — never folded into a named agent, because “we do not know which agent this was” and “it was the refund agent” are different facts, and merging them moves items (and a verdict) onto an agent that never ran them.

If your whole workspace lands under Unattributed, your agent isn’t setting an agent name on its traces — see Send traces. A genuinely single-agent workspace legitimately renders one row; that is honest, not a bug.

Unclassified regressions

The drill-down picks one failure signal for the whole response and says which:

One signal, never a blend, so the same regression can never be counted twice under two names. And regressed sessions that carry no classification are reported in an explicit unclassified count — never dropped, because hiding them shrinks the denominator until the modes that are shown look like the whole story.

A void sweep refuses to name a winner

If a sweep’s arms did not run under identical conditions it is void, and the comparison declines: comparable: false, verdict.outcome: "unavailable", no ranking, no winner. The arms’ own numbers are still shown — they are facts about each run — but no cross-arm claim is drawn from them, in the UI, in the CLI, or in the export.

There is deliberately no way to “just show it anyway”. A comparison of arms that measured different things is the most convincing wrong answer this feature could produce.

”No data” is never 0%

An arm with nothing scored reports rate: null and renders no data. An arm still running is excluded as not_terminal rather than judged on the runs it happens to have finished.

Where the cheap model breaks

Clicking an arm’s regression count opens the drill-down, or call it directly:

curl -sf "$NEENS_BASE_URL/model-sweeps/{sweep_id}/arms/{arm_id}/regressions?limit=50" \
  -H "Authorization: Bearer $NEENS_API_KEY"

A regression is precisely: the baseline arm passed this item on every one of its k runs, and this arm — having been scored at least once — did not. An item this arm never captured is missing evidence, not a regression, and is not counted. The inverse is reported too, as newPasses.

The response gives you three levels, in the order you want them:

1. failureModes — the pinned signal’s labels with counts, each linking to the failure mode when the signal resolved to real taxonomy rows, plus unclassified for the rest.
2. items — up to limit (default 50, max 200) failing golden prompts: the frozen input and expected output, the baseline’s and candidate’s passedRuns/scoredRuns, the candidate’s score, the answer it actually gave, and the judge’s reason.
3. sessionIds on each item — the captured traces themselves, for a full trace view.

totalRegressions is the true total and truncated says when the item list was cut to limit; the per-mode counts always describe the whole regression set, not just the page.

Decide from CI

neens sweep decide reads the comparison and prints the server’s verdict — the same sentence, the same leaderboard, the same per-agent verdicts the UI renders. The ranking is never re-derived client-side: two rankings that can disagree is worse than one.

export NEENS_BASE_URL="https://neens.example.com"
export NEENS_API_KEY="nk_live_..."
 
# Decide a finished sweep against its own pinned bar.
neens sweep decide --sweep-id msw-…
 
# Or against a bar you set here, comparing to a specific incumbent, and wait for it to finish.
neens sweep decide --sweep-id msw-… --bar 0.9 --baseline-arm msa-… --wait
 
# Fail the build unless some arm clears the bar.
neens sweep decide --sweep-id msw-… --bar 0.9 --require-winner --json

export NEENS_BASE_URL="https://neens.example.com"
export NEENS_API_KEY="nk_live_..."
 
npx neens-eval sweep-decide --sweep-id msw-…
 
npx neens-eval sweep-decide --sweep-id msw-… --bar 0.9 --baseline-arm msa-… --wait
 
npx neens-eval sweep-decide --sweep-id msw-… --bar 0.9 --require-winner --json

Flags

Exit codes

Gate on cost, not just quality

A sweep’s pinned gate — and any pre-prod run’s gate — accepts a cost family, evaluated server-side, spelled exactly as the SDK’s client-side cost rule spells it:

{
  "max_regressions": 0,
  "min_pass_rate": 0.9,
  "cost": { "max_delta_pct": -50 }
}

That gate is literally “cheaper and still hits the number”: every child run must hold a 90% pass rate and come in at least 50% below the baseline’s average cost per session. See the server-side cost gate for the full rule set, and for the rule that matters most: an unpriced run is reported skipped, never passed.

Share the decision

The comparison exports as a read-only, redacted, expiring public link — the artifact you paste into the doc that justifies a model switch, without granting anyone access to Neens.

Press Share (redacted) on the comparison page, or:

curl -sf -X POST "$NEENS_BASE_URL/share-links" \
  -H "Authorization: Bearer $NEENS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"resourceType": "sweep_decision", "resourceId": "msw-…", "ttlHours": 168}'

It must be enabled for the agent before it shows anything: turn on shareable links for the agent — PUT /share-links/settings with {"enabled": true}, which requires an admin. Until then the button is hidden rather than broken, and minting a link returns 403.

Drive it from a coding agent (MCP)

The tool description hands the agent the same four rules a human reads: comparable: false means void so never rank; costPerCaseUsd: null means unpriced, never free; rate: null means no data, not 0%; and an item counts as passed only when it passed every repeat. See MCP server.

Reference

Troubleshooting

Related

• Model sweeps — how the evidence is produced, and what void means
• Cost & model pricing — where a cost per case comes from, and how to price a model
• Model comparison — the same question asked of production traffic
• Pre-prod evaluations — the child runs a sweep launches, and the cost gate
• Issues & failure modes — the taxonomy the drill-down groups by
• MCP server — reading the verdict from a coding agent