Model sweeps

A model sweep replays one frozen golden dataset against several models at once — each model behind its own agent endpoint — under identical conditions, and requires every model to pass k independent runs before it is called a pass. It answers the question a single pre-prod evaluation cannot: which of these models is good enough for this job, and what does each one cost?

Where a pre-prod eval asks “is this candidate version safe to ship”, a sweep asks “which model should this agent run on” — and it refuses to answer at all if the comparison it would draw is not a fair one.

At a glance

When to reach for one

• Model selection. You are deciding between a frontier model, a cheaper hosted model and a self-hosted one for the same agent. Run all three against the same golden set and read the pass rates side by side.
• A vendor price or model change. A new model version lands, or a price moves. Re-run last month’s sweep and see whether the answer changed.
• Monthly in CI. A sweep is cheap enough to schedule and expensive enough to want a budget ceiling on. The CLI’s preview-then-launch flow is built for exactly this (see Run a sweep from CI).
• Against your own failures, not a public benchmark. The strongest golden set for this decision is a stress-test suite generated from your confirmed failure modes. A public benchmark tells you how a model does on somebody else’s problems; a suite built from last quarter’s incidents tells you whether it survives yours. Pass the suite id when you create the sweep and Neens uses its frozen synthetic dataset version.

Arms, k, and what stays fixed

A sweep is made of arms. One arm = one model, reached through one Agent endpoint connection. Arms differ in exactly one thing — the endpoint (and therefore the model). Every other input is pinned once on the sweep and reused verbatim for every child run:

Two arms may not point at the same endpoint connection: that is not a comparison, it is the same model twice, and it would render as two independent results that look like corroboration.

pass^k is per arm — not per sweep

A single green run can be luck: a non-deterministic model, a flaky tool call, a judge that graded generously on the day. pass^k requires an arm to pass k independent runs — all green, zero regressions — before that model is reported as passing.

The critical detail is that k is evaluated per arm, not across the sweep:

A cheap model that passes once and fails twice has not hit the number. It reports 1/3 green, passed: false — not “passed, with variance”.

This is the same rule the eval-verified fix engine uses to decide whether a fix may open a pull request, applied to a model choice instead of a patch. k = 3 is the default and is what defeats a flaky green; a run-to-run difference that a single run would have hidden shows up as an arm that cannot get to 3/3.

An arm with fewer than k finished runs has no verdict at all — it stays in progress with passed unset rather than reporting the greens it has so far as a result. A partially finished arm never borrows the benefit of the doubt.

Identical conditions, and what void means

The whole value of a sweep is that the only difference between the arms is the model. If that stops being true, the comparison is worthless — and a worthless comparison that still renders a chart is worse than no chart, because somebody will screenshot it into a decision doc.

So Neens pins the controlled variables at create time as a conditions fingerprint (the dataset version, the sorted judge set, k, the version label, and the gate policy — deliberately not the model, which is the independent variable), and re-derives that fingerprint from each child run’s actual persisted values when it rolls the sweep up.

If any arm’s real conditions diverge from the pinned ones, the sweep becomes void:

• The status is terminal — a void sweep never recovers into a comparable one.
• The detail page shows a prominent banner naming which arms diverged, and suppresses every cross-arm comparison affordance. No chart, no ranking, no “winner”.
• A child run that vanished, or was never created, counts as divergence — not as a pass.

Cost: estimate first, then decide

Every arm multiplies the bill: an arm costs k × item_count model calls, and a sweep costs that again for every arm. So the cost pre-flight is a first-class step, not a nicety.

The budget ceiling and the run caps

A fully unpriced estimate does not refuse on budget — Neens will not block on a number it does not have. It surfaces partial and leaves the decision with you.

Per-arm failure isolation

A dead endpoint fails its own arm and nothing else. If one model’s endpoint is unreachable, times out, or returns nothing usable:

• That arm goes failed with the error recorded on it.
• Every other arm keeps running and is reported normally.
• The sweep finishes as completed_with_failures — every arm terminal, at least one failed, at least one completed — which is a different, and honest, status from completed.

Partial results are the normal case, not an error state: each arm carries its own status and its own progress, so a sweep detail page shows three arms in three different states rather than one spinner over the whole page.

Create a sweep

curl -sf -X POST "$NEENS_BASE_URL/model-sweeps/preview" \
  -H "Authorization: Bearer $NEENS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "datasetId": "ds_golden",
    "passK": 3,
    "arms": [
      {"label": "sonnet",    "agentConnectionId": "conn_sonnet"},
      {"label": "mini",      "agentConnectionId": "conn_mini"},
      {"label": "self-host", "agentConnectionId": "conn_selfhost"}
    ]
  }'

curl -sf -X POST "$NEENS_BASE_URL/model-sweeps" \
  -H "Authorization: Bearer $NEENS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Q3 model selection",
    "versionLabel": "2026-07-candidate",
    "datasetId": "ds_golden",
    "passK": 3,
    "budgetUsd": 10,
    "arms": [
      {"label": "sonnet",    "agentConnectionId": "conn_sonnet"},
      {"label": "mini",      "agentConnectionId": "conn_mini"},
      {"label": "self-host", "agentConnectionId": "conn_selfhost"}
    ]
  }'

curl -sf "$NEENS_BASE_URL/model-sweeps/{sweep_id}" \
  -H "Authorization: Bearer $NEENS_API_KEY"

curl -sf -X POST "$NEENS_BASE_URL/model-sweeps/{sweep_id}/cancel" \
  -H "Authorization: Bearer $NEENS_API_KEY"

Then read the answer

A finished sweep is evidence, not a decision. Press Compare arms on the sweep detail page to get the verdict — which model to ship, at what bar, at what cost per case, and per agent, with the sample size, confidence interval and unpriced arms shown rather than rounded away:

curl -sf "$NEENS_BASE_URL/model-sweeps/{sweep_id}/comparison" \
  -H "Authorization: Bearer $NEENS_API_KEY"

That surface has its own guide: Sweep decisions. It also covers the regression drill-down (where a cheaper arm breaks), the neens sweep decide CI verb, and the redacted share link you can paste into a decision doc.

The Sweeps tab also renders the finished sweep as a cost–quality frontier — each arm plotted by cost per case against its pass rate, with the cheapest arm that clears your bar highlighted as the recommended move. It’s the same chart the Cost tab uses for judge scorer models, and a fast way to see which cheaper model still clears the quality bar. A void sweep draws no frontier, for the same reason it renders no comparison.

Run a sweep from CI

The neens-eval SDK (Python and TypeScript, same flags, same exit codes) has a sweep group beside eval. The CI story is estimate, then decide:

export NEENS_BASE_URL="https://neens.example.com"
export NEENS_API_KEY="nk_live_..."
 
# 1. What will this cost? Writes nothing, spends nothing.
neens sweep preview \
  --dataset-id ds_golden \
  --arm "sonnet=conn_sonnet" \
  --arm "mini=conn_mini" \
  --arm "self-host=conn_selfhost" \
  --pass-k 3 --json
 
# 2. Launch it and wait for every arm.
neens sweep start \
  --name "monthly model sweep" \
  --dataset-id ds_golden \
  --version-label "$GIT_SHA" \
  --arm "sonnet=conn_sonnet" \
  --arm "mini=conn_mini" \
  --arm "self-host=conn_selfhost" \
  --pass-k 3 --budget-usd 10 --wait
 
# 3. Or poll one later.
neens sweep get msw-… --json
 
# 4. Print the verdict: which arm to ship, at what bar and cost, per agent.
neens sweep decide --sweep-id msw-… --bar 0.9

export NEENS_BASE_URL="https://neens.example.com"
export NEENS_API_KEY="nk_live_..."
 
npx neens-eval sweep-preview \
  --dataset-id ds_golden \
  --arm "sonnet=conn_sonnet" \
  --arm "mini=conn_mini" \
  --pass-k 3 --json
 
npx neens-eval sweep-start \
  --name "monthly model sweep" \
  --dataset-id ds_golden \
  --version-label "$GIT_SHA" \
  --arm "sonnet=conn_sonnet" \
  --arm "mini=conn_mini" \
  --pass-k 3 --budget-usd 10 --wait
 
npx neens-eval sweep-get msw-… --json
 
npx neens-eval sweep-decide --sweep-id msw-… --bar 0.9

Exit codes

• A sweep is informational by default. Arms that did not all pass do not fail your build — a sweep is a model-selection decision, not a release gate. Use eval gates and pre-prod evals to block a release.
• Pass --require-all-arms when you do want a non-zero exit unless every arm passed pass^k.
• A void sweep always exits non-zero, with the void reason printed — a comparison that cannot be trusted must never look like a pass.
• neens sweep decide follows the same shape: informational by default, non-zero only for a void or failed sweep — or with --require-winner. See its exit codes.

Drive it from a coding agent (MCP)

Three MCP tools expose the same flow to an agent:

How it works

Troubleshooting