Cost–quality frontier

The proof tabs on Cost & Quality rest on one picture: a cost–quality frontier. Every candidate model is a dot — the cheaper it is, the further left; the better it scores, the higher up. The line through the best dots is the efficient frontier: the most quality you can buy at each price. It answers the question underneath both proof tabs — are we overpaying for quality, and what is the single move that saves money without moving the quality needle?

The same maths drives two tabs, pointed at two different models:

• The Judge model tab ranks your LLM judges’ scorer models — quality is how well each scorer agrees with your human ground truth. See Cost & Quality and Cost Optimization.
• The Agent model tab ranks candidate agent models — quality is a stable pass rate on a frozen golden suite. See Model sweeps and Sweep decisions.

Both read the same frontier, so “clears the bar”, “recommended”, and “no measurable quality loss” mean exactly one thing across the two tabs.

At a glance

How to read the frontier

The axes are always the same shape — cost on the X-axis (right = more expensive), quality on the Y-axis (up = better) — and every dot falls into one role.

The acceptance bar

The acceptance bar is the minimum quality a model must clear to be a candidate at all — a horizontal line on the chart. A model below it is a quality risk no matter how cheap.

• Cost tab (judges): the bar is agreement with your experts, default 80%. Override it per view with the bar control (or ?bar= on the API, a 0–1 rate).
• Sweeps tab (agent models): the bar is the sweep’s pass-rate gate — the one you committed to when you launched it, defaulting to the 90% deployment success threshold. See Set the quality bar.

The noise band — why “cheaper” needs a confidence interval

Quality is measured on a sample, so it is never a single exact number — it’s a point estimate with a confidence interval around it (a 95% Wilson interval, the same statistic the sweep comparison uses). That interval is the noise band.

This is what makes an honest “no quality loss” claim possible:

• If the cheaper model’s interval overlaps your current model’s, the two are not statistically distinguishable at this sample size — switching is a no measurable quality loss move. That is the honest version of “cheaper, without compromising quality.”
• If the intervals don’t overlap, Neens shows the real, signed quality delta instead of hiding it — the cheaper model either measurably improves or measurably regresses.
• A model that is distinguishably worse is never recommended, however much it saves.

Cost tab: the judge frontier

On the Cost tab, each dot is a scorer model one of your judges can run on, and quality is that scorer’s agreement with your experts. The chart answers: can this judge do its grading on a cheaper model without disagreeing with my humans any more than it already does?

There is one independent frontier per judge — you can only swap a judge’s scorer model within that judge, so a recommendation never proposes moving one judge’s grading onto a different judge’s model.

You need gold labels first

To give the Cost frontier a quality axis, produce expert truth:

Find and apply a “switch & save”

An honest limitation you should know

Sweeps tab: the agent-model frontier

On the Sweeps tab, each dot is a model sweep arm — a candidate model for your agent — and quality is the arm’s stable-pass rate (the composite judge score over the frozen eval set). The chart answers: which cheaper model still clears the bar for this agent?

Because a sweep already measures quality with a Wilson interval, cost per case, and a pass-rate gate, the frontier reuses those numbers directly — it recomputes nothing. Read it exactly like the judge frontier:

• The recommended arm is the cheapest one that beats your baseline (the declared incumbent — the arm you listed first, overridable), clears the bar, and is inside the baseline’s noise band.
• Because a sweep is an experiment, not production traffic, the recommendation reports a per-case saving and, honestly, no monthly figure — inventing production volume for an experiment would be a lie.
• An arm with no data (its endpoint was unreachable) is shown as such, never as 0%.

For the full leaderboard, per-agent verdicts, regression drill-down, and the shareable decision, use the comparison view — see Sweep decisions.

How it works

• Cost frontier (GET /cost-optimization/frontier) joins the per-scorer cost from Cost Optimization with judge↔expert agreement from alignment, one frontier per judge, over a window (?window=7d by default, or ?start=&end=). Agreement carries a Wilson confidence interval — the noise band.
• Sweeps frontier (GET /model-sweeps/{id}/frontier) is derived entirely from the sweep’s own comparison — same reconcile, same bar, same baseline — and recomputes no quality.
• The recommended move is, in both cases, the cheapest priced model that is strictly cheaper than your current one, clears the acceptance bar, and whose quality interval overlaps the current one’s (or is better). Overlapping intervals ⇒ no measurable loss; a measurable gap ⇒ the real delta; a distinguishably worse model ⇒ no recommendation.
• Everything is computed on read. Changing the window or the bar recomputes the frontier; there’s nothing to refresh or rebuild, and nothing is spent.

Troubleshooting

Related

• Cost Optimization — the Cost tab: what your agent and judges spend, and how to add a cheaper scorer
• Model sweeps / Sweep decisions — the Sweeps tab: running the comparison and reading the verdict
• Annotations & review — how to produce the gold labels the Cost frontier needs
• Cost & model pricing — where a price comes from, and how unpriced models are handled
• Pre-prod evaluations — how to ship a model switch behind a gate that blocks a real regression