Cost & Quality

Cost & Quality puts spend and quality on the same screen and answers one question for each of your two big LLM bills: could a cheaper model do this job without losing quality I can actually measure? It leads with the single recommended move and the one button that matters — Test before switching — and keeps the raw sweeps and scorer comparisons a click away as evidence.

The page has three tabs:

At a glance

The version slice

A Neens project is one agent, so “which agent?” is never a question this page asks. The useful sub-slice is which version of that agent produced the trace, read from the neens.version_label attribute your traces carry:

span.set_attribute("neens.version_label", "v2.4.0")

• Every label you have sent appears in the version selector, newest first. The label with the most recent trace is live, and it is the default selection on the tabs that have a version control — you land on the build you are actually running, not on an “all versions” blend.
• Traces with no label form their own unversioned row. They are never folded into the live version: “we don’t know which build this was” and “this was the current build” are different facts.
• Nothing breaks if you never send the attribute — you get one unversioned row and everything else works. Adding it later starts the split from that point forward; it does not rewrite history.

Where the version control appears differs by tab, because the useful slice is different:

Overview

Overview merges spend and recommendations into one visual screen: a hero move, three stat tiles, a spend trend, a spend breakdown, and the ranked list of moves.

The hero move

The top of the tab is the single most valuable move we can prove, presented as the page’s main call to action. If a cheaper model clears the bar, this is a Switch card whose button reads Test before switching (see Test before switching). The copy is explicit that pressing it does not change your agent — it opens a gated test first. If there is nothing to recommend, the hero says so honestly rather than inventing an action.

The three tiles

Two of these need reading carefully.

Judge spend leads with the real number — what your judges actually spent — because that is what lands on the bill. Underneath it, “At full coverage ~$X” answers a different question: what would it cost if this judge scored every trace instead of sampling a share of them? The coverage chip tells you how much of your traffic the judge is currently scoring, so you can read the two together: a low actual cost next to a high full-coverage projection just means the judge is sampling, not that it is cheap to run everywhere.

Savings available is the number the earlier version of this page got wrong, so it is worth being precise about what it now means.

Spend trend and breakdown

Spend trend stacks judge spend on agent spend, one bar per complete UTC day. Today’s bucket is marked partial rather than shown as a finished day that happens to look cheap.

Spend by version | model breaks the same total down as bars — the total dollars spent per agent version, or per model. It is deliberately kept at the dollars-you-paid level; there is no per-trace or per-1,000 framing here, because for a “where did the money go” view the absolute total is the honest unit.

Recommended moves

Below the charts is the ranked list of every move we found — at most one per version and one per judge — sorted by projected monthly saving:

Agent Model

On this tab, quality means a stable pass rate on your golden suite. Neens replays the same frozen dataset against each candidate model k times and judges every run with your judges. “Stable” is the key word: a model that passes on two of three replays is not counted as passing. This pass^k reading is what stops a lucky single run from recommending a model that is actually flaky — you are switching on behaviour that repeats, not on one good roll.

Pick a version from the in-body dropdown (it defaults to your live one) and the tab shows the newest finished, comparable sweep for it.

• Candidate bars — one row per model, named exactly as it appears in your traces with the provider underneath. The quality axis is the stable pass rate on your golden suite, and a pass-rate bar is drawn across the chart at the threshold the candidate has to clear. Models that fall below the bar are drawn visually distinct so a failing candidate can never be mistaken for a passing one. The model you run today is chipped Current; the frontier’s pick is chipped Recommended.
• Verdict card — the plain-language read of the comparison: how much a switch saves per month, the change in pass rate (with an honest not distinguishable at n=… when the sample is too small to separate two models), and whether the recommended model clears the bar.
• Dataset version — a link straight to the dataset version the sweep replayed, so you can open the exact items behind the number instead of taking the pass rate on faith.
• Evidence — the underlying sweep, a Share report link a colleague can open without an account, and the option to launch a new sweep.

A worked example, from the shipped demo — a 30-item golden set replayed 3× with a 90% bar:

Two rules make the recommendation trustworthy rather than merely confident, and the demo ships an example of each:

1. An unpriced model can never win. A model with no price shows — for cost, never $0, and is never recommended — you cannot claim a saving you cannot compute. A model priced at a real $0 (self-hosted, where you pay for the GPU not for tokens) can win, but only if it also clears the bar.
2. A measurement carries its uncertainty. “Not measurably worse” means the candidate’s confidence interval overlaps the current model’s — not that its point estimate happened to land higher. Too small a sample to tell them apart reads as not distinguishable, not as a winner.

Judge Model

On this tab there are two quality numbers, and which one you can act on depends on whether you have gold labels. This is the single most important thing to understand about the Judge Model tab, so read it before the table.

The distinction matters because the two can look similar and mean opposite things. A judge with a high flag rate is failing a lot of your traffic — which is fine if your traffic really is failing, and a problem if the judge is trigger-happy. You cannot tell which from the flag rate alone. Only the fail rate vs gold — comparing the judge to a human on the same sessions — tells you whether the judge is right. That is why:

The lens toggle

A toggle at the top-right of the table switches which number both quality columns show:

• Fail rate (default) — 1 − agreement with gold. This is the lens that drives recommendations. Judges without gold read —.
• Flag rate — the fail-verdict rate on traffic, available for every judge. In this lens the recommended-model column reads —, because a model you have not deployed has no live traffic to flag; the recommendation itself stays gold-based whichever lens you are viewing.

All judges at a glance

The table lists every judge with its current model beside the cheapest model that still holds the bar, so the comparison is one row:

Read the first two rows together: the same cheap model is the right answer for one judge and the wrong answer for another. That is exactly why this is measured per judge rather than decided once for the whole workspace. The recommendation is always the cheapest scorer model whose fail-rate-vs-gold is not distinguishably worse and still clears the bar.

The docked comparison chart

The full per-model comparison for a single judge is docked at the bottom of the tab and only opens when you select a judge row. Selecting a row draws that judge’s candidate bars — one per scorer model — with the agreement threshold bar (default 0.80) drawn across them; models below the bar are visually distinct, and cost is shown as a whole-dollar label. A close control hides the dock again. A judge with no gold labels shows an add-gold-labels empty state in the dock instead of bars, because there is no agreement axis to plot.

To compare scorer models you have not run yet, the Run a scorer comparison button at the top of the tab opens the launch dialog directly — the same targets scored by several models side by side, which is how you get gold-grounded evidence for a judge before you switch it.

Test before switching

Every recommended switch on this page leads to the same button — Test before switching — and never to a button that flips the model for you. Neens does not edit your agent or redeploy your judge; it hands you a gated test and gets out of the way.

For an agent-model switch, that test is a pre-prod evaluation run against the candidate model:

For a judge-model switch, the equivalent proof is a scorer comparison against your gold labels — the same “measure before you move” principle, because a judge you cannot check against a human is a judge you cannot trust to be cheaper and right.

How-tos

Cut judge cost without losing agreement

Prove a cheaper agent model on your golden set before switching

Troubleshooting

Related

• Cost & model pricing — where every rate on this page comes from, and what unpriced means
• Cost–quality frontier — the statistics behind “clears the bar” and “not measurably worse”
• Model sweeps / Sweep decisions — the agent-model evidence
• Judges — deployments, sampling modes and scorer models
• Annotations & review — the gold labels that give the judge tab its recommendation
• Pre-prod evaluations / Eval gates — how a switch is tested before it ships