Cost Optimization

Cost Optimization answers a running operational question: what is this agent spending on LLM calls, and how much of that is evaluation? It splits spend into two halves — what your agent spends producing answers, and what your LLM judges spend scoring them — over a time window you choose, alongside how much of your traffic actually got scored. It holds your agent and its model fixed: this is “what does the current setup cost”, not a model experiment (for that, see the Sweeps tab, Model sweeps, and Model comparison).

At a glance

What the page shows

The KPI row

Four tiles summarize the window:

The breakdowns

• Cost by judge — one row per judge and the scorer model it ran on, with the judge’s cost, the share of traces it scored, its number of scores, and the distribution of its verdicts. Deterministic (classifier) and external-API judges appear with no scorer cost — they don’t call an LLM to score, so there’s nothing to charge. That blank is honest, not missing data.
• Agent cost by model and LLM-judge cost by model — the same spend split by the individual models involved, with input/output token counts. This is where you see, for example, that most of your judge spend is on one scorer model and you could move some judges to a cheaper one.

The judge model is not the agent model

A score carries two independent models, and Cost Optimization keeps them apart:

• the agent model — the one that produced the answer being graded (this drives Agent cost), and
• the scorer model — the one the judge used to do the grading (this drives LLM-judge cost).

They’re tracked separately on purpose. Switching a judge to a cheaper scorer model changes your LLM-judge cost without touching your agent or its answers at all — which is exactly the lever this page exists to surface.

Reading unpriced and partial results

Neens never invents a price. When a model isn’t in your price table, its spend shows as — and the affected total is flagged partial rather than silently under-counted.

• A single — in a by-model or by-judge row means that model has no price yet.
• A partial total means at least one contributing model is unpriced, so the number you see is a floor — real spend is that much plus whatever the unpriced models cost.

To turn a — into a number, add a rate for that model in Settings → Model pricing. The next time the page loads, that model’s spend is priced and folds into the totals. Self-hosted / open-weight models are a legitimately priced $0 (you run them yourself) — that’s a real zero, distinct from an unpriced —. The full model of where prices come from, including your own negotiated overrides, is in Cost & model pricing.

Add more scorer LLMs out of the box

The cheapest way to lower LLM-judge cost is often to run judges on a cheaper scorer model. Neens speaks the OpenAI-compatible API, so you can point judges at aggregators and proxies that expose hundreds of models behind one endpoint — OpenRouter and a LiteLLM proxy are both first-class API formats. Add one as an LLM connection, then select it on a judge.

Point a judge at the new scorer, then compare

Once the connection exists, open a judge, assign it the new connection, and let it score. Come back to Cost Optimization and the Cost by judge breakdown now shows that judge on the new scorer model — with its own cost line, so you can read the before/after directly.

Worked example. Say your Helpfulness judge runs on a premium scorer and shows up in Cost by judge at a few dollars for the week, scoring 70% of your traces. You add an OpenRouter connection (steps above), pick openai/gpt-4o-mini, and reassign Helpfulness to it. Next week its row shows the cheaper scorer model and a much smaller Cost — while Traces scored for that judge stays around 70%, because you changed what does the grading, not how much grading happens. If gpt-4o-mini isn’t in your price table yet, its cost shows as — (partial) until you add a rate in Settings → Model pricing — after which the savings are quantified, not just implied.

Compare scorer models side by side

Reassigning a judge to a cheaper scorer (above) tells you the new cost, but not whether the cheaper model agrees with the one you trust. A scorer comparison answers both at once: it holds your agent and your judge’s prompt fixed and varies only the scorer LLM, scoring the same window of traces with two or more candidate models and laying their results out side by side — distributions, cost, and coverage in one view. There’s no reference “correct” judge and no automatic winner: you read the columns and decide.

What it is

Pick one judge (its rubric and prompt are frozen — every candidate grades with the identical instructions), a time window of traces, and two or more scorer LLM connections to put head-to-head. Neens scores each trace with each candidate and shows you, per candidate:

• its score distribution — the same verdict breakdown you see on the Scores page (pass/fail for a numeric judge, or the category split for a labelled one);
• its cost for the window — priced exactly like everywhere else, and shown as — when the model has no price yet (never $0);
• its coverage — how many of the window’s traces it scored, as a percentage;
• shared with all candidates: the shared-targets count — the number of traces every candidate scored, which is the honest apples-to-apples base for reading the columns against each other.

How to create one

Reading the result — a worked example

Say your Response Risk judge (it labels each answer none / low / medium / high risk) currently runs on a premium scorer, and you want to know whether gpt-4o-mini could take over. You compare the two over the last week’s traces. The result:

Both models scored all 40 traces in the window, so shared targets is 40 — the whole set is common ground and the two columns are directly comparable.

Read it like this:

• Do the distributions agree? Over the 40 shared traces the two models land on nearly the same risk breakdown — a handful of traces shift by one level, none swing from none to high. For a triage signal that’s close enough to trust the cheaper model.
• What do you save? The premium column shows a real dollar figure; the gpt-4o-mini column shows — because that model isn’t in your price table yet. Add a rate in Settings → Model pricing and re-open the comparison to turn that — into a number and quantify the saving.
• Is the comparison fair? Both models scored all 40 traces, so shared targets is 40 and you are comparing like with like. If one model had only covered 30, you’d read the columns knowing the distributions rest on different populations.

If the cheaper model’s distribution tracks the incumbent’s and its cost is lower, switch that judge to it (see Point a judge at the new scorer above) and the saving shows up on your LLM-judge cost the next time the page loads. If the distributions diverge in a way that matters, you’ve caught it before changing what grades your production traffic.

How it works

• Agent cost is summed from the input/output token counts already on your ingested traces, grouped by the model each call used, then priced. Nothing extra is recorded — it’s derived at read time from data you already send.
• LLM-judge cost comes from each score a judge writes: Neens records the scorer model and the tokens it spent, so judge spend can be summed per judge and per scorer model and priced the same way. Classifier and external-API judges record no scorer tokens — they don’t call an LLM — so they carry no judge cost.
• Traces scored counts distinct traces in the window that have at least one score, over the total number of traces in the window.
• Changing the window recomputes everything for that range; there’s nothing to refresh or rebuild.

Troubleshooting

Related

• Cost & model pricing — where prices come from and how to set your own
• LLM connections — every API format and how credentials are handled
• Judges and Continuous evaluation — what produces judge cost; the Comparison tab on that page is the scorer comparison documented above
• Cost & Quality — the page this tab lives on: recommended moves, the unit toggle, and both proof tabs
• Cost–quality frontier — the statistics behind “clears the bar”, and where the “switch & save” recommendation comes from
• Model sweeps (the Agent model tab, next door on Cost & Quality) / Model comparison — when you do want to change the agent’s model