Model comparison

Swapping an agent onto a cheaper model is easy to justify on cost and hard to justify on quality. Neens closes that gap: every score records which model produced the answer it graded and how that was known, so eval_pass_rate can be broken down per model — and per agent × model — from the same judges you already run in production.

The other half of the argument is on Cost & model pricing: that page tells you what each model costs, this one tells you whether it still hits the bar.

At a glance

Answer: “does the cheaper model still hit our bar?”

Where a score’s model comes from

Every score carries a Model attribution value naming the evidence used. It is a closed vocabulary, listed here in precedence order — the first one that resolves wins:

The left column is the value stored on the score and used in API filters; the middle column is how it’s worded on a chart.

Only spans that actually name a model vote. A tool call or a retrieval step carries no model, so it never supplies one and never makes a trace look “mixed”.

Reading Mixed and Unknown — and what to do about them

These are the two buckets people want to make disappear. Both are answerable.

Mixed

Mixed means the graded trace ran on more than one model — a supervisor on one model with workers on another is the common shape. Neens does not pick a winner between them (see why), so the trace gets its own bucket and the score row keeps the list of distinct models it saw.

What to do:

• If it’s expected (a genuine multi-model pipeline), compare at a level where the answer is single-model: score spans, or split the pipeline into separate agents and use the agent × model cross-tab below.
• If it’s not expected, it usually means a fallback or retry silently switched models. The Agent Map keys its LLM nodes by model, so a trace that used two shows two nodes — that’s the fastest way to see which step defected.

Unknown

Unknown means no model was derivable. There are two causes and they have different fixes:

Until then, exclude them: a pass rate computed over rows whose model you don’t know is not a statement about any model.

Filter to the attributions you trust

Model attribution exists so you can do exactly that. Add the Scores by model attribution preset first — it answers “how much of this comparison rests on evidence I trust?” — then filter.

The widget builder’s Custom tab groups by one dimension and does not author filters, so a filtered tile is created through the API — the same call the gallery makes, with a filters object:

curl -X POST https://your-neens-host/api/dashboards/dsh_your_dashboard/widgets \
  -H "Authorization: Bearer nk_sess_your_session_token" \
  -H "Content-Type: application/json" \
  -d '{
        "type": "bar",
        "measure": "eval_pass_rate",
        "dimensions": ["model"],
        "filters": {
          "model_source": ["preprod_run", "span", "session_uniform"],
          "metric_key": "primary_score",
          "score_source": "llm_judge"
        },
        "range": "30d"
      }'

That widget reads: pass rate on the Primary Score metric, per model, over the last 30 days, counting only scores whose model attribution is a single known model. Filter values are always bound as parameters, and an invalid measure × dimension combination is rejected with 422 — a widget can never query outside the catalogue.

Fetch its rows with GET /dashboards/{id}/widgets/{wid}/data:

{
  "rows": [
    { "model": "claude-haiku-4-5", "eval_pass_rate": 0.94 },
    { "model": "claude-sonnet-4-5", "eval_pass_rate": 0.96 }
  ]
}

Worked example: agent × model

The per-agent half of the comparison is a cross-tab — agents down the side, models across the top — so you can see that a cheaper model is fine for one agent and not for another. Any widget with two dimensions renders this way, whichever visualization it asks for: a bar chart keyed on the agent alone would show the same agent several times with conflicting values.

Add the Pass rate by agent × model preset from the Quality tab, or build it from the API:

curl -X POST https://your-neens-host/api/dashboards/dsh_your_dashboard/widgets \
  -H "Authorization: Bearer nk_sess_your_session_token" \
  -H "Content-Type: application/json" \
  -d '{
        "type": "table",
        "measure": "eval_pass_rate",
        "dimensions": ["agent", "model"],
        "filters": { "metric_key": "primary_score" },
        "range": "30d"
      }'

The rows come back one per pair:

{
  "rows": [
    { "agent": "support-triage",  "model": "claude-haiku-4-5",  "eval_pass_rate": 0.95 },
    { "agent": "support-triage",  "model": "claude-sonnet-4-5", "eval_pass_rate": 0.96 },
    { "agent": "refund-resolver", "model": "claude-haiku-4-5",  "eval_pass_rate": 0.71 },
    { "agent": "refund-resolver", "model": "claude-sonnet-4-5", "eval_pass_rate": 0.93 },
    { "agent": "refund-resolver", "model": "Mixed",             "eval_pass_rate": 0.80 }
  ]
}

…and render as one cell per pair:

Read that as a decision, not a score: support-triage loses a point of pass rate on the cheaper model and can move; refund-resolver loses twenty-two and cannot. That is the rollout plan the aggregate “94% vs 96%” would have hidden — and it is the reason agent and model are recorded on the same row rather than left on two different tables.

An empty cell renders as an em dash: that pair has no scores in the window — it is not a zero, and it is not a failure. An agent value of Unknown means the score’s target has no resolvable trace (so no agent name), the same explicit bucket the model side uses.

Pre-prod runs record the model they ran on

A pre-prod evaluation is the cleanest model comparison available, because you control both sides: replay the same golden dataset against the same code on two connections and the only difference is the model.

When a run is created, Neens copies the model and provider off the run’s agent connection and freezes them on the run. The run header carries a Model badge showing it — or an explicit Model not recorded when there is nothing honest to show. The run detail (GET /preprod-evals/{id}) carries the same values:

{
  "id": "ppr_…",
  "versionLabel": "pr-482",
  "runnerMode": "push",
  "model": "claude-haiku-4-5",
  "modelProvider": "agent_http",
  "status": "completed"
}

Two honest details:

• model is null for a run with no declared model — most commonly a run you drive yourself, where your own harness serves the replay and Neens genuinely does not know what served it. Those scores fall back to the captured trace’s own spans, and are Unknown if the trace recorded no model either.
• modelProvider is the connection’s provider, i.e. the transport Neens used. When Neens calls your agent at an HTTP endpoint that is agent_http, not an LLM vendor — the vendor is not knowable from an HTTP endpoint, and recording what was actually used beats recording a guess.

Why Neens refuses to pick a dominant model

A trace that used two models has no single correct model, and the tempting fix — attribute it to whichever model produced the most tokens — is the bug this feature exists to prevent. Under token-weighted attribution, a supervisor on an expensive model coordinating workers on a cheap one becomes “a cheap-model trace”, and the cheap model inherits a pass rate it did not earn. That number then survives every review, because nothing about it looks wrong.

So the rules are fixed:

• Two or more models is its own bucket. Mixed is a real, chartable value, not a rounding step.
• Unknown is visible, never blank. A score with no derivable model says Unknown and is excluded from a model comparison rather than quietly merged into one — which would inflate whichever model it landed next to.
• Only model-bearing spans vote. A tool span is not evidence of a model.
• Nothing is re-derived at read time. The attribution and the evidence behind it are recorded together, so two queries over the same window can never disagree about which model produced what.

This is the same posture Neens takes with unpriced models: a visibly missing number beats a confidently wrong one, because you can act on the first and you will plan against the second.

Troubleshooting

Related

• Cost & model pricing — what each model costs, and why an unpriced model is reported rather than guessed at. The other half of “cheaper and still passing”.
• Metrics catalogue — every measure and dimension, including the score-grain slices used here.
• Dashboards — building and sharing the boards these widgets live on.
• Scores — where scores come from and what a threshold means.
• Pre-prod evaluations — the controlled comparison: same dataset, same code, one model changed.
• Sweep decisions — the same question asked of several candidate models at once, answered with a verdict per agent instead of a chart to interpret.
• Agent Map — LLM nodes keyed by model, for seeing a mixed trace step by step.