Pre-prod evaluations

A pre-prod evaluation runs a candidate version of your agent against a frozen golden dataset before it ships, scores the results with the same judges you run in production, and compares them to a baseline — a previous candidate or your live production quality. Items that passed the baseline but fail the candidate are regressions, and the run’s gate turns them into a pass/fail verdict your CI can block on.

Where eval gates guard a single known failure, a pre-prod eval sweeps a whole candidate version across your curated set — so a quality drop is caught before users see it, not after.

At a glance

Two ways to run

Both ways produce the same run, comparison, and gate — they differ only in who invokes your agent:

• Neens calls my agent. You register your agent’s HTTP endpoint once; Neens calls it with every golden prompt, records each response as a session, and scores it. No harness, no exporter wiring — the lowest-effort path.
• I run my agent. You drive your agent over the golden prompts yourself — by hand or wired into your CI pipeline — and the traces it emits carry correlation attributes that link them back to the run. Use this when your agent already runs in a harness, or isn’t reachable from Neens. The neens eval run CLI does the busywork.

Authentication — one key, set once

Every pre-prod route accepts the nk_live_ agent API key you already mint to send traces (Settings → API keys), sent as Authorization: Bearer <key>. It resolves to a company + agent on its own, so:

• No nk_sess_ session/login token.
• No --project-id — the key already scopes the run to an agent (pass one only to override).

For both the CLI and CI, set two environment variables and you’re done:

export NEENS_BASE_URL="https://neens.example.com"   # your Neens origin
export NEENS_API_KEY="nk_live_..."                   # your agent (ingest) API key

Prerequisites

• A golden dataset. A dataset version marked golden is the reference set — each item carries the frozen prompt (and optional expected answer) the candidate is asked to run. Build one from human-labeled failures or a filter — see Datasets and Annotations & review.
• Judges. A run defaults to your agent’s Primary Score — the overall quality gate — and you can pick any other enabled judge(s) to score the replay. LLM-prompt, managed, and composite judges (like Primary Score) all participate. A classifier judge only assigns a category label rather than a pass/fail score, so it isn’t offered here.
• If you run your agent: an agent that exports OpenTelemetry traces to Neens (Send traces). If Neens calls your agent: an HTTP endpoint Neens can reach.

Create a run

In Pre-prod Evals → New pre-prod eval, everything except the version label has a smart default:

| Judges | Which judges score the replay. Defaults to Primary Score; pick any enabled scoreable judge(s). |

The API accepts an explicit judge_deployment_ids list; omit it to default to Primary Score. Only scoreable judge types (LLM-prompt, managed, composite) are accepted — a classifier-only selection is rejected.

Creating the run snapshots the chosen version’s items immediately, so the exact prompt set is frozen for the run even if the dataset is edited later. A dataset with no golden version is rejected unless you pin a version explicitly.

Via the API:

curl -sf -X POST "$NEENS_BASE_URL/preprod-evals" \
  -H "Authorization: Bearer $NEENS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "release-candidate",
    "dataset_id": "ds_golden",
    "version_label": "'"$GIT_SHA"'",
    "baseline": {"kind": "prod_window", "range": "7d"},
    "gate": {"max_regressions": 0, "min_pass_rate": 0.9}
  }'

Optional fields: dataset_version_id (pin a snapshot), runner_mode ("runner" = you run the agent, the default, or "push" = Neens calls your agent, with agent_connection_id), judge_deployment_ids, and baseline: {"kind": "preprod_run", "value": "<prior run id>"}.

The gate object can also carry a cost budget — see the server-side cost gate.

The candidate’s model is frozen at create time

When Neens calls your agent, the run’s Agent endpoint connection declares a model. Neens copies that model (and the connection’s provider) onto the run when the run is created, and never re-reads it — so “Haiku passed at 94% on the June release” keeps meaning the same thing after somebody edits the connection in July. Every score the run produces is attributed to that frozen model, which is what makes a pre-prod run the cleanest model comparison available: same golden prompts, same code, one model changed. The run header shows it as a Model badge — or an explicit Model not recorded rather than a blank that would read as a model name.

The run detail carries the same values:

{
  "id": "ppr_…",
  "versionLabel": "pr-482",
  "runnerMode": "push",
  "model": "claude-haiku-4-5",
  "modelProvider": "agent_http"
}

model is null when there is nothing honest to record — most often a run you drive yourself, where Neens does not know what served the replay, or a connection that declares no model. Those scores fall back to the model recorded on the captured trace’s own spans, and are reported as Unknown if the trace recorded none. Nothing is guessed at. modelProvider is the connection’s provider — for an agent endpoint that is agent_http, the transport Neens dialled, not an LLM vendor (which an HTTP endpoint does not reveal).

I run my agent: neens eval run

The neens-eval SDK (dependency-free) drives the whole flow for you: it fetches the frozen prompts, invokes your agent command once per item with the correlation attributes injected (via OTEL_RESOURCE_ATTRIBUTES), starts the run, polls it to a verdict, and exits with the gate result. Run it by hand on your laptop or wire it into your CI pipeline — the command is the same.

It ships in two flavors with identical flags and exit-code contract — pick the one that matches your stack:

pip install neens-eval    # installs the `neens` console script
# or, from a Neens source checkout: pip install ./sdk

npm install --save-dev @neens/eval   # installs the `neens-eval` bin

Ad-hoc: create and run in one command

With NEENS_BASE_URL and NEENS_API_KEY exported (see Authentication), --create makes a run against your dataset’s golden version and drives it in a single command:

neens eval run --create \
  --dataset ds_golden \
  --version-label local-test \
  -- python -m myagent

npx neens-eval run --create \
  --dataset ds_golden \
  --version-label local-test \
  -- node dist/myagent.js

Everything after -- is your agent command; it’s invoked once per frozen prompt with the prompt piped to stdin (and available as $NEENS_ITEM_INPUT). No --project-id, no session token — the nk_live_ key scopes and authenticates the whole run.

To drive a run you already created, swap --create --dataset … for --run-id <id>.

In CI: a GitHub Actions gate

Store your Neens origin and agent key as repo secrets, then add one step. The exit code is the gate verdict, so the step fails the build automatically when the candidate regresses. Using ${{ github.sha }} as the version label ties the run to the commit under test:

- name: Pre-prod eval gate
  env:
    NEENS_BASE_URL: ${{ secrets.NEENS_BASE_URL }}
    NEENS_API_KEY:  ${{ secrets.NEENS_API_KEY }}   # a nk_live_… agent key
    # Point YOUR agent's OTel exporter at the Neens receiver:
    OTEL_EXPORTER_OTLP_TRACES_ENDPOINT: ${{ secrets.NEENS_BASE_URL }}/v1/traces
  run: |
    neens eval run --create \
      --dataset ds_golden \
      --version-label "${{ github.sha }}" \
      --baseline prod:7d \
      --min-pass-rate 0.9 --max-regressions 0 \
      -- python -m myagent

Because --create self-creates the run each time, the only sticky identifiers in CI are the dataset id and the NEENS_API_KEY secret — there’s no run id to copy between builds.

The exit-code gate contract

The process exit code is the gate verdict, so a plain CI step blocks a bad release:

--no-gate still fetches and prints the gate but always exits 0 (report without blocking CI). --json emits a machine-readable result to stdout (human logs go to stderr) for a step to parse.

Gate-as-code: a richer policy file

--min-pass-rate and --max-regressions are the thresholds the server gate owns from the command line — and the run’s stored gate can carry a cost budget alongside them. On top of those you can commit a gate policy — a JSON (or YAML, if PyYAML is installed) file that declares budgets and floors the CLI evaluates client-side over the comparison, per-metric rollup, and run-detail data Neens already returns. Point the CLI at it with --gate-policy PATH (or set NEENS_GATE_POLICY); it works with both neens eval run and neens eval push.

{
  "gate": { "max_regressions": 0, "min_pass_rate": 0.9 },
  "rules": [
    { "cost":    { "max_avg_usd": 0.05, "max_delta_pct": 20 }, "severity": "warn" },
    { "latency": { "max_avg_ms": 3000, "max_delta_pct": 25 }, "severity": "warn" },
    { "steps":   { "max_avg": 8, "max_delta_pct": 50 }, "severity": "block" },
    { "max_abs_failures": 3, "severity": "block" },
    { "min_avg_score": 0.8, "severity": "block" },
    { "metric": "faithfulness", "min_avg": 0.8, "severity": "block" },
    { "metric": "toxicity_safety", "min_avg": 0.9, "severity": "warn" }
  ],
  "on_missing_baseline": "warn"
}

The gate block is handed to the server at run-create time, so Neens stays authoritative for max_regressions/min_pass_rate. The rules list is what the CLI evaluates. Each rule carries exactly one dimension key plus an optional severity:

Two severity tiers. severity: "block" (the default) fails the build — a breached block rule sets the CLI exit code to 1 on top of the server gate. severity: "warn" is reported in the summary but never changes the exit code.

Missing baseline. on_missing_baseline governs baseline-relative rules (any *_delta / *_delta_pct budget, and max_regressions) when the run has no baseline cohort to compare against — warn (the default) reports without blocking, block honors each rule’s own severity, and pass skips them entirely. Absolute rules (max_avg_*, max_abs_failures, min_pass_rate, min_avg_score, and metric floors) always evaluate — they need no baseline.

For a quick one-off gate you don’t need a file: four convenience flags inject the equivalent block rules straight from the command line, overriding any same-dimension rule in the file — --max-cost-delta-pct, --max-latency-delta-pct, --min-avg-score, and --max-abs-failures.

The metric rule reads GET /preprod-evals/{id}/metrics — a per-metric rollup of the run’s captured, scored sessions. It returns one row per judge metric with its normalized average score (avgScore), pass rate (passRate), and the number of sessions scored, letting the gate enforce a floor the run’s single aggregate score can’t (e.g. faithfulness must stay ≥ 0.8). A run with nothing scored yet returns an empty list rather than erroring.

The server-side cost gate

A run’s stored gate — the gate object you pass to POST /preprod-evals, and the one a model sweep pins onto every child run — accepts a cost family alongside max_regressions and min_pass_rate, evaluated by Neens rather than by the CLI:

{
  "max_regressions": 0,
  "min_pass_rate": 0.9,
  "cost": { "max_avg_usd": 0.004, "max_delta_pct": -50 }
}

The three sub-keys are spelled exactly as the SDK’s client-side cost rule spells them, so one vocabulary covers both sides:

GET /preprod-evals/{run_id}/gate returns a costRules array whenever the gate declares the family — one row per declared sub-key, each with rule, status, observed, threshold and a plain-English reason. The key is absent (not an empty array) when there is no cost family, so every gate written before this existed returns exactly what it always did.

{
  "passed": false,
  "reasons": ["cost avg 0.0062 USD exceeds budget 0.004 USD"],
  "regressions": 0,
  "passRate": 0.95,
  "gate": { "max_regressions": 0, "min_pass_rate": 0.9, "cost": { "max_avg_usd": 0.004 } },
  "costRules": [
    { "rule": "max_avg_usd", "status": "fail", "observed": 0.0062, "threshold": 0.004,
      "reason": "cost avg 0.0062 USD exceeds budget 0.004 USD" }
  ]
}

{
  "gate": {
    "min_pass_rate": 0.9,
    "cost": { "max_delta_pct": -50 }
  }
}

That policy is literally “cheaper and still hits the number” — a candidate must pass at least 90% of items and cost at least 50% less than the baseline — enforced in CI and visible in the product, from one file.

The correlation contract

Neens links each trace your agent emits to a pre-prod item by three attributes, readable from either span attributes or resource attributes (both OpenTelemetry and OpenInference conventions work — resource attributes are flattened onto spans at ingest):

These three keys are a stable wire contract, held fixed across releases by a CI guard, and they are matched at exactly this spelling — an attribute under any other name is stored as ordinary trace metadata and the run stays in awaiting_traces with no error.

The neens eval run CLI injects all three for you via OTEL_RESOURCE_ATTRIBUTES — you don’t touch your agent code. If you drive the run without the CLI, tag them yourself:

export OTEL_RESOURCE_ATTRIBUTES="neens.eval_run_id=ppr_1234,neens.dataset_item_id=<item_id>,neens.version_label=release-2.1"
python my_agent.py   # exporter pointed at the Neens /v1/traces as usual

with tracer.start_as_current_span("agent.run") as span:
    span.set_attribute("neens.eval_run_id", "ppr_1234")
    span.set_attribute("neens.dataset_item_id", item_id)
    span.set_attribute("neens.version_label", "release-2.1")
    ...

{
  "session": { "id": "trace-eval-001", "agent_name": "my-agent" },
  "spans": [
    {
      "id": "span-1", "name": "agent.run", "kind": "agent",
      "input": "<golden prompt>", "output": "<answer>",
      "attributes": {
        "neens.eval_run_id": "ppr_1234",
        "neens.dataset_item_id": "<item_id>",
        "neens.version_label": "release-2.1"
      }
    }
  ]
}

GET /preprod-evals/{run_id}/items returns the frozen prompt list to drive your agent with:

{ "items": [ { "item_id": "…", "input": "<golden prompt>", "expected_output": "…" } ] }

How linking behaves, exactly:

• With neens.dataset_item_id: the trace links only to that item. If the id matches no still-pending item (a duplicate replay of an already-captured prompt, or a typo), the trace is not linked to some other prompt — mislinking would corrupt that prompt’s score.
• Without it: the trace links to the first still-captured-less item in frozen order. Fine for a strictly sequential harness; tag the item id if you run prompts concurrently or retry.
• A trace whose run id belongs to another agent, or to a cancelled run, is ignored.

The first captured trace moves the run from awaiting_traces to running; once every item has a captured trace, scoring is dispatched automatically (exactly once, even if the last traces land simultaneously).

Neens calls my agent

When Neens calls your agent, it HTTP-calls your agent’s endpoint once per golden prompt, records each response as a session, links it to its item, and scores the run — no harness on your side.

{ "model": "your-model", "messages": [{ "role": "user", "content": "<golden prompt>" }] }

{
  "input": "<golden prompt>",
  "eval_run_id": "ppr_…",
  "dataset_item_id": "…",
  "version_label": "agent-v2.3"
}

{ "output": "<answer>" }

neens eval push \
  --dataset ds_golden --version-label "$GIT_SHA" \
  --agent-connection "$AGENT_CONNECTION_ID" \
  --min-pass-rate 0.9 --max-regressions 0

Guardrails on the outbound call

When Neens calls an endpoint you own, every call is bounded and every failure is honest — Neens never fabricates an output or a score:

• Per-call timeout — 60 s. A slow or hung endpoint errors that one item; the rest of the run continues.
• Prompt truncation — the prompt sent to your endpoint is truncated at 200,000 characters, so a pathological golden item can’t blow up the request body.
• Response cap — 8 MiB; an oversized body errors the item instead of exhausting worker memory.
• Per-item failures stay per-item — a timeout, connection failure, non-2xx status, empty body, or a reply with no answer at the expected location marks that item error with the reason recorded. If every item fails (e.g. the endpoint was down), the run is marked failed — never “completed” with nothing scored.

How scoring works

When a run’s items are captured (either way of running), the dedicated pre-prod worker scores each captured session with every judge on the run:

1. Each judge scores the session and its verdict is persisted as a normal score row — but tagged source: preprod, so pre-prod results appear in the score catalogue as their own score type (a separate card, never merged into the production one) and on the captured trace. The tag is also what lets any dashboard widget keep them out: slice or filter by Score source (score_source) on the score-grain measures. A widget that doesn’t will count pre-prod scores alongside production ones.
2. The item’s score is the mean of its judge scores; the item passes when that mean is at or above 0.7.
3. An item whose judges all errored is marked error (with the reason) — never silently scored.

Then the run’s baseline is resolved and each item’s baseline_passed is stamped:

• Previous pre-prod run (preprod_run) — the prior run’s verdict for the same frozen prompt (both runs snapshot the same golden version, so the prompt id is a stable join key).
• Production window (prod_window) — production scores (any source except preprod) over the window, matched to each golden item by input text: a prod session whose input equals the item’s frozen prompt supplies the verdict (its newest score wins; pass = a pass label, or score ≥ 0.7). Items whose prompt never ran in production get no baseline verdict — they can’t regress, but they still count toward the pass rate.

A regression is an item where the baseline passed and the candidate failed. Any regressions put the run in completed_with_regressions and emit a Pre-prod regression insight — so a blocked release shows up in your feeds (and notification sinks) with the run attached, not just as a red CI check.

Reviewing a run

Open a run on Pre-prod Evals to see:

• Per-item results — each frozen prompt’s status, score, pass/fail, baseline verdict, and captured trace (click through to the full trace), plus the recorded error for failed items.
• The comparison (GET /preprod-evals/{id}/comparison) — a per-item pass/fail matrix, the regression set, new passes (items the candidate fixed), and average cost / latency / step-count deltas between the candidate’s captured sessions and the baseline cohort. Each row also carries the candidate’s agent output and the item’s expected output. Average cost covers the priced sessions only; if a side has no priced session the cost delta is null rather than a comparison against a made-up rate — see Cost & model pricing.
• The gate (GET /preprod-evals/{id}/gate) — pass/fail with reasons.

The gate never reports green before the candidate has actually been evaluated: a run that is failed or cancelled, still in progress, or finished with zero scored items fails the gate outright. Only then are the thresholds checked — max regressions and min pass rate.

Comparing multiple versions

Compare Versions lines up N runs that snapshot the same golden version side by side — one row per prompt, a cell per version, with per-version pass rate, cost, latency, and step count. Pick any run as the baseline to highlight which versions regressed against it. Via the API: GET /preprod-compare?runs=ppr_a,ppr_b,ppr_c&baseline=ppr_a (mixing golden versions returns 422).

Expand a prompt row to read each run’s agent output (the real final answer that version produced) alongside the item’s expected output, so a red cell is “here’s what it actually said” — not just a failed check. A run with nothing captured for that prompt shows no output captured.

For any single prompt, the trajectory diff (GET /preprod-compare/trajectory?run=…&baseline=…&item=…) aligns the two versions’ step sequences and marks each step added / removed / changed / same — turning “v3 regressed on this prompt” into “v3 stopped calling lookup_order.”

Run lifecycle

Item statuses: pending → captured → scored, or error (invocation or scoring failed — the reason is recorded on the item).

Troubleshooting