Stress tests

A stress test turns your failure-mode taxonomy from a backward-looking record into a forward-looking asset: pick the failure modes your agent already got caught on, and Neens generates fresh synthetic test scenarios grounded in those real failures — then hands them straight to the pre-prod evaluation gate so you can attack your next candidate with last quarter’s failures before it ships.

Where a pre-prod eval replays a golden dataset you curated by hand, a stress test builds that golden dataset for you from the sessions where the agent actually failed — so the set that guards a release grows automatically as your taxonomy does.

At a glance

How it fits together

A stress test is three moving parts, and the last one is just a pre-prod eval:

1. A scenario suite is generated from failure modes you pick. Each scenario is a user input plus an expected_output — a behavioral assertion of what a correct response must do or avoid (a synthetic prompt has no ground-truth answer, so the judges grade behavior, not a fixed string).
2. The suite’s scenarios are materialized as an immutable golden dataset version — the same frozen-golden contract a hand-curated dataset uses.
3. Launching the suite creates a pre-prod eval run over that golden version. From there everything is a normal pre-prod eval — Neens calls your agent, or your own harness emits the traces — scored by your judges, compared to a baseline, and gated on regressions and pass rate.

So a stress test adds no new scoring, comparison, or gate machinery — it feeds the pre-prod gate you already know.

Generate a suite

Via the API — the whole body except name and failure_mode_ids has a default:

curl -sf -X POST "$NEENS_BASE_URL/scenario-suites" \
  -H "Authorization: Bearer $NEENS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Refund-policy regressions",
    "failure_mode_ids": ["fm_abc", "fm_def"],
    "per_mode": 5,
    "strategies": ["paraphrase", "escalate", "boundary"],
    "include_replay": true
  }'

Optional fields: per_mode (LLM scenarios per mode, 1–25), strategies (subset of paraphrase/escalate/boundary), include_replay (default true), and connection_id to pin a specific LLM connection for generation.

Review the scenarios

A ready suite lists its generated scenarios. Each one carries:

• Input — the synthetic end-user prompt.
• Expected output — the behavioral assertion the judges grade against (“a correct response must not exhibit the ’…’ failure mode…”).
• Strategy — which kind of scenario it is (replay, paraphrase, escalate, or boundary).
• Source failure mode — which mode it was generated from, so every scenario is traceable back to the real failure that inspired it.

Not happy with the set — added more exemplars, or just configured an LLM connection? Regenerate the suite (POST /scenario-suites/{id}/regenerate) to rebuild it, optionally overriding per_mode, strategies, include_replay, or connection_id. Regeneration replaces the suite’s scenarios and snapshots a fresh golden version.

The strategies

Each scenario is generated with one strategy. Three are LLM-authored variations grounded in the mode’s real exemplars; the fourth is a deterministic floor that needs no model:

Launch a stress test

A ready suite launches as a pre-prod eval run in one step.

Via the API, POST /scenario-suites/{id}/launch delegates to the pre-prod eval create endpoint and returns the created run:

curl -sf -X POST "$NEENS_BASE_URL/scenario-suites/$SUITE_ID/launch" \
  -H "Authorization: Bearer $NEENS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "version_label": "'"$GIT_SHA"'",
    "baseline": {"kind": "prod_window", "range": "7d"},
    "gate": {"max_regressions": 0, "min_pass_rate": 0.9}
  }'

The launch body accepts the same knobs a pre-prod run does — version_label (required), runner_mode ("push"/"runner", defaulted for you), agent_connection_id, judge_deployment_ids, baseline, and gate. Omit judge_deployment_ids to score with your agent’s Primary Score, like any pre-prod run.

Read the gate

Because a launched stress test is a pre-prod eval run, everything in Pre-prod evaluations applies unchanged:

• Each scenario’s captured trace is scored by your judges; scores are tagged source: preprod so they stay out of production metrics and dashboards.
• A scenario passes when the mean of its judge scores is at or above 0.7.
• The run is compared to a baseline (a previous run or a production window), and any item that passed the baseline but fails the candidate is a regression.
• The gate turns the run into a pass/fail verdict on your max regressions and min pass rate thresholds — and emits a Pre-prod regression insight when something regresses.

Review it on the run’s page, or drive and read the gate from CI (below).

In CI: generate → launch → gate

Because every route accepts the nk_live_ agent API key you already use to send traces, a CI job can build a stress-test suite from your live taxonomy and gate a release on it with a single credential — no login, no session token. Set the shared pre-prod environment variables once (see Pre-prod evals → Authentication):

export NEENS_BASE_URL="https://neens.example.com"
export NEENS_API_KEY="nk_live_..."   # your agent (ingest) API key

Then generate a suite, wait for it to become ready, and launch it against the commit under test:

# 1. Generate a suite from the confirmed failure modes you want to guard against.
SUITE=$(curl -sf -X POST "$NEENS_BASE_URL/scenario-suites" \
  -H "Authorization: Bearer $NEENS_API_KEY" -H "Content-Type: application/json" \
  -d '{"name":"CI stress test","failure_mode_ids":["fm_abc","fm_def"]}' | jq -r .id)
 
# 2. Poll until the suite is ready (generation runs on the worker fleet).
until [ "$(curl -sf -H "Authorization: Bearer $NEENS_API_KEY" \
  "$NEENS_BASE_URL/scenario-suites/$SUITE" | jq -r .status)" = "ready" ]; do sleep 5; done
 
# 3. Launch it as a pre-prod run against the commit under test, gating on regressions.
curl -sf -X POST "$NEENS_BASE_URL/scenario-suites/$SUITE/launch" \
  -H "Authorization: Bearer $NEENS_API_KEY" -H "Content-Type: application/json" \
  -d '{"version_label":"'"$GITHUB_SHA"'","gate":{"max_regressions":0,"min_pass_rate":0.9}}'

The launch returns a pre-prod run; from there the same exit-code gate contract and neens eval CLI described in Pre-prod evaluations turn the verdict into a passing or failing build step.

Run lifecycle

Troubleshooting