GuidesPrompt optimization

Prompt optimization

The Neens prompt optimizer searches for a better system prompt for one confirmed failure mode. It replays that failure’s real historical traces against candidate prompts, grades every replay with your judges, reflects on the graded results to write a better candidate, and keeps searching — then proves the winner on a held-out set of traces it never optimized against. The winner leaves the optimizer as an ordinary remediation, so it earns a pull request the same way every other fix does.

This is not “the agent rewrites its own prompt in production”. Every rollout is a replay of a trace that already happened, and the result is a proposal a human reviews and merges.

The optimizer sits on top of things you already have: a confirmed failure mode with a remediation rail, an eval gate derived from that failure, the pre-prod eval runner that verifies candidates, and the eval-verified PR gate the winner exits through. If you can already open an eval-verified PR for a failure mode, you can optimize its prompt.

At a glance

WhereThe Optimizer tab on the Prompts page
Key APIPOST /prompt-optimization/runs · GET /prompt-optimization/runs/{id} · GET /prompt-optimization/preview · POST /prompt-optimization/runs/{id}/emit-remediation
MCP toolsstart_prompt_optimization (launch) · get_prompt_optimization (poll)
What it needsA confirmed failure mode (or cluster) with at least 3 replayable failing traces, judges that grade them, and your agent’s LLM connection
What it producesA prompt_change remediation carrying the winning prompt, the before/after diff, and the held-out proof — plus the full candidate lineage you can inspect
Merges?Never. The winner is a proposal; it goes through the same pass^k verification and human merge as every other fix

The optimizer lives on the Optimizer tab of the Prompts page, alongside prompt versions and deploy tags — it used to be its own page under Fix. An old /prompt-optimizer link still works: it redirects to the new tab.

How it works

The loop is deliberately small and bounded. Each step is either a replay of something that already happened or a single LLM call, and both are capped.

Seed from a confirmed failure

A run always starts from a failure mode (preferred) or a cluster — never from “all traffic”. Neens takes that failure’s failing traces as the task set and reads the baseline prompt: the system prompt those traces actually ran under. Grounding is mandatory, because a prompt tuned against undifferentiated traffic produces a weakly targeted repair.

Split train vs. held-out

The task set is split deterministically into a train split the optimizer may optimize against and a held-out split it may not touch until the end. Reporting a train-split gain is exactly how a prompt optimizer fools itself, so the split is not optional.

Replay and grade

Each rollout re-runs one historical request against a candidate prompt, with the agent’s tool results served from that trace’s recorded tool spans, and grades the result with your judges. A rollout that can’t be replayed (the fixed prompt calls a tool the trace never recorded, the provider errors) scores 0 and is recorded as a failure — never dropped, so a candidate can’t win by breaking replay on the tasks it is worst at.

Reflect

Neens shows the current prompt, the worst rollouts, and — critically — the judges’ reasoning for each one to your LLM connection, and asks for a complete replacement prompt. Reflecting on why a rollout was graded down is what makes this affordable: a bare score would need orders of magnitude more rollouts to learn the same thing.

Accept or discard, cheaply

A child prompt is re-run on the same small batch its parent was judged on. It earns a full evaluation only if it beats its parent there; a tie is discarded. This is where the rollout budget is saved.

Keep the Pareto frontier

Surviving candidates are kept as a Pareto frontier: every candidate that is best on at least one task stays, not just the one with the best average. The next parent is sampled from that frontier, weighted by how many tasks it wins. A mean-score hill-climb collapses to one lineage and throws away the candidate that solved the one hard task nobody else did.

Prove on held-out, then emit

When the budget runs out, every frontier candidate and the baseline are rolled out on the held-out split. The winner is the best held-out mean — and it becomes a remediation only if it beats the baseline there by a real margin (default 0.05) over at least 2 held-out tasks.

A run that never beats the baseline finishes normally and emits nothing. That is a result, not an error: the honest answer is “prompt changes of this shape didn’t fix this failure.”

Preview a run before you spend anything

Every rollout is a billable LLM call on your connection, so check what a run would cost first. GET /prompt-optimization/preview resolves the task set, the baseline prompt, the graders and the estimated rollout count without calling a model. The launch form on Prompts → Optimizer shows the same estimate live as you change the budget.

curl -G https://your-neens/api/prompt-optimization/preview \
  -H "Authorization: Bearer nk_live_…" \
  --data-urlencode "failureModeId=fm-2a91c4d80f13"
{
  "failureModeId": "fm-2a91c4d80f13",
  "clusterId": null,
  "modeName": "Unsupported order-status claims",
  "rootCause": "The prompt never tells the agent to ground order claims in tool output.",
  "seedSource": "failure_mode",
  "tasks": ["ses-1a2b", "ses-3c4d", "…"],
  "trainTasks": ["ses-1a2b", "…"],
  "valTasks": ["ses-9x8y", "…"],
  "warnings": [],
  "baselinePrompt": "You are Acme Support. Answer the customer's question…",
  "hasBaselinePrompt": true,
  "judgeIds": ["jdg-7f10c2"],
  "judgeNames": ["Faithfulness"],
  "judgeSource": "deployments",
  "estimatedRollouts": 216,
  "maxRollouts": 240,
  "runnable": true,
  "provable": true
}

runnable is the one field to branch on: it is false when the seed resolves to fewer than 3 in-scope sessions (no honest held-out split is possible) or when none of them carries a recoverable system prompt — either way POST /prompt-optimization/runs will refuse with a 422. provable is a softer warning: the run can optimize but has too few held-out tasks to prove a gain, so it will complete and emit nothing.

The preview also accepts maxTasks, maxIterations, minibatchSize and valFraction, so you can see how a smaller budget changes the split and the estimate before you launch.

Launch a run

Pick the failure to optimize against

Open Prompts → Optimizer, click New optimization in the top right to open the launch form, and choose the failure mode you want a better prompt for. Prefer a failure mode over a bare cluster: a failure mode carries the eval gate that becomes the emitted remediation’s proof gate, and only a remediation with a failure mode can start an eval-verified fix run.

Choose the graders and the budget

The graders default to the judges already deployed for that agent; you can narrow them. The budget caps how many rollouts and reflection iterations the run may spend — the defaults (240 rollouts, 12 iterations) are the ceiling, and a per-run value is clamped to that cap, never above it.

Launch and let it run

Click New optimization on Prompts → Optimizer, fill in the launch form, and start the run. It appears in the run list as queued, then running; the page polls until it reaches a terminal state, so you can leave it open and watch candidates appear generation by generation.

Poll until it finishes

curl https://your-neens/api/prompt-optimization/runs/opt-9c41ab77e2d0 \
  -H "Authorization: Bearer nk_live_…"
{
  "id": "opt-9c41ab77e2d0",
  "status": "completed",
  "baselineScore": 0.412,
  "bestScore": 0.703,
  "improvement": 0.291,
  "rolloutsUsed": 154,
  "iterationsUsed": 9,
  "winnerCandidateId": "optc-31f0aa92b7c4",
  "remediationId": "rem-88d1e4c07a29",
  "outcome": {
    "emit": true,
    "winnerId": "optc-31f0aa92b7c4",
    "baselineVal": 0.412,
    "winnerVal": 0.703,
    "improvement": 0.291,
    "improvedTasks": ["sess-4417…", "sess-c8ab…", "sess-1f70…"],
    "regressedTasks": ["sess-b294…"],
    "reason": "candidate beat baseline on held-out rollouts: 0.703 vs 0.412 (Δ +0.291) over 8 task(s); improved 5, regressed 1.",
    "warnings": [
      "the winning candidate REGRESSES 1 held-out task(s) the baseline handled; the reviewer sees this on the remediation."
    ]
  },
  "candidates": [
    {
      "id": "optc-31f0aa92b7c4",
      "generation": 3,
      "parentId": "optc-0b7d5518e2fa",
      "trainMean": 0.688,
      "valMean": 0.703,
      "accepted": true,
      "onFrontier": true,
      "isWinner": true,
      "rolloutsUsed": 24,
      "rationale": "Requires the agent to quote the tool's returned status verbatim and to say it doesn't know when no status was returned."
    }
  ],
  "frontier": ["optc-31f0aa92b7c4", "optc-0b7d5518e2fa"]
}

Statuses are queued, running, completed, failed and cancelled. POST /prompt-optimization/runs/{id}/cancel stops a queued or running run; the optimizer checks for cancellation between iterations, so a cancelled run stops spending budget and never writes a result. GET /prompt-optimization/runs/{id}/candidates/{candidateId} returns one candidate with its individual rollouts.

Read the results

The run detail view leads with the numbers that decide whether the candidate is real.

Held-out improvement is the only number that matters

The KPI row shows the baseline’s held-out score, the winner’s held-out score, and the Δ between them. That Δ — measured on traces the optimizer was never allowed to optimize against — is the claim. A candidate’s train-split score will almost always look better than the baseline’s, because that is what it was fitted to; a train gain that doesn’t survive the held-out split is an optimizer fooling itself, and Neens will not emit it.

The Pareto frontier and the lineage

Candidates are shown by generation with their parent, their train mean, their held-out mean, and badges for accepted, frontier and winner. Being on the frontier means the candidate is the best of all candidates on at least one individual task, even if its average is unremarkable — that’s why it was kept and why it could be sampled as the next parent. Dominated candidates (best at nothing) are still listed, so you can see what the search tried and rejected.

”Regresses N held-out tasks”

A candidate can raise the average while breaking a task the baseline handled. Neens reports that count explicitly and carries it onto the remediation instead of hiding it in the mean. Treat it as a review item: open the regressed tasks and decide whether the trade is acceptable for your product before you accept the remediation. A run with a large regression count is often better re-run with a narrower objective.

The prompt diff

The winner is shown as a before/after diff against the baseline prompt — the same typed prompt_change artifact the remediation carries, so what you review here is exactly what a fix run would apply.

What happens to the winner

When a run clears the bar, Neens writes a prompt_change remediation in the proposed state against the same failure mode, carrying the winning prompt, the diff, the held-out proof, and a View remediation link from the run. From there it is an ordinary remediation: you review it, accept it, and it can start an eval-verified fix run that verifies the prompt with pass^k pre-prod runs against the failure’s proof gate plus your accumulated regression set, and opens a pull request only if every run is green.

You can also emit the winner by hand — the Emit remediation button on a finished run, or POST /prompt-optimization/runs/{id}/emit-remediation. Emitting twice returns 409; emitting a run that never cleared the held-out bar returns 422.

⚠️

The judges that optimized the prompt cannot verify it. Neens records which judges a candidate was optimized against and excludes them from verifying the resulting fix — a candidate graded by its own optimization target is overfit, not verified. The consequence is real and intended: if the failure mode’s only human-aligned judge is the one the optimizer used, no eligible verifier remains, and the fix engine lands a draft PR instead of opening a clean one. A real PR for an optimizer-authored fix needs a second, human-κ-anchored judge. Deploy one and keep labelling gold items so its alignment stays measured — see Annotations & review.

Offline by design

The framing matters, because it is what makes this safe to run against a production agent’s prompt:

  • It is offline. Every rollout is a record-and-replay re-run of a historical trace. The agent’s tool results are served from that trace’s recorded tool spans — no tool is actually called, no customer request is affected, and no live traffic is scored or steered.
  • It never deploys. The winner does not go live. It becomes a proposal in your remediation queue.
  • It has one trust mechanic. An optimizer-authored fix goes through exactly the same gate as a hand-written one: pass^k pre-prod verification against the failure-derived proof dataset and the accumulated regression set, verified by judges it did not optimize against, and merged by a human. There is no separate, faster path for machine-written prompts.
  • It is grounded upstream. A run is seeded from a diagnosed failure mode and its root cause, not from aggregate traffic, so the change it proposes targets a failure you already confirmed.

LLM connection and cost

The optimizer runs entirely on your agent’s configured LLM connection (Settings → LLM providers) — Neens has no model access of its own. Both kinds of call it makes are billed to that connection: each rollout replays and grades one trace, and each reflection is one call that writes the next candidate prompt.

An agent with no LLM connection cannot run the optimizer: launching returns 400 and the launch form tells you to add one. Add a connection in Settings → LLM providers — see Getting started.

Because a run is expensive by construction, it is bounded on every axis:

  • a hard rollout cap (default 240 per run) that a per-run value can lower but never raise;
  • an iteration cap (default 12 reflections);
  • a per-call timeout on each reflection (90s) and each rollout (60s), so one hung call fails that item instead of the run;
  • a per-caller launch throttle (default 10 runs per hour).

Use the preview endpoint to see the estimated rollout count before you commit, and lower maxRollouts or maxTasks for a cheaper first pass.

Troubleshooting

  • The run completed but emitted nothing. The best candidate didn’t beat the baseline on the held-out split by the required margin (default 0.05). Read the run’s verdict — it states the two held-out scores and the gap. This is the expected outcome when the failure isn’t actually caused by the prompt; look at the failure mode’s root cause and consider a tool or retrieval fix instead (Remediations).
  • “Too few sessions to optimize.” The seed resolved to fewer than 3 replayable failing traces, so there is no honest held-out split and nothing could be proved. Widen the seed — a larger failure mode or cluster, a longer time range — or pass an explicit sessionIds list (Clustering).
  • The run is rejected with “grounding required.” No failureModeId, clusterId or sessionIds was given. The optimizer will not optimize against undifferentiated traffic; start from a confirmed failure (Issues & failure modes).
  • No LLM connection. Launching returns 400 because the agent has no connection to run rollouts and reflections on. Add one in Settings → LLM providers — see Getting started.
  • The fix engine drafted a PR instead of opening one. Every eligible verifying judge was excluded because the optimizer optimized against it. Deploy a second, human-aligned judge for that failure mode and label enough gold items for its alignment to be measured (Annotations & review, Eval-verified PR).
  • The run is stuck at queued. Nothing is draining the scoring queue. Optimization runs on the same worker fleet as judges and eval runs, so if judge runs are also stuck, the worker is the problem — see Judges and your deployment’s worker configuration.
  • The emitted remediation can’t start a fix run. It was seeded from a cluster rather than a failure mode, or its failure mode has no proof gate. Generate one for the failure mode (Eval gates) and re-open the remediation.
Autonomy levelsFix outcomes