# Neens documentation — full text > The complete Neens product documentation, concatenated for LLM context. ================================================================================ # Neens Source: /docs/ ================================================================================ ⬡ Observe · Diagnose · Evaluate · Fix Make every agent failure its last. Neens turns your agent's traces into answers: see every run, find why it failed, score it with automated judges, and close the loop with fixes that can't regress. Get started → Send your first trace Observe → Diagnose → Evaluate → Fix 👁️ Observe Inspect every trace and session — spans, tool calls, inputs, outputs, latency, and cost. 🩺 Diagnose Recurring failures are clustered into failure modes and tracked as Issues you can manage. ⚖️ Evaluate Score traces with automated judges, curate golden datasets, and align judges to human labels. 🔧 Fix Turn failures into typed remediations, simulate a fix before shipping, and gate releases on your own evals. Start here 🚀 Getting started Sign in, create an agent, and send your first trace in a few minutes. 📖 Core concepts Traces, sessions, judges, scores, failure modes — the vocabulary everything builds on. 🔌 Send traces The ingestion API, with copy-paste examples for OpenTelemetry, Python, and curl. Go deeper 🧪 Pre-prod evaluations Replay a golden dataset against a candidate agent version and catch regressions before they ship. 📊 Dashboards & insights Slice quality, cost, and volume with the metrics catalogue, and let detectors surface anomalies. 🏢 Administration Orgs, agents, members and roles, API keys, LLM connections, and the audit log. ================================================================================ # Getting started Source: /docs/getting-started/ ================================================================================ # Getting started This guide takes you from a fresh account to your first scored trace: activate and sign in, set up your workspace, connect an LLM, create an ingest API key, and send a trace. It takes a few minutes once your agent is emitting traces. ## At a glance | Step | Where | Needs | | --- | --- | --- | | Activate & sign in | Emailed link → `/activate`, then `/login` | Your invite email | | Workspace checklist | **Getting started** page | Company admin role | | Agent setup | **Set up trace tracking** step / **Settings → Agent** | `configure judges` permission | | LLM connection | **Settings → LLM providers** | A provider API key (Anthropic, OpenAI, …) | | Ingest API key | **Settings → API keys** | — | | First trace | Your agent / curl | The `nk_live_…` key | ### Activate your account and sign in Neens accounts are created by invitation. The first admin of a new company receives an activation email when the workspace is provisioned; everyone else is invited by a workspace admin from **Settings → Members**. The email contains a one-time activation link (`/activate?token=…`). Open it, set a password (at least 12 characters by default, and not one that appears in public breach lists — the form states the exact rules) and optionally your display name, and you're signed in immediately — no separate login needed the first time. Invitation links expire after 14 days; if yours has expired, ask an admin to resend it. After that, sign in at `/login` with your email and password. If you've turned on [two-factor authentication](/administration/account#two-factor-authentication) — required on operator accounts, optional for everyone else — sign-in asks for your 6-digit code as a second step, and a completed password reset asks for it too. Forgot your password? **Forgot password** emails you a reset link that's valid for 60 minutes. If activation or reset emails aren't arriving, contact your admin — see the [FAQ](/faq#emails-arent-arriving). ### Work through the workspace checklist (admins) The first company admin lands on the **Getting started** page — a five-step checklist that sets up the workspace. Only admins see it; you can complete it in any order, or **Skip for now** and return later via **Getting started** in the sidebar. 1. **Name your workspace** — rename the seeded default organization and agent to names your team will recognize (changeable any time). 2. **Invite your team** — teammates get a one-time activation link by email; assign each a role (**admin**, **member**, or **viewer**). 3. **Connect an LLM** — required for scoring and every other AI feature; see the next step. 4. **Assign personas to your team** — give each member a starting view tailored to how they work. A persona changes layout emphasis only, never access. 5. **Set up trace tracking** — hands off to the agent setup below. ### Connect an LLM Every AI feature — judge scoring, cluster labels, insight summaries, topic classification, enrichments, remediations, and the in-app assistant — runs on an LLM connection **you** configure. Neens ships with no built-in model access. 1. Open **Settings → LLM providers** and click to add a connection. 2. Pick an **API format** — **Anthropic**, **OpenAI-compatible** (which covers DeepSeek, Together, Fireworks, vLLM, and most gateways, plus a local **Ollama**), **Google Gemini**, or one of the other formats — then set the model, an optional base URL for compatible/local endpoints, and your API key. 3. Optionally scope it: a connection can be visible company-wide (default), or restricted to specific orgs or agents. Mark one as the **default** for its scope. 4. Use the connection test to verify it's reachable — this makes one real (tiny) model call. Credentials are encrypted at rest and never shown again after you save them. **No connection, no AI features.** Without a visible LLM connection, traces still ingest and display fine, but nothing gets scored, clusters stay unlabeled, and the assistant can't answer. Features degrade gracefully rather than erroring — see the [FAQ](/faq#ai-features-arent-working). ### Run agent setup Everything in Neens lives inside an **Agent** (formerly called a Project) — traces, judges, datasets, and dashboards are all agent-scoped. Pick your agent from the agent switcher in the sidebar, then run the one-click agent setup (the **Set up trace tracking** checklist step, or **Settings → Agent**). It provisions, in one go: - an **ingest API key** for the agent (shown once — copy it), - three default judges plus the **Primary Score** composite, configured to continuously score a 5% sample of incoming traces, and then tracks your first trace, first score, and first failure cluster as they happen. ### Create an ingest API key [#3-get-an-ingest-api-key] Agent setup already minted a key, but you can manage keys any time — for example a separate key per environment. 1. Open **Settings → API keys**. 2. Click create, give the key a name (e.g. `production-ingest`), and copy the value. Keys are prefixed `nk_live_` and each key is bound to exactly one agent: anything sent with it lands in that agent, and it can't write anywhere else. **The key is shown once.** Neens stores only a hash. If you lose a key, revoke it and create a new one — revocation takes effect immediately. ### Send your first trace Neens ingests standard **OpenTelemetry** traces — if your agent is already instrumented, just point the OTLP exporter at Neens and send your key as a bearer token. ```bash # then run your agent as usual — spans export to Neens over OTLP/HTTP python my_agent.py ``` ```bash curl -X POST "https:///ingest/raw" \ -H "Authorization: Bearer nk_live_your_key_here" \ -H "Content-Type: application/json" \ -d '{ "session": {"id": "trace-001", "agent_name": "my-agent"}, "spans": [ {"id": "span-1", "name": "llm.call", "kind": "llm", "input": "What is the capital of France?", "output": "The capital of France is Paris."} ] }' ``` Ingestion is asynchronous: Neens accepts the trace and returns `202 Accepted` immediately, then processes it in the background. Add `?sync=true` to the URL to process inline while you're testing. For every supported format (OTLP, OpenInference, raw JSON, batch) and richer examples, see **[Send traces](/guides/send-traces)**. ### See it in Neens Open **Traces** in the sidebar — your trace appears within a few seconds. Click it to inspect spans, tool calls, inputs, outputs, latency, and cost. Related traces that share a conversation are rolled up on the **Sessions** page (the conversation-level view) — see [Traces & sessions](/guides/traces-and-sessions). If nothing shows up, work through the [FAQ checklist](/faq#my-traces-arent-showing-up). ### Watch the first scores arrive With an LLM connection configured, the **Primary Score** judge scores a sample of incoming traces automatically — open **Scores** to watch quality signal accumulate, or **Judges** to add more evaluators. See [Continuous evaluation](/guides/continuous-evaluation). ## Where to go next - **[Core concepts](/concepts)** — the vocabulary the rest of the product uses. - **[Send traces](/guides/send-traces)** — every ingestion format, with code examples. - **[Judges](/guides/judges)** and **[Scores](/guides/scores)** — automated evaluation. - **[Clustering](/guides/clustering)** and **[Issues & failure modes](/guides/issues-and-failure-modes)** — see failure patterns, not one-off errors. - **[Dashboards](/guides/dashboards)** and **[Insights](/guides/insights)** — track quality, cost, and anomalies over time. - **[FAQ](/faq)** — troubleshooting and common questions. ================================================================================ # Core concepts Source: /docs/concepts/ ================================================================================ # Core concepts A short glossary of the terms Neens uses. Each entry is a definition plus a link to the guide that covers it in depth — everything else in the docs builds on these. ## Traces & the data model **Trace** — A single end-to-end run of your agent, made of spans. One trace is one interaction: a request comes in, your agent works, a response goes out. This is the raw unit you send to Neens; the **Traces** page lists them individually. See [Traces & sessions](/guides/traces-and-sessions). **Session** — A conversation-level rollup of traces. Traces that share a conversation id (captured at ingest from standard attributes like `gen_ai.conversation.id`, `session.id`, `conversation.id`, or `thread.id`) are grouped into one session, so a multi-turn chat reads as one unit on the **Sessions** page. A trace with no conversation id is its own single-trace session. See [Traces & sessions](/guides/traces-and-sessions). **Span** — One step inside a trace: an LLM call, a tool call, a retrieval, a guardrail check, an agent step, or a custom operation. Spans nest to form the shape of the run, and each carries inputs, outputs, timing, a status (ok / error), and — for model calls — the model name and token usage that Neens turns into cost (tokens × that model's price — see [Cost & model pricing](/guides/cost-and-model-pricing)). See [Send traces](/guides/send-traces). **Tool call** — A tool or function invocation your agent made, extracted from its span. Neens captures the tool name, the arguments, the result, the duration, and any error. See [Traces & sessions](/guides/traces-and-sessions). **Agent** — The named producer of a trace, read off each incoming trace. A single **Agent** workspace (see [the workspace hierarchy](#workspace--configuration) below) can hold traces from several such agent names (or several versions of one) and slice metrics by name. See [Metrics](/guides/metrics). ## Evaluation **Judge** — An automated evaluator that scores traces against a metric such as faithfulness, answer relevancy, or coherence. Judges are usually LLM-based (they run on your configured [LLM connection](#llm-connection)) and can also be composed into composites. A judge has versions and deployments, is visible at platform, org, or agent level, and runs on demand, on a schedule, or continuously on incoming traffic. See [Judges](/guides/judges). **Score** — The output of a judge for one target (a trace or a span): a metric key, a numeric value, an optional pass/fail label against a threshold, and a rationale explaining the verdict. The **Scores** page is the catalogue of every metric being produced. See [Scores](/guides/scores). **Primary Score** — Your agent's headline quality metric: a composite judge set up during agent onboarding that continuously scores a sample of incoming traces (5% by default), so you have quality signal from day one without configuring anything. See [Continuous evaluation](/guides/continuous-evaluation). **Enrichment** — An LLM step that adds structured metadata to traces — a category, a sentiment, extracted fields — rather than a quality score. Enrichments run manually or automatically on ingest (capped per day), and their outputs become filters and dataset criteria. See [Enrichments](/guides/enrichments). **Topic** — A subject-matter grouping of what your users actually ask about. Topics are proposed automatically and curatable by hand, and each shows volume and failure rate so you can see *where* quality problems concentrate. See [Topics](/guides/topics). **Dataset** — A curated collection of examples (inputs and expected outputs) built manually, from a filter, from a cluster, or synced continuously from live traffic. A dataset can be snapshotted into immutable **versions**, and a version can be marked **golden** — the frozen reference set that eval gates and pre-prod evaluations replay. See [Datasets](/guides/datasets). **Annotation** — A human label on a trace: a pass/fail verdict with an optional critique. Annotations are the ground truth Neens aligns judges against — the review queue, judge-alignment reports, and golden datasets are all built from them. See [Annotations & review](/guides/annotations-and-review). ## Diagnosis **Cluster** — An automatically discovered group of similar failing traces. Neens selects the failure set (by default, traces whose primary score falls below 0.5 — configurable to all traces or errors only), embeds them, clusters them, and labels each cluster using your LLM connection. Clusters are `active` until resolved. See [Clustering](/guides/clustering). **Failure mode** — A named category in your failure taxonomy — the durable "kind of failure" that clusters and classified traces roll up into. Neens ships a platform taxonomy and you can add custom modes. See [Issues & failure modes](/guides/issues-and-failure-modes). **Issue** — A failure mode being actively tracked as work: it has a lifecycle (`open` → `acknowledged` → `investigating` → `mitigating` → `resolved`, plus `muted`), carries evidence sessions, and can own remediations. Issues are how you manage failures over time instead of rediscovering them. See [Issues & failure modes](/guides/issues-and-failure-modes). **Insight** — A detected signal worth your attention: an anomaly (error-rate or latency jump), an evaluation regression, or an issue volume spike. Insights are deduplicated, tracked through new → recurred → resolved, routable to Slack/webhooks, and mutable with feedback. See [Insights](/guides/insights). ## Fixing **Remediation** — A typed, tracked fix for a failure — for example a prompt change or a tool fix — generated from real trace evidence. It moves `proposed` → `accepted` → `applied` → `verified` (or `regressed`), separate from your own triage lane (to do / in progress / done). You can **simulate** a remediation to preview its effect on real failing traces before shipping anything, and after a Neens-opened fix PR merges, Neens measures whether the failure actually dropped in production to set the `verified` / `regressed` verdict. Applying a fix requires a **bound proof** (a verification run or an eval gate) — Neens won't mark a fix shipped on a hunch. See [Remediations](/guides/remediations) and [Fix outcomes](/guides/post-merge-efficacy). **Failure locus** — *Where* a failure actually lives, decided from the trace evidence rather than the cluster label: your agent's reasoning, a tool contract, a downstream/upstream service, a quality issue, a control working as intended, or an incoherent/unknown grouping. The locus sets a remediation's **actionability** — an in-repo fix, an **advisory** you route to a service owner (e.g. *"this is a downstream `503`, not your agent"*), or **no action** when a guardrail fired correctly. A proposal with no concrete change is held as **needs grounding** and kept out of the actionable backlog. See [Remediations](/guides/remediations#where-the-failure-lives-the-failure-locus). **Prompt optimization run** — An *offline* search for a better system prompt for one confirmed failure mode. Neens replays that failure's historical traces against candidate prompts (tool results are served from each recorded trace), grades every replay with your judges, and reflects on the graded results to write the next candidate — keeping the candidates that are best on at least one trace rather than only the best average. The winner has to beat the original prompt on a **held-out** set of traces it was never optimized against; only then does it become a `prompt_change` remediation, which still goes through the same eval-verified PR gate and human merge as any other fix. See [Prompt optimization](/guides/prompt-optimization). **Eval gate** — An evaluation built *from your own failures* and used as a regression guard. A gate ties a failure mode to a dataset of its real examples and a judge, and tracks the pass rate against a baseline — turning "we fixed this" into "this can't quietly come back". See [Eval gates](/guides/eval-gates). **Pre-prod eval run** — A whole-release gate. Before a candidate agent version ships, it replays a **golden dataset** against that version, scores the results with your existing judges, and compares against a **baseline** (a prior run or your live production window), flagging regressions. Two ways to run it: **you run your agent** and Neens correlates the emitted traces, or **Neens calls your agent** at an HTTP endpoint you register, once for every golden prompt. See [Pre-prod evaluations](/guides/preprod-evals). ## Workspace & configuration **Company / Org / Agent** — The workspace hierarchy. **Agents were formerly called Projects** — same concept, new name. Your **company** is the isolated tenant; it contains **orgs** (team- or product-level groupings), which contain **agents**. The agent is the working scope: traces, judges, datasets, and dashboards all live in an agent, and ingest API keys are bound to exactly one agent. Admins see everything; members see the orgs and agents they've been given. See the Administration section. **Persona** — A *lens*, not a permission. A persona (Executive, Developer, Product manager, Compliance, Finance, …) changes which navigation items are emphasized, your landing page, and default time ranges — it never grants or removes access, and everything stays reachable. Admins can assign personas and author custom ones. See the Administration section. **LLM connection** — Your configured model provider: Anthropic, OpenAI (or any OpenAI-compatible endpoint, including local Ollama), or AWS Bedrock, with a model and an encrypted credential. Every AI feature in Neens — judges, cluster labels, insights, topics, enrichments, remediations, the assistant — resolves the agent's visible connection; there is no built-in model access. Connections are scoped company-wide or to specific orgs/agents, with one default per scope. Set up in **Settings → LLM providers** — see [Getting started](/getting-started#connect-an-llm). ## Putting it together ```mermaid flowchart TD T[Traces] --> S[Sessions] T --> J[Judges → Scores] T --> C[Clusters → Failure modes → Issues] J --> D[Datasets & golden versions] A[Human annotations] --> D C --> R[Remediations] D --> G[Eval gates & pre-prod evals] R --> G ``` You send **traces**, Neens rolls them into **sessions**, scores them with **judges**, clusters failures into **issues**, and gives you **remediations**, **eval gates**, and **pre-prod evaluations** to fix problems and keep them fixed — with **datasets** and **annotations** keeping the loop grounded in real, human-verified data. ================================================================================ # Send traces Source: /docs/guides/send-traces/ ================================================================================ # Send traces Neens ingests your agent's traces over HTTP. It speaks standard **OpenTelemetry (OTLP)** and **OpenInference**, and also accepts a simple Neens-native raw JSON format. Pick whichever matches how your agent is already instrumented — all paths land in the same trace store. **Already using Langfuse or Braintrust?** You don't have to re-instrument. A [trace inlet](/guides/connectors/trace-inlets) pulls your existing traces into Neens on a schedule — keep your tracing tool, add the Neens brain. ## At a glance | | | | --- | --- | | **Endpoints** | `POST /v1/traces` (OTLP protobuf + JSON), `POST /ingest/openinference`, `POST /ingest/raw`, `POST /ingest/batch` | | **Auth** | `Authorization: Bearer nk_live_…` agent API key (**Settings → API keys**) | | **Default response** | `202 Accepted` — parsing and persistence happen asynchronously | | **Compression** | `Content-Encoding: gzip` or `deflate` accepted on every endpoint | | **Size limits** | 1 MiB per single request · 4 MiB OTLP · 8 MiB batch | | **Sync mode** | `?sync=true` processes in-request and returns `200` with the persisted session | ## Authentication Every request should authenticate with an agent **API key** (`nk_live_…`) as a bearer token: ```http Authorization: Bearer nk_live_your_key_here ``` The key alone scopes the request: it resolves to exactly one company and agent, so you don't need to send an agent ID — and an authenticated request **cannot** be redirected to another agent by any header or body field. Create keys in **Settings → API keys** (see [Getting started](/getting-started#3-get-an-ingest-api-key)). How a trace is attributed to an agent, in precedence order: 1. **The API key's agent** — always wins when a key is supplied. 2. **`X-Neens-Project-Id` header** — only consulted when no key was supplied (keyless local/dev setups). 3. **`project_id` field in the request body** — same keyless fallback, lowest priority. 4. Otherwise the trace lands in the server's default agent. **A bad credential is always rejected.** A supplied credential that doesn't resolve — an edited, invalid, or revoked key, or an unrecognized bearer token — returns `401 Unauthorized`. It is never silently accepted, so a trace can never land in the wrong agent because of a typo in the key. Whether a request with **no credential at all** is accepted depends on the deployment, which derives it from its storage backend: a **Postgres** deployment requires a key (keyless requests get `401`), while a **SQLite** single-tenant dev install still accepts keyless ingest. An admin can override that in either direction for your deployment. Upgrading a Postgres install that used to post keyless? See [A keyless request started returning 401](#a-keyless-request-started-returning-401) below. **Base URL.** Replace `https://` in the examples with your Neens URL. In the app, the **Traces** page shows ready-to-copy snippets with your host and key already filled in. ## Choosing a format | Your setup | Use | Endpoint | | --- | --- | --- | | Already using OpenTelemetry | **OTLP** | `POST /v1/traces` | | Using OpenInference instrumentation | **OpenInference** | `POST /ingest/openinference` | | No tracing yet / custom pipeline | **Raw JSON** | `POST /ingest/raw` | | Many traces in one request | **Batch** | `POST /ingest/batch` | All endpoints transparently decompress `Content-Encoding: gzip` (or `deflate`) bodies — the default for OTLP exporters and collectors. `zstd` is not supported and returns `415 Unsupported Media Type`; fall back to gzip. ## OpenTelemetry (recommended) If your agent is instrumented with OpenTelemetry, point its OTLP/HTTP exporter at Neens and add your key as a header. No Neens-specific code required. **Using a framework?** The [framework quickstarts](/guides/quickstarts) have copy-paste OpenTelemetry config for LangGraph, CrewAI, OpenAI Agents SDK, Pydantic AI, Claude Agent SDK, and the Vercel AI SDK — install the instrumentation, point it here, done. `POST /v1/traces` accepts both OTLP encodings: - **protobuf** (`Content-Type: application/x-protobuf`) — the exporter default, - **JSON** (`Content-Type: application/json`) — the OTLP/JSON shape. ```bash python my_agent.py ``` Standard exporters append `/v1/traces` to the endpoint automatically. ```python from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter exporter = OTLPSpanExporter( endpoint="https:///v1/traces", headers={"Authorization": "Bearer nk_live_your_key_here"}, ) # wire this exporter into your tracer provider as usual ``` ```bash curl -X POST "https:///v1/traces" \ -H "Authorization: Bearer nk_live_your_key_here" \ -H "Content-Type: application/json" \ -d '{ "resourceSpans": [{ "resource": {"attributes": [ {"key": "service.name", "value": {"stringValue": "support-bot"}} ]}, "scopeSpans": [{"spans": [{ "traceId": "W47/95gDgQPSabYzgT/GDA==", "spanId": "7uGbfsPBsXM=", "name": "chat gpt-4o", "startTimeUnixNano": "1752660000000000000", "endTimeUnixNano": "1752660002000000000", "status": {"code": 1}, "attributes": [ {"key": "gen_ai.operation.name", "value": {"stringValue": "chat"}}, {"key": "gen_ai.request.model", "value": {"stringValue": "gpt-4o"}}, {"key": "gen_ai.usage.input_tokens", "value": {"intValue": "120"}}, {"key": "gen_ai.usage.output_tokens", "value": {"intValue": "35"}} ] }]}] }] }' ``` A few OTLP-specific behaviors, all standards-compliant: - The response is the OTLP `ExportTraceServiceResponse` envelope (protobuf or `{}` JSON, matching the request encoding) — `202` on the default async path, so a stock exporter or collector is happy without any Neens-specific handling. - One OTLP export may carry spans from **many traces** (that's how exporters batch); Neens groups spans by `traceId` and assembles one trace per id. The agent name comes from the resource's `service.name`. - An export with no `resourceSpans` (e.g. a collector heartbeat) is acknowledged as a successful no-op. - The OTLP payload cap is **4 MiB** (a `413` includes a `Retry-After` header so collectors back off and split batches). Neens reads model, inputs/outputs, messages, token usage, tool calls, and conversation IDs from standard **OTel GenAI** *and* **OpenInference** attributes — per field, first match wins — so mixed instrumentation resolves correctly. See the [attribute reference](#which-attributes-neens-reads) below. ## OpenInference If you export OpenInference span dicts directly (rather than over OTLP), post them to `/ingest/openinference`. The payload is an object with a top-level `spans` array; spans from multiple traces may be mixed in one payload and are grouped by `context.trace_id`. ```bash curl -X POST "https:///ingest/openinference" \ -H "Authorization: Bearer nk_live_your_key_here" \ -H "Content-Type: application/json" \ -d '{ "spans": [{ "context": {"trace_id": "trace-001", "span_id": "span-1"}, "name": "ChatCompletion", "parent_id": null, "start_time": "2026-07-16T10:00:00Z", "end_time": "2026-07-16T10:00:02Z", "status_code": "OK", "attributes": { "openinference.span.kind": "LLM", "llm.model_name": "gpt-4o", "llm.token_count.prompt": 120, "llm.token_count.completion": 35, "llm.input_messages.0.message.role": "user", "llm.input_messages.0.message.content": "Where is my order?", "llm.output_messages.0.message.role": "assistant", "llm.output_messages.0.message.content": "Let me check that for you.", "session.id": "chat-42", "service.name": "support-bot" } }] }' ``` Each span needs `context.trace_id`, `context.span_id`, `start_time`, and `end_time` (ISO-8601). `status_code` is `OK`, `ERROR`, or unset; on errors, `exception.message` or `error.message` in `attributes` becomes the span's error message. The agent name is taken from a `service.name` attribute. Spans with `openinference.span.kind: "TOOL"` also produce tool-call records automatically (tool name = span name, args/result from `input.value`/`output.value`). OpenInference and OTel GenAI are attribute vocabularies, not different wire formats — if your OpenInference instrumentation already exports over OTLP, just use `/v1/traces`. Both endpoints resolve both vocabularies with the same per-field logic. ## Raw JSON (Neens-native) The simplest option when you're not using a tracing SDK. Post a session (one trace) and its spans directly. Only `session.id` is required — everything else has sensible defaults. ```bash curl -X POST "https:///ingest/raw" \ -H "Authorization: Bearer nk_live_your_key_here" \ -H "Content-Type: application/json" \ -d '{ "session": {"id": "trace-001", "agent_name": "my-agent"}, "spans": [ {"id": "span-1", "name": "llm.call", "kind": "llm", "input": "Hello?", "output": "Hi!", "model": "gpt-4o", "input_tokens": 12, "output_tokens": 4} ] }' ``` ```python requests.post( "https:///ingest/raw", headers={"Authorization": "Bearer nk_live_your_key_here"}, json={ "session": {"id": "trace-001", "agent_name": "my-agent"}, "spans": [ {"id": "span-1", "name": "llm.call", "kind": "llm", "input": "Hello?", "output": "Hi!", "model": "gpt-4o", "input_tokens": 12, "output_tokens": 4}, ], }, timeout=10, ) ``` ### Raw payload reference **`session`** — the trace header. Fields you supply are taken verbatim. | Field | Required | Default | Description | | --- | --- | --- | --- | | `id` | ✅ | — | Unique trace ID. Re-sending the same ID updates the trace (idempotent), never duplicates it. | | `agent_name` | | `""` | Name of the agent that produced the trace. | | `conversation_id` | | — | Groups multiple traces into one **session** (see [below](#grouping-traces-into-sessions)). | | `status` | | `"ok"` | `ok` or `error`. | | `started_at` / `ended_at` | | now / — | ISO-8601 timestamps (e.g. `2026-07-16T10:00:00Z`). A malformed value is a `422` naming the field. | | `duration_ms` | | `0` | Total trace duration in milliseconds. | | `input_tokens` / `output_tokens` | | `0` | Token usage for cost roll-ups. | | `turn_count` | | `0` | Number of conversational turns. | | `metadata` | | `{}` | Arbitrary key/value object, usable for filtering. | | `source` | | `"raw"` | Free-form label for where the trace came from. | **`spans`** — the steps inside the trace (optional array). Only `id` is required per span. | Field | Required | Default | Description | | --- | --- | --- | --- | | `id` | ✅ | — | Unique span ID. | | `name` | | `""` | Span name (e.g. `llm.call`, `retrieve`). | | `kind` | | `"custom"` | `llm`, `agent`, `tool`, `chain`, `retrieval`, `guardrail`, or `custom`. | | `parent_id` | | — | Parent span ID, to nest spans into a tree. | | `input` / `output` | | — | The span's input and output (string; message lists as JSON). | | `model` | | — | Model name for LLM spans. | | `input_tokens` / `output_tokens` | | — | Per-span token usage. | | `status` | | `"ok"` | `ok` or `error`; add `error_message` on failures. | | `error_message` | | — | Error detail for failed spans. | | `started_at` | | session start | ISO-8601 timestamp. | | `duration_ms` | | `0` | Span duration in milliseconds. | | `attributes` | | `{}` | Flat key/value dict of semantic-convention attributes (see below). | **Attribute fallback.** When a top-level span field (`input`, `output`, `model`, `input_tokens`, `output_tokens`, `kind`) is absent, Neens fills it from the span's `attributes` using the same OTel-GenAI/OpenInference resolvers as the OTLP path. So an OpenInference-shaped span (data under `input.value`, `llm.model_name`, `llm.token_count.*`, `openinference.span.kind`) keeps its semantics through `/ingest/raw` too. Fields you set explicitly always win. **`tool_calls`** — optional array of tool/function invocations. | Field | Required | Default | Description | | --- | --- | --- | --- | | `span_id` | ✅ | — | The span this call belongs to. | | `tool_name` | ✅ | — | Name of the tool/function. | | `args` | | `{}` | Arguments passed (object). | | `result` | | — | Return value. | | `error` | | — | Error string if the call failed. | | `duration_ms` | | `0` | Call duration in milliseconds. | ## Batch To send many traces in one request, post an array of raw payloads — or `{"items": [...]}` — to `/ingest/batch`. All items are attributed to the API key's agent and buffered as one unit. ```bash curl -X POST "https:///ingest/batch" \ -H "Authorization: Bearer nk_live_your_key_here" \ -H "Content-Type: application/json" \ -d '[ {"session": {"id": "trace-001", "agent_name": "my-agent"}, "spans": []}, {"session": {"id": "trace-002", "agent_name": "my-agent"}, "spans": []} ]' ``` A batch may be up to **8 MiB**; each item is validated up front — a missing `session` or `session.id` returns `422` naming the offending item (e.g. `batch item[1] session is missing required field 'id'`). The batch must be non-empty. With `?sync=true` the batch is processed in-request and returns `200 {"session_ids": [...], "session_count": n, "span_count": n, "warnings": [...]}`. ## Grouping traces into sessions Neens rolls individual traces up into a **session** when they share a `conversation_id`. Set the same ID on each turn of a multi-turn conversation and the turns appear together on the **Sessions** page, while remaining individually visible under **Traces**. - **Raw JSON**: set `session.conversation_id`. - **OTLP / OpenInference**: set any of `gen_ai.conversation.id`, `session.id`, `conversation.id`, or `thread.id` as a span attribute (first one found across the trace's spans wins; empty strings are ignored). ```json {"session": {"id": "turn-2", "conversation_id": "chat-42", "agent_name": "support-bot"}} ``` A trace with no conversation ID is simply its own single-trace session. See [Core concepts](/concepts) for the trace/session distinction. ## How it works Ingestion is **asynchronous by default** so your agent's latency never depends on the Neens write path: 1. The edge authenticates the request, resolves the agent from the credential, and runs cheap structural validation (required top-level fields, size caps). 2. The payload is durably buffered and the request returns `202` immediately — for the `/ingest/*` endpoints the body is `{"batch_id": "ing_…", "accepted": true}`; `/v1/traces` answers with the standard OTLP response envelope instead. 3. A worker parses the payload (one session per trace ID), resolves semantic-convention attributes, and persists spans and tool calls. Traces appear in the UI within a few seconds. 4. If the buffer is unavailable the request is **not** acknowledged — you get `503` with `Retry-After`, and your exporter/collector retries from its own queue. Nothing is silently dropped. Re-sending a trace is safe: sessions and spans are keyed by their IDs and upserted, so retries and duplicate deliveries update rather than duplicate. You may also supply your own `Idempotency-Key` header; otherwise a hash of the payload is used. **Retrying safely.** On a `429` the response carries a `Retry-After` header — honor it and back off exponentially rather than hammering the edge. Re-sending a batch under the **same** `Idempotency-Key` is deduplicated: the ingest worker reads the key and skips a batch it has already processed (a cheap skip instead of a re-parse), so a client-side retry never double-counts. The dedup window is 7 days by default. Add `?sync=true` to any ingest endpoint to process in-request instead — useful for tests, first-trace verification, and low-volume backfills: ```bash curl -X POST "https:///ingest/raw?sync=true" ... # → 200 {"session_id": "trace-001", "span_count": 1, "warnings": []} ``` In sync mode you also get immediate `422`s for deeper semantic problems (e.g. a malformed timestamp) that the async path would only detect after acceptance. ## Reference ### Response codes | Status | Meaning | | --- | --- | | `202 Accepted` | Trace accepted and queued (default). `/ingest/*` return `{"batch_id": …, "accepted": true}`; `/v1/traces` returns the OTLP response envelope. | | `200 OK` | `?sync=true` — processed in-request; body includes the persisted `session_id`(s) and `span_count`. Also a no-op OTLP export (empty `resourceSpans`). | | `400 Bad Request` | Invalid OTLP JSON/protobuf, or a corrupt compressed body. | | `401 Unauthorized` | Invalid, edited, or revoked API key; unrecognized bearer token — or a keyless request when the server requires auth (the default on self-hosted Postgres; see [A keyless request started returning 401](#a-keyless-request-started-returning-401)). Check the `Authorization: Bearer nk_live_…` header. | | `413 Payload Too Large` | Body exceeds the size limit — 1 MiB single (`/ingest/otlp`, `/ingest/openinference`, `/ingest/raw`), 4 MiB OTLP (`/v1/traces`), 8 MiB batch. Limits apply to the **decompressed** size, so gzip can't smuggle an oversized payload. | | `415 Unsupported Media Type` | Unsupported `Content-Encoding` (use gzip, deflate, or none). | | `422 Unprocessable Entity` | Malformed payload — the `detail` names the offending field, e.g. `missing required field 'resourceSpans'`, `session is missing required field 'id'`, `span[2]: raw span is missing required field 'id'`, `invalid span started_at timestamp '…': expected ISO-8601`. | | `429 Too Many Requests` | Backpressure — your agent's ingest queue is saturated. Honor the `Retry-After` header (default **5 seconds**) and retry; standard collectors/SDKs do this automatically. | | `503 Service Unavailable` | The ingest buffer is temporarily unavailable. Not an ack — retry after `Retry-After`. | Backpressure is per-agent: only the saturated agent is throttled, so a burst on one agent never slows another. The `Retry-After` value defaults to **5 seconds**. ### Which attributes Neens reads Neens resolves each semantic field by probing the OTel GenAI and OpenInference vocabularies **per field, in order — first non-empty value wins**. Mixing vocabularies within one payload, or even one span, is fine. Full attribute table (OTel GenAI + OpenInference) | Field | Keys probed, in order | | --- | --- | | Model | `gen_ai.response.model` → `gen_ai.request.model` → `llm.model_name` | | Input tokens | `gen_ai.usage.input_tokens` → `gen_ai.usage.prompt_tokens` → `llm.token_count.prompt` | | Output tokens | `gen_ai.usage.output_tokens` → `gen_ai.usage.completion_tokens` → `llm.token_count.completion` | | Conversation ID | `gen_ai.conversation.id` → `session.id` → `conversation.id` → `thread.id` | | Span kind | `openinference.span.kind` (`LLM`/`AGENT`/`TOOL`/`CHAIN`/`RETRIEVAL`/`GUARDRAIL`) → `gen_ai.operation.name` (tool/function → tool, embeddings → retrieval, chat etc. → llm) → heuristic (`gen_ai.tool.name` present → tool; a model attribute present → llm) → `custom` | | Input messages | `llm.input_messages.{i}.message.role` / `.content` (indexed, in order) → `gen_ai.prompt` (JSON message list, or plain text) → `input.value` (raw fallback) | | Output messages | `llm.output_messages.{i}.message.role` / `.content` → `gen_ai.completion` → `output.value` | | Tool name (tool spans) | `gen_ai.tool.name` → span name | | Tool args / result (tool spans) | span input / output (`input.value` / `output.value` on OpenInference) — JSON strings are parsed | | Error message | OTLP: `gen_ai.error.message` → span status message · OpenInference: `exception.message` → `error.message` | | Agent name | OTLP: resource `service.name` · OpenInference: span attribute `service.name` | **Message content cleanup.** Message `content` values may be plain strings, serialized LangChain messages (unwrapped to the inner content), or lists of content blocks (concatenated). Neens normalizes them all to clean `{role, content}` lists. **Serialized-blob fallback.** Some frameworks (LangChain/LangGraph LLM nodes) bury the model name and token usage inside the serialized `output` blob instead of span attributes. When the attribute chains above come up empty, Neens parses that blob and recovers `model_name`/`token_usage`/`usage_metadata` best-effort. Attributes always win when present. **Neens-specific attributes.** | Key | Purpose | | --- | --- | | `neens.version_label` | Stamps the trace with the agent version that produced it — surfaced in trace metadata and used to ground pre-prod comparisons. | | `neens.eval_run_id` | Correlates a replayed trace with a [pre-prod evaluation run](/guides/preprod-evals). | | `neens.dataset_item_id` | Pins the trace to the exact frozen golden prompt within that run. | Endpoint size limits and encodings | Endpoint | Formats | Max size (decompressed) | | --- | --- | --- | | `POST /v1/traces` | OTLP protobuf (`application/x-protobuf`), OTLP JSON | 4 MiB | | `POST /ingest/openinference` | JSON (`{"spans": [...]}`) | 1 MiB | | `POST /ingest/raw` | JSON (`{"session": …, "spans": […], "tool_calls": […]}`) | 1 MiB | | `POST /ingest/batch` | JSON array of raw items, or `{"items": [...]}` | 8 MiB | | `POST /ingest/otlp` | OTLP JSON only (legacy; prefer `/v1/traces`) | 1 MiB | All accept `Content-Encoding: gzip` or `deflate`. The compressed body is also capped at the same limit, and decompression bombs are rejected with `413`. ## Troubleshooting | Symptom | Likely cause | Fix | | --- | --- | --- | | `401` on every request | Key typo'd, revoked, or missing — or the server requires auth and you sent none | Re-copy the key; send it as `Authorization: Bearer nk_live_…` | | `202` but nothing in the UI | Looking at the wrong agent, or async processing hasn't finished | Switch to the key's agent; wait a few seconds; test with `?sync=true` | | `422` with a field name | Malformed payload — the `detail` says exactly which field | Fix the named field (e.g. add `session.id`, use ISO-8601 timestamps) | | `429` under load | Your agent's ingest queue is saturated | Back off for `Retry-After` seconds and retry; batch smaller | | Tokens/model missing on spans | Attributes not in a recognized vocabulary | Use the keys in the [attribute table](#which-attributes-neens-reads), or set top-level fields on raw spans | ### A keyless request started returning 401 Ingest that used to be accepted with no credential now fails. This happens on a **self-hosted Postgres** deployment: a keyless write is refused rather than being filed into the unscoped default agent. (SQLite single-tenant dev installs are unaffected — keyless ingest still works there.) The request that now fails: ```bash curl -i -X POST "https:///ingest/raw" \ -H "Content-Type: application/json" \ -d '{"session": {"id": "trace-001"}, "spans": []}' ``` and the response: ```http HTTP/1.1 401 Unauthorized Content-Type: application/json {"detail": "Ingest requires an API key. Provide 'Authorization: Bearer nk_live_…'."} ``` **The fix: send an agent API key.** Mint one in the app under **Settings → API keys** — click **Generate key** and copy it immediately, it is shown once (full walkthrough: [API keys](/administration/api-keys)). Then send it as a bearer token: ```bash curl -i -X POST "https:///ingest/raw" \ -H "Authorization: Bearer nk_live_your_key_here" \ -H "Content-Type: application/json" \ -d '{"session": {"id": "trace-001"}, "spans": []}' ``` Expect `202 Accepted` with a `batch_id`. The same header works on `/v1/traces`, `/ingest/openinference` and `/ingest/batch` — for an OTLP exporter or collector, set it via `OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer nk_live_your_key_here"`. The key carries the agent scope, so once you send one you can drop any `X-Neens-Project-Id` header or `project_id` body field you were relying on — the key wins over both. Still stuck? See [My traces aren't showing up](/faq#my-traces-arent-showing-up) in the FAQ. ## Verify Open **Traces** in the app. Your traces appear within a few seconds; click one to inspect its spans, tool calls, inputs, outputs, latency, and cost. Once traces flow, continue to [Traces & sessions](/guides/traces-and-sessions) and [Judges](/guides/judges). ================================================================================ # Framework quickstarts Source: /docs/guides/quickstarts/ ================================================================================ # Framework quickstarts Neens is a **neutral inlet**: it ingests standard [OpenTelemetry](/guides/send-traces) — there is no Neens SDK to adopt and nothing to fork. To connect a framework you already use, you install its OpenTelemetry instrumentation (most often the community [OpenInference](https://github.com/Arize-ai/openinference) instrumentors, or the framework's own built-in OTel support) and point the exporter at the Neens OTLP endpoint. That's it — the same wiring works for any OTel-instrumented app. Each recipe below is copy-paste: install, point the exporter at `https:///v1/traces` with your `nk_live_…` agent key, run your agent, and watch traces land under **Traces**. ## The common contract Every recipe wires the same three things — only the instrumentation package changes: | | | | --- | --- | | **Endpoint** | `POST https:///v1/traces` (OTLP/HTTP). Standard exporters append `/v1/traces` to `OTEL_EXPORTER_OTLP_ENDPOINT` automatically. | | **Auth** | `Authorization: Bearer nk_live_…` — an agent API key from **Settings → API keys**. | | **Attributes** | Neens reads standard **OTel-GenAI** *and* **OpenInference** attributes (model, tokens, prompt/response messages, tool calls, conversation id) — per field, first match wins. | Not using one of these frameworks? Any OpenTelemetry-instrumented app exports to Neens the same way — see [Send traces](/guides/send-traces) for the endpoints, size limits, response codes, and the full semantic-convention attribute reference. Prefer to send raw JSON instead of OTLP? That's supported too. ## Correlating pre-prod eval traces Running these traces through a **[pre-prod evaluation](/guides/preprod-evals)** gate in CI? The [`neens-eval`](/guides/eval-gates) runner (Python and TypeScript) injects the `neens.eval_run_id` / `neens.dataset_item_id` / `neens.version_label` correlation attributes into `OTEL_RESOURCE_ATTRIBUTES` for you, so the traces your instrumented agent emits above are automatically tied to the right run. Those three attribute names are a stable wire contract — if you tag traces yourself instead of using the runner, spell them exactly as listed above. ================================================================================ # LangGraph Source: /docs/guides/quickstarts/langgraph/ ================================================================================ # LangGraph [LangGraph](https://langchain-ai.github.io/langgraph/) is LangChain's graph-based framework for building stateful, multi-actor agents. Neens ingests its traces through standard OpenTelemetry with zero Neens-specific code — you point an OTLP exporter at Neens and instrument LangChain once at startup. ## At a glance | | | | --- | --- | | **Language** | Python | | **Instrumentation** | `openinference-instrumentation-langchain` (OpenInference LangChain instrumentor — community-maintained by Arize) | | **Endpoint** | `POST /v1/traces` (OTLP/HTTP) | | **Auth** | `Authorization: Bearer nk_live_…` agent key | ## Instrument your agent ### Install the instrumentation LangGraph runs on LangChain, so the OpenInference **LangChain** instrumentor captures LangGraph spans automatically. ```bash pip install openinference-instrumentation-langchain opentelemetry-sdk opentelemetry-exporter-otlp ``` ### Point the exporter at Neens ```python from openinference.instrumentation.langchain import LangChainInstrumentor from opentelemetry import trace from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchSpanProcessor from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter exporter = OTLPSpanExporter( endpoint="https:///v1/traces", headers={"Authorization": "Bearer nk_live_your_key_here"}, ) provider = TracerProvider() provider.add_span_processor(BatchSpanProcessor(exporter)) trace.set_tracer_provider(provider) # Instrument once, before you build or invoke your graph. LangChainInstrumentor().instrument() ``` ```bash ``` You still call the instrumentor once at startup — the exporter reads these env vars, so you can drop the explicit `endpoint`/`headers`: ```python from openinference.instrumentation.langchain import LangChainInstrumentor from opentelemetry import trace from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchSpanProcessor from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter provider = TracerProvider() provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter())) trace.set_tracer_provider(provider) LangChainInstrumentor().instrument() ``` ### Run your agent Run your LangGraph app as usual — invoke or stream your compiled graph. Spans export to Neens in the background. ## What Neens captures The instrumentor emits spans for each LLM call (model, token counts, prompt/response messages), every tool call, and the graph's node and chain spans — so you see the full agent trajectory. For multi-turn agents, set a conversation/session id on your runs so Neens groups them into one session (see [Traces & sessions](/guides/traces-and-sessions)). **Name your final answer in the graph state.** OpenInference maps every LangGraph node and graph to a `chain` span, whose payload is *state*, not dialogue. Neens reads a chain span as a conversational turn only when the state names its content — the user's question under `message`/`query`/`question`/`prompt`, and the final reply under `answer`/`final_answer`/`final_output` (among others). If your last node returns a bare string or an unrecognized key, the **Conversation** tab shows the question and no reply — and the same derivation is what a golden dataset item and a pre-prod baseline record. See [Conversation transcript](/guides/conversation-transcript). Neens speaks standard OTLP — this is not a Neens SDK fork. Any OpenTelemetry-instrumented LangGraph app exports to Neens by pointing its exporter at the endpoint above. See [Send traces](/guides/send-traces) for the full attribute reference, size limits, and response codes. ## Verify Open **Traces** in Neens; your LangGraph runs appear within a few seconds. Continue to [Traces & sessions](/guides/traces-and-sessions). ================================================================================ # CrewAI Source: /docs/guides/quickstarts/crewai/ ================================================================================ # CrewAI [CrewAI](https://docs.crewai.com/) is a Python framework for orchestrating role-playing, multi-agent crews. Neens ingests its traces through standard OpenTelemetry with zero Neens-specific code — you instrument CrewAI once at startup and point an OTLP exporter at Neens. CrewAI ships its own built-in **telemetry** (anonymous usage stats) — that is unrelated to trace export. The OpenTelemetry trace path below is a separate, third-party instrumentor. ## At a glance | | | | --- | --- | | **Language** | Python | | **Instrumentation** | `openinference-instrumentation-crewai` + an LLM-level instrumentor, e.g. `openinference-instrumentation-litellm` (OpenInference instrumentors — community-maintained by Arize) | | **Endpoint** | `POST /v1/traces` (OTLP/HTTP) | | **Auth** | `Authorization: Bearer nk_live_…` agent key | ## Instrument your agent ### Install the instrumentation The OpenInference **CrewAI** instrumentor captures Crew / Agent / Task spans. It does **not** trace the underlying LLM calls on its own, so pair it with an LLM-level instrumentor. CrewAI routes most models through **LiteLLM**, so `openinference-instrumentation-litellm` is the common pairing; if your model string routes to a native SDK (e.g. `openai/…`), install that provider's instrumentor instead (`openinference-instrumentation-openai`, `openinference-instrumentation-anthropic`, …). ```bash pip install openinference-instrumentation-crewai openinference-instrumentation-litellm opentelemetry-sdk opentelemetry-exporter-otlp ``` ### Point the exporter at Neens ```python from openinference.instrumentation.crewai import CrewAIInstrumentor from openinference.instrumentation.litellm import LiteLLMInstrumentor from opentelemetry import trace from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchSpanProcessor from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter exporter = OTLPSpanExporter( endpoint="https:///v1/traces", headers={"Authorization": "Bearer nk_live_your_key_here"}, ) provider = TracerProvider() provider.add_span_processor(BatchSpanProcessor(exporter)) trace.set_tracer_provider(provider) # Instrument once, before you build or kick off your crew. CrewAIInstrumentor().instrument(tracer_provider=provider) LiteLLMInstrumentor().instrument(tracer_provider=provider) ``` ```bash ``` You still call the instrumentors once at startup — the exporter reads these env vars, so you can drop the explicit `endpoint`/`headers`: ```python from openinference.instrumentation.crewai import CrewAIInstrumentor from openinference.instrumentation.litellm import LiteLLMInstrumentor from opentelemetry import trace from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchSpanProcessor from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter provider = TracerProvider() provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter())) trace.set_tracer_provider(provider) CrewAIInstrumentor().instrument(tracer_provider=provider) LiteLLMInstrumentor().instrument(tracer_provider=provider) ``` ### Run your crew Run your crew as usual — `crew.kickoff()`. Spans export to Neens in the background. ## What Neens captures The instrumentors emit spans for each **agent** and **task** in the crew, every **LLM call** (model, token counts, prompt/response messages), and every **tool call** — so you see the full multi-agent trajectory. For multi-turn crews, set a conversation/session id on your runs so Neens groups them into one session (see [Traces & sessions](/guides/traces-and-sessions)). Neens speaks standard OTLP — this is not a Neens SDK fork. Any OpenTelemetry-instrumented CrewAI app exports to Neens by pointing its exporter at the endpoint above. See [Send traces](/guides/send-traces) for the full attribute reference, size limits, and response codes. ## Verify Open **Traces** in Neens; your CrewAI runs appear within a few seconds. Continue to [Traces & sessions](/guides/traces-and-sessions). ================================================================================ # OpenAI Agents SDK Source: /docs/guides/quickstarts/openai-agents/ ================================================================================ # OpenAI Agents SDK The [OpenAI Agents SDK](https://openai.github.io/openai-agents-python/) (the `openai-agents` Python package) is a lightweight framework for building multi-agent workflows. Neens ingests its traces through standard OpenTelemetry with zero Neens-specific code — you point an OTLP exporter at Neens and instrument the SDK once at startup. ## At a glance | | | | --- | --- | | **Language** | Python | | **Instrumentation** | `openinference-instrumentation-openai-agents` (OpenInference OpenAI Agents instrumentor — community-maintained by Arize) | | **Endpoint** | `POST /v1/traces` (OTLP/HTTP) | | **Auth** | `Authorization: Bearer nk_live_…` agent key | ## Instrument your agent ### Install the instrumentation The OpenInference instrumentor hooks the Agents SDK's own tracing runtime and re-emits it as OpenTelemetry spans. ```bash pip install openinference-instrumentation-openai-agents opentelemetry-sdk opentelemetry-exporter-otlp ``` This assumes you already have the SDK itself installed (`pip install openai-agents`). ### Point the exporter at Neens ```python from openinference.instrumentation.openai_agents import OpenAIAgentsInstrumentor from opentelemetry import trace from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchSpanProcessor from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter exporter = OTLPSpanExporter( endpoint="https:///v1/traces", headers={"Authorization": "Bearer nk_live_your_key_here"}, ) provider = TracerProvider() provider.add_span_processor(BatchSpanProcessor(exporter)) trace.set_tracer_provider(provider) # Instrument once, before you run your agent. OpenAIAgentsInstrumentor().instrument() ``` ```bash ``` You still call the instrumentor once at startup — the exporter reads these env vars, so you can drop the explicit `endpoint`/`headers`: ```python from openinference.instrumentation.openai_agents import OpenAIAgentsInstrumentor from opentelemetry import trace from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchSpanProcessor from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter provider = TracerProvider() provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter())) trace.set_tracer_provider(provider) OpenAIAgentsInstrumentor().instrument() ``` ### Run your agent Run your agent as usual — call `Runner.run(...)` or `Runner.run_sync(...)`. Spans export to Neens in the background. ## What Neens captures The instrumentor emits spans for each agent run, every LLM call (model, token counts, prompt/response messages), and every tool/function call — so you see the full agent trajectory. For multi-turn agents, set a conversation/session id on your runs so Neens groups them into one session (see [Traces & sessions](/guides/traces-and-sessions)). Neens speaks standard OTLP — this is not a Neens SDK fork. Any OpenTelemetry-instrumented Agents SDK app exports to Neens by pointing its exporter at the endpoint above. See [Send traces](/guides/send-traces) for the full attribute reference, size limits, and response codes. **Alternative: the SDK's built-in tracing.** The Agents SDK ships [its own tracing runtime](https://openai.github.io/openai-agents-python/tracing/), which [Pydantic Logfire](https://pydantic.dev/logfire) can bridge to OpenTelemetry via `logfire.instrument_openai_agents()` (community, not first-party OpenAI). If you already run Logfire, configure it to export OTLP at the endpoint above instead of installing the OpenInference instrumentor. Neens reads either path — it ingests OTel-GenAI and OpenInference attributes alike. We recommend the OpenInference instrumentor above for consistency with the other Neens framework quickstarts. ## Verify Open **Traces** in Neens; your agent runs appear within a few seconds. Continue to [Traces & sessions](/guides/traces-and-sessions). ================================================================================ # Pydantic AI Source: /docs/guides/quickstarts/pydantic-ai/ ================================================================================ # Pydantic AI [Pydantic AI](https://ai.pydantic.dev/) (from the Pydantic team) has **first-party, built-in OpenTelemetry instrumentation** — it emits OTel GenAI spans natively, so there is no third-party instrumentor to install. Point a standard OTLP exporter at Neens and flip on instrumentation with one flag. ## At a glance | | | | --- | --- | | **Language** | Python | | **Instrumentation** | Built-in (first-party OTel GenAI) | | **Endpoint** | `POST /v1/traces` (OTLP/HTTP) | | **Auth** | `Authorization: Bearer nk_live_…` agent key | ## Instrument your agent ### Install Install Pydantic AI and the standard OpenTelemetry SDK + OTLP/HTTP exporter. No Neens SDK, no third-party instrumentor. ```bash pip install pydantic-ai opentelemetry-sdk opentelemetry-exporter-otlp ``` ### Point OpenTelemetry at Neens Configure a standard OTel `TracerProvider` with an OTLP exporter aimed at Neens, then enable Pydantic AI's built-in instrumentation with `instrument=True`. No Logfire account required. ```python from opentelemetry import trace from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchSpanProcessor from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter from pydantic_ai import Agent exporter = OTLPSpanExporter( endpoint="https:///v1/traces", headers={"Authorization": "Bearer nk_live_your_key_here"}, ) provider = TracerProvider() provider.add_span_processor(BatchSpanProcessor(exporter)) trace.set_tracer_provider(provider) # Enable Pydantic AI's built-in OTel instrumentation on this agent. agent = Agent("openai:gpt-4o", instrument=True) ``` To instrument every agent in your process at once, call `Agent.instrument_all()` after setting the tracer provider instead of passing `instrument=True` per agent. ```bash ``` With the endpoint and headers in the environment, the OTLP exporter reads them automatically — you only wire the provider and turn on instrumentation: ```python from opentelemetry import trace from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchSpanProcessor from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter from pydantic_ai import Agent provider = TracerProvider() provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter())) trace.set_tracer_provider(provider) agent = Agent("openai:gpt-4o", instrument=True) ``` The exporter appends `/v1/traces` to `OTEL_EXPORTER_OTLP_ENDPOINT`, so set the host **without** the path. ### Run your agent Run your agent as usual — `agent.run_sync("...")` (or `await agent.run("...")`). Spans export to Neens in the background. ## What Neens captures Pydantic AI emits a span per agent run, with native OTel GenAI spans for every model call — model name, input/output token counts, and the prompt/response messages — plus a span per tool call. Together they give you the full agent trajectory. For multi-turn agents, set a conversation/session id on your runs so Neens groups them into one session (see [Traces & sessions](/guides/traces-and-sessions)). Neens speaks standard OTLP — this is not a Neens SDK fork. Any OpenTelemetry-configured Pydantic AI app exports to Neens by pointing its exporter at the endpoint above. Prefer [Pydantic Logfire](https://logfire.pydantic.dev/)? It's an optional convenience: call `logfire.configure()` (which sets the global tracer provider) with the same OTLP endpoint and headers, then `logfire.instrument_pydantic_ai()` — Neens reads either path, since it ingests OTel-GenAI and OpenInference attributes alike. See [Send traces](/guides/send-traces) for the full attribute reference, size limits, and response codes. ## Verify Open **Traces** in Neens; your agent runs appear within a few seconds. Continue to [Traces & sessions](/guides/traces-and-sessions). ================================================================================ # Claude Agent SDK Source: /docs/guides/quickstarts/claude-agent-sdk/ ================================================================================ # Claude Agent SDK The [Claude Agent SDK](https://code.claude.com/docs/en/agent-sdk/overview) from Anthropic (`claude-agent-sdk` in Python, `@anthropic-ai/claude-agent-sdk` in TypeScript — formerly the "Claude Code SDK") wraps Claude models plus tool use into an agent loop. Neens ingests its traces through standard OpenTelemetry with zero Neens-specific code — you point an OTLP exporter at Neens and instrument once at startup. ## At a glance | | | | --- | --- | | **Language** | Python & TypeScript | | **Instrumentation** | `openinference-instrumentation-anthropic` (instruments the underlying Anthropic Messages API; community-maintained by Arize) — see the note below for the dedicated agent-SDK instrumentor and Anthropic's native OTLP export | | **Endpoint** | `POST /v1/traces` (OTLP/HTTP) | | **Auth** | `Authorization: Bearer nk_live_…` agent key | ## Instrument your agent ### Install the instrumentation The OpenInference Anthropic instrumentor hooks the Anthropic SDK's `messages` client — every Claude call the agent makes becomes an OpenTelemetry LLM span. ```bash pip install openinference-instrumentation-anthropic opentelemetry-sdk opentelemetry-exporter-otlp ``` This assumes you already have the agent SDK itself installed (`pip install claude-agent-sdk`). ### Point the exporter at Neens ```python from openinference.instrumentation.anthropic import AnthropicInstrumentor from opentelemetry import trace from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchSpanProcessor from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter exporter = OTLPSpanExporter( endpoint="https:///v1/traces", headers={"Authorization": "Bearer nk_live_your_key_here"}, ) provider = TracerProvider() provider.add_span_processor(BatchSpanProcessor(exporter)) trace.set_tracer_provider(provider) # Instrument once, before you run your agent. AnthropicInstrumentor().instrument() ``` ```bash ``` The exporter appends `/v1/traces` to `OTEL_EXPORTER_OTLP_ENDPOINT`, so leave the path off here. You still call the instrumentor once at startup — the exporter reads these env vars, so you can drop the explicit `endpoint`/`headers`: ```python from openinference.instrumentation.anthropic import AnthropicInstrumentor from opentelemetry import trace from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchSpanProcessor from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter provider = TracerProvider() provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter())) trace.set_tracer_provider(provider) AnthropicInstrumentor().instrument() ``` ### Run your agent Run your agent as usual — call `query(...)` or drive a `ClaudeSDKClient`. Spans export to Neens in the background. ## What Neens captures The instrumentor emits a span for every Claude model call the agent makes — model name, input/output token counts, and the prompt/response messages — plus a span for each tool use, so you see the full agent trajectory. For multi-turn agents, set a conversation/session id so Neens groups the turns into one session (see [Traces & sessions](/guides/traces-and-sessions)). Neens speaks standard OTLP — this is not a Neens SDK fork. Any OpenTelemetry-instrumented Claude Agent SDK app exports to Neens by pointing its exporter at the endpoint above. See [Send traces](/guides/send-traces) for the full attribute reference, size limits, and response codes. **Two other real trace paths — Neens reads all of them.** ① A dedicated agent-SDK instrumentor, `openinference-instrumentation-claude-agent-sdk` (also community/Arize), traces `query()` / `ClaudeSDKClient` as agent-level spans with child tool spans — install it instead of `-anthropic` and call `ClaudeAgentSDKInstrumentor().instrument()` for agent-shaped traces rather than raw LLM calls. ② The Agent SDK has a **native** OTLP export (it runs the Claude Code CLI, which has OpenTelemetry built in): set `CLAUDE_CODE_ENABLE_TELEMETRY=1`, `OTEL_TRACES_EXPORTER=otlp`, and `CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1` (**trace export is beta**; metrics and log events are GA), then point `OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf` + `OTEL_EXPORTER_OTLP_ENDPOINT`/`OTEL_EXPORTER_OTLP_HEADERS` at Neens as above — it emits `claude_code.interaction`, `claude_code.llm_request`, and `claude_code.tool` spans. This needs no OpenInference package, but the trace schema is beta and may change between releases. We lead with the OpenInference instrumentor for a stable, GA trace shape consistent with the other Neens quickstarts. ## Verify Open **Traces** in Neens; your agent runs appear within a few seconds. Continue to [Traces & sessions](/guides/traces-and-sessions). ================================================================================ # Vercel AI SDK Source: /docs/guides/quickstarts/vercel-ai-sdk/ ================================================================================ # Vercel AI SDK The [Vercel AI SDK](https://ai-sdk.dev/) (the `ai` package) has **first-party, built-in OpenTelemetry telemetry** — you flip it on per call with `experimental_telemetry`, and the SDK emits OTel spans for every generation. Neens ingests those spans through standard OTLP with zero Neens-specific code: you set up an OpenTelemetry exporter in your TypeScript/Node app and point it at Neens. ## At a glance | | | | --- | --- | | **Language** | TypeScript / Node.js | | **Instrumentation** | Built-in (`experimental_telemetry`) + OpenTelemetry Node SDK | | **Endpoint** | `POST /v1/traces` (OTLP/HTTP) | | **Auth** | `Authorization: Bearer nk_live_…` agent key | ## Instrument your agent ### Install the OpenTelemetry packages The AI SDK produces the spans; you supply an OpenTelemetry SDK to export them. This assumes you already have the AI SDK itself installed (`npm install ai`). ```bash npm install @opentelemetry/sdk-node \ @opentelemetry/exporter-trace-otlp-http \ @opentelemetry/auto-instrumentations-node ``` For the Next.js path below, install `@vercel/otel` instead: ```bash npm install @vercel/otel @opentelemetry/api ``` ### Point the exporter at Neens Create an `instrumentation.ts` and start the Node SDK **before** you import or call the AI SDK — load it with `node --import ./instrumentation.js your-app.js` (or `require('./instrumentation')` at the very top of your entry file). ```typescript // instrumentation.ts const sdk = new NodeSDK({ traceExporter: new OTLPTraceExporter({ url: 'https:///v1/traces', headers: { Authorization: 'Bearer nk_live_your_key_here' }, }), instrumentations: [getNodeAutoInstrumentations()], }) sdk.start() // Flush spans on shutdown so nothing is lost. process.on('SIGTERM', () => { sdk.shutdown().finally(() => process.exit(0)) }) ``` Prefer env vars? The OTLP exporter reads them, so you can drop the explicit `url`/`headers` and construct `new OTLPTraceExporter()`. The exporter auto-appends `/v1/traces` to the endpoint: ```bash ``` In a Next.js app, create `instrumentation.ts` in your project root — Next.js calls its `register()` export automatically at startup. ```typescript // instrumentation.ts registerOTel({ serviceName: 'my-ai-app', traceExporter: new OTLPHttpJsonTraceExporter({ url: 'https:///v1/traces', headers: { Authorization: 'Bearer nk_live_your_key_here' }, }), }) } ``` `@vercel/otel` also honors `OTEL_EXPORTER_OTLP_ENDPOINT` / `OTEL_EXPORTER_OTLP_HEADERS`, so you can omit `traceExporter` and configure Neens through the env vars shown in the Node tab instead. ### Enable telemetry on your calls Telemetry is opt-in per call. Pass `experimental_telemetry: { isEnabled: true }` to any `generateText`, `streamText`, `generateObject`, or `streamObject` call. A `functionId` labels the span so you can tell your agent's steps apart. ```typescript const { text } = await generateText({ model: openai('gpt-4o'), prompt: 'Summarize the latest support ticket.', experimental_telemetry: { isEnabled: true, functionId: 'summarize-ticket', // recordInputs / recordOutputs default to true; // set false to keep prompts/responses out of spans. }, }) ``` ## What Neens captures With telemetry enabled, the AI SDK emits a span per generation carrying the model, token usage, the prompt, the response text, and any tool calls — so you see the full trajectory. The SDK writes its own `ai.*` attributes (e.g. `ai.prompt`, `ai.response.text`, `ai.toolCall.*`) **and** the OpenTelemetry GenAI `gen_ai.*` attributes; Neens reads the standard `gen_ai.*` model and token fields (`gen_ai.request.model`, `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`) alongside the `ai.*` prompt/response and tool-call attributes. For multi-turn agents, propagate a conversation/session id across calls so Neens groups them into one session (see [Traces & sessions](/guides/traces-and-sessions)). Neens speaks standard OTLP — this is not a Neens SDK fork. Any OpenTelemetry-instrumented app exports to Neens by pointing its exporter at the endpoint above. See [Send traces](/guides/send-traces) for the full attribute reference, size limits, and response codes. ## Verify Open **Traces** in Neens; your generations appear within a few seconds. Continue to [Traces & sessions](/guides/traces-and-sessions). ================================================================================ # Trace inlets Source: /docs/guides/connectors/trace-inlets/ ================================================================================ # Trace inlets A **trace inlet** is a connector that **pulls** trace data out of a tracing tool you already run — **Langfuse** or **Braintrust** — into Neens on a schedule. Each external trace is mapped into the Neens Session / Span / Tool-call model and fed through the [normal ingest path](/guides/send-traces), so clustering, judges, scores, remediations, and every other Neens feature run on top of it with no change to how your agent is instrumented. This is the **brownfield** on-ramp: *keep your tracing tool, add the Neens brain.* You don't re-instrument your agent or move off your incumbent observability stack — you point Neens at it and Neens starts diagnosing failures from the traces that already exist. **Inlets pull in; [external API scoring](/guides/external-api-scoring) reaches out.** These are opposite directions and easy to confuse. An **inlet** is *inbound* — Neens fetches traces **from** Langfuse/Braintrust into your tenant. An **external API** judge/enrichment is *outbound* — Neens calls **out** to a third-party HTTP API to score a trace it already has. Use an inlet to get traces in; use external API scoring to attach a signal. ## At a glance | | | | --- | --- | | **Where** | **Settings → Trace inlets** (admin-only tab) | | **Providers** | **Langfuse**, **Braintrust** | | **Key API** | `GET/POST /trace-inlets`, `PATCH/DELETE /trace-inlets/{id}`, `POST /trace-inlets/{id}/test`, `POST /trace-inlets/{id}/sync` | | **Direction** | Inbound — Neens polls the provider and ingests new traces on a schedule | | **Auth to the provider** | Langfuse: Public key + Secret key · Braintrust: API key + Project ID (encrypted at rest) | | **Scope** | Agent-scoped — traces land in the connector's agent, exactly like a direct ingest | | **Egress safety** | Every fetch is SSRF-gated; provider cloud hosts work out of the box | ## Add a connector Adding a connector is admin-only — inlet credentials are sensitive external-system secrets, so the tab and its API require the **admin** role. ### Open Settings → Trace inlets Go to **Settings → Trace inlets** and click **Add**. Pick the provider — **Langfuse** or **Braintrust**. ### Fill in the common fields Every connector, whatever the provider, has: | Field | Meaning | | --- | --- | | **Name** | A label for this connector in the list. | | **Base URL** | The provider API origin (see the per-provider setup below). | | **Poll interval (minutes)** | How often Neens syncs this connector. Default `15`. | | **Backfill days** | How far back the **first** sync reaches. Default `7`. | | **Enabled** | Off by default — a new connector is created disabled so you can **Test** it before it starts syncing. | ### Add the provider credentials Enter the provider-specific keys (below). Credentials are **write-only**: they're encrypted the moment you save and are never shown again. When you **edit** a connector later, leave a credential field blank to keep the stored secret; type a new value only to replace it. ### Test before you enable Click **Test**. Neens makes one bounded, SSRF-gated call to the provider and reports an honest result — `ok` with a small sample count, or the real error (`401` for a bad key, a timeout, or a blocked host). Fix any problem, then turn **Enabled** on. ### Enable and let it sync — or sync now Once enabled, the connector syncs automatically on its poll interval. To pull immediately, click **Sync now**; Neens runs one bounded sync and shows how many traces were ingested. ## Langfuse setup | Field | Value | | --- | --- | | **Base URL** | `https://cloud.langfuse.com` for Langfuse Cloud, or your **self-hosted** Langfuse origin (e.g. `https://langfuse.internal`). | | **Public key** | Your Langfuse project's **Public Key** (`pk-lf-…`). | | **Secret key** | Your Langfuse project's **Secret Key** (`sk-lf-…`). | Find both keys in Langfuse under **Project Settings → API Keys**. Neens authenticates to the Langfuse public API with HTTP Basic auth (public key as username, secret key as password) and reads traces and their observations from the `/api/public/traces` endpoints. **What maps.** Each Langfuse **trace** becomes a Neens trace; each **observation** becomes a span: - A `GENERATION` observation → an **LLM** span, carrying the model name and prompt/completion token counts. - A tool-like `SPAN` observation → a **TOOL** span. - Other observations → **CHAIN** / **AGENT** spans. - An observation at `ERROR` level marks its span as failed and carries the status message as the error. A **self-hosted** Langfuse URL resolves to a private/internal address, so it's refused by the SSRF gate — see [Security](#security). Langfuse Cloud (`cloud.langfuse.com`) is a public host and works with no extra configuration. ## Braintrust setup | Field | Value | | --- | --- | | **Base URL** | `https://api.braintrust.dev` (Braintrust Cloud), or your self-hosted Braintrust API origin. | | **API key** | A Braintrust API key. | | **Project ID** | The Braintrust **project** whose logs to pull. | Neens authenticates with a bearer token (`Authorization: Bearer `) and fetches from the project-logs API (`/v1/project_logs/{project_id}/fetch`). **What maps.** Braintrust log **events** are grouped by their root span id into traces, and each event becomes a span: - An event whose type is `llm` → an **LLM** span, with model and token counts from its metrics. - An event whose type is `tool` or `function` → a **TOOL** span. - Other event types → **CHAIN** spans. ## How syncing works Neens periodically syncs the enabled connectors that are **due** — whose last run is older than their poll interval (default `15` minutes, set per connector). - **First sync — backfill.** The first sync of a connector reaches back **backfill days** (default `7`) and pulls the traces in that window. - **Incremental syncs.** After that, each sync fetches only traces newer than the last successful high-water mark (with a small overlap so nothing on the boundary is missed), advancing an opaque provider cursor as it goes. A failed sync does **not** advance the cursor, so the next run retries the same window — you never lose traces to a transient error. - **Idempotent re-ingest.** Ingested traces keep their stable provider trace/span ids, and the Neens ingest path is idempotent on those ids. Re-pulling an overlapping window (or re-running a sync) never duplicates data — the same trace resolves to the same session. - **What lands.** Ingested traces appear on the [Traces & sessions](/guides/traces-and-sessions) page and flow into [clustering](/guides/clustering), [continuous evaluation](/guides/continuous-evaluation), and everything downstream — indistinguishable from traces sent directly. The connector list shows each connector's last status (`ok` / `error` / `never`), when it last synced, how many traces the last run ingested, and the last error inline if a sync failed. ## Security Inlets are built to be a **zero-migration, zero-new-egress-risk** on-ramp: - **Credentials encrypted at rest.** Every provider secret (the Langfuse secret key, the Braintrust API key) is Fernet-encrypted before it's stored and is **never** returned by any API — the connector responses report only *whether* a credential is set, never its value. - **SSRF egress gate.** Every outbound fetch (and the **Test** probe) is validated against the same gate as the other Neens outbound integrations: the host is resolved and pinned, redirects are not followed, and a private, loopback, link-local, or reserved address is **refused**. Provider cloud hosts (`cloud.langfuse.com`, `api.braintrust.dev`) are public and reachable out of the box; a self-hosted provider on a private address is gated by default and an operator must allowlist its host. - **No data leaves your tenant.** An inlet only *pulls in*. Traces are ingested straight into the connector's agent inside your tenant; nothing about your traces is sent back out to the provider or anywhere else. ## Limits Each sync is bounded so a single connector can never overwhelm a worker or the broker: a connector ingests up to **500** traces per run (a larger backlog drains across several runs), each provider response body is capped at **8 MiB**, and a single outbound HTTP call is allowed **30 seconds**. ## Troubleshooting | Symptom | Likely cause | Fix | | --- | --- | --- | | **Test** returns `401` | Wrong or revoked provider keys. | Re-copy the keys from the provider's project settings and re-save (leave a field blank to keep the stored one). | | **Test** says the URL is blocked | A self-hosted provider resolves to a private address. | Ask an operator to allowlist the host — a provider on a private address is gated by default. | | Connector is enabled but nothing ingests | It isn't due yet, or the first backfill window predates your traces. | Click **Sync now**, or raise **Backfill days** and sync again. | | The **Trace inlets** tab isn't there | You're not an admin. | Ask an admin. | | A sync shows `error` with a real message | A transient provider/network failure. | The cursor didn't advance — the next sync retries the same window; fix the root cause shown in the last error. | See also [Send traces](/guides/send-traces) for direct ingest and the [framework quickstarts](/guides/quickstarts) if you'd rather instrument your agent to send to Neens natively. ================================================================================ # Connect your helpdesk Source: /docs/guides/connectors/outcome-inlets/ ================================================================================ # Connect your helpdesk An **outcome inlet** is a connector that **pulls** case outcomes out of your system of record — **Zendesk**, **Intercom**, **Salesforce** or **Jira Service Management** — into Neens on a schedule. Each closed ticket becomes one or more [business outcomes](/guides/business-outcomes) (`resolution`, `escalation`, `csat`, `resolution_time`, …) keyed to the Neens case the agent actually handled. It's the pull half of the outcome inlet. If you already run an ETL or a warehouse job, pushing to `POST /outcomes` is simpler and gives you full control over the mapping — see [Business outcomes](/guides/business-outcomes#push-an-outcome). Use a connector when you'd rather not build that job. **Outcome inlets bring in *results*; [trace inlets](/guides/connectors/trace-inlets) bring in *traces*.** Different data, different providers, same shape of connector — you can run both. ## At a glance | | | | --- | --- | | **Where** | The **Data sources** tab of the **[Business KPIs](/guides/business-kpis)** page (connectors are admin-only), the **Helpdesk outcome inlets** section | | **Providers** | **Zendesk**, **Intercom**, **Salesforce**, **Jira Service Management** | | **Key API** | `GET/POST /outcome-inlets`, `PATCH/DELETE /outcome-inlets/{id}`, `POST /outcome-inlets/{id}/test`, `POST /outcome-inlets/{id}/sync` | | **Direction** | Inbound — Neens polls the provider and records new outcomes on a schedule | | **Scope** | Agent-scoped: outcomes land in the connector's agent | | **Egress safety** | Every fetch is SSRF-gated; provider clouds work out of the box | | **LLM** | None. Provider fields are mapped deterministically; nothing is inferred | ## Add a connector Every route on this tab is **admin-only** — a connector holds a credential for an external system and its base URL is dialled server-side. ### Open the Data sources tab On the **[Business KPIs](/guides/business-kpis)** page, open the **Data sources** tab. Under **Helpdesk outcome inlets**, click **Add inlet** and pick the **Provider**. ### Fill in the common fields | Field | Meaning | | --- | --- | | **Name** | A label for this connector in the list. | | **Base URL** | The provider API origin (see the per-provider setup below). | | **Account email** | Zendesk and Jira Service Management only — the account the API token belongs to. | | **Poll interval (minutes)** | How often Neens syncs this connector. Default `15`. | | **Backfill window (days)** | How far back the **first** sync reaches. Default `7`. | | **Enabled** | Off by default — create it disabled, **Test** it, then turn it on. | ### Set the correlation contract This is the part that decides whether your outcomes match anything, and it is worth two minutes of thought. Two settings, shared by all four providers: | Field | Meaning | | --- | --- | | **Correlation type** | `conversation` (default) · `session` · `external` — how Neens matches an inbound record to a case. See [choosing a correlation type](/guides/business-outcomes#choose-a-correlation-type). | | **Correlation field (path on the provider record)** | The path **on the provider's record** whose value becomes the correlation key. Leave blank for the provider default. | The two combine like this: - **Your agent uses the ticket id as its conversation id.** Correlation type `conversation`, correlation field the provider's own record id (`id` for Zendesk/Intercom, `Id` for Salesforce, `key` for Jira). This is the default and the cleanest setup. - **Your agent writes the ticket id into its trace metadata.** Correlation type `external`, and the correlation field is still the provider's ticket id — Neens then looks that value up in your sessions' metadata. - **The provider record stores the Neens conversation id in a custom field.** Correlation type `conversation`, correlation field that custom field (`external_id` on Zendesk, `custom_attributes.neens_conversation` on Intercom, a `customfield_…` on Jira, a custom field on Salesforce). If the correlation field resolves to nothing on a record, that record is **skipped and counted** — never guessed into a match. The connector reports `partial` with the number skipped and the field name, so a typo is visible on the very first sync instead of silently halving your denominator. ### Add the credential Credentials are **write-only**: encrypted the moment you save, never returned by any API (responses report only whether one is set). When you edit a connector later, leave the field blank to keep the stored secret; type a value only to replace it. ### Test before you enable Click **Test**. Neens makes one bounded, SSRF-gated call and reports an honest result — `reachable — N recent ticket(s) visible`, or the real error (`401`, a timeout, a blocked host). Fix anything broken, then enable. ### Enable, or sync now Once enabled the connector syncs on its poll interval. **Sync now** runs one bounded sync immediately and reports how many outcomes were recorded and how many matched. ## Zendesk | Field | Value | | --- | --- | | **Base URL** | `https://.zendesk.com` | | **Email** | The Zendesk account the API token belongs to (required). | | **Credential** | A Zendesk **API token** (Admin Center → Apps and integrations → APIs → Zendesk API). | | **Default correlation field** | `id` (the ticket id) | Neens authenticates with HTTP Basic using Zendesk's API-token scheme (`{email}/token` as the username) and reads the **cursor-based Incremental Ticket Export** (`/api/v2/incremental/tickets/cursor`), side-loading Ticket Metrics so resolution and reply times are available. **What maps:** | Outcome | From | | --- | --- | | `resolution` | `status` — `solved` or `closed` is `true`. Emitted for every status, so an open ticket is an honest `false` that later restates to `true`. | | `outcome_label` | The raw `status`. | | `csat` | `satisfaction_rating.score` — `good` → `1.0`, `bad` → `0.0`, a numeric score passed through. `offered`/`unoffered` is **omitted**: an unanswered survey is not a zero. | | `resolution_time` | The metric set's full resolution time (calendar), else `solved_at − created_at`. | | `first_response_time` | The metric set's reply time (calendar). | | `reopened` | The metric set's reopen count `> 0`. | | `transfer_count` | Assignee stations minus one. | | `escalation` | An **escalation tag** you configure, if present in the ticket's `tags`; otherwise "the ticket moved between groups". With neither signal the kind is omitted rather than assumed `false`. | Other options: **Escalation tag (optional)** — the tag your team applies when a ticket goes to a human. To correlate on a Zendesk custom field, set the correlation field to `custom_fields.`. ## Intercom | Field | Value | | --- | --- | | **Base URL** | `https://api.intercom.io` (or the EU/AU regional host your workspace uses) | | **Credential** | An Intercom **access token**. | | **Default correlation field** | `id` (the conversation id) | Neens authenticates with `Authorization: Bearer …` plus a pinned `Intercom-Version` header (default `2.11`, configurable) so a future breaking API version can't silently reshape the payload, and reads `POST /conversations/search`. **What maps:** | Outcome | From | | --- | --- | | `resolution` | `state == "closed"` (`open`/`snoozed` are an honest `false`). | | `outcome_label` | The raw `state`. | | `csat` | `conversation_rating.rating` (Intercom's 1–5 integer). | | `first_response_time` | `statistics.time_to_admin_reply`. | | `resolution_time` | `statistics.time_to_last_close`, else `time_to_first_close`. | | `reopened` | `statistics.count_reopens > 0`. | | `transfer_count` | `statistics.count_assignments` minus one. | | `escalation` | Only from a configured **escalation tag**. Intercom has no unambiguous "handed to a human" flag (Fin is itself an admin), so with no tag configured the kind is omitted rather than inferred. | Other options: **API version (optional)** and **Escalation tag (optional)**. A `stateFilter` (`open`/`closed`/`snoozed`) to sync only closed conversations is available through the API's `options` object. To correlate on a custom attribute, set the correlation field to `custom_attributes.`. Intercom's search API documents no sort order, so a sync that hits its record budget does **not** advance the time high-water mark — it hands back Intercom's own cursor and the next run resumes the identical query mid-scan. Only a scan that completes moves the window forward. Nothing is skipped. ## Salesforce | Field | Value | | --- | --- | | **Base URL** | Your org's instance URL — `https://.my.salesforce.com` | | **Credential** | A Salesforce **access token**. | | **Default correlation field** | `Id` (the Case id) | Neens authenticates with `Authorization: Bearer …` and runs a SOQL query over the standard `Case` object through the REST Query resource (default API version `v60.0`). **What maps:** | Outcome | From | | --- | --- | | `resolution` | The standard `IsClosed` boolean, timestamped at `ClosedDate`. | | `outcome_label` | `Status`. | | `escalation` | The standard `IsEscalated` boolean. | | `resolution_time` | `ClosedDate − CreatedDate`. | | `csat` | Only from a **CSAT field** you configure. Salesforce ships no standard CSAT field on `Case`, so Neens never invents one. | Other options: **API version (optional)**, **CSAT field (optional)** and **Extra WHERE clause (optional)** to narrow the export. A different `object` (if you use a custom case object) and `extraFields` to widen the `SELECT` are available through the API's `options` object. Neens stores whatever token you give it. A short-lived token simply fails the next sync with an honest `401` — the cursor doesn't advance and no outcome is fabricated. ## Jira Service Management | Field | Value | | --- | --- | | **Base URL** | `https://.atlassian.net` for Cloud, or your Data Center origin | | **Email** | The Atlassian account the API token belongs to (required). | | **Credential** | An Atlassian **API token**. | | **Default correlation field** | `key` (the issue key) | Neens authenticates with HTTP Basic (`{email}:{api token}`) and runs a JQL search. Atlassian split its search endpoints, so Neens drives Jira Cloud's JQL enhanced search by default and falls back once to the classic Data Center endpoint when Cloud answers `404`/`410`, then sticks with whichever works. Pin it explicitly by setting the `apiVariant` option (`cloud` or `datacenter`) through the API. **What maps:** | Outcome | From | | --- | --- | | `resolution` | `fields.resolution` is non-null, timestamped at `resolutiondate`. | | `outcome_label` | The resolution name when resolved, else the status name. | | `resolution_time` | `resolutiondate − created`. | | `escalation` | Only from an `escalationField` (a `customfield_…` path) configured through the API's `options` object, optionally compared against an `escalationValue`. Jira ships no standard escalation flag. | | `csat` | Only from a configured **CSAT field (optional)**. JSM's satisfaction rating lives in a site-specific `customfield_…`, so there is no default to hard-code. | Other options: **Project key (optional)** and a **JQL filter (optional)** to scope the export. JQL date comparisons are interpreted in the **credential user's** Jira time zone, not UTC. The five-minute re-fetch overlap plus content dedup absorbs the skew, but set the connector's Jira user to UTC if you want the window to be exact. ## How syncing works A scheduler periodically syncs the enabled connectors that are **due** — whose last run is older than their poll interval (default `15` minutes, set per connector). - **First sync — backfill.** The first sync reaches back **backfill days** (default `7`). - **Incremental syncs.** After that each sync fetches only records updated since the last successful high-water mark, with a five-minute overlap so nothing on the boundary is missed. - **A failure never advances anything.** On any error — a `401`, a timeout, a blocked host, a database problem — the cursor and the high-water mark are left exactly where they were, so the next run retries the same window. Nothing is invented to fill the gap. - **Re-fetching is free.** Records go through the same content-hash dedup as pushed outcomes: an unchanged re-poll is deduplicated, and a genuinely changed value becomes a new revision. A ticket reopening is a restatement you can see, not a duplicate row. ## Read the match counters Every connector row in the list shows its own honesty numbers after each run: | Column | What it shows | | --- | --- | | **Name** | The connector name, with its correlation contract underneath — *keyed by external · ticket_id*. | | **Status** | `never` · `ok` · `partial` · `error`, plus how long ago the last run was, and the real error string inline when a run failed. `partial` means records were skipped because the correlation field resolved to nothing — the message says how many and which field. | | **Last sync** | The match counters: *N outcomes · X% matched*, and below it *M matched, U unresolved*. `unresolved` folds in `ambiguous`, because "did not resolve to exactly one case" is the number that matters. A non-zero unresolved count is highlighted. | | **Enabled** | The on/off toggle for the scheduled sync. | The same row carries the **Test**, **Sync now**, **Edit** and **Delete** actions. A connector whose matched percentage is far below 100% is a correlation-contract problem, not a provider problem. Open the **Match coverage** panel at the top of the same tab, read the per-correlation-type breakdown and the recent unmatched sample, and work through [Why your match rate is not 100%](/guides/business-outcomes#why-your-match-rate-is-not-100). ## Security - **Credentials encrypted at rest.** Every provider secret is Fernet-encrypted before storage and is never returned by any API — responses report only *whether* a credential is set. - **SSRF egress gate.** Every outbound fetch and every **Test** probe resolves and pins the host, refuses redirects, and caps the response body. A private, loopback, link-local or reserved address is **refused**. A literal private IP is also rejected at save time. Provider clouds (`*.zendesk.com`, `api.intercom.io`, `*.my.salesforce.com`, `*.atlassian.net`) are public and reachable out of the box. - **Nothing is written back.** An inlet only pulls. Neens never modifies a ticket, and no trace content is sent to the provider. - **Admin-only.** Creating, editing, testing and syncing a connector all require the admin role, and every change is written to the [audit log](/administration/audit-log). ## Limits Each sync is bounded so a single connector can't overwhelm a worker: a connector records up to **500** outcomes per run (a larger backlog drains across successive runs), each provider response body is capped at **8 MiB**, and a single outbound call is allowed **30 seconds**. The per-connector **poll interval** (default 15 minutes) and **backfill window** (default 7 days) are set on the connector itself when you create it. ## Troubleshooting | Symptom | Likely cause | Fix | | --- | --- | --- | | **Test** returns `401` | Wrong or revoked credential, or the wrong email for a Basic-auth provider. | Re-copy the token; for Zendesk and Jira check the **email** matches the account that owns it. | | **Test** says the URL is blocked | A self-hosted instance resolves to a private address. | Ask an operator to allowlist the host — a provider on a private address is gated by default. | | Last status is `partial` | The correlation field resolved to nothing on some records. | Read the message — it names the field. Fix the field path, or narrow the export to records that carry it. | | Records arrive but almost nothing matches | The correlation *type* is wrong for how your agent is instrumented. | See [Choose a correlation type](/guides/business-outcomes#choose-a-correlation-type) and the [match-rate troubleshooting table](/guides/business-outcomes#why-your-match-rate-is-not-100). | | Enabled but nothing syncs | Not due yet, or the first backfill window predates your closed tickets. | Click **Sync now**, or raise **Backfill days**. | | A sync shows `error` | A transient provider or network failure. | The cursor didn't advance — the next run retries the same window. Fix the cause shown in the last error. | | The tab isn't there | You're not an admin. | Ask an admin. | See also [Business outcomes](/guides/business-outcomes) for the push API, the outcome catalogue and the coverage panel. ================================================================================ # Traces & Sessions Source: /docs/guides/traces-and-sessions/ ================================================================================ # Traces & Sessions **Traces** and **Sessions** are where you see what your agents actually did. A *trace* is one agent run — a request that came in, the steps the agent took, and the response it produced. A *session* is a conversation: one or more traces grouped together. Start here to inspect a single run, follow a multi-turn interaction, or hunt down the runs behind a failure. If you haven't sent any data yet, head to [Send traces](/guides/send-traces) first — the Traces page will walk you through it. ## At a glance | | | |---|---| | **Where** | Sidebar → **Observe** → **Traces** / **Sessions** | | **Key API routes** | `GET /sessions` (list traces), `GET /sessions/{id}` (trace detail), `GET /conversations` (session rollup), `GET /conversations/{id}`, `GET /agents` | | **Scope** | Always scoped to the current agent; you can only open traces in agents you belong to | | **Needs** | Ingested trace data — nothing else. No LLM connection required to browse | ## The trend header Above the filters on both the **Traces** and **Sessions** pages sits a **trend header** — a rich, at-a-glance read on how your volume, token usage, and cost are moving over time, *before* you touch a single row. Use it to spot a spike or a creeping trend first, then dig into the table to find the runs behind it. The header has three parts, plus one control that ties everything below it together. ### The KPI strip Four tiles run across the top. Each shows the value for the selected time window, a **period-over-period delta**, and a **sparkline** of the trend: | Tile | What it shows | |---|---| | **Traces/day** (Sessions page: **Sessions/day**) | The average number of runs ingested per day in the window. | | **Total tokens** | Input + output tokens across every run in the window. | | **p50 latency** | The median end-to-end run duration — half your runs finished faster, half slower. | | **Est. spend** | Estimated cost across the window, derived from tokens × each run's model price (see [Token usage & cost](#token-usage--cost)). | **Reading the delta.** Each tile compares the selected window to the **immediately-preceding window of equal length** — so on **7d**, "this week vs. last week"; on **24h**, "today vs. yesterday". The delta is the change between the two. A rising **Traces/day** or **Total tokens** means more activity; a rising **p50 latency** or **Est. spend** is usually worth a look. The direction is what matters — read it as "up since last period" or "down since last period", not as an absolute score. **A blank metric means "no data", never zero.** If a window has nothing to compute a tile from — for example **Est. spend** when every run used an unpriced model, or a delta with no preceding period to compare against — the tile shows **—**. Neens never shows a fake `0` in place of a number it doesn't have. ### The token trend chart Below the tiles is a **stacked-area chart** of token throughput over the selected window, split into two bands: - **Input tokens** — tokens sent *to* the models (prompts, context, tool results). - **Output tokens** — tokens the models generated *back*. The two bands stack, so the top edge of the chart is your **total** token volume over time — the same figure the **Total tokens** tile reports for the window — and the split shows the **input/output mix** behind it. That mix is worth watching because **output tokens usually cost more** than input tokens, so a run whose output share is climbing gets more expensive faster than its raw volume suggests. Hover any point to read the exact **Input** and **Output** counts at that moment. ### Segment the trend by model, agent, status or source Use the **Group by** control in the header to break the volume into **stacked series** instead of the input/output token split. Pick a dimension: - **Model** — one band per model, so you can see a migration land or a model's share grow. - **Agent** — one band per agent, to compare how much traffic each is handling. - **Status** — `ok` vs. `error` vs. `unknown` volume side by side. - **Source** — one band per ingestion source. Each band is a **count** over time, and the bands stack to the same total volume as the ungrouped view — so the shape of the whole is unchanged, only its composition is revealed. When a dimension has many values, the busiest few are shown individually and the rest fold into a single **Other** band so the chart stays readable; the legend names every band. Set **Group by** back to **None** to return to the token chart. Your choice is remembered per page. ### Overlay the eval pass-rate (semantic health) When judges or scorers have graded runs in the window, the header offers a **Pass rate** view — the share of scored runs that pass their threshold, which is the meaningful *"is quality holding up over time?"* signal (and **not** the same as operational status: a run can complete cleanly yet still be a *wrong* answer). - A **Pass rate** KPI tile appears in the strip, with its period-over-period delta. - Click **Pass-rate trend** to reveal a small **pass-rate % line** beneath the main chart. It sits on its **own chart with its own 0–100% axis** — a percentage and a token count can't share a scale, so it is never overlaid as a second axis. A period with no graded scores shows a **gap** in the line rather than a misleading 0%. If nothing has been scored in the window, the tile, the toggle and the line are simply not shown — there's no empty health claim to read into. ### The time-window control — one window, one truth A single **time-window control** in the header scopes the chart **and** the list table below it **together**. Change it in one place and everything on the page — the sparklines, the KPI deltas, the token chart, and the rows in the table — moves to the same window. There is no way for the chart and the table to disagree about which slice of time you're looking at. The presets are **Today**, **24h**, **7d**, **30d**, and **All**, plus **Custom** for an explicit from/to range. (See [Time ranges](#time-ranges) for exactly what each preset covers.) ### Drag across the chart to zoom You don't have to reach for the picker to narrow the window. **Drag horizontally across the token chart** to select a span of buckets, and on release the page zooms to exactly that range — the chart, the KPI tiles, and the list table below all snap to it together, because the selection sets the page's one time window (as a **Custom** range). The selection includes the full width of the last bucket you drag over, so dragging across two days gives you both days end-to-end. A plain click (without dragging) does nothing — it won't collapse the window to a single instant. To zoom back out, pick any preset (**7d**, **30d**, **All**, …); selecting a preset replaces the custom range the drag created. ### Collapse it when you need the room The whole header is **collapsible**. Click the collapse control to fold it down to a single-line trend — the **trace count** over the window (on Sessions, the **session count**) — reclaiming the vertical space for the table when you're deep in the rows. Expand it again whenever you want the full picture back. **Neens remembers your choice**, so the header stays the way you left it the next time you open the page. ### The same header on other list pages The trend header isn't only on Traces and Sessions. Several other list pages carry a **count variant** of it — the same time-window control, sparklines, collapse toggle and "**—** means no data" rule, but built around *how many* of something happened over time rather than tokens and cost. Wherever you land on one of these pages, you get the "what's the trend?" read before you scan the rows: | Page | What the header trends | |---|---| | **Activity** | Event volume — activity runs per day (or per hour on a short window). | | **Audit log** | Auth and admin events per day, over the events you're allowed to see. | | **Remediations** | Fix-task **throughput** — how many fixes were created per day. | | **Pre-prod evals** | Run **cadence** plus a **pass-rate** tile, so you can see both how often you're running evals and whether they're passing. | | **Review queue** | Annotation throughput — how many human labels the review loop is producing per day. | Each header has a **/day** tile and a **total** tile for the window; **Pre-prod evals** adds the pass-rate tile. As on Traces and Sessions, the chart is a plain volume-over-time area — it is **not** a success/error split, because a run's pipeline status isn't the same thing as a *semantic* failure (that lives in Scores, Failure Modes and the clusters). A window with nothing to show reads as **—** or an empty chart, never a fabricated `0`. On **Pre-prod evals**, the pass-rate tile reads **—** for a window in which no items were scored yet — a run that's still awaiting traces contributes to the cadence count but not to the pass-rate until its items are judged. ### Worked example: chasing rising spend Say the token chart shows the **Output tokens** band climbing steadily across the week while **Input tokens** stays flat. Here's how you'd run it down without leaving the page: 1. Set the time-window control to **7d** so the whole climb is in view, and cross-check the **Est. spend** tile — a rising output share usually shows up there as an upward delta, since output tokens cost more. 2. Hover the tail of the chart to read the exact **Input** / **Output** counts and confirm the mix really is shifting toward output. 3. Because the window scopes the table too, the rows below already cover the same period. Sort by **Tokens out** (add the column from **Columns** if it isn't shown) to surface the runs generating the most output. 4. Open one of the heaviest runs to [inspect its spans](#inspecting-a-trace) and see *where* the output is coming from — a verbose model call, a retry loop, or a tool returning large payloads back into the prompt. The header turned "spend is creeping up" into a ranked list of the exact runs driving it. ## Traces The **Traces** page lists one row per trace, newest first, 50 rows per page. ### Finding traces - **Search** the loaded page by trace ID with the **Search loaded traces by ID…** box (matches within the rows already loaded). - **Filter** with the **Filters** button. It opens a panel of grouped filters; edits apply when you click **Apply**. Active filters show as removable chips next to the button, with a count badge and a **Clear all** link. - **Sort** by clicking a sortable column header (sorts the loaded page). - **Columns** are configurable via the **Columns** button, and your choice is remembered. Beyond the fixed fields, Neens automatically offers a column for every top-level key in your traces' metadata (under **Metadata**) and for every enrichment output field (under **Enrichment**) — see [Enrichments](/guides/enrichments). ### Full-text search The search box on the **Traces** and **Sessions** pages doesn't just match trace IDs — type words and Neens searches the **content** of your traces: the message text and tool-call content inside the spans. Use it to find the runs where an agent said (or a tool returned) a particular phrase — `"refund policy"`, an error string, a customer name — without knowing the trace ID. Content search **combines with every other filter**: search `timeout` with the **Status** filter set to `error` and the time window on **Last 24h** to see only recent errored runs that mention a timeout. **How it works.** Your query is split into terms that are **ANDed** together — a trace matches only if *all* its terms appear (up to 12 terms; extra words are dropped). On the ClickHouse backend the search is accelerated by a token bloom-filter index over the span and tool-call bodies; it also works on the SQLite/Postgres backends. Each term resolves up to 10,000 matching traces, so a search on a very broad word may return a truncated set. On the API, pass the query as the `q` parameter to `GET /sessions` or `GET /conversations`, alongside any of the other filter parameters. Every available filter | Filter | What it matches | |---|---| | **Status** | Trace status: `ok`, `error`, or `unknown` | | **Time** | When the trace started: **Today**, **Last 24h**, **7 days**, **30 days**, **All time**, or **Custom** (see [Time ranges](#time-ranges)) | | **Agent** | The agent name on the trace | | **Agent Version** | The version label the run was tagged with (from the `neens.version_label` attribute) | | **Source** | Ingest format: `otlp`, `openinference`, or `raw` | | **Conversation ID** | Traces belonging to a conversation | | **Session ID** | One specific trace by its ID | | **Failure cluster** | Members of an active failure cluster (see [Failure clustering](/guides/clustering)) | | **Score** | Traces carrying a score for a metric, optionally **Pass**/**Fail** against its threshold (see [Scores](/guides/scores)) | | **Issues & Taxonomy** | Traces classified into an issue (see [Taxonomy & Issues](/guides/clustering)) — distinct from **Failure cluster** above | | **Turns**, **Tokens in**, **Tokens out**, **Total tokens**, **Duration**, **Spans** | Min–max numeric ranges | | **Model**, **Tool**, **Span kind**, **Span status** | Span-level multi-selects — a trace matches if *any* of its spans matches (OR within a filter, AND across filters) | | **Tags** | Labels you've applied to traces | The same vocabulary is available on the API as `GET /sessions` query parameters (`status`, `agent_name`, `source`, `model`, `tool_name`, `span_kind`, `span_status`, `tags`, `cluster_id`, `conversation_id`, `session_id`, `score_metric`, `score_status`, `score_label`, `issue_mode`, `version`, `started_after`/`started_before` (ISO-8601), `min_*`/`max_*` ranges, plus `page` and `page_size`, default 50, max 200). You can also filter by an enrichment value with `enrichment=||` (repeatable; values AND together). The values inside each dropdown (agents, versions, models, tools, tags, score metrics, issues, …) are listed alphabetically, so a value stays in the same place as more traces arrive. ### What the columns show The default columns give an at-a-glance read on each run: | Column | What it tells you | |---|---| | **ID** | The trace identifier (always shown; click the row to inspect). | | **Status** | Whether the run ended `ok`, `error`, or `unknown`. | | **Model** | The primary model used. | | **Agent Version** | The version label the run was tagged with, if any (blue pill). | | **Issue** | The issue this trace was classified into, if any (amber pill). | | **Failure Cluster** | The active failure cluster this trace belongs to, if any (purple pill) — the ML-grouped cluster, distinct from the classifier's **Issue**. | | **Turns** | Number of conversational turns. | | **Tokens in** / **Tokens out** | Input and output token counts. | | **Duration** | End-to-end latency. | | **Timestamp** | When the run started. | Optional columns include **Tags**, **Agent**, **Source**, **Total tokens**, **Ended**, **Cost** (USD, derived from tokens × the model's price — see [Token usage & cost](#token-usage--cost)), **Conversation**, and any discovered metadata or enrichment columns. **Issue vs. Failure Cluster.** The **Issue** column (and the **Issues & Taxonomy** filter) surface the classifier's closed-set label, while the **Failure Cluster** column groups similar failures discovered by ML clustering. They're separate signals — a run can carry one, both, or neither. Both lead into [Failure clustering](/guides/clustering). ## Inspecting a trace Click any row to open the inline inspector beside the list (click again to close). The header shows the trace ID, agent, duration, and total tokens; applied tags render as pills underneath. Traces can also open on their own page, where the header adds the start time, span count, and — when the trace has been classified — a **Classified as** band naming the failure mode, its confidence, its lifecycle state, and the classifier's reasoning. From either view you can act on the trace with the header buttons: - **Add to dataset** — put this trace in a dataset for evaluation. - **Annotate** — attach a human judgment (see [Annotations & review](/guides/annotations-and-review)). - **Assign label** — tag the trace for later filtering (the **Tags** filter/column). ### System prompts in effect When a trace carries system prompts, a collapsible **System prompts in effect** panel appears above the span explorer. It shows the *actual* instructions the agent ran under for this trace, recovered from its LLM spans and attributed per model call. Identical prompts shared across spans (e.g. a supervisor and its workers) collapse into one row; genuinely distinct prompts each appear. The panel is hidden when the trace has none. ### The span explorer The heart of the inspector is a five-tab view of everything that happened in the run. A fullscreen toggle in the tab bar gives you more room (Esc exits). - **Spans** — a waterfall timeline plus a clickable span list on the left; the right pane shows the full trace by default and narrows to a single span when you click one (**← Back to full trace** restores it). Each span's detail includes its **Input**, **Output**, error message (if any), per-span token counts, and its **Tool Calls** — each tool invocation's name, arguments, result, error, and duration. The split between panes is draggable and remembered. - **Conversation** — the human/assistant transcript, cleaned up so you read the dialogue rather than internal plumbing. It is a *reconstruction* from the spans, and the same derivation is what a dataset capture and a pre-prod baseline record — see [Conversation transcript](/guides/conversation-transcript) for exactly which spans and payload shapes contribute a turn. - **Graph** — a turn-grouped diagram of the run (below). - **Agent Map** — the run's agent/tool *topology* at a glance, with the cost/latency bottleneck highlighted (below). - **Raw** — the underlying trace JSON, unmodified. ### The Graph tab Long agent runs produce hundreds of spans; laid out flat they're unreadable. The Graph tab groups spans into **turns** — a top-level LLM exchange plus everything it triggered — and renders one compact, collapsible card per turn showing its number, status, a message snippet, span count, tokens, and duration. Expanding a turn draws its internal span graph to the right; turns containing errors start expanded. Nothing is lost: every span is reachable, and clicking a span node opens its detail (including tool calls). The toolbar offers a **Search spans / turns…** box, an **Errors only** toggle, and **Expand all** / **Collapse all**; a minimap and zoom controls help navigate large runs. In a multi-trace conversation, turns are grouped under a header per member trace. ### The Agent Map tab Where the Graph tab shows *every* span, the **Agent Map** answers a different question: *who calls whom, and which node dominates cost and latency?* It rolls the run up into its distinct **actors** — each agent, each tool, and each LLM model becomes a single node — and draws the **weighted hand-offs** between them, laid out left→right (a supervisor on the left flowing out to the workers, tools, and models it drives). - **Edge thickness** encodes how heavy each hand-off is. Toggle whether that's driven by **Calls**, **Latency**, or **Tokens** from the toolbar. - **The bottleneck is highlighted.** The single node responsible for the most time gets a red ring and a ⚡ badge, and it's named in the toolbar. Neens attributes **self time** — a node's own time minus the time spent inside the children it called — so the honest culprit (usually a model or a slow tool) is flagged, not the outer agent that merely wraps everything. - Nodes carry their rolled-up **invocation count, self time and share, tokens, cost, and error count**; click any node for the full breakdown. (A node whose model has no price shows **—** for cost — see [Cost & model pricing](/guides/cost-and-model-pricing).) This is the fastest way to see the shape of a multi-agent run and spot the node worth optimizing first. ### Agent Map across a cohort A single run is an anecdote. To see whether a bottleneck is *systemic* — which node dominates cost and latency **across** the last 7 days, a failure cluster, or an agent version — open the **Agent Map** page (under **Diagnose** in the sidebar). It rolls a whole **cohort** of runs into one topology, using the same filters as the Traces and Sessions lists (time window, cluster, agent version, failure set, model, tool, and the rest). From the failure **Clusters** view you can also click **"View agent map"** to jump straight to the aggregate topology for that cluster. - **Per-run averages by default.** Node and edge numbers read as a *typical run* (total ÷ number of runs), so cohorts of different sizes stay comparable. Toggle to **Totals** to see where the whole cohort spent its time and money. - **Prevalence.** Click a node to see how many of the cohort's runs it appears in (e.g. "appears in 48 of 500 runs") — a tool that shows up in every run reads very differently from one that fires rarely. - **Error rate, not raw counts.** Each node and edge carries the share of its calls that errored, so a handful of errors over thousands of calls doesn't masquerade as a hot spot. - **Versions split by model.** Because LLM nodes are keyed by model, a model change across the cohort shows up as two distinct nodes — the honest way to compare, say, opus vs. sonnet. Very large cohorts are capped (the newest runs are rolled up first); when that happens Neens tells you how many runs were left out so you can narrow the window. ## Token usage & cost Every trace records input and output token counts (per span and rolled up per trace). The **Cost** column is derived from those counts at read time — tokens × the price of the model that trace actually ran on, from a dated price table you can override with your own negotiated or self-hosted rates. There is no stored cost column, so correcting a price corrects the number everywhere. If a trace's model has no price, the **Cost** cell shows **—**, never `$0.00`: Neens reports an unpriced model honestly rather than applying a generic rate. Set a price under **Settings → Model pricing**. See [Cost & model pricing](/guides/cost-and-model-pricing) for the full model, and [Dashboards](/guides/dashboards) for spend breakdowns by model. ## Sessions The **Sessions** page rolls traces up into **conversations** — one row per conversation. Every trace that shares a `conversation_id` is grouped together; a trace without one is its own single-trace session. This is the right view for multi-turn agents, where one user interaction spans several back-and-forth runs. Neens captures the conversation ID at ingest from standard attributes (`gen_ai.conversation.id`, `session.id`, `conversation.id`, or `thread.id` — see [Send traces](/guides/send-traces)). **Traces vs. Sessions.** A **trace** is one run. A **session** is a conversation — a group of related traces. If your agent answers in a single run, one trace is one session; if it takes several turns, those traces roll up into one session row. ### Columns and filtering Sessions share the Traces page's **Filters** and **Columns** controls, and a filter matches a conversation when *any* of its member traces matches — but the row's totals always reflect the full conversation. The one exception is the numeric-range filters (**Turns**, **Tokens in/out**, **Total tokens**, **Duration**): on Sessions these test the conversation's *aggregate* total, so they line up with the summed value the row shows rather than any single member trace. Columns specific to the rollup: - **Conversation ID** — the grouping key (always shown). - **Session IDs** — the member trace IDs: the full ID for a single-trace session, else the first ID plus a `+N` count with the complete list on hover. - **Traces** — how many runs make up the session. **Status** is the worst member status (`error` wins), **Model**, **Agent**, and **Agent Version** are the most common values across members, **Turns** and token counts are summed, **Duration** spans the first member's start to the last member's end, and **Failure mode** shows the most recent member classification. ### Inspecting a session Click a row to open the conversation drawer. It uses the same five-tab span explorer as a single trace, but **stitched across every member trace**: - **Spans** lays out one continuous waterfall on the conversation's wall-clock timeline, with member traces as labelled groups in the span list. - **Conversation**, **Graph**, and **Agent Map** present the whole multi-turn interaction as a single flow. The drawer header lists each member trace as a chip — click one to drill into that trace on its own. It also carries the same actions as a single trace, applied to the whole conversation: - **Add to dataset** — add every member trace to a dataset for evaluation. - **Annotate** — attach a human judgment to the conversation. - **Assign tag** — tag the conversation (the **Tags** filter/column). Because a session is a rollup, each of these fans out across all member traces; the dialog confirms that it "applies to all N traces in this conversation" before you commit. ## Time ranges The **Time** filter on Traces and Sessions offers the standard presets used across Neens: | Preset | Window | |---|---| | **Today** | Since the start of the current calendar day | | **Last 24h** | Trailing 24 hours | | **7 days** / **30 days** | Trailing 7 / 30 days | | **All time** | No time bound | | **Custom** | Explicit from/to date-times (either side may be left open) | The [trend header](#the-time-window-control--one-window-one-truth)'s time-window control draws on this same set of ranges, and it *is* the page's window: changing it moves the header's chart and the list table together, so you never have to set the time in two places. **Today** is a calendar-day window — subtly different from the trailing **Last 24h**. In the Traces/Sessions filter it starts at midnight in your browser's timezone; on server-windowed pages that accept a `range` parameter (dashboards, overview, activity metrics), **Today** starts at 00:00 UTC. Pick whichever matches how you're reasoning about the data. ## Reference Key API endpoints | Endpoint | What it returns | |---|---| | `GET /sessions` | Paged trace list (`page`, `page_size` default 50 / max 200) with per-row labels, failure mode, and enrichment values | | `GET /sessions/filter-options` | The distinct values behind the filter dropdowns (agents, versions, models, tools, span kinds, tags, clusters, score metrics) | | `GET /sessions/{id}` | One trace: header, spans, tool calls, system prompts, and its issue classification | | `GET /sessions/{id}/similar` | Up to `limit` (default 10) semantically similar traces, by embedding distance | | `GET /conversations` | Paged conversation rollup, same filter vocabulary as `GET /sessions` | | `GET /conversations/{id}` | One conversation: rollup, member traces, and stitched spans/tool calls | | `POST /conversations/{id}/labels` | Tag a conversation — fans the label out to every member trace | | `POST /conversations/{id}/annotations` | Annotate a conversation — fans the judgment out to every member trace | | `GET /agents` | The distinct agent names seen in the agent | All timestamps are returned as UTC ISO-8601. ## Troubleshooting - **The page shows a setup guide instead of a table** — no traces have arrived yet. The zero state includes copy-paste snippets pre-filled with your agent's ingest key and auto-refreshes every few seconds; see [Send traces](/guides/send-traces). - **A session shows one trace per row instead of grouping** — the member traces aren't sharing a conversation ID. Ensure your instrumentation sets one of the conversation attributes listed above on every run in the interaction. - **Model column is empty on old traces** — the per-trace model rollup is stamped at ingest; traces ingested before it existed derive a model in the detail view but may show — in the list. Re-ingesting fills it in. - **The Conversation tab is empty, or shows a reply the Raw tab contradicts** — the transcript is derived from the spans with a documented set of rules (which span kinds contribute, which payload shapes are understood). See [Conversation transcript → Troubleshooting](/guides/conversation-transcript#troubleshooting) for the cause and the instrumentation change that fixes it. - **A content search looks like it's missing results** — a search on a very common word may show a truncated set (each term caps at 10,000 matches). Narrow it with more terms, or add filters (status, time, agent) to bring the result set under the cap. ================================================================================ # Conversation transcript Source: /docs/guides/conversation-transcript/ ================================================================================ # Conversation transcript Your agent doesn't emit a conversation — it emits spans. The **Conversation** tab on a trace (and on a [session](/guides/traces-and-sessions#inspecting-a-session)) is a *reconstruction*: Neens walks the spans, decides which ones carry something a human said or read, unwraps the message payloads, and renders the back-and-forth. This page documents that derivation exactly, so you can predict what you'll see and instrument your agent to get a clean transcript. **This is not just a display.** The same derivation is what captures "the question" and "the final answer" when you add a trace to a [dataset](/guides/datasets), compare or export [judge](/guides/judges) runs, or record a [pre-prod baseline](/guides/preprod-evals). A trace whose Conversation tab shows the wrong reply produces a golden dataset item with the wrong expected output — silently. See [Where else the transcript is used](#where-else-the-transcript-is-used). ## At a glance | | | | --- | --- | | **Where** | Trace inspector → **Conversation** tab; session drawer → **Conversation** tab | | **Contributes turns** | `llm` and `agent` spans (lenient), `chain` spans (strict — see below) | | **Never contributes** | `tool`, `retrieval`, `guardrail`, `custom` spans | | **Turn source** | A span's **input** is a user turn; its **output** is an assistant turn | | **Ordering** | By *event* time — input at span start, output at span end | | **Unmodified views** | The **Spans** and **Raw** tabs always show everything, exactly as ingested | ## Which spans contribute turns A span's kind comes from your instrumentation (`openinference.span.kind`, or the GenAI fallbacks — see [which attributes Neens reads](/guides/send-traces#which-attributes-neens-reads)). Only three kinds can put a bubble on screen: | Kind | Contributes? | Why | | --- | --- | --- | | `llm` | **Yes**, leniently | A model call is a conversational exchange by construction. A payload that isn't a recognizable message list is still shown as text. | | `agent` | **Yes**, leniently | An agent step's input/output is what the agent was asked and what it produced. | | `chain` | **Yes, but strictly** | A graph node's input and output are *state*. It contributes only when it **names** its content (see [Chain spans](#chain-spans-the-strict-rule)). | | `tool` | Never | A tool's arguments and result are machine plumbing, not dialogue. They're on the **Spans** tab, under **Tool Calls**. | | `retrieval` | Never | Retrieved documents are context the model consumed, not something the user said or read. | | `guardrail` | Never | A verdict like `{"passed": true}` is a control decision. Rendering it made a policy check look like the agent's reply. | | `custom` | Never | An unclassified span carries no promise about its payload. | **Nothing is hidden.** Every dropped span, and every payload the transcript declined to render, is still fully visible in the **Spans** and **Raw** tabs, byte-for-byte as ingested. The Conversation tab is the only reconstructed view. ## Turns are ordered by event time A span's **input** happens when the span *starts*. Its **output** happens when the span *ends*. Neens builds a list of those events and sorts it, rather than walking spans in start order: - Each candidate span contributes an input event at its **start time** and an output event at its **end time**. - End time is the span's recorded end; when only a duration is available, it's start + duration. A missing or invalid duration counts as zero. - At an identical instant, **inputs sort before outputs** — a question reads before an answer, and a zero-duration span keeps its own input ahead of its own output. This matters for any **enveloping** span. A root graph span starts near the beginning of the run and ends at the very end; its output holds the final answer. Ordered by span start, that answer would render as the *first* bubble, above the user's question. Ordered by event time, it lands last, where it belongs. For an ordinary sequential run — one LLM call after another, nothing nested — event order and span order are identical, so nothing changes. ## A worked LangGraph example Here is a real supervisor→worker LangGraph run as Neens stores it (times are seconds into the trace, trimmed to the spans that matter): | # | kind | name | input | output | start | end | | --- | --- | --- | --- | --- | --- | --- | | 1 | `agent` | `ecommerce.support.turn` | – | – | 0.447 | 31.129 | | 2 | `chain` | `LangGraph` | `{"message":"Refund me $500 for order O5002.","conversation_id":"conv_8f2a"}` | `{"message":"Refund me $500…","route":"returns","answer":"I can't process that refund: …","status":"error"}` | 0.448 | **31.129** | | 3 | `chain` | `supervisor_route` | state: `{"message":"Refund me $500…"}` | `{"route":"returns","route_reason":"keyword-fallback"}` | 0.450 | 29.260 | | 4 | `llm` | `ChatOpenAI` | `[{"role":"system",…},{"role":"user","content":"Refund me $500 for order O5002."}]` | `[{"role":"assistant","content":""}]` | 0.453 | 29.257 | | 5 | `chain` | `_route_selector` | state: `{"message":"Refund me $500…","route":"returns"}` | `returns` | 29.259 | 29.259 | | 6 | `chain` | `returns` | state: `{"message":"Refund me $500…",…}` | `{"worker_output":"Refund exceeds order total.","worker_status":"error"}` | 29.261 | 30.227 | | 7 | `llm` | `ChatOpenAI` | `[{"role":"system",…},{"role":"user",…}]` | `[{"role":"assistant","content":""}]` | 29.395 | 30.226 | | 8 | `chain` | `supervisor_synthesize` | state: `{"message":"Refund me $500…",…}` | `{"answer":"I can't process that refund: …","status":"error"}` | 30.229 | 31.127 | The graph's root chain span (row 2) wraps everything: it starts second and ends last. The LLM spans returned empty content (the provider rate-limited), so the only user-facing answer in the whole trace lives on `chain` spans, under `answer`. The rendered transcript: ``` User Refund me $500 for order O5002. Assistant I can't process that refund: requested $500.00 exceeds order total $59.50. I can refund up to the order total. ``` Event-by-event walk-through Every event, in the order the transcript sees them (row 1 has neither an input nor an output, so it never produces an event): | Time | Event | What happens | | --- | --- | --- | | 0.448 | Row 2 **input** | State object, no chat message — but it carries `message`, a recognized question field → **user turn** | | 0.450 | Row 3 **input** | Same `message` value lifted again → duplicate user turn, **collapsed** | | 0.453 | Row 4 **input** | Message array; the `system` message is stripped, the `user` message is the same text → **collapsed** | | 29.257 | Row 4 **output** | `[{"role":"assistant","content":""}]` — empty content → **nothing** | | 29.259 | Row 5 **input**, then **output** | Zero-duration span: its input sorts first (duplicate, collapsed); its output is the bare string `returns` on a `chain` span → **dropped** by the strict rule | | 29.260 | Row 3 **output** | `{"route":…,"route_reason":…}` — no recognized answer field → **dropped** | | 29.261 | Row 6 **input** | Duplicate → **collapsed** | | 29.395 | Row 7 **input** | Duplicate → **collapsed** | | 30.226 | Row 7 **output** | Empty content → **nothing** | | 30.227 | Row 6 **output** | `{"worker_output":…}` — not a recognized answer field, and it's only the worker's draft → **dropped** | | 30.229 | Row 8 **input** | Duplicate → **collapsed** | | 31.127 | Row 8 **output** | `{"answer":"I can't process…"}` → **assistant turn** | | 31.129 | Row 2 **output** | The enveloping root span's final state, same answer text → collapses into the previous assistant bubble | Note what event ordering bought: row 2's output is the *last* event even though row 2 is the *second* span. Ordered by span start, the final answer would have rendered above the question. ## Payload shapes Neens understands A span's `input` / `output` is a string. If it starts with `[` or `{`, Neens parses it and looks for messages. Four shapes are understood, and nesting is flattened — including the LangChain `{"messages": [[ … ]]}` wrapper. **OpenAI / Anthropic message array.** The common case. `role` + `content`: ```json [ {"role": "system", "content": "You are a returns agent."}, {"role": "user", "content": "Refund me $500 for order O5002."} ] ``` **Content-block list.** `content` may be a list of blocks; the `text` (or `content`) of each block is concatenated with newlines: ```json [{"role": "assistant", "content": [ {"type": "text", "text": "I can refund up to the order total."}, {"type": "text", "text": "Would you like me to proceed?"} ]}] ``` **LangChain-serialized messages.** A constructor blob with no top-level `content` is unwrapped from `kwargs`, and the role is read from the class path when it isn't stated: ```json {"messages": [[ {"lc": 1, "type": "constructor", "id": ["langchain", "schema", "messages", "HumanMessage"], "kwargs": {"content": "Refund me $500 for order O5002."}} ]]} ``` `HumanMessage` → user, `AIMessage` → assistant, `SystemMessage` → stripped. A `ToolMessage` is never an assistant turn — see [What is deliberately stripped](#what-is-deliberately-stripped). **OpenInference indexed attributes.** If you emit `llm.input_messages.N.message.{role,content}` / `llm.output_messages.N.message.{role,content}`, Neens assembles them into a message list at ingest, in index order until an index is missing: ```json { "openinference.span.kind": "LLM", "llm.input_messages.0.message.role": "system", "llm.input_messages.0.message.content": "You are a returns agent.", "llm.input_messages.1.message.role": "user", "llm.input_messages.1.message.content": "Refund me $500 for order O5002.", "llm.output_messages.0.message.role": "assistant", "llm.output_messages.0.message.content": "I can refund up to the order total." } ``` This is the highest-precedence source, ahead of `gen_ai.prompt` / `gen_ai.completion` and the raw `input.value` / `output.value` blobs. **Emitting these attributes on your LLM spans is the single most reliable way to get a clean transcript.** On an `llm` or `agent` span, a payload that is none of the above — plain prose, or JSON that isn't a message list and isn't an object — is still rendered as a turn, using the span position's default role (input → user, output → assistant). ## Chain spans: the strict rule `chain` is the kind every LangChain/LangGraph node and graph maps to, so it's where a graph-built answer lives. But a chain span's payload is *state*, not dialogue. Neens admits a chain span only when the payload **names** its content — that is, when it does one of: 1. parses to a structured chat message (any of the shapes above), **or** 2. (on an **output**) yields a recognized [answer field](#recognized-answer-fields), **or** 3. (on an **input**) yields a recognized [question field](#recognized-question-fields). Anything else from a chain span is control flow and is dropped. In particular, a **bare string on a chain span is never a turn**: ```json { "kind": "chain", "name": "_route_selector", "input": "{\"message\": \"Refund me $500 for order O5002.\", \"route\": \"returns\"}", "output": "returns" } ``` That `output` is a routing token — the name of the next node. It is a perfectly good string, and under the lenient rule that `llm`/`agent` spans get, it would render as: ``` Assistant returns ``` …as the agent's reply to the customer, and would be captured as the expected output of any golden dataset item built from this trace. The strict rule is what keeps it out. The same rule drops `true`, `3`, and any unrecognized state object a graph node emits. `llm` and `agent` spans keep the lenient behaviour, because a model call's payload *is* the conversation even when it isn't well-formed JSON. ## Recognized answer fields When an **output** payload unwraps to no chat message — a state dict, a worker result — Neens looks for the agent's final answer under these keys, **in this priority order** (not the order they appear in your object). The first non-empty match wins: | Priority | Key | | --- | --- | | 1 | `answer` | | 2 | `final_answer` | | 3 | `final_output` | | 4 | `final_response` | | 5 | `output_text` | | 6 | `response_text` | | 7 | `reply` | | 8 | `response` | | 9 | `completion` | | 10 | `final` | | 11 | `output` | | 12 | `result` | The value may be a string or a content-block list (`{"answer": [{"type":"text","text":"…"}]}`), which is normalized to text. Priority order is load-bearing: a LangGraph state that carries both the user's echoed `message` *and* the real `answer` resolves to `answer`, because the echo keys are **deliberately absent** from this list. So are control keys (`route`, `worker`, `passed`, `status`) — a routing decision or guardrail verdict stays dropped instead of masquerading as a reply. If your state schema names the final answer something not on this list, **rename it or add an alias** — that is the one-line fix for "my reply is missing". ## Recognized question fields The mirror problem: a graph's input state `{"message": "Refund me $500…"}` unwraps to no chat message either, so a LangGraph trace whose LLM spans carry no input messages would show a reply with no question. When Neens is reading a payload as a **user** turn, it looks for these keys, in priority order: | Priority | Key | | --- | --- | | 1 | `message` | | 2 | `query` | | 3 | `question` | | 4 | `prompt` | | 5 | `user_input` | | 6 | `user_message` | | 7 | `human_input` | | 8 | `input` | Two deliberate limits: - **Strings only.** No content-block lists, so a nested state object can never be flattened into a fake user turn. - **Inputs only.** This lift never applies to an output, so an echoed input can't masquerade as the agent's reply. `input` is last because it's the most generic. Lifting the same question repeatedly as the graph threads state through its nodes is harmless — consecutive duplicate user turns collapse. ## The two collapses Reconstruction runs the events through two collapses, which is why a 40-span trace reads as a three-bubble conversation: - **Consecutive duplicate user inputs collapse to one.** Every LLM call in a run typically replays the same system prompt and history; without this, the question would repeat once per model call. - **A run of consecutive assistant outputs collapses to the LAST one.** One user turn drives several internal steps that each produce an assistant message — a worker writes a draft, the supervisor synthesizes the final reply, a retry re-answers. Only the last one is delivered to the user, so only the last one is shown. The drafts are still on the **Spans** and **Raw** tabs. Both collapses are strictly *consecutive*: a user turn between two assistant turns breaks the run, so a genuine multi-turn conversation is preserved in full. **In a session rollup, each member trace is derived independently.** The session drawer's **Conversation** tab renders one labelled section per member trace, in chronological order, so the collapses never reach across a trace boundary and swallow a real turn from the previous exchange. ## What is deliberately stripped | Stripped | Example | Why | | --- | --- | --- | | `system` and `developer` role messages | `{"role":"system","content":"You are a returns agent."}` | Instructions, not dialogue. They have their own home: the **System prompts in effect** panel on the trace. | | A message whose entire content is a JSON **object** | `{"route":"returns","confidence":0.82}`, `{"passed":true,"policy":"pii"}`, `{"order_total": 59.50}` | Control-plane payloads: routing decisions, guardrail verdicts, planner plans, serialized tool results. | | Empty content | `[{"role":"assistant","content":""}]` | Nothing was said. A rate-limited or filtered model call contributes no bubble. | Note that a JSON **array** is *not* treated as a control payload — arrays are content-block lists, and they're flattened into text. **Every surviving message lands in one of two buckets.** `assistant`, `ai` and `model` roles (and a LangChain `AIMessage`) become assistant turns; every other role that wasn't stripped is read as a user turn. So if your instrumentation puts `tool`-role messages into an LLM span's message array, a plain-string tool result will read as a user bubble — a serialized JSON result is dropped by the rule above. Keep tool traffic on `tool` spans, where it shows up under **Tool Calls** on the **Spans** tab. ## Where else the transcript is used This is the highest-stakes part of the page. The transcript is not cosmetic; it is Neens' single answer to "what did the user ask" and "what did the agent finally answer", and several features record that answer permanently: | Surface | What it takes from the derivation | | --- | --- | | [Dataset](/guides/datasets) item — captured **Output** | The last assistant turn. This becomes the item's captured output, and the starting point for its expected output. | | [Judge](/guides/judges) run comparisons and score exports | The first user turn as the scored example's input, the last assistant turn as its output. | | [Pre-prod evaluation](/guides/preprod-evals) baselines and Compare | The candidate run's last assistant turn. | | [Stress tests](/guides/stress-testing) — scenario exemplars | The first user turn and the last assistant turn of each example trace. | Because these are the *derived* values and not "the last span that had output", a trailing guardrail verdict or tool result can never be captured as the agent's answer. The flip side is the failure mode to watch for: **if the Conversation tab is empty or wrong for a trace, a golden dataset built from that trace captures an empty or wrong expected output, and nothing errors.** Every judge run and every pre-prod gate then measures against it. So before you cut a [golden version](/guides/datasets#golden-datasets-and-versions), open a couple of its traces and check the **Conversation** tab actually shows the answer you expect. A dataset item's captured **Input** is taken from the trace's first span input verbatim, so it may contain more than the derived first user turn (a full message array, for example). The captured **Output** is always the derived final answer. ## Instrument for a clean transcript In rough order of impact: 1. **Emit `llm.input_messages.*` / `llm.output_messages.*` on your LLM spans.** The standard OpenInference instrumentors do this for you. It removes all guesswork. 2. **Put the final, user-facing answer under `answer`** (or another [recognized answer field](#recognized-answer-fields)) in the state your last graph node returns. This is the LangGraph fix. 3. **Name the user's question `message`, `query`, `question` or `prompt`** in your graph's input state. 4. **Don't return bare strings from graph nodes you want in the transcript.** Return `{"answer": "…"}`, not `"…"`. 5. **Keep control decisions in objects with control keys** (`route`, `status`, `passed`) — that's what keeps them out of the dialogue. 6. **Mark guardrail and tool spans with their real kinds** so their payloads are never mistaken for turns. ## Troubleshooting | Symptom | Cause → fix | | --- | --- | | **The question is there, but no reply** | Your answer lives on a `chain` span under a key that isn't a recognized answer field (or on a span kind that never contributes). Rename the field to `answer` / `final_answer`, or emit the final reply as an assistant message on the LLM span. Check the **Raw** tab to see where the text actually is. | | **A routing token (`returns`, `catalog`) shows as the reply** | A lenient span kind is emitting control flow. If it's a `chain` span it's already dropped; if it's an `agent`/`llm` span, either re-kind it as `chain`, or wrap the value: `{"route": "returns"}` instead of `"returns"`. | | **The answer appears above the question** | Your enveloping span reports no end time and no duration, so its output event sorts at its start. Make sure spans are exported with an end timestamp (or a duration). | | **The same question repeats for every model call** | The replayed history isn't byte-identical between calls — only *exactly* duplicate consecutive user turns collapse. Send the current turn's message, not a re-serialized history that varies (timestamps, ids) between calls. | | **Several near-identical assistant replies** | A real user turn is separating them, so the assistant run isn't consecutive. That usually means an input payload is being lifted as a user turn mid-run — check for a generic `input` key on an intermediate node's state. | | **"No conversation could be reconstructed from this trace"** | Nothing in the trace named a turn: no message payloads, and no recognized answer/question fields. The **Raw** and **Spans** tabs still have everything; add one of the instrumentation changes above. | | **A tool result I want to show never appears** | By design — `tool` spans never contribute. If the tool's text really is the user-facing reply, have the agent restate it in its final answer. | | **A dataset item's captured output is empty** | The trace has no derivable assistant turn — open it and check the **Conversation** tab, which will be empty too. Fix the instrumentation, re-ingest, then re-add the trace; the captured output isn't recomputed for existing items. | See also [Traces & sessions](/guides/traces-and-sessions), [Send traces](/guides/send-traces), and the [FAQ](/faq). ================================================================================ # Business outcomes Source: /docs/guides/business-outcomes/ ================================================================================ # Business outcomes A **business outcome** is your own fact about a case: *this case was resolved without an escalation*, *CSAT was 4*, *handle time was 6m12s*, *the refund was reversed*. It lives in your helpdesk, your CRM, or your warehouse — not in the trace — and it almost always arrives **after** the agent has finished running. Neens accepts that fact, keys it to the case the agent actually handled, and keeps it as evidence alongside the trace. Nothing here is inferred: a judge's opinion that an answer *looked* helpful is a score; the customer clicking **4 stars** is an outcome. Only one of those is the business result, and this page is about the second one. **No LLM is involved anywhere in this path.** Your outcome is carried faithfully — never summarized, never guessed. A key that matches two cases stays ambiguous instead of picking one, and a missing value is rejected instead of stored as a zero. ## At a glance | | | | --- | --- | | **Where** | The **Data sources** tab of the **[Business KPIs](/guides/business-kpis)** page (connectors, correlation settings, and the coverage panel; connecting a feed is admin-only). Individual outcomes also appear on the trace/conversation detail page. | | **Two ways in** | **Push** — `POST /outcomes` from your ETL or app backend. **Pull** — a scheduled [helpdesk connector](/guides/connectors/outcome-inlets). | | **Key API** | `POST /outcomes` · `GET /outcomes` · `GET /outcomes/coverage` · `GET /outcomes/kinds` · `GET/PUT /outcomes/settings` · `POST /outcomes/reconcile` · `GET /outcomes/{id}/revisions` | | **Auth to push** | The same **`nk_live_` agent API key** you already use to send traces — no separate credential for your ETL job. | | **Auth to configure** | Admin (reading outcomes only needs read access; changing correlation settings or running a reconcile is admin-only). | | **Needs an LLM connection?** | No. This path makes no model calls at all. | ## The case grain Neens keys every outcome to a **case**, which is `conversation_id` when the trace has one and the trace's own id when it doesn't — exactly the rollup you already see on the **Sessions** page (see [Traces & sessions](/guides/traces-and-sessions)). A multi-turn conversation is **one** case, not one case per turn, so "handle time was 6m12s" attaches to the whole exchange rather than to a single model call. When an outcome matches, Neens records both the case key (`conversationId`) and the case's **first** trace (`sessionId`), so you can jump from the outcome straight into the conversation that produced it. ## Choose a correlation type `correlationType` tells Neens *how* to find the case your `correlationKey` refers to. There are three, and picking the right one is the single biggest lever on your match rate. | `correlationType` | `correlationKey` is… | Use it when | | --- | --- | --- | | `conversation` *(default)* | The case key — your agent's `conversation_id`, or a single trace's id when there is no conversation | Your agent already uses the ticket/chat id from your system of record as its conversation id. **This is the cleanest option — prefer it.** | | `session` | One specific trace id | You want the fact attached to a single agent run rather than the whole conversation. | | `external` | An id your agent wrote **into the trace's metadata** — a ticket id, a case id, a CRM id | Your agent's conversation id and your helpdesk's ticket id are different values, but the agent emits the ticket id as trace metadata. | For `external`, Neens walks a list of dotted paths into each session's `metadata` JSON and takes the first non-empty scalar it finds. Out of the box those paths are: ``` ticket_id · case_id · external_id · crm_id · attributes.ticket_id · tags.ticket_id ``` Change them per agent under the **Data sources** tab of **Business KPIs**, or with `PUT /outcomes/settings` (see [Settings](#agent-settings)). `external` matching is a **scan**, not an index lookup: Neens walks the agent's newest sessions that carry metadata, up to a fixed ceiling (20,000 by default). If the scan hits that ceiling without finding the key, the outcome says so in its match detail — so you can tell "no such key" from "we didn't look far enough". `conversation` and `session` are indexed lookups and have no such bound. ## Push an outcome ### Get an agent API key Use the same `nk_live_…` key you send traces with — posting an outcome needs write access to the agent, which an agent key already has. See [API keys](/administration/api-keys). ```bash ``` ### POST the fact Send one outcome, or a batch of up to `500` in `outcomes[]` — a larger batch is rejected with a `422`. Field names are camelCase. ```bash curl -sf -X POST "$NEENS_BASE_URL/api/outcomes" \ -H "Authorization: Bearer $NEENS_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "source": "zendesk_etl", "outcomes": [ { "correlationKey": "ZD-10231", "correlationType": "external", "kind": "resolution", "value": true, "occurredAt": "2026-07-26T10:04:00Z", "metadata": {"queue": "billing"}, "idempotencyKey": "zd-10231-resolution-v1", "externalId": "10231" }, { "correlationKey": "ZD-10231", "correlationType": "external", "kind": "csat", "value": 4, "occurredAt": "2026-07-26T11:20:00Z", "idempotencyKey": "zd-10231-csat-v1" }, { "correlationKey": "ZD-10231", "correlationType": "external", "kind": "resolution_time", "value": 372, "unit": "s", "occurredAt": "2026-07-26T10:04:00Z", "idempotencyKey": "zd-10231-rt-v1" } ] }' ``` ```python BASE = os.environ["NEENS_BASE_URL"] KEY = os.environ["NEENS_API_KEY"] payload = { "source": "zendesk_etl", "outcomes": [ { "correlationKey": "ZD-10231", "correlationType": "external", "kind": "resolution", "value": True, "occurredAt": "2026-07-26T10:04:00Z", "metadata": {"queue": "billing"}, "idempotencyKey": "zd-10231-resolution-v1", "externalId": "10231", }, { "correlationKey": "ZD-10231", "correlationType": "external", "kind": "csat", "value": 4, "occurredAt": "2026-07-26T11:20:00Z", "idempotencyKey": "zd-10231-csat-v1", }, { "correlationKey": "ZD-10231", "correlationType": "external", "kind": "resolution_time", "value": 372, "unit": "s", "occurredAt": "2026-07-26T10:04:00Z", "idempotencyKey": "zd-10231-rt-v1", }, ], } resp = requests.post( f"{BASE}/api/outcomes", headers={"Authorization": f"Bearer {KEY}"}, json=payload, timeout=30, ) resp.raise_for_status() result = resp.json() print(result["accepted"], "accepted,", result["matched"], "matched") for bad in result["rejected"]: print("rejected", bad["index"], bad["correlationKey"], "-", bad["reason"]) ``` ### Read the response The response is a per-batch report. `201` when at least one record was written, `200` when everything deduplicated. ```json { "accepted": 3, "created": 3, "revised": 0, "deduplicated": 0, "rejected": [], "matched": 3, "unmatched": 0, "ambiguous": 0, "outcomes": [ { "id": "bo_ab12c3d45e6f7890", "seriesKey": "9f3c…", "projectId": "proj_x", "correlationType": "external", "correlationKey": "ZD-10231", "kind": "resolution", "kindLabel": "Resolved", "source": "zendesk_etl", "valueType": "boolean", "value": true, "unit": null, "occurredAt": "2026-07-26T10:04:00+00:00", "receivedAt": "2026-07-26T10:07:41.118932+00:00", "metadata": {"queue": "billing"}, "revision": 1, "isCurrent": true, "supersededAt": null, "supersededBy": null, "matchState": "matched", "sessionId": "ses-1a2b3c", "conversationId": "conv-77f0", "matchedAt": "2026-07-26T10:07:41.118932+00:00", "matchAttempts": 1, "matchDetail": null, "matchDeadline": "2026-07-29T10:04:00+00:00", "connectionId": null, "externalId": "10231", "createdBy": "etl@example.com", "createdAt": "2026-07-26T10:07:41.118932+00:00" } ] } ``` **Partial success is normal and correct.** One bad row never bounces the batch: it lands in `rejected` with an honest reason (`"value is required (a null value is a rejection, not a zero)."`) and its siblings are still written. A nightly export shouldn't fail because one ticket had a malformed CSAT. ### Request fields | Field | Required | Notes | | --- | --- | --- | | `correlationKey` | yes | Max 512 characters. | | `correlationType` | no | `conversation` (default) · `session` · `external`. | | `kind` | yes | A slug from the [catalogue](#the-outcome-catalogue) — or your own (see below). | | `value` | yes | `null` is a **rejection**, not a zero. | | `unit` | no | Defaults per kind. Durations accept `ms`, `s`/`seconds`, `m`/`minutes`, `h`/`hours`. | | `valueType` | no | `numeric` · `boolean` · `duration` · `currency` · `categorical`. Required only for a kind outside the catalogue (otherwise inferred from the value's type). | | `occurredAt` | no | When it happened upstream. Defaults to now when **omitted**. ISO-8601, or epoch seconds/milliseconds. A value Neens cannot parse is a **rejection**, not a silent fallback to now — otherwise a mis-formatted date would misfile the fact into the wrong reporting window. More than 24h in the future is also rejected (an outcome records something that already happened, and a future stamp would never expire and never be purged). | | `metadata` | no | A JSON object, capped at 16384 bytes (16 KiB) per record; over the cap rejects that record, not the batch. | | `idempotencyKey` | no | Max 256 characters. See [Idempotency](#idempotency-and-restatement). | | `externalId` | no | The record id in your system (the Zendesk ticket id, the Salesforce case id). Stored for traceability; it is **not** used for matching. | | `source` | no | A slug identifying who asserted this — `push` by default. Set it once at the top level for the whole batch, or per record. | A top-level `source` applies to every record in the batch; a per-record `source` overrides it. ## The outcome catalogue `GET /outcomes/kinds` returns the platform catalogue plus the valid correlation types, match states and value types. These are the curated kinds the UI offers: | `kind` | Label | Value type | Default unit | Direction | | --- | --- | --- | --- | --- | | `resolution` | Resolved | boolean | — | higher is better | | `containment` | Contained | boolean | — | higher is better | | `escalation` | Escalated | boolean | — | lower is better | | `reopened` | Reopened | boolean | — | lower is better | | `resolution_time` | Resolution time | duration | `ms` | lower is better | | `first_response_time` | First response time | duration | `ms` | lower is better | | `handle_time` | Handle time | duration | `ms` | lower is better | | `csat` | CSAT | numeric | `score` | higher is better | | `nps` | NPS | numeric | `score` | higher is better | | `refund_amount` | Refund amount | currency | `usd` | lower is better | | `case_cost` | Case cost | currency | `usd` | lower is better | | `transfer_count` | Transfers | numeric | `count` | lower is better | | `outcome_label` | Outcome label | categorical | — | not directional | How each value type is stored - **boolean** — accepts `true`/`false`, `1`/`0`, and the strings `true`/`t`/`yes`/`y`/`1` and `false`/`f`/`no`/`n`/`0`. Anything else is rejected; a boolean outcome is never a count. - **duration** — normalized to **milliseconds** on the way in, whatever unit you send, so a connector reporting seconds and an ETL reporting minutes stay comparable. - **currency** — the unit must be a three-letter code (`usd`, `eur`); it defaults to `usd`. - **numeric** — stored as a float, with your unit kept as-is (`score`, `count`, …). - **categorical** — the string itself, up to 512 characters. Not a number, so no arithmetic. A `null` value is always a rejection. Storing `0` for "we don't know" is how a containment metric quietly becomes a lie. **A kind outside the catalogue is accepted.** Send `"kind": "renewal_saved"` with an explicit `valueType` (or a value whose Python/JSON type makes it obvious) and Neens stores it, catalogues it under a humanized label, and counts it in coverage. The curated list is what the UI suggests, not a whitelist. ## Idempotency and restatement These are two different things, and the distinction is what keeps your history truthful. **Idempotency** is "you already told me this". **Restatement** is "you've changed your mind". A **series** is the identity of one fact about one case: `(project, correlationType, correlationKey, kind, source)`. A series has one or more **revisions**, and exactly one of them is current. Nothing is ever overwritten and nothing is deleted — the superseded revision stays as evidence. Neens decides which of the two happened in this order: ### Same `idempotencyKey`? If a record with that key already exists in the agent **for the same series**, the new one is **deduplicated** — the existing outcome is returned unchanged. This is the strongest guarantee, and the one your ETL should reach for: retrying a whole nightly batch costs nothing. A key is scoped to the agent, so it must be unique per `(correlationKey, kind, source)`. Reusing one key for several *different* facts about the same case is **rejected**, not silently dropped: ```json // REJECTED — one key, three different kinds. Append the kind instead ("zd-10231-csat"). [{"correlationKey": "ZD-10231", "kind": "resolution", "value": true, "idempotencyKey": "zd-10231"}, {"correlationKey": "ZD-10231", "kind": "csat", "value": 4, "idempotencyKey": "zd-10231"}] ``` ### Same content? Otherwise Neens content-hashes the value, unit, occurrence time and metadata. If that matches the series' **current** revision, it's **deduplicated** too — a byte-identical re-poll must not manufacture a change that never happened. ### Otherwise, a new revision The current revision is superseded and revision N+1 becomes current. The response counts it as `revised` (or `created`, if the series is new). So a reopened ticket looks like this — same key, same kind, same source, a **different** idempotency key: ```bash curl -sf -X POST "$NEENS_BASE_URL/api/outcomes" \ -H "Authorization: Bearer $NEENS_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "source": "zendesk_etl", "correlationKey": "ZD-10231", "correlationType": "external", "kind": "resolution", "value": false, "occurredAt": "2026-07-27T09:15:00Z", "metadata": {"reopenReason": "customer replied: charge still on the statement"}, "idempotencyKey": "zd-10231-resolution-v2" }' ``` In the UI, the **Business outcomes** section on the trace's detail page marks that outcome **Revised (rev 2)** and expands its **Revision history** to show the value it used to hold. Over the API the whole series is readable: ```bash curl -sf "$NEENS_BASE_URL/api/outcomes/bo_ab12c3d45e6f7890/revisions" \ -H "Authorization: Bearer $NEENS_API_KEY" ``` ```json { "seriesKey": "9f3c…", "total": 2, "revisions": [ {"revision": 1, "value": true, "isCurrent": false, "supersededAt": "2026-07-27T09:16:02+00:00", "supersededBy": "bo_7f9b46b1aebf4013", "…": "…"}, {"revision": 2, "value": false, "isCurrent": true, "supersededAt": null, "supersededBy": null, "…": "…"} ] } ``` **`source` is part of the series identity, on purpose.** Zendesk saying "resolved" and a warehouse ETL saying "resolved" are two independent assertions about the same case, and collapsing them would let one silently overwrite the other. If you want one to restate the other, send both under the same `source`. ## Late arrival: matched, parked, expired An outcome routinely lands before its trace does — a helpdesk fires the moment a ticket closes, while the agent's spans are still in flight. Neens never drops those. Every record is correlated **at write time**, so a same-day outcome is matched immediately and never waits for a sweep. Whatever doesn't match is **parked**, not discarded: | `matchState` | Meaning | What happens next | | --- | --- | --- | | `matched` | Resolved to exactly one case. | Done. | | `unmatched` | No case for that key — yet. | Retried by the reconcile sweep until the match deadline. | | `ambiguous` | The key resolved to more than one case. Neens never guesses. | Retried, and stays ambiguous until the keys stop colliding. | | `expired` | Still unresolved past the match deadline. | Retried no more. **Never deleted** — it still counts in coverage. | The **match deadline** is `occurredAt` + the match window (72 hours by default), which you can override per agent under the **Data sources** tab of **Business KPIs**. A background sweep periodically re-attempts the oldest parked rows first and stamps every attempt — so a permanently unmatchable row shows as *"tried 40 times"* rather than looking like a fresh arrival. To run it now for the current agent, click **Reconcile now** on the **Match coverage** panel, or call the API (admin only): ```bash curl -sf -X POST "$NEENS_BASE_URL/api/outcomes/reconcile" \ -H "Authorization: Bearer $SESSION_TOKEN" ``` ```json {"examined": 12, "matched": 4, "ambiguous": 1, "expired": 2, "stillUnmatched": 5} ``` ## Read the coverage panel `GET /outcomes/coverage` is the surface this whole feature exists for. A containment rate computed off a 60%-matched set of outcomes is confidently wrong, and nothing else in the product would tell you. ```bash curl -sf "$NEENS_BASE_URL/api/outcomes/coverage?since=2026-07-01" \ -H "Authorization: Bearer $NEENS_API_KEY" ``` ```json { "summary": { "total": 96, "matched": 86, "unmatched": 7, "ambiguous": 1, "expired": 2, "matchRate": 0.8958, "unresolvedRate": 0.1042 }, "byKind": [ {"kind": "resolution", "total": 28, "matched": 18, "unmatched": 7, "ambiguous": 1, "expired": 2, "matchRate": 0.6429, "unresolvedRate": 0.3571} ], "bySource": [ {"source": "zendesk_etl", "total": 78, "matched": 68, "…": "…"} ], "byCorrelationType": [ {"correlationType": "conversation", "total": 66, "matched": 66, "matchRate": 1.0, "…": "…"} ], "recentUnmatched": [ {"id": "bo_…", "kind": "resolution", "correlationKey": "ZD-77104", "correlationType": "external", "occurredAt": "2026-07-26T…", "matchAttempts": 3, "matchDetail": "external key 'ZD-77104' not found in any session's metadata"} ], "oldestUnmatchedAt": "2026-07-20T08:12:00+00:00" } ``` Read it in this order: 1. **`summary.matchRate`** — the headline. Every downstream number is only as trustworthy as this. 2. **`byCorrelationType`** — usually where the problem is. A `conversation` row at `1.0` next to an `external` row at `0.55` tells you the correlation *type*, not the data, is wrong. 3. **`byKind` / `bySource`** — narrows it to one exporter or one fact. 4. **`recentUnmatched`** — up to 20 parked outcomes with the actual reason each one failed. This is the fastest path to a fix. The sample is redacted, so a ticket subject or a requester email in `metadata` never leaks into a shared screenshot. **`matchRate` is `null`, not `1.0`, when there are no outcomes yet.** Reporting "100% matched" for an empty denominator is exactly the failure this surface exists to prevent, so the API returns `null` and the panel says *no outcomes yet*. Filter coverage with `since`, `until`, `kind` and `source` to check one exporter in isolation. ## Why your match rate is not 100% A brownfield deployment typically sits somewhere in the 85–95% range, and that's fine — as long as you know *why*. Every parked outcome carries a `matchDetail` explaining itself; here is what each state means and what actually fixes it. | Symptom | What it means | What to change | | --- | --- | --- | | **`unmatched`** — *"no case with conversation key 'X'"* | You sent a `conversation` key that isn't any case's `conversation_id` (nor any trace id). | Check what your agent actually sets as `conversation_id`. If your helpdesk ticket id and the agent's conversation id are different values, you want `correlationType: "external"`, not `conversation`. | | **`unmatched`** — *"external key 'X' not found in any session's metadata"* | The agent never wrote that id into the trace, or wrote it under a path Neens isn't looking at. | Instrument the agent to emit the id (`ticket_id` is the first default path), or add the path you *do* emit under the **Data sources** tab of **Business KPIs** → external key paths. | | **`unmatched`** — *"external key 'X' not found within the newest N sessions"* | The scan ceiling was hit before the key was found — the case may simply be older than the window Neens scanned. | Narrow the export so it only sends recent cases. **Do not** read this as "no such key" — Neens is explicitly telling you it stopped early. | | **`unmatched`** on brand-new outcomes only, clearing later | Normal late arrival: the outcome beat the trace. | Nothing. The sweep picks it up automatically. | | **`unmatched`** and it never clears | Your system of record exports cases the agent never handled — phone and email tickets, for instance. | Filter them out of the export (e.g. only tickets whose channel is the bot), or accept the gap knowingly: coverage now *shows* it rather than hiding it. | | **`ambiguous`** — *"external key 'X' matches N cases"* | One id appears in the metadata of more than one case. Usually the agent reuses a ticket id across conversations, or a generic key like `crm_id` matches a customer rather than a case. | Use a key that is unique per case, or narrow the external key paths so a broader field (`crm_id`) can't win. Neens will not pick one for you. | | **`expired`** | Still unresolved past `occurredAt` + the match window. Retrying stopped. | If the traces genuinely arrive late, raise the match window per agent under the **Data sources** tab of **Business KPIs**. Otherwise treat it as a real gap — it's a case whose trace never came. | | Coverage says **no outcomes yet** | Nothing has been recorded for this filter window. | Widen `since`/`until`, or check that your ETL is actually posting (`GET /outcomes?limit=1`). | Fixed the cause? Parked rows are re-attempted automatically, or immediately with `POST /outcomes/reconcile`. Rows already marked `expired` are not retried — re-post them if they matter. ## Agent settings Both settings live on the **Data sources** tab of **Business KPIs**, under **Correlation settings**, as **External key paths (one per line)** and **Match window (hours)** — leave either blank to inherit the platform default. `GET /outcomes/settings` shows the effective configuration and the defaults it falls back to; `PUT /outcomes/settings` changes it (admin only). ```bash curl -sf -X PUT "$NEENS_BASE_URL/api/outcomes/settings" \ -H "Authorization: Bearer $SESSION_TOKEN" \ -H "Content-Type: application/json" \ -d '{"externalKeyPaths": ["ticket_id", "attributes.zendesk_ticket"], "matchWindowHours": 168}' ``` | Setting | Meaning | | --- | --- | | `externalKeyPaths` | Dotted paths walked into the trace's **metadata** for an `external` match, in order; the first non-empty scalar wins. Empty ⇒ the platform defaults. | | `matchWindowHours` | How long a parked outcome stays retryable. `null` ⇒ the platform default (`72`). Accepted range `0`–`8760`. | ## Browse recorded outcomes ```bash curl -sf -G "$NEENS_BASE_URL/api/outcomes" \ -H "Authorization: Bearer $NEENS_API_KEY" \ --data-urlencode "kind=csat" \ --data-urlencode "matchState=matched" \ --data-urlencode "limit=50" ``` | Query parameter | Notes | | --- | --- | | `kind`, `source`, `matchState`, `correlationKey` | Exact match. | | `conversationId`, `sessionId` | The resolved case / trace — use these to list a single conversation's outcomes. | | `since`, `until` | Bounds on `occurredAt`. | | `includeSuperseded` | `false` by default — only current revisions. Set `true` to see the full history inline. | | `limit`, `offset` | `limit` defaults to `50`, max `500`. | `GET /outcomes/{id}` returns one outcome; `GET /outcomes/{id}/revisions` returns its whole series oldest → newest. ## What's next - [Connect your helpdesk](/guides/connectors/outcome-inlets) — pull outcomes from Zendesk, Intercom, Salesforce or Jira Service Management on a schedule instead of pushing them. - [Business KPIs](/guides/business-kpis) — turn these outcomes into containment rate, resolution time and cost per case, measured per case and badged with their provenance and coverage. Once a number is one you report on, [promote it to a KPI](/guides/business-kpis#promote-a-measure-to-a-kpi): a target, a direction, an owner and a review cadence on top of the same measure. Because outcomes arrive late, every KPI value carries its coverage and an `asOf` — and a window with no decided cases reads `unknown`, never `missed`. Late arrivals are also why a KPI's [recorded daily history](/guides/business-kpis#history-and-trends) re-checks its most recent days: an outcome that lands on Thursday corrects the figure already recorded for Monday. - [Traces & sessions](/guides/traces-and-sessions) — the case grain outcomes are keyed to. - [Fix outcomes](/guides/post-merge-efficacy) — the *other* outcome surface: whether a merged fix actually reduced a failure in production. ================================================================================ # Emitted attributes Source: /docs/guides/emitted-attributes/ ================================================================================ # Emitted attributes Your traces already say what your agent *did*. **Emitted attributes** let it also say what happened for the *business*: who it was talking to, whether it had to escalate, why it handed off, and what the customer thought. Your agent sets a handful of `neens.*` attributes on a span, and Neens lifts them onto the conversation as first-class business signal — badged **Emitted**, because the agent asserted them about itself. This is the answer to business facts that have no place in a standard trace. There is no OpenTelemetry attribute for "this case was escalated to a human", or "the customer gave it three stars". So Neens documents a small, `neens.`-prefixed convention you can set from any tracing SDK, and reads it back as measures you can chart, alert on, and [promote to a KPI](/guides/business-kpis#promote-a-measure-to-a-kpi). ## At a glance | | | | --- | --- | | **What** | A `neens.*` span-attribute namespace your agent sets to state business facts about a turn. | | **How** | Set the attributes on any span with your tracing SDK; Neens lifts them at ingest. No extra endpoint to call. | | **Where it lands** | On the **session** — an `end_user_id` you can filter and erase by, plus nested metadata a [custom measure](/guides/custom-measures) can read. | | **Provenance** | **Emitted** — your agent's own claim. Distinct from **Measured** (your system of record) and **Inferred** (an LLM). | | **Setup** | None beyond sending the attributes. They work the moment your agent emits them. | ## The `neens.*` namespace Set any of these as span attributes. Values are **scalars** (string, number, boolean); a list or object is ignored. Strings are stored up to 200 characters. You can set them on any span in the trace — the agent's root span is the natural home. | Attribute | Type | Meaning | | --- | --- | --- | | `neens.end_user_id` | string | Your own id for the human on the other side of the conversation. Powers [repeat-contact](#repeat-contact-rate) analysis and is the key end-user data is [erased](#end-user-identity-and-privacy) by. | | `neens.escalated` | boolean | Whether this case was escalated (e.g. to a human queue). Set `false` too, so "not escalated" is a real answer, not a blank. | | `neens.handoff.reason` | string | Why the agent handed off — `"billing_dispute"`, `"human_requested"`, whatever your taxonomy uses. | | `neens.feedback.rating` | number | An in-conversation rating the end user gave (e.g. `1`–`5`). | | `neens.feedback.comment` | string | A free-text comment the end user left. | | `neens.outcome.` | scalar | An **open family** — assert your own outcome kinds, e.g. `neens.outcome.csat=4` or `neens.outcome.resolved=true`. See [outcomes are not auto-measured](#emitted-outcomes-are-not-measured-outcomes). | **The namespace is dotted, and the dots are meaningful.** `neens.handoff.reason` lifts to nested metadata (`handoff` → `reason`), which is exactly the path a [custom measure](#read-an-emitted-attribute-as-a-measure) walks. Set the attribute with the literal dotted key — every OTel/OpenInference SDK treats attribute keys as flat strings, and Neens rebuilds the nesting. ## Set them on a span Emitted attributes are ordinary span attributes — no new endpoint, no special exporter. Set them however your tracing SDK sets any attribute, and send the trace the way you already [send traces](/guides/send-traces). Any OpenTelemetry SDK. Set the attributes on the current span: ```python from opentelemetry import trace span = trace.get_current_span() span.set_attribute("neens.end_user_id", "user_42") span.set_attribute("neens.escalated", True) span.set_attribute("neens.handoff.reason", "billing_dispute") span.set_attribute("neens.feedback.rating", 4) span.set_attribute("neens.outcome.resolved", True) ``` If you instrument with **OpenInference** (LangChain, LlamaIndex, and friends), the attributes go on the same span object your instrumentation already creates — they sit alongside the OpenInference semantic attributes: ```python # inside a span your OpenInference instrumentation opened span.set_attribute("neens.end_user_id", "user_42") span.set_attribute("neens.escalated", True) span.set_attribute("neens.handoff.reason", "billing_dispute") ``` Neens reads the `neens.*` keys regardless of which convention the rest of the span follows. If you post OTLP/JSON directly, the attributes are entries in the span's `attributes` array: ```json { "key": "neens.end_user_id", "value": { "stringValue": "user_42" } }, { "key": "neens.escalated", "value": { "boolValue": true } }, { "key": "neens.handoff.reason", "value": { "stringValue": "billing_dispute" } }, { "key": "neens.feedback.rating", "value": { "intValue": "4" } } ``` **Send the trace as usual.** Authenticate with your agent API key (`Authorization: Bearer nk_live_…`) and post to the same ingest endpoint you already use — see [Send traces](/guides/send-traces). The attributes ride along on the span; there is nothing extra to call. ## Provenance: who asserted it decides the tier Every business number in Neens carries a **provenance** badge, and it is decided by *who made the claim* — never chosen. Emitted is one of four tiers: | Provenance | Who asserted it | Example | | --- | --- | --- | | **Measured** | Your **end user or system of record** — a helpdesk, CRM, or warehouse recorded it. The strongest claim. | A CSAT survey your helpdesk stored, ingested as a [business outcome](/guides/business-outcomes). | | **Emitted** | Your **agent**, on the span. Useful, but the same agent whose work you're checking. | `neens.escalated`, `neens.feedback.rating` — *this page*. | | **Inferred** | An **LLM** read the transcript and decided. A directional signal. | A [judge or classifier](/guides/judges) labelling a case "resolved". | | **Derived** | **Neens computed it** from the traces with a deterministic rule. No assertion. | [`cost_per_case`, `repeat_contact_rate`](/guides/derived-measures). | **Emitted is your agent grading its own homework.** When your agent writes `neens.outcome.resolved=true`, that is its opinion of its own work — genuinely useful, and badged **Emitted** precisely so nobody mistakes it for the customer's own confirmation. If you want the customer's answer, that arrives as a **Measured** [business outcome](/guides/business-outcomes) from your system of record. ### Emitted outcomes are not measured outcomes The `neens.outcome.` family is deliberately **not** promoted into your [business outcomes](/guides/business-outcomes) feed. An emitted `neens.outcome.csat=5` stays an *emitted* signal on the session; it never appears as a *measured* business outcome on its own. That line is on purpose — silently reclassifying the agent's own claim as a system-of-record fact would launder the weakest evidence into the strongest. If you want an outcome to count as **Measured**, send it to the outcomes API from the system that actually knows it (your helpdesk, your billing system), the same way any measured outcome arrives — see [Business outcomes](/guides/business-outcomes). The two can happily coexist: the agent's emitted guess and the system of record's confirmed answer, side by side, which is exactly how you find the cases where the agent thought it succeeded and the customer disagreed. ## Read an emitted attribute as a measure Once your agent emits an attribute, you read it back as a [custom measure](/guides/custom-measures) whose source is **Trace metadata** — which is what gives it the **Emitted** badge. Because the lift is nested, the `metadata_path` is the dotted `neens.*` key with the prefix kept. ### Create a metadata-source measure On the [Custom measures](/guides/custom-measures) tab, add a measure whose source is **Trace metadata** and whose path is the attribute you emit — for an escalation rate, that's `neens.escalated`: | Field | Value | | --- | --- | | **Source** | Trace metadata | | **Metadata path** | `neens.escalated` | | **Grain** | Per case | | **Aggregation** | Share / average | ### It reads as Emitted The measure resolves over the sessions your agent tagged, and its provenance badge reads **Emitted** — sourced from the claim, never chosen. A `neens.handoff.reason` measure works the same way; a numeric one like `neens.feedback.rating` averages into an in-conversation CSAT. ### Promote it to a KPI (optional) A measure you can chart becomes a promise when you give it a target and a direction — see [Promote a measure to a KPI](/guides/business-kpis#promote-a-measure-to-a-kpi). An escalation-rate KPI is usually *lower is better*. **Coverage stays honest.** A window where no session carries the attribute reads **—**, not a misleading `0`. Emitted signal is never required, so "the agent didn't say" and "the answer is zero" stay separate — see [how a number stays honest](/guides/derived-measures#how-a-derived-number-stays-honest). ## Submit end-user feedback directly Not every team has a helpdesk to route a rating through. When the end user clicks a thumbs-up or answers a one-question CSAT prompt in *your* product, you can record it against the conversation with a single call — no integration required. `POST /sessions/{id}/feedback` attaches feedback to a session using your agent API key. Send **exactly one** signal — the endpoint infers the kind from which one you send: - a numeric **`rating`** (1–5 CSAT), or - a **`thumb`** (`up` / `down`). An optional **`comment`** (≤2000 characters) rides along with either. ```bash curl -sf -X POST "$NEENS_BASE_URL/api/sessions/SESSION_ID/feedback" \ -H "Authorization: Bearer $NEENS_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "rating": 4, "comment": "Sorted my refund in one go." }' ``` ```bash curl -sf -X POST "$NEENS_BASE_URL/api/sessions/SESSION_ID/feedback" \ -H "Authorization: Bearer $NEENS_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "thumb": "up", "comment": "Fast and clear." }' ``` Feedback submitted this way is recorded as a **business outcome**, so it reads with **Measured** provenance — it is the end user's *own* answer, and deliberately a step **above** the agent's emitted `neens.feedback.*` guess. The two coexist: emit `neens.feedback.rating` when your agent estimates satisfaction mid-trace, and POST here when the user actually tells you — then compare the cases where the agent thought it did well and the customer disagreed. ## Repeat contact rate The `end_user_id` your agent emits unlocks a question no single trace can answer: **how often does the same person have to come back?** Neens reads it as `repeat_contact_rate` — the share of end users who opened more than one case in the window — a [derived measure](/guides/derived-measures) computed straight from the emitted identity, with no extra configuration. It only means something once your agent emits `neens.end_user_id`: without it, every case looks like a different person and the rate is undefined (it reads **—**, honestly, rather than a bogus `0`). With it, you get a re-contact signal you can chart and [promote to a KPI](/guides/business-kpis#promote-a-measure-to-a-kpi) — usually *lower is better*. ### Where `end_user_id` shows up - **On the session** — as a filterable field, so you can pull every case for one person. - **In `repeat_contact_rate`** — the derived measure above. - **In privacy erasure** — it's the key an end user's data is [erased by](#end-user-identity-and-privacy). ## Map onto standards Emitted attributes are a Neens convention, but they sit comfortably next to the emerging standards where those cover the same ground: - **`gen_ai.evaluation.result`** — where the OpenTelemetry GenAI conventions carry an evaluation result on a span, Neens reads it as evaluation signal. Use `neens.feedback.*` for the end-user-supplied rating that those conventions don't yet standardise. - **OpenInference session annotations** — if you already annotate sessions via OpenInference, keep doing so; add the `neens.*` keys for the business facts (escalation, handoff, end-user identity) that have no annotation of their own. The `neens.*` namespace fills the gaps rather than competing: business facts about a conversation that no cross-vendor convention has landed on yet. ## Related - [Send traces](/guides/send-traces) — how to get spans (and these attributes) into Neens. - [Custom measures](/guides/custom-measures) — turn an emitted attribute into a measure you can chart. - [Business outcomes](/guides/business-outcomes) — the **Measured** tier, for facts your system of record confirms. - [Derived measures](/guides/derived-measures) — `repeat_contact_rate` and the rest of the zero-setup tier. - [Business KPIs](/guides/business-kpis) — promote any of these to a tracked commitment. ## End-user identity and privacy `neens.end_user_id` is personal data — it names a real person. Neens stores it as a queryable field on the session so you can both analyse re-contact and honour a deletion request: an end user's data is erased by that id. Emit a **stable, non-guessable** id (your own user key, not an email address in the clear) so the same person links across cases without exposing their contact details in your traces. ================================================================================ # Agent Map Source: /docs/guides/agent-map/ ================================================================================ # Agent Map The **Agent Map** answers a question the raw span waterfall can't: *who calls whom, and which node dominates cost and latency?* It rolls the spans and tool calls of a run — or an entire cohort of runs — into a compact **topology**: each agent, tool, and LLM model becomes a single **node**, the **hand-offs** between them become weighted **edges**, and the one node responsible for the most time gets flagged as the bottleneck. Use it to see the shape of a multi-agent run at a glance and find the node worth optimizing first. There are two ways in: the **Agent Map** tab on a single trace or conversation, and the standalone **Agent Map** page that aggregates a whole cohort. ## At a glance | | | |---|---| | **Where** | The **Agent Map** tab in the trace/conversation inspector (Observe → **Traces** / **Sessions**), and the standalone **Agent Map** page (sidebar → **Diagnose**) | | **Key API routes** | `GET /sessions/{id}/agent-graph` (one run), `GET /conversations/{id}/agent-graph` (one conversation), `GET /agent-graph` (a cohort) | | **Scope** | Always scoped to the current agent; you can only map traces in agents you belong to | | **Needs** | Ingested trace data — nothing else. No LLM connection required | ## Map a single run Open any trace from the [Traces](/guides/traces-and-sessions) list and pick the **Agent Map** tab in the span explorer (alongside **Spans**, **Conversation**, **Graph**, and **Raw**). Where the **Graph** tab shows *every* span grouped by turn, the **Agent Map** collapses the run into its distinct **actors** and draws the flow between them, laid out left→right — a supervisor on the left flowing out to the workers, tools, and models it drives. - **Nodes** are colored by kind: **agent**, **tool**, **LLM** (keyed by model), plus retrieval and guardrail spans where present. - **Edges** are the observed hand-offs. Use the **Edge weight** toggle in the toolbar to drive edge thickness by **Calls**, **Latency**, or **Tokens**. - **The bottleneck is highlighted.** The single node responsible for the most time gets a red ring and a ⚡ **Bottleneck** badge, and it's named in the toolbar. - **Click any node** for its full breakdown: invocations, self time and self-time share, total (inclusive) time, model, tokens, cost, and errors. Node cost is tokens × that model's price; a node whose model has no price shows **—** and the map's totals are flagged as partial — see [Cost & model pricing](/guides/cost-and-model-pricing). **Self time, not wall-clock, decides the bottleneck.** Neens attributes each node its *self* time — its own time minus the time spent inside the children it called. That way the honest culprit (usually a model or a slow tool) is flagged, not the outer agent that merely wraps everything below it. ## Map a conversation A [session](/guides/traces-and-sessions#sessions) is a conversation — one or more traces sharing a `conversation_id`. Open a session from the **Sessions** page and its **Agent Map** tab stitches every member trace into one topology, so a multi-turn interaction reads as a single flow rather than one map per turn. Because a conversation is genuinely several runs, this view is *cohort-shaped* — it carries per-run (here, per-turn) averages and the **Show** toggle described below. ## Map a cohort A single run is an anecdote. To see whether a bottleneck is *systemic* — which node dominates cost and latency **across** the last 7 days, a failure cluster, or an agent version — open the standalone **Agent Map** page under **Diagnose** in the sidebar. ### Pick the cohort The page uses the same filters as the Traces and Sessions lists — time window, agent, agent version, failure cluster, model, tool, span filters, tags, scores, and full-text search. The cohort is exactly the set the Sessions list would show for those filters. You can also arrive pre-filtered: from a failure **Clusters** detail, click **View agent map for this scope** to jump straight to the aggregate topology for that cluster. ### Read per-run averages By default node and edge numbers read as a *typical run* — the cohort total divided by the number of runs — so cohorts of different sizes stay comparable. Use the **Show** toggle to switch between **Per-run avg** and **Totals** (where the whole cohort spent its time and money). ### Drill into a node Click any node to see how many of the cohort's runs it appears in (**Appears in N of M runs** — prevalence), its per-run and total invocations, self time, cost, and its **Error rate** — the share of its calls that errored, so a handful of errors over thousands of calls doesn't masquerade as a hot spot. **Versions split by model.** LLM nodes are keyed by model, so a model change across the cohort shows up as two distinct nodes — the honest way to compare, say, opus vs. sonnet on the same workload. ## How it works Neens reads a session's spans and tool calls and folds them into nodes and edges in-process — no LLM call is involved, so the Agent Map works whether or not an agent has a connection configured. - **Nodes** are the run's distinct actors. Agent, tool, and retrieval spans each become a node; LLM spans are keyed by model. Every invocation of the same actor rolls into one node. - **Edges** are the parent→child hand-offs between those actors, summed across the runs in scope. - **Self time** is a node's own duration minus the time spent inside its children. The node with the greatest self time is the bottleneck. ### Aggregating a cohort The cohort page and the conversation tab share one aggregation. Each node carries values **summed across runs**, alongside **per-run means**, its **prevalence** (in how many runs it appears), and its **error rate** (share of calls that errored) — the same for each edge. That's why per-run averages are the default: totals reward big cohorts, but a per-run mean reads as one representative run. ### The cohort cap A cohort filter — a 30-day window, a large failure cluster — can select tens of thousands of runs, and each contributes spans and tool calls Neens reads and merges. To keep the read tier safe, the matched set is capped at the **newest 2,000 runs**. When a cohort exceeds the cap, the page shows an amber banner: > This cohort matched **N** runs; showing the newest **2,000** (cap 2,000). **N−2,000** older runs were > left out — narrow the time window or filter to include them. Narrow the time window or tighten the filters to bring the whole cohort under the cap. The per-run and per-conversation maps are never capped — their run counts are small. Aggregate maps are cache-eligible on the read tier, so a repeated cohort view is served from a short-lived, tenant-keyed cache — toggling **Edge weight** or **Show** stays fast. ## Reference API endpoints | Endpoint | What it returns | |---|---| | `GET /sessions/{id}/agent-graph` | The topology for one trace: nodes, weighted edges, and the bottleneck node id | | `GET /conversations/{id}/agent-graph` | The topology for a whole conversation, rolling up its member traces (per-run/per-turn means + prevalence + error rate) | | `GET /agent-graph` | The aggregate topology for a cohort, driven by the full `GET /sessions` filter vocabulary; includes a `cohort` block with `matched_total`, `rolled_up`, `cap`, `capped`, and `dropped` | The cohort endpoint accepts the same query parameters as `GET /sessions` (see [Traces & Sessions](/guides/traces-and-sessions)) — `status`, `agent_name`, `version`, `cluster_id`, `failure_set`, `model`, `tool_name`, `span_kind`, `started_after`/`started_before`, `score_metric`, `enrichment`, `q`, and the numeric ranges. ## Troubleshooting | Symptom | Cause | Fix | |---|---|---| | **"No agents or tools to graph for this run/conversation."** | The trace has no agent, tool, or LLM spans to connect | Instrument agent/tool spans with a supported semantic convention (see [Send traces](/guides/send-traces)) | | **The cohort map shows an amber "showing the newest 2,000" banner** | The cohort matched more runs than the cap | Narrow the time window or tighten the filters to bring the whole cohort under the cap | | **A model change isn't obvious in the map** | LLM nodes are keyed by model, so a version that switched models shows as two separate nodes | Filter by **Agent Version** to compare one version's map at a time | | **The bottleneck isn't the node I expected** | The bottleneck is chosen by *self* time, not inclusive time — the outer agent's total includes its children | Click the flagged node to confirm its self-time share; the wrapping agent's time is mostly its children's | ================================================================================ # Activity Source: /docs/guides/activity/ ================================================================================ # Activity **Activity** is the live feed of background work Neens runs over your data: **scoring** runs (judges evaluating traces), **enrichment** runs, and **clustering** runs. Use it to confirm an evaluation finished, watch a live run's progress, or dig into why a run failed. ## At a glance | | | |---|---| | **Where** | Sidebar → **Observe** → **Activity** | | **Key API routes** | `GET /activity/runs`, `GET /eval-runs/{id}/metrics` | | **Scope** | Scoped to the current agent | | **Live behavior** | Polls every few seconds while runs are active; idle otherwise | ## What appears in the feed | Type | What it is | Where it comes from | |---|---|---| | **Scores** | A judge evaluating a set of targets — a manual **Run now**, a scheduled run, or continuous on-ingest scoring | [Scores](/guides/scores) and [Judges](/guides/judges) | | **Enrichments** | An enrichment computing attributes over traces — manual runs and continuous on-ingest ones | [Enrichments](/guides/enrichments) | | **Clustering** | A failure-clustering run grouping problem traces — nightly, readiness-triggered, or manual | [Failure clustering](/guides/clustering) | Each row shows the run's **Name**, **Type**, **Status**, **Target** (what kind of item it processes), a **Progress** bar showing how the run's targets split across succeeded / failed / running (see [Reading the Progress bar](#reading-the-progress-bar)), **Triggered by**, **Started**, and **Duration**. Columns are configurable with the **Columns** button and sortable by clicking headers; the feed lists the most recent runs (up to 100). ### Focusing the feed - A summary row counts **Total**, **Active**, **Running**, **Queued**, **Historic**, and **Failed** runs across the agent. - Toggle **All / Live / History** to show everything, only in-flight work (queued + running), or only finished runs. - The **Filters** button narrows by **Type** (**Clustering**, **Scores**, **Enrichments**), **Status** (**Queued**, **Running**, **Completed**, **Degraded**, **Failed**, **Skipped**), and **Started** (a date range on when the run began). ## Run statuses Runs move through **queued → running** to a terminal state: | Status | Meaning | |---|---| | `queued` | Accepted, not yet started. | | `running` | In flight — the progress bar updates live. | | `completed` | Finished successfully. | | `completed_with_failures` | A scoring run finished, but some targets failed (see below). | | `degraded` | A clustering run finished with a limited result (e.g. it hit its time budget). | | `skipped` | A scoring run finished having scored **nothing** — no target succeeded and none failed. Neutral, not an error: the [skip reason](#why-a-run-was-skipped) says whether it was legitimate. | | `cancelled` | The run was stopped before it finished — either you cancelled it, or its processing was interrupted and it was moved out of `running` automatically. Its partial results are kept but never graded into a verdict — see [Cancel or recover a run](/guides/judges#cancel-or-recover-a-run). | | `failed` | The run did not succeed. | A scoring run is graded by its **success rate** — completed targets over targets that actually ran. It's not realistic to expect 100% against external LLM providers, so by default a run is `completed` at ≥ 90% success, `failed` below 10%, and `completed_with_failures` in between (a judge can override its own green cutoff). Individual target failures retry automatically on transient errors — up to 3 attempts by default — before counting against the run. ### Why a run was skipped A `skipped` run scored nothing, and Neens records **why** — per target and rolled up on the run. The Activity row's error summary and the **Details** drawer both show a machine reason code (`no_llm_connection`, `insufficient_components`, `target_not_found`, `no_taxonomy`, …), the sentence it means, whether it will be retried, and what to do about it. If Neens cannot classify the cause it says `unknown` out loud rather than showing a blank. Skips caused by a **transient** condition — the LLM connection could not be resolved on that pass, a component judge's call kept erroring, the trace hadn't finished landing — are **re-dispatched automatically** as a new run over exactly those targets, bounded to a couple of attempts (2 by default) with a doubling backoff. Permanent causes are never retried; they need a change from you. The drawer links the two runs in both directions, so "was this retried?" is answerable without reading a log. The full code table, and a step-by-step for debugging one, live in [Continuous evaluation → When a run is Skipped](/guides/continuous-evaluation#when-a-run-is-skipped). **Nothing shows "running" forever.** If the processing behind a run is interrupted while it is still in flight, the feed self-heals: the row is automatically moved to a terminal state instead of sitting at `running`, so the scorer never gets wedged and you can simply re-run it. A stalled clustering run is marked `failed` with an explanatory error; an interrupted scoring run is marked `cancelled`. ## Reading the Progress bar The **Progress** cell is a *distribution*, not a completion meter. It shows what happened to every target the run was given, colour-coded, on a single bar: | Segment | Colour | Meaning | |---|---|---| | **Succeeded** | Green | Targets the run processed successfully. | | **Failed** | Red | Targets that were attempted and errored — after their automatic retries. | | **Running** | Aqua (brand) | Targets in flight right now. | | **Not run** | The exposed grey track | Targets the run never attempted: skipped, or not reached yet. | Segments are drawn left to right in that order, so the green tip is always in the same place and two rows are comparable at a glance. Beside the bar is a text readout — `120/165 · 45 failed`. The failure clause appears **only when something actually failed**; a clean run reads `165/165` with nothing after it. Colour is never the only signal: the readout and the [status pill](#run-statuses) both state the outcome in words. ### Worked example — a finished scoring run A judge ran against **165** sessions. 120 were scored; 45 errored against the provider. - The bar is about three-quarters green, then a red block, and **no grey** — 120 + 45 = 165, so every target was attempted. - The readout reads `120/165 · 45 failed`. The leading number is the **succeeded** count, not "how far along" the run is. That distinction is the point: a finished run at `120/165` used to look like a bar frozen short of the end, with nothing on screen saying that the missing 45 had failed. - Click **Details** and the drawer leads with **73%** success rate — *120 of 165 tasks succeeded, 45 failed* — over a thicker copy of the same bar, with a legend giving each category's count and share: **Succeeded 120 · 73%**, **Failed 45 · 27%** (and a **Not run** row whenever targets were never attempted). - 73% is `120 ÷ (120 + 45)`. With the default cutoffs that sits between 10% and 90%, so the run is graded **`completed_with_failures`**. ### Worked example — a run still in flight The same judge over **900** targets, checked mid-run: 600 succeeded, 30 failed, 24 currently executing. All four categories are visible at once — green 600, red 30, aqua 24, and 246 targets of exposed grey track that the run has not reached yet. The readout reads `600/900 · 30 failed`. While the run is live the grey shrinks as work is dispatched, and the aqua segment moves along in front of the green. The success rate *so far* is `600 ÷ 630` = 95%, but no verdict exists yet — the pill stays `running` until the run reaches a terminal state. ### Skipped targets don't drag the rate down A run's success rate is measured over **attempted** targets only: ``` success rate = succeeded ÷ (succeeded + failed) ``` Targets that were skipped never ran, so they are not in the denominator — they appear as **Not run** grey. If the run above had 185 targets with 120 succeeded, 45 failed and 20 skipped, the success rate would still be **73%**, and the extra 20 would show as grey. This is exactly the rule the [status pill](#run-statuses) is graded on, so the bar and the pill can never disagree about what counted. ### An unknown is not a zero Some runs report no failure count at all: - a run recorded **before** the distribution bar existed — that history only ever kept a completed/total pair; - a run kind with **no per-target failure notion** — a clustering run is a single refit over the failure set, not a fan-out of independently failing tasks. Those rows render the plain single-fill bar, with no colour split and no failure clause. Neens deliberately shows **nothing** rather than `0 failed` for a run whose failure count was never measured. Printing a zero would be a success claim nobody made. To judge one of these runs, read its status and its **Details** summary instead of the bar. ### The pill is the verdict; the bar is the breakdown The **Status** pill is *not* derived from these counts. It is the server's graded verdict, using the success-rate bands in [Run statuses](#run-statuses); the bar is the evidence behind that verdict. So the two can look like they disagree, and shouldn't surprise you: - A run graded **`completed`** can still show a thin red segment — the green cutoff is **90%** by default, not 100%. - A **`skipped`** run shows an all-grey bar: nothing was attempted, so there is nothing to colour. The [skip reason](#why-a-run-was-skipped) says whether that was legitimate. - A **`cancelled`** run's bar shows how far it got before it was stopped; those partial counts are never graded into a verdict. Cancelling takes effect promptly — the run stops scoring within seconds rather than draining its queued work — and it reads **Cancelled** here *and* on the scorer's run history on the [Judges](/guides/judges#cancel-or-recover-a-run) page, so the two pages never disagree. ### What to do when you see red 1. **Expand the row.** A scoring run's inline metrics include **Failure rate** and **Top errors** — the real error messages, grouped and counted. That usually names the cause outright (provider rate limit, timeout, revoked credential). 2. **Open Details.** The per-target breakdown lists every failing target with its error, duration and a `retried N×` marker where a target needed several attempts. Transient errors were already retried automatically — up to 3 attempts by default — before counting as failed, so a red segment is a *persistent* failure, not a blip. 3. **For grey, look for a skip reason.** A skipped target carries a typed reason code, the sentence it means, whether it will be retried, and the remedy — see [Why a run was skipped](#why-a-run-was-skipped). Grey that is simply "not reached yet" needs nothing from you. 4. **Then act on the cause, not the run.** Re-running a judge against a dead LLM connection reproduces the same red bar; fix the connection, quota or judge configuration first. ## Who triggered a run The **Triggered by** column separates human-initiated from automated work: - A **person's name** with avatar initials — someone kicked the run off manually (e.g. **Run now** on a judge or enrichment). Attribution is stamped when the run is created and survives later display-name changes. - A **Scheduled** chip — the Neens scheduler started it (e.g. nightly clustering). - An **Automated** chip — the platform triggered it in response to data, such as continuous on-ingest scoring or on-ingest enrichment. - An **API key** chip — a machine credential (a data-plane API key) triggered it. - A **System** chip or **—** — attribution is unknown (including runs recorded before attribution existed). ## Run details ### Scoring runs Click a **Scores** row to expand its pipeline metrics inline: - **Throughput** (targets/second) and **ETA** while running. - **Failure rate** and **Top errors** — the actual error messages, grouped and counted. - **Workers** / **Concurrency** — how many judge calls run in parallel. - **Per-connection (req/min)** — the request rate against the LLM connection backing the run. The **Details** button opens the drawer. It leads with the run's **success rate** as a large percentage, a full-width copy of the [distribution bar](#reading-the-progress-bar), and a legend giving each category's count and share, then lists the per-target breakdown: each target's status, score, pass/fail label, duration, any error, and a `retried N×` marker when a target needed multiple attempts. A chip row totals the targets by status (completed / failed / skipped / running / queued / cancelled). A **skipped** target additionally shows its reason code, the sentence it means, whether it will be retried, and the remedy — see [Why a run was skipped](#why-a-run-was-skipped). ### Clustering runs Clustering rows have a **Details** action summarizing the run — **Failure traces analyzed**, **Duration**, **Started**, and a note explaining the outcome — with a link into [Failure clustering](/guides/clustering) to explore the resulting clusters. A clustering run's Progress bar has **no colour split**: the run is one refit over the whole failure set, so there is no per-target success/failure to break down. Its verdict lives in the status (`completed`, `degraded`, `failed`) — see [An unknown is not a zero](#an-unknown-is-not-a-zero). ## Reference GET /activity/runs query parameters | Parameter | Values | Meaning | |---|---|---| | `kind` | `score`, `enrichment`, `cluster`, `topic`, `all` | Run type | | `state` | `live`, `history` | In-flight vs. finished (mirrors the **All / Live / History** toggle) | | `status` | any status above, or `all` | Exact status | | `started_after` / `started_before` | ISO-8601 | Window on when the run started | | `sort_by` / `sort_dir` | column key / `asc`\|`desc` | Server-side sort | | `limit` | 1–500, default 100 | Max rows returned | The response includes the run list plus the summary counts shown at the top of the page. ## Troubleshooting - **A run sits in `queued`** — no worker is consuming its queue yet (on a distributed deployment) or earlier work is still draining. It starts as soon as capacity frees up. - **A scoring run is `completed_with_failures`** — its Progress bar carries a red segment sized to the failures. Expand the row and check **Top errors**, then open **Details** for the failing targets. Provider rate limits and timeouts are the usual culprits; failed targets were already retried automatically. Full walkthrough: [What to do when you see red](#what-to-do-when-you-see-red). - **A finished run's bar isn't full, and nothing is red** — the grey remainder is **Not run**: targets that were skipped or never reached. Open **Details** and read the skip reason. - **A scoring run is `skipped`** — it scored nothing. Open **Details**: the header names the reason(s). `no_llm_connection` means add a connection; `insufficient_components` means the composite couldn't gather enough component scores; `target_not_found` and `llm_connection_error` are transient and already scheduled for retry. - **An enrichment or judge you expected isn't producing runs** — continuous work only triggers on *new* traces, and the judge/enrichment must be enabled. See [Scores](/guides/scores) and [Enrichments](/guides/enrichments). ================================================================================ # Failure clustering Source: /docs/guides/clustering/ ================================================================================ # Failure clustering Failure clustering is how Neens turns a pile of failing traces into a short, named list of failure modes. It embeds each failing session's behavior — what the agent did, where it errored, which quality checks it failed — and groups similar sessions together, so a thousand bad traces become a dozen patterns you can actually triage. The results power the **Failure Modes** page (see [Issues & failure modes](/guides/issues-and-failure-modes) for the taxonomy built on top). ## At a glance | | | |---|---| | **Where it lives** | Diagnose → **Failure Modes** (cards + **Scatter** view); each cluster has a detail page | | **Key API routes** | `POST /clusters/run`, `GET /clusters/failure-modes?range=`, `GET /clusters/configs`, `POST /clusters/configs/{id}/activate`, `POST /clusters/assign`, `GET /clusters/{id}/exemplars`, `POST /clusters/{id}/merge`, `PATCH /clusters/{id}`, `POST /taxonomy/failure-modes/{id}/links` | | **What it needs** | Nothing to cluster — embedding runs locally on the server. An agent LLM connection (**Settings** → **LLM connections**) to *label* clusters and draft root causes | | **Scope** | Per agent — settings, runs, and results never cross agents | ## Reading the Failure Modes tab The **Failure Modes** tab groups clusters into buckets (Timeouts, Tool failures, and so on — see [Issues & failure modes](/guides/issues-and-failure-modes)) and, within each bucket, sorts cards by classified trace count descending — the highest-impact failure mode in the bucket is always first. Use **Expand all** / **Collapse all** at the top of the tab to open or close every bucket group at once, or a group's own header to toggle just that one. A summary strip above the groups gives you the shape of the page at a glance: **total modes**, **affected traces** (the sum of each mode's trace count in the window), how many **buckets** are represented, how many modes are **eroding a business KPI** (see [Business KPIs](/guides/business-kpis)), and a breakdown of fix status across every mode — how many already have an **applied** fix, how many have one **proposed** and in flight, and how many have **no fix** yet. ### The time window The same time-range picker used on the Issues & Taxonomy tab (**Today**, **24h**, **7d**, **30d**, **All time**, or custom) also scopes the Failure Modes tab: it re-derives each mode's displayed trace count, trend sparkline, and the summary's affected-traces total to the window you pick. It does **not** re-cluster — the clusters themselves (their membership, labels, and root causes) come from the last completed analysis run regardless of the window; the picker only changes which of that run's members are counted as "in window" for display. Leave it at the default to see the same window the last analysis used, or narrow it to see what's happening *right now* within an existing clustering. ## What counts as a failure Clustering runs over the agent's **failure set**, and you control what's in it. The **Analyzing:** chip at the top of the **Failure Modes** page shows the current criteria in plain language; click **Configure clustering** on the same page to change them and every other knob (see [Clustering configurations](#clustering-configurations) below). | Selection mode | Sessions included | |---|---| | **Score** (default) | Sessions with an `error` status, **or** a failing / below-threshold score, **or** a human "fail" annotation | | All | Every session in the analysis window (no failure filter) | | Errors only | Only sessions whose status is `error` | In **Score** mode you also pick which score is consulted: | Metric option | Meaning | |---|---| | **Primary Score (recommended)** | The default — the session's primary quality composite must be at or above the threshold | | **Any score** | Any below-threshold score on the session marks it a failure | | A specific metric | Only that metric's scores are consulted | The **Score threshold (0–1)** defaults to `0.5`: a session scoring below it counts as a failure even if it finished without an error. **Why score-based is the default.** Most agent failures are *silent*: the trace completes with status `ok`, but the answer was wrong, irrelevant, or unhelpful. Clustering only errored traces misses all of them. The default criteria — low primary score, plus hard errors, plus human "fail" labels — catch quality failures your [judges](/guides/judges) detect, not just crashes. In **Score** mode, hard errors and human-flagged failures are always included regardless of their scores. The failure set is also windowed and capped so runs stay fast at volume: by default the last **30 days**, at most **5,000 sessions** per run (sampled with day-and-agent stratification when the set is larger — the page then labels counts as a *sample-based estimate*). ## When clustering runs - **Nightly** — a full clustering pass runs at 2:00 AM (server timezone) for every agent. - **First-run readiness sweep** — every 30 minutes, Neens checks agents that have never been analyzed and triggers their first run as soon as they cross the readiness threshold (**30** failure traces by default, shown as a progress meter on the page). A low-traffic agent older than a week gets its first run at 15 failures so the feature is never invisible. - **On demand** — click **Run analysis now** (or **Re-analyze** once results exist), or call `POST /clusters/run`. A manual run uses exactly the same per-agent settings as the nightly job, so the button and the scheduler always produce the same clustering of the same data. Only one run per agent executes at a time; a crashed run stops blocking after an hour. The run returns an honest outcome — `completed`, `skipped_insufficient_data` (with the count it found vs. the minimum it needs), `queued` (the analysis was handed to a background worker), or `failed` — and the page shows **Last analyzed** freshness from the most recent run. ## How it works ### Build a structured failure summary per session For each session in the failure set, Neens composes a deterministic summary of the agent name, status, span names, error messages, failed tool calls, and every judge score verdict. Folding score verdicts in is what lets *silent* quality failures cluster together even when their surface text differs. ### Embed locally Summaries are embedded with a local sentence-transformer model on the server — no external API call, and no LLM is needed for this step. Embeddings are cached per session so repeat runs only embed new sessions. ### Cluster Neens clusters the embeddings with **HDBSCAN** (variable density, no fixed radius, a native noise bucket) and falls back to **DBSCAN** automatically when HDBSCAN isn't installed. The minimum cluster size defaults to `3` and scales up automatically on large failure sets (0.5% of the fitted set) so a 5,000-session run yields dozens of meaningful modes instead of hundreds of micro-clusters. ### Keep identities stable Fresh clusters are stitched to the previous run's clusters by centroid similarity and member overlap, so a recurring failure mode keeps the **same id, label, and root cause** night after night instead of appearing as a new duplicate. Near-identical fragments within one run are merged before labeling. Each run is written atomically — a failure mid-run leaves no partial clusters behind — and clusters superseded by a new run are marked `resolved` (kept as history), never deleted. ### Label and diagnose new clusters Genuinely new clusters are labeled and diagnosed with an LLM (see below). Stitched clusters reuse last run's label and root cause, so recurring modes cost nothing to re-label. ### Surface the leftovers as novelty Sessions that fit no cluster aren't dropped: any that sit far from every known cluster are flagged as **novel failure candidates** and raise a novelty insight on your Insights feed — an early warning that a new kind of failure is emerging. ## Labeling and root cause — the LLM connection Cluster labeling, cluster-level root-cause hypotheses, and draft remediations all use the agent's default LLM connection, configured in **Settings** → **LLM connections** (Anthropic, Bedrock, or any OpenAI-compatible endpoint). There is no platform-wide key. Labels are pushed to be *specific*: if the model returns something generic ("Agent Error"), Neens retries with an explicit correction, then falls back to a deterministic label derived from the dominant failing tool or failed metric (for example `get_order_status tool failure` or `Low answer relevancy score`). **No LLM connection? Clustering still runs.** Without a resolvable connection, clusters are produced *unlabeled* (shown as "Unlabeled cluster", with no root-cause hypothesis) — grouping, counts, trends, and exemplars all still work. Configure a connection and the next run backfills labels and root causes for existing clusters automatically. ## Online assignment, novelty, and "more like this" Between full runs, Neens classifies *new* failing sessions against the fitted clusters without re-clustering: - **Intraday assignment** — every 30 minutes, recently arrived failure sessions (the last hour, up to 500) are matched to the nearest cluster, so fresh failures land in existing modes and novel ones raise insights the same day, not the next morning. - **On demand** — `POST /clusters/assign` with a list of session ids returns a status per session: `assigned` (joins a known cluster), `novel` (far from everything — a candidate new failure mode, which also fires a novelty insight), `unassigned` (in between), or `no_model` (no clustering run has completed yet). - **Similar sessions** — `GET /sessions/{id}/similar` returns the sessions semantically closest to a given one (nearest-neighbor over the same embeddings), useful for "show me more traces like this failure". ## Exemplars, status, and the taxonomy - **Exemplars** — `GET /clusters/{id}/exemplars` returns a cluster's most representative members (closest to its center; 5 by default, up to 50). The UI uses these when you promote a cluster: **Confirm candidate** requires reviewing at least 3 exemplar traces first, so the taxonomy stays grounded in real evidence. - **Status** — clusters are `active` (current) or `resolved` (superseded by a later run or no longer recurring). Pickers and the Failure Modes page show active clusters only. - **Failure modes** — clusters are the machine-discovered evidence behind the human-curated taxonomy. Confirming a cluster creates a failure mode linked to it, and after each run Neens auto-proposes cluster-to-mode links wherever a clear majority of a cluster's members were classified into one mode. See [Issues & failure modes](/guides/issues-and-failure-modes), and [Remediations](/guides/remediations) for drafting a fix from a cluster. ## From a cluster to action Every cluster card carries the same fix path as an Issue, plus one action of its own: - **View traces** — opens the sessions behind the cluster in the selected window. - **Generate fix** → **View fix** — the same stateful action as on an Issue card: **Generate fix** drafts a typed remediation grounded in the cluster's evidence when none exists yet; once one does, the same spot becomes **View fix**. See [Remediations](/guides/remediations). - **Promote to issue** — link the cluster into your curated taxonomy as evidence for a named issue, instead of **Create eval** (that action lives on Issue cards, once a mode already has evidence behind it). ### Promote a cluster to an issue #### Click **Promote to issue** on the cluster card Review the exemplar traces first — the same **3-exemplar** grounding **Confirm candidate** requires, so you're not promoting a cluster you haven't actually looked at. #### Pick a target Choose an **existing** failure mode from the picker to add this cluster as more evidence for it, or create a **new** one on the spot (the same as **+ New issue type**, pre-filled from the cluster's label). #### Confirm Neens links the cluster to the mode (idempotent — promoting the same cluster to the same mode twice doesn't duplicate the link). The mode's Issue card on the **Issues & Taxonomy** tab now counts this cluster's sessions, and **Cluster evidence** in the mode's detail drawer lists the link. **Promote to issue vs. Confirm candidate.** Both ground a taxonomy entry in real cluster evidence. **Confirm candidate** is the quick path for a *brand-new* auto-discovered cluster that doesn't map to anything you've named yet. **Promote to issue** is the general form: it also lets you fold a cluster into an issue that **already exists** — useful once your taxonomy has grown and a new cluster turns out to be more evidence for a failure mode you already track. ## Keeping failure modes clean Over time an agent can end up with near-duplicate failure modes — two cards with the *same* name, or one broad mode that should really be several. Neens keeps the list tidy in two ways: it **dedupes duplicate names automatically** on every run, and it lets you **merge** and **rename** modes by hand from a card. ### Automatic dedup of duplicate names Cluster names are auto-generated (from the dominant failing tool or metric), so two unrelated modes can land on the *same* label — e.g. two different clusters both named `Low faithfulness score`. When that happens, the dedup pass resolves the collision one of two ways: - **Disambiguate** — if the two modes are genuinely distinct, Neens keeps them separate and appends a distinguishing suffix so each name is unique, e.g. **Low faithfulness score (checkout)** and **Low faithfulness score (search)**. - **Coalesce** — if the two modes are effectively the same failure, Neens folds them into one so you don't triage the same pattern twice. This runs on every clustering pass and is controlled by the **Dedup duplicate labels** toggle in the [clustering config](#clustering-configurations) (Advanced section), **on by default**. Turn it off if you'd rather see the raw, un-disambiguated names. A name you set yourself (see [Rename](#rename-a-failure-mode-sticky-names) below) is never touched by this pass. ### Merge a failure mode When two cards describe the same problem, merge one into the other from its card. ### Open the mode's actions On a failure-mode card, open the actions menu (**Failure mode actions**) and choose **Merge**. ### Pick a target In **Merge failure mode**, use the **Merge into** picker to choose another mode. The picker lists only the *other* modes in the **same bucket**, so you can't merge across unrelated groups. Confirm with **Merge**. The traces from the mode you started on move into the target, and the now-empty source mode is marked `resolved` (kept as history, not deleted). Behind the card this calls `POST /clusters/{id}/merge` against the target, with the source mode's id. If a bucket has no other modes, the dialog says so and there's nothing to merge into. ### Rename a failure mode (sticky names) Auto-generated names are a starting point — rename any mode to something your team recognizes. ### Choose Rename From the card's actions menu, choose **Rename**. ### Set the name In **Rename failure mode**, type a new **Name** and confirm. A renamed mode is **pinned**: the name you set *sticks* across every future analysis run, so the nightly pass and every **Re-analyze** keep your wording instead of reverting to the auto-generated label. Pinned modes show a small pin marker next to their name (hover it for "This name was set manually — future analysis runs won't rename it."). Renaming a mode also exempts it from automatic dedup. Behind the card this is `PATCH /clusters/{id}` with the new label. **Example.** After a run you see two cards both named `get_order_status tool failure`. You open the second card, **Rename** it to *Order status: auth timeout* (now pinned, with a pin marker), and **Merge** a third near-identical card into it. Tomorrow's nightly run keeps your name and doesn't resurrect the merged duplicate. ## Clustering configurations Everything above — what counts as a failure, which score is consulted, how fine the clusters are, how much data a run reads — is captured in a **clustering configuration**. Configs work like [judges](/guides/judges): an agent keeps a *collection* of named configs, but exactly **one is active**, and only the active config drives analysis. Open the collection from **Configure clustering** on the **Failure Modes** page. **Out of the box, there's nothing to set up.** Every agent starts with one active config named **Default** that mirrors the sane defaults in the reference table below. You only create more configs when you want to try a different way of grouping failures. ### The knobs Every config carries the same set of tunables, grouped in the editor: **Selection criteria** — what goes into the failure set: - **What gets clustered** (`selectionMode`) — **Score** (default), **All traces**, or **Errors only**. See [What counts as a failure](#what-counts-as-a-failure) for the full behavior of each. - **Score** metric and **Score threshold (0–1)** — in Score mode, which score is consulted (**Primary Score (recommended)**, **Any score**, or a specific metric) and the cutoff below which a session counts as a failure (default `0.5`). **Advanced** — how the run behaves: - **Min cluster size** — the granularity lever, and the one worth tuning first. It's the fewest sessions that can form a cluster; **smaller = more, finer failure modes**, larger = fewer, broader ones. Default `3`, floor `2`. (Clustering is density-based, so there's no "number of clusters" to set — you steer granularity with this instead.) - **Min traces to run** (`minFailureSessions`) — how many failing traces must exist before the first run starts (default `30`). Lower it for a low-volume agent so analysis begins sooner. - **Sampling %**, **Window (days)**, and **Max traces per run** — down-sample the eligible set, set the rolling look-back window, and cap how many sessions a single run reads (defaults `100`, `30`, `5000`). Together they keep runs fast at volume. - **Dedup duplicate labels** — automatically disambiguate or merge failure modes that get the same auto-generated name (default **on**). See [Automatic dedup of duplicate names](#automatic-dedup-of-duplicate-names). - **Enable this config for scheduled clustering** — turn scheduled and manual runs off without deleting the config. As you edit, a **live estimate** shows how many traces match the current criteria in the window — e.g. *"142 eligible traces · meets the 30-trace minimum"* — so you can see the effect before you save. It's a quick count, not a trial run, so it returns instantly. ### Lifecycle: Experimental, Finalized, Archived Each config carries a **lifecycle stage** — a colored badge of **Experimental** (amber), **Finalized** (green), or **Archived** (gray). It's a **curation label only**: it tells your team how settled a config is and controls which configs show by default. It never changes what runs — only [activating](#activate-a-config) a config does that. - A config you create starts **Experimental**. The seeded **Default** config is **Finalized**. - The config list defaults to the **Finalized** view (plus the active config, which is always shown); switch the filter to **All** to see experimental and archived configs too. - Move a config between stages any time with **Change lifecycle stage** — there's no fixed order and no config is ever frozen. Archive a config you've stopped using to tuck it out of the default view without losing it. ### Activate a config **Activate** is the switch that matters: the active config — and only the active config — drives the nightly run, the readiness sweep, and every **Run analysis now**. Activation is independent of lifecycle stage; you can run an Experimental config while you're still tuning it. ### Create a config to try On the **Failure Modes** page, open **Configure clustering**, click **New config**, name it (e.g. *Strict failures*), adjust the knobs, watch the estimate, and **Create config**. It's saved as Experimental and does **not** run yet — activating is a separate, deliberate step. ### Activate it Click **Activate** on that config. Neens atomically makes it the agent's one active config and deactivates the previous one. The next scheduled or manual run uses it. ### Re-run to see the effect Click **Run analysis now** (or **Re-analyze**) to cluster immediately with the new config, rather than waiting for the nightly pass. **Exactly one config is active per agent.** Activating a config always deactivates the one that was active — you never have to switch the old one off yourself. You can't delete the active config (activate another first), and an agent always keeps at least one config. ## Reference: clustering configs Manage the collection per agent via the config routes; the active config's knobs are also readable through the compatibility route `GET /clusters/settings`. | Route | Purpose | |---|---| | `GET /clusters/configs?lifecycle=finalized\|experimental\|all` | List configs (defaults to finalized, plus the active one) with the active config's id | | `POST /clusters/configs` | Create a config (saved Experimental, inactive) | | `GET` / `PATCH` / `DELETE /clusters/configs/{id}` | Read, edit, or delete a config | | `POST /clusters/configs/{id}/activate` | Make a config the agent's active one | | `PATCH /clusters/configs/{id}/lifecycle` | Change its lifecycle stage | | `POST /clusters/configs/estimate` | Live count of eligible traces for a set of knobs (no run) | Each config's tunable knobs and their defaults: | Setting | Default | Notes | |---|---|---| | `enabled` | `true` | Turns scheduled + manual clustering off for the config | | `selectionMode` | `score` | `score`, `all`, or `error_only` | | `scoreMetricKey` | `null` | `null` = the primary metric; `__any__` = any score; else a specific metric key | | `scoreThreshold` | `0.5` | Failure cutoff for scores, clamped to 0–1 | | `windowDays` | `30` | Rolling analysis window (minimum 1) | | `maxSessions` | `5000` | Per-run cap; larger sets are sampled (stratified by day and agent) | | `samplingPct` | `100` | Optional down-sampling of the windowed set, 0–100 | | `minClusterSize` | `3` | Minimum sessions per cluster (floor 2); auto-scales on large sets | | `minFailureSessions` | `30` | Failure traces needed before the first run (minimum 15) | | `dedupLabels` | `true` | Auto-disambiguate or coalesce failure modes that get the same auto-generated name | ## Troubleshooting | Symptom | Cause | Fix | |---|---|---| | Clusters show "Unlabeled cluster" and no root cause | No default LLM connection resolved for the agent, or its credential is invalid | Configure or fix the connection in **Settings** → **LLM connections**; the next run (or **Re-analyze**) backfills labels | | "Not enough data to cluster yet" | Fewer failure traces in the window than the minimum cluster size / readiness threshold | Send more traffic, widen the window, or loosen the selection criteria | | "Analysis queued — results will appear once the run completes." | This server doesn't run the ML stack; the run was handed to a background worker | Wait a few minutes, then refresh | | Counts marked "est." / sample-based | The windowed failure set exceeded `maxSessions` (or `samplingPct` is below 100), so the run fit on a sample | Raise `maxSessions` if you want exact counts and can afford longer runs | | The failure count doesn't match the Sessions page | The readiness meter counts individual failure *traces*; the Sessions page rolls traces up into conversations | Compare against the Traces view instead | ================================================================================ # Issues & failure modes Source: /docs/guides/issues-and-failure-modes/ ================================================================================ # Issues & failure modes A **failure mode** is a named, defined way your agent goes wrong — "Tool-call argument hallucination", "Refund policy misquoted". The set of them is your agent's **taxonomy**. An **Issue** is a failure mode in operation: the Neens classifier assigns each failing trace to a mode, and the volume rolls up into Issue cards with a lifecycle, a trend, and a path to a fix. Both live on the **Failure Modes** page, under the **Issues & Taxonomy** tab. ## At a glance | | | |---|---| | **Where it lives** | Diagnose → **Failure Modes** → **Issues & Taxonomy** | | **Key API routes** | `GET /taxonomy/failure-modes`, `POST /taxonomy/failure-modes/import`, `POST /taxonomy/failure-modes/import-file`, `GET /taxonomy/issues`, `PATCH /taxonomy/issues/{id}/lifecycle`, `POST /flywheel/failure-modes/{id}/generate-eval`, `POST /taxonomy/failure-modes/{id}/links` | | **What it needs** | A taxonomy (import the starter library or confirm clusters); an agent LLM connection (**Settings** → **LLM connections**) for the classifier | | **Scope** | Per agent | The **Failure Modes** page has two tabs. The **Failure Modes** tab (covered in [Failure clustering](/guides/clustering)) shows what the machine found by clustering raw traces. This page covers the **Issues & Taxonomy** tab: the curated, named catalogue your team owns — plus the operational **Issues** rollup of how often each named failure is actually happening. Neens organizes failures at two levels. A fixed set of about ten broad **buckets** (Timeouts, Rate limits, Auth failures, Guardrails tripped, Upstream service errors, Tool failures, Data/parsing errors, Agent output issues, Other) groups the top-level rollups and chart legends. Underneath, the open-ended **failure modes** are the specific patterns — the level you name, classify against, and manage. ## The taxonomy Every failure mode carries a name, a definition ("what distinguishes this failure mode?"), a curated severity (**High** / **Medium** / **Low**), an optional owner, compliance tags, a curation status, and a provenance badge: | Provenance badge | Meaning | |---|---| | **Platform default** | Shipped with the Neens starter library (or bulk-imported from your own label list) | | **Custom** | Added or confirmed by your team | | **Auto-discovered** | Surfaced by [failure clustering](/guides/clustering) | | Curation status | Meaning | |---|---| | **Candidate** | Suggested, not yet reviewed | | **Confirmed** | Reviewed and adopted by your team | | **Monitoring** | Kept under watch | | **Archived** | Retired — excluded from classification and the Issues rollup | ### Build your taxonomy The **Issues & Taxonomy** tab header carries two buttons for managing the catalogue itself: - **+ New issue type** — the primary button. Opens a dialog to add a single failure mode by hand: a name, definition, severity, optional owner, and compliance tags (`POST /taxonomy/failure-modes`). - **Manage taxonomy** — opens the full **Taxonomy** table: every mode with its provenance, curation status, and bulk actions (import, edit, merge, split, delete — see [Curate it over time](#curate-it-over-time) below). Use this when you're tidying the catalogue rather than adding one issue. Ways to populate the taxonomy: - **Import starter library** — the fastest start. On the empty state (or via `POST /taxonomy/failure-modes/import`), seed a set of common agent failure modes, or post your own label list in the same call. Imports are idempotent: re-importing refreshes definitions without duplicating, and never overwrites a status your team has since changed. - **Confirm candidate / Promote to issue** — turn a discovered cluster on the **Failure Modes** tab into (or link it to) a named issue. You must review at least **3 exemplar traces** from the cluster first, so every mode is grounded in real evidence. See [Promote a cluster to an issue](/guides/clustering#promote-a-cluster-to-an-issue) for the cluster-side how-to. - **Create manually** — the **+ New issue type** button above, or `POST /taxonomy/failure-modes` directly. ### Import a taxonomy file (CSV / JSON) Most teams already keep their failure taxonomy in a spreadsheet. Click **Import** on the **Issues & Taxonomy** tab to upload it as a **CSV** or **JSON** file — a thin path over the same idempotent bulk-import, so re-uploading updates modes in place instead of duplicating. The CSV columns are: | Column | Required | Notes | |---|---|---| | `name` | Yes | The failure mode's name | | `definition` | | What distinguishes this failure mode | | `severity` | | One of `high`, `medium`, `low` (blank is allowed) | | `compliance_tags` | | One or more tags separated by `;` or `,` | | `key` | | A stable id used for idempotent re-import (defaults to a slug of the name) | Common spreadsheet header names are accepted as aliases — for example **Failure Mode** → `name`, **Description** → `definition`, and **Tags** → `compliance_tags`. Grab the exact header row from the **Download template** link in the Import modal (backed by `GET /taxonomy/failure-modes/import-template`). ```csv name,definition,severity,compliance_tags,key Hallucinated Price,"Agent states a product price not present in the catalog tool result",high,accuracy;grounding,hallucinated_price Wrong Order Lookup,"Agent looks up the wrong order id for a returns request",medium,reliability,wrong_order_lookup ``` **Malformed rows never sink the upload.** A row missing `name` or carrying an invalid `severity` is reported back individually — you get a **created / updated / skipped** summary where each skipped row lists its row number and the reason — while every valid row still imports. Nothing is silently dropped. Imported modes land as **Candidate** (`provenance: seeded`) and are idempotent by `key`/name: re-uploading the same file refreshes definitions and bumps each mode's version rather than creating duplicates, and it never overwrites a status your team has since changed. Import from the command line or CI For automation, post the file body directly to the raw-body endpoint (the request body **is** the file — this is not a multipart form upload). Requires the `manage_taxonomy` permission — any **member or admin** can import, edit, and delete taxonomy modes; viewers are read-only: ```bash curl -X POST "$NEENS_BASE_URL/taxonomy/failure-modes/import-file?format=csv" \ -H "Authorization: Bearer $NEENS_API_KEY" \ -H "Content-Type: text/csv" \ --data-binary @failure-modes.csv ``` `format` is optional (`csv` or `json`) — Neens infers it from the `Content-Type` and content when omitted — and `status` defaults to `candidate`. The response is `{created, updated, skipped: [{row, name, reason}], failureModes, format}`. The pre-existing strict-JSON endpoint `POST /taxonomy/failure-modes/import` (body `{"modes": [...]}`, or `{"include_starter_library": true}`) remains the programmatic path when you're generating the list rather than uploading a file. ### Curate it over time Click any row in the **Taxonomy** table to open its detail drawer: - **Edit** name, definition, severity, owner, and status (every change bumps the mode's version). - **Merge** two modes — the source's cluster evidence and exemplars fold into the target and the source is archived. - **Split** a mode into a new one, moving selected cluster links across. - **Delete** a mistaken or obsolete mode outright (`DELETE /taxonomy/failure-modes/{id}`) — distinct from *merge* (which archives the source) and *archive* (a status change); its cluster-evidence links are removed with it. - **Cluster evidence** — the clusters backing a mode. Links marked *auto* were proposed by Neens (after each clustering run, a cluster is linked to a mode when a clear majority of its classified members carry that mode); links you create yourself are never overwritten by the machine. Every taxonomy change — create, import, edit, delete — is recorded in the tenant **audit log** (`GET /audit`, admin-only) with who did it and when. ## How traces get classified Classification is done by the built-in **Issue Classification** judge, provisioned enabled for every agent: - Every 30 minutes, Neens takes the agent's recent **failure-set** sessions (the same selection criteria clustering uses — errors, below-threshold scores, human "fail" labels) that haven't been classified yet. Healthy traffic is never classified, and each session is classified once. - For each session, the classifier shows an LLM the session's evidence (input, output, messages, metadata) and the list of your non-archived failure modes with their definitions (up to 40), and asks for the **single** best-matching mode — or `none` if no listed mode clearly applies. Answers are snapped to exact mode names; the model can never invent a mode. - Each classification is stored as a score (`metric_key` = `issue_class`) carrying the mode name as its label, the classifier's confidence, and a one-sentence reason. - Spend is bounded: at most **500 sessions per day** per agent by default, drawn from the classifier's own daily budget. The classifier uses the agent's LLM connection, like every LLM feature. **With no connection configured, no classification happens** — the taxonomy still exists and is fully editable, but Issues show zero classified traces. With no (non-archived) taxonomy, the classifier doesn't run at all, so it never burns budget with nothing to classify against. ## Issues The **Issues** rollup shows each non-archived failure mode as a card with its lifecycle state, severity, classified trace count, mean classifier confidence, linked clusters, a trend sparkline, and its fix / eval-gate status. A mode with zero classified traces still appears — an Issue *is* a mode, whether or not it's currently occurring. Scope the rollup with the time-range picker (**Today**, **24h**, **7d**, **30d**, **All time**, or a custom range; default **7d**), and filter by search, severity, and lifecycle state. ### The analytics summary A summary strip sits above the cards: **total issues**, **active** vs **watching** counts, **affected traces** (sessions caught by an active issue in the window), and a severity / lifecycle breakdown across the active ones. These numbers are computed on the server over your *entire* taxonomy in the selected window, before the search/severity/state filters are applied — so narrowing the list to `severity=high` never changes what the summary reports. Widening or narrowing the time range does change it, since it re-windows which sessions count as "active". ### Active vs Watching Every issue falls into exactly one of two buckets for the selected window: - **Active** — classified traces > 0 in the window. Active issues render as full cards, sorted by classified trace count descending — the noisiest problem is always first, so triage order tracks actual impact rather than alphabetical order or creation date. - **Watching** — zero classified traces in the window. These are modes in your taxonomy that simply haven't happened (yet, or in this window). They render as a **collapsed**, compact one-line list — a severity dot, the name, its one-line definition, and "0 traces" — so a taxonomy of a hundred named modes doesn't bury the handful that actually need attention. Expand the **Watching** section to see them all, or use **Expand all** / **Collapse all** at the top of the tab to open or close every group (active cards and the watching list) at once. **Example.** Your taxonomy has 30 failure modes. This week, 6 of them classified at least one trace — one dominant mode with hundreds of sessions, the rest much smaller — and the other 24 classified none. The summary reads **6 active · 24 watching**; the 6 active cards are sorted loudest-first, and the 24 watching modes sit collapsed underneath until you expand them or widen the window. A **Watching** row's action is deliberately lighter than an active card's: a muted **Definition** link (so you can remind yourself what it means without opening the drawer) and an accent **Create eval →** link — pre-emptively locking the failure in as a regression test *before* it ever fires in production. **View traces** and **Generate fix** don't apply with zero traces, so they're not shown until the mode goes active. ### Lifecycle | State | Meaning | |---|---| | **Open** | Newly surfaced, not yet triaged | | **Acknowledged** | Someone has seen it and taken ownership | | **Investigating** | Being actively looked into | | **Mitigating** | A fix is in progress | | **Resolved** | Addressed — no longer expected to recur | | **Muted** | Deliberately silenced | Move an Issue between states with the **Lifecycle actions** menu on its card (or `PATCH /taxonomy/issues/{id}/lifecycle`). Any state can transition to any other; the one exception is a mode whose curation status is **Archived** — it's out of the operational loop entirely. **Resolved** and **Muted** Issues are hidden from the default list and suppressed from dashboards and insight highlights; select those states in the filter to bring them back. **Neens reconciles the lifecycle for you.** Every 30 minutes: an Issue whose linked [remediation](/guides/remediations) reaches *verified* (its proof eval passed) is auto-transitioned to **Resolved** (unless you muted it), and a **Resolved** Issue that receives new classified traces is reopened to **Open** — a fixed failure that recurs never stays silently "resolved". ### From an Issue to action An **active** Issue card carries three actions: - **View traces** — opens the traces the classifier assigned to the Issue in the selected window, each with its classification confidence and reason. - **Generate fix** → **View fix** — a stateful action: **Generate fix** drafts a typed remediation for the Issue when it has none; once one exists, the same spot becomes **View fix**, linking to it with its current status. See [Remediations](/guides/remediations). - **Create eval** — locks the failure in as a regression test (below). Once the Issue already has a gate, the card shows the gate's status here instead. A **watching** Issue's row keeps only **Definition** and **Create eval** — see [Active vs Watching](#active-vs-watching) above. ### Create an eval from an Issue #### Click **Create eval** on the Issue card Neens needs real evidence first: if the mode has no classified traces, linked clusters, or exemplars yet, the call is rejected with a clear message rather than creating an eval that would trivially pass forever. #### Neens builds the pieces One guided step (`POST /flywheel/failure-modes/{id}/generate-eval`) materializes a [dataset](/guides/datasets) captured from the failing traces, drafts an LLM [judge](/guides/judges) that scores whether the failure is *absent* (high score = good), and wires them together into an eval gate. #### Enable the gate The gate starts as a draft. Enable it on the **Eval Gates** page to start catching regressions — if the failure creeps back, the gate fails. An Issue that already has a gate shows it on the card instead of the **Create eval** action. ## Troubleshooting | Symptom | Cause | Fix | |---|---|---| | "No issue taxonomy yet" | The agent has no failure modes | Click **Import starter library**, **Import** a CSV/JSON file of your own taxonomy, add one manually, or confirm a cluster | | Issues all show zero traces | No LLM connection (classifier can't run), or no failing traffic in the selected window | Configure **Settings** → **LLM connections**; widen the time range | | "No classified issues in this window" | Taxonomy exists but nothing was classified into it recently | Normal when the agent is healthy; check **View traces** on the Failure Modes cards for unclassified failures | | Everything is in **Watching**, nothing **Active** | No traces were classified into any mode in the current window | Widen the time range, or check the LLM connection / classifier as above | | Summary tiles don't match the cards you can see | The summary counts the whole taxonomy in the window, before your search/severity/state filters | Expected — clear the filters to see the same population the summary describes | | **Create eval** fails with "no evidence sessions" | The mode has no classified traces, linked clusters, or exemplars | Wait for classification, link a cluster, or add exemplars first | | A resolved Issue reopened by itself | New traces were classified into it after it was resolved | That's the recurrence guard working — investigate the new traces | ================================================================================ # Topics Source: /docs/guides/topics/ ================================================================================ # Topics Topics organize your traffic by **what users are asking about** — billing disputes, password resets, refund requests — where [failure modes](/guides/issues-and-failure-modes) organize it by *what goes wrong*. The **Topics** page is a taxonomy atlas: topics grouped into spaces, each with volume, failure-rate, and coverage metrics, so you can see which subjects dominate your traffic and which ones your evaluation coverage is missing. ## At a glance | | | |---|---| | **Where it lives** | **Topics** in the sidebar | | **Key API routes** | `GET /topics/atlas`, `POST /topics`, `POST /topics/spaces`, `PATCH /topics/{id}`, `POST /topics/{id}/merge`, `POST /topics/{id}/split` | | **What it needs** | Nothing — topics involve no LLM calls and no LLM connection | | **Scope** | Per agent | ## Concepts - **Space** — a top-level grouping of topics (e.g. "Support intents"). A topic belongs to one space and displays a path like `Support intents / Billing dispute`. - **Topic** — a named subject with a severity (`high` / `medium` / `low`), an owner, a summary, and free-text instructions (e.g. routing guidance for your team). - **Assignments** — the sessions associated with a topic. Assignments are what the topic metrics and slices are computed from; merging topics moves them. Each topic carries a **source** telling you where it came from: | Source | Meaning | |---|---| | `proposed` | Suggested by the Neens starter atlas (see below) | | `manual` | Created by your team (**New topic**) | | `split` | Split off from another topic | **Topics are a curated taxonomy, not a live classifier.** The first time an agent touches Topics, Neens seeds a starter atlas: if [failure clusters](/guides/clustering) already exist, it proposes one topic per cluster (marked **Proposed**); otherwise it seeds a generic set of support intents, and spreads a sample of existing sessions across them so the metrics render. From there the atlas is yours to curate — Neens does not currently run a continuous LLM topic classifier over incoming traffic, and no LLM connection is required. ## Working with the atlas The header tiles summarize the filtered atlas: **Total topics**, **Spaces**, **Proposed** (suggestions awaiting curation), and **Coverage gaps** (topics whose gap score is 0.5 or higher). Filter by space, severity, source, or a name search; click any column header to sort. - **New space** / **New topic** — create a grouping or a topic (name, severity, owner, summary, instructions). - Click a topic to open its detail panel: metrics, summary, owner, instructions, and up to five example sessions. From there: - **Curate** — edit the topic's fields (`PATCH /topics/{id}`); this is also how you adopt a **Proposed** topic. - **Merge** — fold this topic into another: its session assignments move to the target and the source is archived. - **Split** — carve a new sub-topic off this one (it lands with source `split`, under the parent's path). - **Archive** — hide the topic from the atlas without deleting anything. - **Create dataset** — start a [dataset](/guides/datasets) from the topic's sessions. ## The metrics Every topic row computes its metrics live from its assigned sessions: | Metric | What it is | |---|---| | **Volume** | Number of sessions assigned to the topic | | **Failure rate** | Share of those sessions with an error/failed status | | **Coverage gap** | 0–1 heuristic that rises with failure rate and falls with volume — high-failure, low-volume topics score highest. A topic with no sessions scores 0.8 (unknown = uncovered). Topics at 0.5+ count in the **Coverage gaps** tile | | **Share** | The topic's volume as a fraction of the (filtered) atlas total | Use the coverage gap as a to-do list for your eval program: a topic failing often with little traffic is exactly where a targeted [dataset](/guides/datasets) and [judge](/guides/judges) pay off most. ## Slicing by topic Topics are a first-class dimension across Neens: - **Dashboards** — the metrics catalogue includes topic measures (**Top topics**, **Topic share**), and the *topic* dimension can slice supported measures in custom dashboard widgets. Note that topic assignments carry no timestamp, so the dashboard time range doesn't narrow topic-assignment counts. - **Datasets** — a dataset can be sourced from a topic (via **Create dataset** on the topic, or a topic filter on the dataset source), turning a subject area into a reusable evaluation corpus. See [Datasets](/guides/datasets). ## Troubleshooting | Symptom | Cause | Fix | |---|---|---| | The atlas is full of topics you didn't create | The one-time starter seed proposed them (from your failure clusters, or the generic starter set) | Curate: rename and adopt the useful ones, **Merge** or **Archive** the rest | | A topic's volume is 0 | No sessions are assigned to it | Merge sessions in from an overlapping topic, or treat it as a coverage gap to instrument | | Coverage gap seems high everywhere | Low volume per topic dominates the heuristic | Consolidate near-duplicate topics with **Merge** so volume concentrates | ================================================================================ # Prompt management Source: /docs/guides/prompt-management/ ================================================================================ # Prompt management **Prompt management** gives your prompts the same first-class lifecycle your code already has: a named entry in a registry, an immutable version history, and movable **deploy tags** (like `production` and `staging`) that point at a specific version. A built-in **playground** runs any version against your agent's own [LLM connection](/administration/llm-connections) so you can iterate before you deploy — and a prompt-change [remediation](/guides/remediations) can be turned into a new version with one click. The **Prompts** page also carries an **Optimizer** tab, where Neens searches for a better system prompt for a confirmed failure mode instead of you writing one by hand — see [Prompt optimization](/guides/prompt-optimization). ## Concepts - **Prompt** — a named, agent-scoped entry. Two types: **chat** (a list of `system` / `user` / `assistant` messages) or **text** (a single template string). - **Version** — an **immutable** snapshot of a prompt's content + model config, numbered from 1. Editing never overwrites: saving creates the next version, so history is a complete audit trail. - **Variable** — a `{{name}}` placeholder inside message content. Neens auto-detects the variables in each version so the playground can prompt you to fill them in. - **Deploy tag (label)** — a movable pointer like `production` or `staging` that names a version. Your agent or an SDK asks for "the `production` prompt" and gets whatever version the tag points at today — move the tag to roll forward or back without a code change. `latest` is implicit (it always resolves to the newest version), so it is a reserved name you can't assign by hand. ## Create and version a prompt ### Create a prompt On the **Prompts** page, click **New prompt**. Give it a name, pick **chat** or **text**, write the message(s), and set the model + temperature. Any `{{variable}}` you reference is detected automatically. Saving creates **version 1**. ### Add a version Open a prompt and edit its content, then **Save as new version**. This appends the next version (v2, v3, …) with an optional commit message — the previous version is untouched and stays in the history, exactly like a judge's [versioned definition](/guides/judges). ### Deploy a tag Point a deploy tag at the version you trust. Set `production → v2` to promote it; move the tag back to `v1` to instantly roll back. Deploying a prompt is also recorded as a [what-changed](/guides/what-changed) event, so a later regression can be correlated to the rollout. ## Playground The playground runs a prompt against your agent's **default LLM connection** (or a connection you pick as an override). Fill in the `{{variables}}`, choose the model and sampling parameters, and **Run** — Neens makes a single live model call and shows the output, token counts, and latency. Nothing is persisted: the playground is for iteration, not scoring. The playground makes a real, billable model call, so it is **rate-limited per user** and the output is **token-capped** (2048 tokens by default). If your agent has no LLM connection yet, the run returns a clear prompt to configure one under **Settings → Connections**. ## Resolve a prompt by name The registry is the source of truth an agent or harness can read at runtime. Resolve a prompt by its name and (optionally) a deploy tag or explicit version — with neither, you get the newest version: ``` GET /prompts/by-name/{name} # newest version GET /prompts/by-name/{name}?label=production # whatever `production` points at GET /prompts/by-name/{name}?version=2 # a specific version ``` The response carries the resolved version's content, its detected variables, and the prompt's current tags — so your agent can pull "the production system prompt" without hard-coding it. ## From a remediation to a new version When Neens proposes a **prompt-change** [remediation](/guides/remediations) — a corrected system prompt grounded in real failing traces — you don't have to copy-paste the fix back into your agent. **Save it to the prompt registry** and Neens lifts the remediation's corrected prompt into a new version of the target prompt (or a brand-new prompt). This is the "one-click apply" that closes the loop from a diagnosed failure to a shippable, versioned, deploy-tagged fix. ## Related - [Judges](/guides/judges) — scorers whose prompts follow the same versioned lifecycle. - [Prompt optimization](/guides/prompt-optimization) — the **Optimizer** tab on this page. - [Remediations](/guides/remediations) — where prompt-change fixes are generated from failures. - [What changed](/guides/what-changed) — a prompt deploy is recorded here for regression correlation. ================================================================================ # Judges Source: /docs/guides/judges/ ================================================================================ # Judges A **judge** is a scorer that grades each trace or session against a rubric and produces a quality signal between 0 and 1 — plus a written reason you can audit. Judges are how raw traces become [scores](/guides/scores): enable one, run it, and every graded trace gets a number you can filter, chart, and gate on. ## At a glance | | | | --- | --- | | **Where** | The **Judges** page — **Judges**, **Alignment**, **Run Targets**, and **Comparison** tabs | | **Key API routes** | `POST /judges`, `POST /judges/{id}/versions`, `POST /judges/{id}/deployments`, `PATCH /judges/{id}/lifecycle`, `POST /eval-runs`, `GET /eval-runs/{id}` | | **Needs** | An LLM connection (**Settings → Connections**) for LLM-prompt and composite judges | | **Scope** | Judges can be agent-, org-, or platform-level; enablement and run history are always per agent | | **Produces** | Rows in the [score catalogue](/guides/scores), one metric per judge | ## Judge types | Type | What it does | | --- | --- | | **LLM Prompt** | An LLM reads evidence from the trace and returns `{score, reason}` against your instructions. | | **Composite** | Combines other judges' scores into one number — a weighted sum or an arithmetic expression. | | **External API** | Calls your own scoring endpoint. | | **Custom Python** | Author a Python scoring function. **Authoring only — execution is not yet supported**, so these judges can't run in eval-runs, continuous eval, or pre-prod yet. | Each judge targets either a **trace** (one span tree) or a **session** (a conversation rollup). Neens also ships a built-in judge library — **Faithfulness**, **Answer Relevancy**, **Hallucination**, **Toxicity**, **Bias**, **PII Leakage**, **Correctness**, and more — visible in every agent under **Available judges**. Library judges are visible but score nothing until you enable them. Two scorers are provisioned per agent out of the box: - **Primary Score** — a composite that blends **Faithfulness**, **Answer Relevancy**, and **Coherence** into one headline quality number and scores a sample of incoming traces continuously (see [Continuous evaluation](/guides/continuous-evaluation)). - **Issue Classification** — a classifier that tags failing sessions with the failure mode they exhibit, from your taxonomy. It runs on the failure set only, not on healthy traffic. ## Create a judge ### Define it Click **Create judge**. For an **LLM Prompt** judge you fill in: | Field | Purpose | | --- | --- | | **Name** | The judge's label — its scores group under a metric key derived from it, so it is fixed after creation. | | **Type** | LLM Prompt, Composite, External API, or Custom Python (Custom Python is authoring-only — it can't run yet). | | **Target type** | Score a **trace** or a **session**. | | **Scope** | **Agent** (this agent only), **Org** (all agents in the org), or **All orgs & agents**. | | **System prompt** | The judge's persona / global instruction, sent as the system message. | | **Judge instructions** | Plain-English task for the model — no placeholder syntax to write. | | **Criteria** | Your rubric, appended in a dedicated block so the model grades against it. | | **Evidence to attach** | Which parts of the trace the judge sees (below). | | **Output schema (JSON)** | The JSON verdict shape the model must return (e.g. `score` + `reason`). | | **Output type** | The native scale the judge scores on — a 0–1 score, a numeric range, categorical labels, or binary pass/fail (see [Output types](#output-types)). | | **Max tokens / Temperature** | Optional generation settings (blank → `512` / `0.0`). | ### Pick the evidence **Evidence to attach** controls exactly which blocks of the trace are assembled into the prompt. Tick only what your criteria needs: | Evidence | What it is | | --- | --- | | **Final input** | The user's request to the agent. | | **Final output** | The agent's user-facing answer. | | **Full message trajectory** | Every span in order — reasoning, tool calls, retrievals. For trajectory and tool-use judges. | | **Retrieved context** | Retrieved documents/chunks. For RAG, faithfulness, hallucination. | | **Expected / reference answer** | The gold answer carried on the trace. For correctness and exact-match. | | **Session metadata** | Agent name, status, and custom metadata. | If you tick nothing, the judge gets the default set: final input, final output, the full trajectory, and metadata. Evidence that a trace legitimately lacks (say, no retrieved context) is simply omitted, and library judges are instructed to grade what *is* present rather than penalize the absence. Evidence is injected as clean, labeled text sections — a readable transcript the model grades far more reliably than a raw dump. Tick **Also inject raw JSON** to additionally append the full raw trace payload as JSON alongside that readable evidence. **No placeholders.** You never template evidence into your prompt by hand — the checkboxes decide *what* the judge gets, and Neens assembles the prompt automatically. There is nothing to write like `{{input}}` or `{{output}}`; if your instructions contain such a string it is passed to the model as literal text, not substituted. ### What actually gets injected **Final input** and **Final output** are the *parsed* exchange — the shopper's question and the agent's user-facing reply — not the raw body of some span. Neens derives them from the trace with the same reconstruction that powers the **Conversation** tab on a trace and golden-dataset capture, so what the judge grades is what you see when you open the trace. Concretely, for a trace whose spans were emitted by a LangChain/LangGraph agent, the stored span body looks like this: ```json [{"lc": 1, "type": "constructor", "id": ["langchain", "schema", "messages", "HumanMessage"], "kwargs": {"content": "Can I return a sale dress?"}}] ``` …and what reaches the judge is this: ```text Evidence: INPUT (the user's request / task): Can I return a sale dress? OUTPUT (the agent's response to evaluate): Sale items are final sale. ``` The rules, in full: | Situation | What the judge sees | | --- | --- | | Ordinary single-turn trace | The user's question and the agent's reply, as text. | | **Multi-turn** trace | The **last** exchange — the final reply, paired with the request it answers (not the opening question, which would ask the model to grade an answer to a different question). Tick **Full message trajectory** if the rubric needs the earlier turns too. | | Serialized message payloads (LangChain constructors, OpenAI/Anthropic message arrays, content-block lists) | Unwrapped to plain text. Framework scaffolding never reaches the model. | | The agent **never replied** (guardrail block, a tool-only run, a failed run that ended on a retry decision) | `(the agent produced no user-facing reply in this trace)`. A routing decision, planner output, or guardrail verdict is *not* promoted to "the answer" — grading one of those scores the control flow, not the agent. | | No user turn is recoverable (a scheduled/campaign agent with no human request) | `(no user request could be derived from this trace)`. | | **Full message trajectory** on a tool-heavy trace | Every span in order, with each body parsed the same way. A genuinely structured record — a tool's arguments or result — stays compact JSON, because it *is* data the judge may be grading. | **Prompts are bounded.** Each block is capped so one pathological span body cannot produce a megabyte prompt: 8,000 characters for input/output/expected answer, 12,000 for retrieved context, 24,000 for the trajectory (2,000 per step, 80 steps), 4,000 for metadata. A cut is always stated in the prompt itself — `… [truncated: showing the first 8,000 of 40,000 characters]` — and the model is told not to read a truncation marker as the agent stopping mid-answer. **Also inject raw JSON** is deliberately *not* truncated: if you ask for the raw payload you get all of it. **Changing judge?** Before this behavior existed, **Final output** was the raw body of the last answer-bearing span, which on framework-emitted traces was often a JSON blob. Judges written against that blob — for example a rubric that says "the `action` field must be `create_ticket`" — now see the parsed reply instead. Open the **Live preview** on a real trace to see exactly what your judge gets, and tick **Also inject raw JSON** if a rubric genuinely depends on the raw shape. ### Check the live preview — and test-run it For LLM Prompt judges, the **Live preview** panel renders the *exact* system and user message that a run would send — hydrated on a **real sample trace** from your agent. It renders in three legended blocks — **Your prompt**, **Injected trace data**, and **Output contract** — so it's clear which part is your instruction versus the trace evidence Neens assembles. As you edit instructions or toggle evidence the preview updates; use the sample dropdown to try a different trace. There is no second templating path, so what you preview is byte-for-byte what runs. If the agent has no traces yet, the preview shows the prompt structure only. Click **Test run** to execute the draft prompt against that sample trace immediately, using the agent's default LLM connection. The verdict — the score or label, **PASS**/**FAIL**, the reason, and the raw model response — is shown but **not saved**, so you can tune the rubric before you ever enable the judge. If the agent has no LLM connection, the panel prompts you to set one up. ### Save — and iterate with versions Editing a judge later publishes a new immutable **version** (with an optional change note) via **Save as new version**. The new version becomes current; older versions are kept so score history stays intact, and each enabled scorer records which version it runs. ### Output types By default an LLM Prompt judge returns a **Score 0–1** and Neens asks the model for a `score` float. But many rubrics are naturally a numeric range or a set of labels — so the **Output type** control lets a judge score on its **own** native scale while Neens keeps a normalized 0–1 score under the hood for dashboards, composites, clustering, and pass-rate. The judge's native value is captured alongside for display, and you define what counts as a pass in native terms. | Output type | The model returns | You configure | Pass rule | | --- | --- | --- | --- | | **Score 0–1 (default)** | A `score` float in `[0,1]`. | Nothing — this is the classic behavior. | `score` ≥ the deployment threshold. | | **Numeric range** | A `score` number on your scale. | **Min**, **Max**, optional **Step**, optional **Pass at ≥**, and **Higher is better**. | `score` ≥ **Pass at ≥** (blank → the equivalent of `0.7` normalized). | | **Categorical labels** | A `label` from your ordered list. | **Labels (ordered worst → best)**, optional **Passing labels**, and **Higher is better**. | The label is in the **Passing labels** set (blank → the top half). | | **Binary pass/fail** | A `label` that is your pass or fail word. | **Pass label** and **Fail label**. | The label equals **Pass label**. | The stored score is always the normalized 0–1 value, so every downstream chart and composite keeps working unchanged; the native score/label is what you see on run drill-downs and the [score catalogue](/guides/scores). A judge with no output type set behaves exactly like the classic 0–1 path. **The output type is also what the Scores page reads.** A **Categorical labels** or **Binary pass/fail** judge gets a **label distribution** — a donut cut into one arc per label, with a named legend and the window's scored count in the centre — instead of an average donut, and its drill-down shows the share of each label over time. See [Boolean and categorical scorers](/guides/scores#boolean-and-categorical-scorers). Your ordered **Labels** are the order the legend renders in, and a label that never occurred in the window still shows at 0% in the drill-down. **Composites always emit 0–1.** A composite judge blends its components on the normalized axis, so it can't take a native output type — pick one only on the LLM Prompt (and External API) judges that feed it. A non-continuous component is normalized before it's aggregated. ### Composite judges A composite judge combines other judges. Pick a **method**: - **Weighted sum** — assign a weight to each component; **Normalize to 1.0** rebalances weights over the components that are actually present per target. - **Expression** — arithmetic over component scores (e.g. `0.6*answer_completeness + 0.4*bias`), referencing each component by its variable name. Supported operations: `+ - * / ** %` and `min` / `max` / `abs` / `round`. An invalid expression fails the run up front rather than degrading silently. Set a **Missing policy** — **Skip missing** (a target missing a component score is skipped) or **Fail on missing** — and a **Threshold** for the composite's pass/fail label. When a composite runs, each component's most recent score is reused if one exists; an LLM-prompt component with no score yet is scored inline once (its own score row is written too, so it is cached for next time). **Primary Score** works exactly this way, which is why its component metrics appear in the catalogue without their own enabled scorers. ## Lifecycle stage Every judge carries a **lifecycle stage** — a curation signal that says how far along the prompt is, without changing anything about how the judge scores: | Stage | Meaning | | --- | --- | | **Experimental** | A work-in-progress scorer you're still tuning. This is where **every new judge you create starts**. | | **Finalized** | Reviewed and trusted for everyday use. The built-in library judges (**Faithfulness**, **Primary Score**, and the rest) ship **Finalized**. | | **Archived** | Retired — kept for its score history but no longer part of your working set. | Change a judge's stage from the **Judges** page. The move is one-directional in spirit — experimental → finalized → archived — but you can set any stage at any time (e.g. re-open an archived judge back to experimental). **Stage is curation only.** Promoting or archiving a judge never changes what it scores, whether it runs, or any existing scores. What it *does* change is the default [Scores](/guides/scores) view: that page shows **Finalized** scorers by default, so an experimental judge you're still iterating on doesn't clutter everyone's metrics until you promote it. See [Scorer lifecycle](/guides/scorer-lifecycle) for the whole flow. The change is recorded in the [activity/audit](/administration/audit-log) trail and takes the same permission as configuring a judge, so a viewer can't silently re-stage someone else's scorer. Via the API, `PATCH /judges/{id}/lifecycle` with `{"stage": "experimental" | "finalized" | "archived"}` (any other value is a `422`); a scoped caller can only re-stage a judge visible to its agent. ## Enable it as a scorer Creating (or having) a judge scores nothing by itself. Click **Configure** to enable it as a **scorer** in the current agent, where you choose: - **Version** to run. - **Apply for** — **Manual (run on demand)** or **Continuous (score new traces)**. See [Continuous evaluation](/guides/continuous-evaluation). - **What traces to score** — everything eligible, a fixed count, or a percentage sample. A live estimate shows how many traces are **Eligible** and how many it **Will score**. - **Target source** — **Filter** (traces matched by status/agent/start-date filters) or **Dataset** (score only the members of a [dataset](/guides/datasets); sampling still applies). - **Scoring connections (pool)** — which LLM connection(s) score the traces. Pick several to spread load across providers and dodge rate limits; leave empty to use the agent's default connection. - **Success threshold** — an optional per-scorer minimum success rate for a run to count as fully completed (blank → the platform default, see [Run outcomes](#run-outcomes)). Enabled scorers appear in the **Enabled scorers** table with two independent controls: **Run now** (a one-off run over existing traces) and **Pause / Resume** (continuous) or **Disable / Enable** (manual) for automatic scoring. **Remove** deletes the scorer; removing a platform-default judge sticks — it won't silently re-enable itself. You can also ask the [Assistant](/guides/assistant) to do this — *"turn that judge on and score the last week"* — or drive it from an agent over the [MCP server](/guides/mcp). Neither needs a version id: omit it and the judge's latest version is deployed, and an eval can be started from the judge alone when it has a single enabled deployment. **Connections are chosen at run time, not saved into the judge.** A judge definition is portable; the LLM connection or pool is picked when you enable or run it. If an agent has no visible connection, an LLM-judge run fails cleanly (every target marked failed with the error) — it never borrows another agent's credentials. ## Run it Click **Run now** on an enabled scorer. A dialog pre-fills the scorer's connection pool and success threshold and lets you override both for this run. While the run is live, the table's status, run count, and last-run time update in place every few seconds — no page reload. Click a scorer row to open its run history and drill into per-trace results: each target's score, the judge's reason, raw model output, attempt count, and any error. A running run can be **cancelled** — in-flight calls finish and the rest are dropped, and it then reads **Cancelled** on both this page and the [Activity](/guides/activity) feed. See [Cancel or recover a run](#cancel-or-recover-a-run). ```bash # Preview how many traces a filter would score curl -X POST /eval-targets/estimate \ -H 'Content-Type: application/json' \ -d '{"filter": {"sampling": {"mode": "percentage", "percentage": 10}}}' # → {"eligible": 1240, "sampled": 124} # Start a run for an enabled scorer (deployment) curl -X POST /eval-runs \ -H 'Content-Type: application/json' \ -d '{"deployment_id": "dep-…"}' # → {"id": "run-…", "status": "queued", "total": 124, …} # Cap THIS run at the 200 most-recent eligible traces (the deployment's filter is untouched) curl -X POST /eval-runs \ -H 'Content-Type: application/json' \ -d '{"deployment_id": "dep-…", "sample_size": 200}' # → {"id": "run-…", "status": "queued", "total": 200, …} (total is what was actually queued) # Poll the run: status, per-target tasks, scores + reasons curl /eval-runs/run-…?limit=200 # Stop it curl -X POST /eval-runs/run-…/cancel ``` `POST /eval-runs` accepts optional `pool_connection_ids`, `success_threshold`, and `sample_size` overrides; omitted, the run inherits the scorer's stored configuration. `pool_connection_ids` and `success_threshold` are stored on the run so the verdict is reproducible; `sample_size` caps the run at the N most-recent eligible traces (leaving the deployment's stored sampling filter unchanged), and the returned `total` is the number actually queued. `GET /judges/{id}/runs` lists a judge's full run history, and `GET /judges/run-status` is a lightweight status-only feed suited to frequent polling. ## Cancel or recover a run A scorer run moves through **queued → running** to a terminal state — **completed**, **completed-with-failures**, **failed**, **cancelled**, or **skipped** (see [Run outcomes](#run-outcomes) for the grading bands and the [Activity feed](/guides/activity#run-statuses) for the full status list). Two things are worth knowing before you launch one. **Enabling a scorer doesn't run it.** Turning a scorer on only makes it eligible: a **manual** scorer scores nothing until you click **Run now**, and a **continuous** scorer scores only *new* traces as they arrive — never your existing history. To score the traces you already have, use **Run now**. (The Judges page says the same inline: *"Enabling a scorer doesn't run it automatically. Click Run now…"*) ### Cancel a running run Click **Cancel** on a live **Run now** run — or `POST /eval-runs/{id}/cancel`. Cancelling takes effect promptly: any scoring call already in flight finishes, but the run's remaining queued targets are **dropped rather than drained**, so scoring stops within seconds instead of grinding through work you no longer want. The cancelled run then reads **Cancelled** in **both** places it appears — its scorer's run history on the **Judges** page *and* the run's row on the [**Activity**](/guides/activity) page — so the two never disagree about whether it is still going. Whatever scored before you stopped it keeps its scores, but a cancelled run's partial counts are never graded into a pass/fail verdict. **Example.** You enable **PII Exposure** and click **Run now** over your recent sessions, then change your mind and click **Cancel**. Within a few seconds the run shows **Cancelled** on both the Judges page and the Activity feed, no further sessions are scored, and the sessions already scored keep their scores. When you're ready, just **Run now** again. ### If a run is interrupted If the processing behind a run is interrupted while it is still in flight, the run is **automatically moved to a terminal state** — it reads **Cancelled** rather than being left showing **Running** forever. So a scorer never gets wedged: once the stuck run is closed out, simply **Run now** again to re-score cleanly. This is the same self-healing you see for clustering runs on the [Activity feed](/guides/activity) (a stalled clustering run is marked **failed** there). ## How it works When a run starts, Neens resolves the scorer's judge version and target set (the filter or dataset, sampled as configured), creates one task per target, and scores them: 1. **Evidence assembly** — for each target, the selected evidence blocks are read from the trace, [parsed](#what-actually-gets-injected), and rendered into the prompt. The live preview, the **Test run** button and a real run all call the same assembly and the same renderer, so they cannot drift: what the preview shows is byte-for-byte what the model receives. 2. **Scoring** — the prompt is sent to a connection from the run's pool; targets round-robin across pool members. The model must return strict JSON — `{"score": …, "reason": …}`, or `{"label": …, "reason": …}` for a categorical or binary [output type](#output-types) — which is parsed defensively and normalized to a 0–1 score. 3. **Persistence** — each verdict is written as a score row (metric = the judge, source `llm_judge`, or `composite` for composites) and appears immediately in the [score catalogue](/guides/scores), on run drill-downs, and in [dashboards](/guides/metrics). Judge calls within a run execute sequentially — one LLM call at a time per run. **Transient failures are retried.** A timeout, `429`, or `5xx` from the LLM provider is retried with exponential backoff — up to 3 retries per target by default, with delays growing from 0.5s up to a 30s cap. Terminal failures (an unparseable verdict, a config or auth error) are never retried; that target is marked failed with its error and the run moves on. ### Run outcomes A finished run is graded by its **success rate** — the fraction of *attempted* targets that scored successfully (skipped and cancelled targets don't count against it): | Status | Meaning | | --- | --- | | `completed` | Success rate ≥ **0.90** — healthy (green). | | `completed_with_failures` | Between the two cutoffs — the run finished, but a notable share of targets errored (yellow). | | `failed` | Success rate < **0.10** — effectively broken (red). | | `cancelled` | You stopped the run. | The green cutoff can be overridden per scorer (**Success threshold** in Configure, or per run in the Run dialog). ### Filter a run's scores, compare versions, and export A judge's run history (open it by clicking a scorer row) is also where you audit and compare **ad-hoc** batch runs — the manual **Run now** runs, each tied to one judge [version](#save--and-iterate-with-versions). These flows read each run's own per-target **snapshot**, so a later run overwriting the shared score row never corrupts an older run's numbers. - **Scope the Scores page to one run.** From a specific ad-hoc run you can view exactly the scores *that* run produced, rather than the metric's current live scores. API: `GET /scores?runId=` (returns the run's snapshot — this is meant for ad-hoc runs, not a continuous scorer that overwrites each trace's row every time it re-scores). - **Compare two versions.** Pick two runs of the same judge — typically runs of two different versions — to see, per target, which sessions **improved**, **regressed**, or stayed **unchanged**, plus each side's score and reason and the average delta. API: `GET /judges/{id}/compare?runA=&runB=`; a run from another judge or agent is a `404`. - **Export a run with input and output.** Download a run's (or several runs') scores as **CSV** or **JSON**, including each target's **input** and the agent's **output** alongside the score, label, threshold, and reason — handy for spreadsheets, offline review, or sharing a regression. API: `GET /eval-runs/export?runs=,&format=csv|json`. These read the run's captured snapshot, so version comparison and export stay accurate even after the judge is re-run. They're aimed at bounded ad-hoc runs; a continuous scorer re-grades traces in place, so a single "run" isn't a meaningful export boundary for it. See [Scorer lifecycle](/guides/scorer-lifecycle) for when to reach for each. Definition reference (LLM Prompt judges) | Definition field | Meaning | | --- | --- | | `system_prompt` | System message — the judge's persona and ground rules. | | `prompt_template` | The task instruction (the UI's **Judge instructions**). | | `criteria` | The rubric intent, rendered in a `Criteria:` block. | | `evaluation_steps` | Optional ordered checklist the model must follow (library judges use this G-Eval-style structure). | | `rubric` | Optional anchored score bands, e.g. `[{"range": "0.0-0.3", "when": "…"}]`. | | `required_data` | Evidence keys: `input`, `output`, `messages`, `context`, `expected_output`, `metadata`. Empty → the default set (`input`, `output`, `messages`, `metadata`). `input`/`output` are the [parsed exchange](#what-actually-gets-injected). | | `include_raw_evidence` | When `true`, also append the raw trace payload as JSON alongside the readable evidence (the **Also inject raw JSON** toggle). | | `output_schema` | The verdict shape, e.g. `{"score": "number", "reason": "string"}`. | | `output_spec` | The native output scale (the **Output type**): omitted / `{"type": "continuous_unit"}` for a 0–1 score, `{"type": "numeric_range", "min", "max", "step?", "pass_min?", "higher_is_better"}`, `{"type": "categorical", "labels", "passing?", "higher_is_better"}`, or `{"type": "binary", "pass_label", "fail_label"}`. Neens always stores a normalized 0–1 score plus the native value. | | `target_type` | `trace` or `session`. | Composite definitions carry `method` (`weighted_sum` or `expression`), `components` (each with a `key`, label, and `weight`), `renormalize`, `min_components`, `missing_policy` (`skip` / `fail`), `threshold`, and — for the expression method — `expression`. ## Judge ↔ human alignment An LLM judge is only useful if you can trust it. The **Alignment** tab measures each judge's agreement (precision, recall, Cohen's κ) against human gold labels and lets you drill into disagreements. See [Annotations & review](/guides/annotations-and-review) for the full label-review-align loop, and [Pre-prod evaluations](/guides/preprod-evals) for running your trusted judges against a candidate release. ## The Run Targets and Comparison tabs Two more tabs on this page go one level deeper on *what* a scorer runs on and *which* scorer model to trust: - **Run Targets** — the library of reusable populations (a golden dataset, a live sample, a filtered sweep) that scorers run against, plus each target's **Used by** list. Attaching a scorer to a Run Target is done from that scorer's own **Run Targets** tab. Full guide: [Run Targets](/guides/run-targets). - **Comparison** — holds one judge's rubric fixed and scores the same window of traces with two or more candidate **scorer LLMs**, side by side, so you can see whether a cheaper model agrees closely enough to switch. Full guide, with a worked example: [Cost Optimization → Compare scorer models side by side](/guides/cost-optimization#compare-scorer-models-side-by-side). **Run Targets and Comparison used to be separate pages in the sidebar.** They now live as tabs on **Judges**, alongside **Judges** and **Alignment** — everything about *what a scorer measures* and *what it runs on* is in one place. An old `/scorer-comparison` link still works: it redirects to the **Comparison** tab. ## Troubleshooting | Symptom | Likely cause | Fix | | --- | --- | --- | | Run fails immediately, every target errored | No LLM connection visible to this agent | Add one in **Settings → Connections**, or pick a pool in the Run dialog. | | Scores are 0 with reason "no evidence" | The judge's evidence needs data the traces don't carry (e.g. retrieved context on a non-RAG agent) | Check the live preview against a real trace; trim **Evidence to attach** to what your traces have. | | The prompt shows `(the agent produced no user-facing reply in this trace)` | The trace really has no assistant turn — it ended on a guardrail block, a tool failure, or a planning/retry decision | Expected for failed runs. If your agent *does* reply, its answer is probably on a span kind Neens doesn't treat as a turn — check the trace's **Conversation** tab, and see [Traces & sessions](/guides/traces-and-sessions). | | The prompt shows `(no user request could be derived from this trace)` | No span recorded the user's message — common for scheduled/campaign agents that aren't user-initiated | Record the request (or the trigger) as the input of the first LLM or agent span. | | A judge's scores changed after an upgrade without the rubric changing | The rubric was written against the raw span body, which is now the [parsed reply](#what-actually-gets-injected) | Re-read the **Live preview** on a real trace; tick **Also inject raw JSON** if the rubric genuinely needs the raw payload. | | The prompt ends in `… [truncated: showing the first N of M characters]` | A span body exceeded its block budget | Expected and safe — the model is told it's an excerpt. Narrow **Evidence to attach**, or shorten what the agent logs. | | Run ends `completed_with_failures` | A minority of targets hit provider errors or unparseable verdicts | Open the run drill-down and read per-target errors; transient provider errors were already retried. | | A composite skips most targets | Components have no scores and can't be scored inline (missing connection, or non-LLM components) | Ensure component judges are LLM-prompt judges and a connection is configured. | ================================================================================ # Scores Source: /docs/guides/scores/ ================================================================================ # Scores A **score** is one quality verdict about one trace or session — a number between 0 and 1, plus the reasoning behind it. Scores are what [judges](/guides/judges) produce and what everything downstream consumes: filters, dashboards, failure clustering, and release gates. The **Scores** page is the catalogue where every metric your agent produces lives, with per-user favorites and hiding so you see the metrics you care about first. ## At a glance | | | | --- | --- | | **Where** | The **Scores** page ("Quality metrics from judges across your traces and sessions") | | **Key API routes** | `GET /scores`, `GET /scores/metrics`, `GET /scores/timeseries`, `PUT /scores/prefs`, `GET /scores/display-settings` (all take a `lifecycle` filter) | | **Fed by** | Judge runs, [continuous evaluation](/guides/continuous-evaluation), and [pre-prod evaluations](/guides/preprod-evals) | | **Feeds** | Trace detail, the Traces **Score** filter, [dashboards & metrics](/guides/metrics), [failure clustering](/guides/clustering) | | **Personal prefs** | Favorite / hide per user, per agent — view-only, never deletes data or stops a judge | ## Anatomy of a score Every score row carries: | Field | Meaning | | --- | --- | | `metric_key` | Which metric this is (e.g. `faithfulness`, `primary_score`). Derived from the judge's name. | | `source` | Where the score came from (see below). | | `target_type` | What was graded — a `trace`/`span` or a `session`. | | `target_id` | The specific trace or session that was graded. | | `score` | The numeric verdict, clamped to 0–1. | | `label` | `pass` or `fail` against the row's threshold — except classifier scores, whose label is the detected failure-mode name. | | `threshold` | The pass/fail cutoff for this row (default `0.7`; composite judges carry their own). | | `reason` | The judge's written rationale — every score is auditable. | | `raw_output` | The judge's full raw verdict (JSON), viewable in the score modal. | | `judge` | The judge that produced it. | | `model` / `model_source` | **Which model produced the answer that was graded, and how that was known** — recorded when the score is written, never inferred later. The model may be a real id, the **Mixed** bucket (the graded trace used more than one model), or **Unknown**. | | `agent_name` | The agent behind the graded trace, recorded alongside the model so quality can be read per agent × model. **Unknown** when the score's target has no resolvable trace. | The last two are what make quality sliceable per model; they surface through the [metrics catalogue](/guides/metrics) (as the **Model**, **Agent**, and **Model attribution** dimensions) rather than as columns on the Scores list — see [Model comparison](/guides/model-comparison). A score *type* — one card in the catalogue — is identified by the triple **(`metric_key`, `source`, `target_type`)**. The same metric key can appear more than once if different sources or target types produce it. **One current score per metric per target.** Score ids are deterministic, so re-running a judge on the same trace **overwrites** its previous verdict for that metric instead of accumulating duplicates. Score history in charts reflects when targets were scored, not repeated re-grades of one trace. ### Where scores come from | `source` | Produced by | | --- | --- | | `llm_judge` | LLM Prompt judges — manual **Run now** runs and [continuous evaluation](/guides/continuous-evaluation). | | `composite` | Composite judges (including **Primary Score**), which aggregate component scores into one number. | | `classifier` | The **Issue Classification** judge — its label is a failure-mode name from your taxonomy, not pass/fail. | | `preprod` | [Pre-prod evaluation](/guides/preprod-evals) runs. Tagged separately so candidate-release scores form their own score type — and so a dashboard widget can filter them out by **Score source**. | ## Browse the catalogue The Scores page opens on a card grid — one card per score type — over a selectable time window (default **30d**), showing scores from **Finalized** judges only by default (see [Lifecycle filter](#lifecycle-filter-finalized--experimental--all)). Every card shows how many targets were scored, the pass rate, the fail count, a sparkline, and pills for its source and target type. **What the card leads with depends on what the scorer emits.** A numeric scorer shows its average as a donut gauge and its delta; a boolean or categorical scorer shows a **label distribution** — the same donut shape, cut into one arc per label with a named legend beside it — because a mean of `low` / `medium` / `high` answers nothing. See [Boolean and categorical scorers](#boolean-and-categorical-scorers). Click a card to drill into that metric: - **Summary tiles** — **Pass rate**, **Scored**, **Fails** for the window, led by **Average** for a numeric scorer or **Most common** (the dominant label and its share) for a boolean or categorical one. - **Donut** — the average gauge, or the full label distribution: the same donut, with a legend listing every label's share *and* its count. - **Chart** — the average score over time for a numeric scorer, or the **share of each label over time** for a boolean or categorical one. - **Score rows table** — every individual verdict, each score shown as a donut gauge (with its native label where the judge emits one), with a column picker, sortable headers, and filters for **Result** (pass/fail), **Score** range, and **Source**. Click a row to open the score modal: the score against its threshold, the judge's **Rationale**, the **Raw output**, and a **View trace** button that jumps to the graded trace. ### Reading the donut and scale Every score is stored normalized to **0–1**, but the donut center shows it on a friendlier display scale — **0–100 by default** — so `0.75` reads as **75**. The ring fills in proportion to the underlying 0–1 score and is colored by the metric's threshold bands: **green** at or above the green cutoff, **yellow** between the two cutoffs, and **red** below the yellow cutoff. When a judge carries a native readout (e.g. `7.5 / 10` or `high`), the center shows that value verbatim while the ring still fills on the normalized score. The display scale and color bands are **presentation only** — they never change a stored score, how scores aggregate, or how pass/fail is graded. See [Score display settings](#score-display-settings) to change them for your agent. ## Boolean and categorical scorers Not every scorer produces a number. A guard answers **yes or no**; a classifier picks **one label from a set**. For those, an average is not a weaker summary — it is a meaningless one, so the Scores page renders a **label distribution** instead: a donut cut into one arc per label, with a legend naming each one. | The judge emits | The card shows | The drill-down shows | | --- | --- | --- | | A number (the default) | Average donut + delta + sparkline | Average, progression chart | | A **boolean** verdict | A two-arc donut, `True 88%` / `False 12%` in the legend | The same donut with counts, plus share-over-time | | A **categorical** label | A donut over the **four** largest labels plus an **Other** arc | Up to **24** labels — including declared ones sitting at 0% — plus share-over-time | | **Mixed** (its judges disagree) | A note saying so | The same note; no number is invented | | **No scores in the window** | The card doesn't appear | An empty state | Percentages are always computed **server-side over the whole selected window**, not over the page of rows in the table below — so changing the time range changes the shares, and paging through the rows does not. ### Reading the label donut The label donut is the same object as the numeric gauge next to it in the grid, so a mixed catalogue reads as one board rather than two: - **The centre is the denominator.** It carries the number of scores in the selected window — the count every percentage beside it divides by. `40%` over 5 scores and `40%` over 5,000 are very different evidence, and the legend alone can't tell you which you're looking at. - **The legend is the chart.** Each row is a swatch, the label, and its share; the drill-down adds the raw count. Labels render in the judge's **declared order** (worst → best), not by popularity, so a severity scale always reads the same way from window to window. - **Colour means nothing beyond "different slice".** A boolean's two verdicts are deliberately *not* painted green and red: which verdict is the good one is only knowable from a declared [output spec](/guides/judges#output-types), and colouring a guess would be a quality claim the data doesn't support. Pass/fail lives in the **Pass rate** tile, which comes from your own `passing` / `pass_label` rule. Every slice is named in text, so nothing is carried by colour alone. - **A rare label still shows.** A slice worth a fraction of a percent keeps a hairline arc rather than vanishing, and the legend carries its real number. ### Other and No label Two slices are generated rather than emitted by the judge, and both exist so that nothing is silently dropped: | Slice | What it is | What to do about it | | --- | --- | --- | | **Other** | The tail folded out by the card's four-label cap, with the combined count of everything in it. Hovering it names how many further labels it holds. | Open the drill-down — it lists up to 24 labels individually. | | **No label** | Scores whose judge returned no parseable label at all. A real, countable outcome, never hidden. | Open those rows' **Raw output** in the score modal; usually the model answered in prose instead of the requested JSON. | **A declared label that never occurred stays in the legend at 0%.** It has no arc — that is exactly the point. "Nothing was rated `high` this week" is a finding, and a row that disappeared when its count hit zero would hide it. The card omits zero-count labels to stay readable; the drill-down is where the complete declared vocabulary lives. A card and everything under it always describe the **same** scores: whatever a card counts, its breakdown covers and its drill-down lists. **No scores in this window yet.** therefore means exactly that — the selected window holds no scores for that scorer — and never appears beside a card that is counting some. ### Declare the output type on the judge The page reads the type from the judge's **output spec** — the same [`output_spec`](/guides/judges#output-types) that lets a judge honour its own rubric. Set it when you create or edit the judge; you don't configure anything on the Scores page. **A boolean judge.** `pass_label` and `fail_label` are the literal words the judge returns, and they define which verdict counts as a pass: ```json { "criteria": "Does the answer expose personally identifiable information the user had not already shared?", "required_data": ["input", "output"], "output_spec": { "type": "binary", "pass_label": "false", "fail_label": "true" } } ``` The Scores card then draws a two-arc donut whose legend reads **False 88%** and **True 12%**, with the window's scored count in the centre. Neens localizes the generic verdict words (`true`/`false`, `yes`/`no`, `pass`/`fail`); a judge that names its own verdicts — `safe`/`unsafe` — has them rendered verbatim, which is usually the more readable choice for a safety guard. **A categorical judge.** `labels` is an **ordered** list, worst to best, and that order is the order the legend and the breakdown render in — so a severity scale reads top to bottom in your own order rather than by popularity: ```json { "criteria": "Rate the business risk this answer creates for the company.", "required_data": ["input", "output", "context"], "output_spec": { "type": "categorical", "labels": ["none", "low", "medium", "high"], "higher_is_better": false, "passing": ["none", "low"] } } ``` The card draws four arcs and a legend reading **none 37%**, **low 30%**, **medium 18%**, **high 15%** — in that order, because that is the order you declared, not the order the window happened to produce them in. Four labels is exactly what a card renders, so nothing folds into **Other** here. In a window where `high` never occurred, the card drops to three arcs and the drill-down still lists `high` at **0%**. **Pass rate still works, and it comes from your criteria.** `passing` (categorical) and `pass_label` (boolean) define the verdict, so the **Pass rate** tile and the Traces page's **Pass**/**Fail** filter agree with the judge's own rule rather than with a 0.7 cutoff on a number the judge never produced. See [Output specs](/guides/judges#output-types). ### Many labels: a worked example A wide label set — a triage judge over your agent's failure taxonomy — is folded on the card. Declare the vocabulary the same way: ```json { "criteria": "Which failure class best explains why this trace went wrong? Choose exactly one.", "required_data": ["input", "output", "messages"], "output_spec": { "type": "categorical", "labels": [ "tool_selection_error", "tool_execution_error", "missing_context", "hallucinated_fact", "instruction_ignored", "unsafe_output", "format_violation", "retry_loop", "no_failure" ], "higher_is_better": false, "passing": ["no_failure"] } } ``` Over a week of 1,240 classified traces you get: 1. **The card** draws five arcs — the four largest labels (`tool_selection_error`, `missing_context`, `hallucinated_fact`, `no_failure`) plus **Other**, whose tooltip reads *Grouped remainder — 4 further labels*. `1,240` sits in the centre. 2. **Click the card.** The drill-down redraws the same donut with every label listed individually, each with its share and its count, and `unsafe_output` still listed at **0%** because you declared it and this week produced none. A remainder past 24 labels would again fold into **Other**. 3. **The chart underneath** switches from the numeric progression to **share over time** — one stacked 0–100% band per label, using the same colour each label has in the donut above it. It answers the categorical question ("is the mix shifting?") rather than the numeric one, and a bucket where nothing was scored is drawn as a **gap** in the bands, not as a row of zeroes — "nothing happened this hour" and "everything dropped to zero" are different findings. 4. **The rows table** below lists the individual verdicts. Its paging never moves the percentages above it: those are a server-side aggregate over the whole window. ### The classifier is categorical too The built-in **Issue Classification** judge writes the detected failure-mode name as its label, so its card is a distribution over your [taxonomy](/guides/issues-and-failure-modes) — which failure mode is actually most common, as a share of everything classified in the window. You do not declare an output spec for it, and you do not need to. A classification judge picks a label from your taxonomy and has no number to average, so it is **categorical by construction** — its taxonomy is the vocabulary, and that taxonomy changes whenever you edit it, which is why the labels are not frozen into a spec. The number it does store is the classifier's **confidence** in the label it chose, and a mean of that answers nothing about which failure modes you have. If you do give a classification judge an explicit `output_spec` — a fixed triage vocabulary, say — that declaration wins, exactly like any other judge's. ### When the type can't be declared Neens works out a score type in this order, and tells you which it used: 1. **Declared** — the producing judge's current `output_spec`. This is the normal case, and it is what you should rely on. A judge with no spec is declaring the default: a plain 0–1 number. 2. **Recorded** — the output type stamped on each score row when it was written. This answers for rows whose judge was since deleted or renamed. 3. **Classifier** — `source = classifier` rows are categorical by construction. 4. **Inferred** — a last resort, read from the labels the rows actually carry. A **classification** judge is the one exception to step 1: leaving its spec unset declares nothing, because it has no number to fall back to, so it is answered at step 2 or step 3 instead. Every other kind of judge — and any classification judge that *does* declare a spec — is answered at step 1. **Inference is deliberately conservative, and that is why declaring the spec matters.** Two observed labels do not make a scorer boolean — `low` and `high` are a three-label scale whose window happened to miss `medium`, so an undeclared scorer showing two labels is treated as categorical. And a scorer whose rows disagree about their own type reads as **Mixed** and gets no summary at all. Declare `output_spec` and none of this applies. ### Lifecycle filter (Finalized / Experimental / All) Judges carry a [lifecycle stage](/guides/judges#lifecycle-stage) — **experimental**, **finalized**, or **archived** — and the Scores page uses it to keep the catalogue focused. A **Finalized / Experimental / All** control switches which scores the whole page (cards, drill-downs, and sparklines) shows: | Filter | Shows scores from | | --- | --- | | **Finalized** *(default)* | Judges staged **finalized** — the trusted, everyday set. Experimental and archived scorers are hidden. | | **Experimental** | Only judges staged **experimental** — the work-in-progress scorers you're still tuning. | | **All** | Every judge, regardless of stage. | This is the point of lifecycle staging: an ad-hoc experimental judge doesn't flood the main view until you promote it to **Finalized** on the [Judges](/guides/judges#lifecycle-stage) page. The filter is view-only — it never changes what any judge scores. A score whose judge can't be resolved (legacy rows, built-in `ragas` metrics) is treated as **Finalized**, so relaxing the filter only ever *adds* experimental rows; it never hides a legitimate metric. API: pass `?lifecycle=finalized|experimental|all` to `GET /scores`, `/scores/metrics`, and `/scores/timeseries` (default `finalized`). ### Scope to a single run For an **ad-hoc** batch [eval run](/guides/judges#run-it), you can view exactly the scores that one run produced — its captured snapshot — instead of the metric's current live scores. Open it from a judge's run history on the [Judges](/guides/judges#filter-a-runs-scores-compare-versions-and-export) page, or via `GET /scores?runId=`. This is meant for bounded ad-hoc runs; a continuous scorer overwrites each trace's score row as it re-grades, so a single run isn't a meaningful boundary for it. The run history is also where you [compare two versions](/guides/judges#filter-a-runs-scores-compare-versions-and-export) of a judge and [export a run](/guides/judges#filter-a-runs-scores-compare-versions-and-export) with each target's input and output. ## Favorites and hiding Each catalogue card has a star (favorite) and an eye (hide) toggle. These are **personal view preferences** — stored per user, per agent — and are saved via `PUT /scores/prefs` keyed on the (`metric_key`, `source`, `target_type`) triple: - **Favorites** float into a **Favorites** section at the top of the page. - **Hidden** metrics collapse into a **Hidden (n)** reveal at the bottom. A hidden metric can't be favorited until you unhide it. Favoriting and hiding never change which judges run and never delete score data — they only change what *you* see on this page. ### Hiding vs. disabling the judge Hiding is view-only: if an enabled judge is still producing new scores for the metric you just hid, Neens shows a banner naming that judge and offering **Disable judge** as an explicit second step. Disabling stops new scores from being produced (for everyone); existing scores are kept. You can always re-enable the judge from the [Judges](/guides/judges) page. ## Score display settings The display scale and color bands are configured **per agent** under **Settings → Score display**. The card shows a live preview of three sample scores rendered with your saved settings, so you can see the effect before applying it. Editing is **admin-only**; everyone else sees it read-only. It is presentation only — it never changes how a score is stored or how pass/fail is graded. | Setting | What it does | Platform default | | --- | --- | --- | | **Display scale (max)** | The denominator the donut center is rescaled onto — `100` shows `0.75` as **75**, `10` shows it as **7.5**. | `100` | | **Green at/above (0–1)** | Normalized score at or above which a value reads green. | `0.85` | | **Yellow at/above (0–1)** | Normalized score at or above which a value reads yellow; below this it reads red. | `0.60` | | **Per-metric thresholds** | Optional per-metric override of the green/yellow cutoffs — tighten or loosen the bands for one metric (e.g. a stricter toxicity guard) without affecting the rest. Leave blank to use the agent defaults. | none | Thresholds always stay on the underlying 0–1 scale; a score at or above the green line reads green, between the two lines yellow, and below the yellow line red. Neens keeps the bands non-inverted (the yellow cutoff can't sit above the green one). Changing the display scale or color bands affects **every viewer on the agent**, which is why it's admin-only. Read access is open to anyone who can read the agent — the UI needs it to render every score consistently. Agents without saved settings fall back to the platform defaults above. ## Primary Score **Primary Score** (`primary_score`) is the headline quality number every new agent gets out of the box — a composite that blends **Faithfulness**, **Answer Relevancy**, and **Coherence** into one session-level score. It plays two special roles: - **Continuous evaluation.** It ships as the agent's one enabled continuous scorer, sampling newly ingested traces automatically. Its component metrics appear in the catalogue as a byproduct of each composite run — they don't need their own enabled scorers. See [Continuous evaluation](/guides/continuous-evaluation). - **Failure clustering.** The default selection criterion for [failure clustering](/guides/clustering) is a low Primary Score — which is how silent failures (bad answers with no hard error) end up in the failure set. ## Where scores surface - **Trace detail** — from any score's modal, **View trace** opens the graded trace; classifier scores also surface there as the trace's issue chip. - **Traces page** — the **Score** filter selects traces carrying a score for a metric, optionally **Pass**/**Fail** against its threshold (URL params `score_metric`, `score_status`, `score_label`). See [Traces & sessions](/guides/traces-and-sessions). - **Dashboards & metrics** — the score-grain catalogue measures `eval_pass_rate`, `avg_score`, and `score_count`, sliceable by `judge`, `score_source`, `score_label`, `target_type`, `score_bucket`, and — because every score records the model and agent behind the answer it graded — by `model`, `agent`, and `model_source`. See the [Metrics catalogue](/guides/metrics) and [Model comparison](/guides/model-comparison). - **Judge run drill-downs** — each eval run lists its per-target scores, reasons, and errors on the [Judges](/guides/judges) page. - **Pre-prod comparisons** — candidate-vs-baseline verdicts in [pre-prod evaluations](/guides/preprod-evals) are built from `preprod`-sourced scores. API reference | Route | Purpose | | --- | --- | | `GET /scores/metrics?range=&lifecycle=` | The catalogue: one entry per (`metric_key`, `source`, `target_type`) with average, delta, pass rate, sparkline, and your favorite/hidden flags. Also `valueType` (`numeric`/`boolean`/`categorical`/`mixed`/`unknown`), `valueTypeBasis` (how that was determined), and — for a label-typed scorer — a compact `distribution` (top 4 buckets plus an `other` remainder). `lifecycle` (`finalized`/`experimental`/`all`, default `finalized`) filters by the producing judge's [stage](/guides/judges#lifecycle-stage). | | `GET /scores/timeseries?metric_key=&range=&lifecycle=` | Bucketed average-score series for one metric; honors the same `lifecycle` filter. | | `GET /scores/distribution?metric_key=&source=&targetType=&range=&lifecycle=&buckets=` | The label breakdown for one score type: `valueType`, `valueTypeBasis`, the judge's `declaredLabels`, a full `distribution` (up to 24 buckets, plus `unlabeledCount` and an `other` remainder), and a `series` of per-bucket label counts for the share-over-time chart. Every share is over the whole filtered window. A `numeric`, `mixed` or `unknown` type returns no buckets and no series rather than a fabricated split. | | `GET /scores` | Individual score rows. Filters: `metric_key`, `targetId`, `targetType`, `status` (`pass`/`fail` against each row's own threshold), `min_score`/`max_score`, `source`, `lifecycle`, and `runId` (return one ad-hoc run's captured snapshot); paginated and sortable. | | `PUT /scores/prefs` | Upsert your favorite/hide preference for one score type: `{"metricKey", "source", "targetType", "favorite", "hidden"}`. When a hide leaves an enabled judge still producing the metric, the response lists those `producingJudges` so the UI can offer to disable them. | | `GET /scores/display-settings` | The agent's resolved score-display config: `scaleMax`, `passThreshold`, `warnThreshold`, `metricOverrides`, and `isCustom`. Readable by anyone who can read the agent; falls back to the platform defaults when nothing is saved. | | `PATCH /scores/display-settings` | Update the display scale and/or color bands. Admin-only. Body (all optional): `{"scaleMax", "passThreshold", "warnThreshold", "metricOverrides": {"": {"pass", "warn"}}}`. Values are clamped to safe ranges and bands kept non-inverted. | ## Troubleshooting | Symptom | Likely cause | Fix | | --- | --- | --- | | Catalogue is empty ("No scores yet") | No judge has ever run in this agent | **Set up judges** — enable a judge and run it, or wait for [continuous evaluation](/guides/continuous-evaluation) to score arriving traces. | | A metric stopped moving | The producing judge was paused or disabled | Check its state on the [Judges](/guides/judges) page — hiding a card here never causes this. | | A card shows no delta | The window is **All time** or an open-ended custom range | Deltas compare against the preceding window of equal length, which needs a bounded range. | | A metric you hid keeps growing | Hiding is view-only | Use the banner's **Disable judge** offer (or disable it on the Judges page) to stop new scores. | | A label-based scorer still shows an average donut | Its judge has no `output_spec`, so its scores are plain numbers | Set the **Output type** on the judge ([Output types](/guides/judges#output-types)). Existing rows keep their old shape; rows written after the change carry the new one. | | A card says **No scores in this window yet.** | The selected range holds no scores for that scorer, or its judge is filtered out by the lifecycle lens | Widen the time range, or switch the lifecycle filter to **All**. A card never shows this note while counting scores beside it. | | A card says the scorer is **Mixed** | Two visible judges of that name declare different output types, or the type changed mid-window | Check for a duplicate judge name across agent/org/platform scope on the [Judges](/guides/judges) page, or narrow the time range to a window written under one spec. | | A declared label never appears | It genuinely has no scores in this window | The drill-down lists it at **0%** — the card omits zero-count labels to stay readable. Widen the range if you expect it. | | The card's donut has an **Other** slice | The scorer emits more than four labels, so the tail is folded on the card | Click the card. The drill-down lists up to 24 labels individually, each with its own count. | | Some rows land in **No label** | The judge returned no parseable label for them | Open those rows' **Raw output** in the score modal — usually the model answered with prose instead of the requested JSON. | | The share-over-time chart has gaps | Those buckets hold no scores at all | A gap is deliberate — it means nothing was scored then, which a flat 0% line would misreport as every label collapsing. | ================================================================================ # Scorer lifecycle Source: /docs/guides/scorer-lifecycle/ ================================================================================ # Scorer lifecycle A **scorer lifecycle** is a lightweight way to curate your [judges](/guides/judges) as they mature — from a prompt you're still tuning, to a trusted metric everyone relies on, to one you've retired. It's a curation signal only: staging a judge never changes what it scores. What it *does* change is which scores show up by default, so an agent's [Scores](/guides/scores) page stays focused on the metrics that matter instead of every ad-hoc experiment. ## At a glance | | | | --- | --- | | **Where** | Stage a judge on the **Judges** page; the **Finalized / Experimental / All** filter and per-run views live on the **Scores** page and each judge's run history | | **Key API routes** | `PATCH /judges/{id}/lifecycle`, `GET /scores?lifecycle=`, `GET /scores?runId=`, `GET /judges/{id}/compare`, `GET /eval-runs/export` | | **Scope** | Agent-scoped — you can only stage a judge visible to your agent; comparisons and exports are validated against your agent | | **Changes scores?** | **No.** Every capability here is curation, filtering, comparison, or export — it never re-grades a trace or edits a stored score | ## The stages A judge moves through three stages: | Stage | What it means | Default view | | --- | --- | --- | | **Experimental** | A work-in-progress scorer you're still tuning. **New judges you create start here.** | Hidden from the Scores page by default | | **Finalized** | Reviewed and trusted for everyday use. The built-in library judges (**Faithfulness**, **Primary Score**, and the rest) ship **Finalized**. | Shown by default | | **Archived** | Retired — its score history is kept, but it's out of your working set. | Hidden from the Scores page by default | Staging is a signal, not a switch. It never enables, disables, pauses, or re-runs a judge, and it never touches existing scores — it only drives which scores the [Scores](/guides/scores) page shows by default. Enabling and running a judge is a separate concern; see [Judges](/guides/judges#enable-it-as-a-scorer). ## Change a judge's stage ### Open the Judges page Find the judge you want to promote or retire on the **Judges** page. ### Set its stage Change the judge's lifecycle stage — typically promoting an **experimental** scorer to **finalized** once you trust its verdicts, or **archiving** one you no longer use. The move is one-directional in spirit (experimental → finalized → archived), but you can set any stage at any time, including re-opening an archived judge back to experimental. The change is recorded in the [activity/audit](/administration/audit-log) trail and takes the same permission as configuring a judge, so a viewer can't silently re-stage someone else's scorer. Via the API, `PATCH /judges/{id}/lifecycle` with a body of `{"stage": "experimental" | "finalized" | "archived"}` (any other value is a `422`). A agent-scoped caller can only re-stage a judge visible to its agent; anything else is a `404`. ## Keep the Scores page focused Because new judges start **experimental**, the [Scores](/guides/scores) page defaults to showing **Finalized** scorers only — so an in-progress judge doesn't clutter everyone's metrics until you promote it. A **Finalized / Experimental / All** filter switches the whole page: - **Finalized** *(default)* — trusted, everyday metrics. - **Experimental** — only the work-in-progress scorers you're tuning. - **All** — every judge, regardless of stage. A score whose judge can't be resolved (legacy rows, built-in `ragas` metrics) is treated as **Finalized**, so relaxing the filter only ever *adds* experimental rows — it never hides a legitimate metric. See [Scores → Lifecycle filter](/guides/scores#lifecycle-filter-finalized--experimental--all). ## Compare, scope, and export runs For **ad-hoc** batch [eval runs](/guides/judges#run-it) — the manual **Run now** runs, each tied to one judge [version](/guides/judges#save--and-iterate-with-versions) — a judge's run history lets you audit and compare what individual runs produced. Each of these reads the run's own captured **snapshot**, so a later run overwriting the shared score row never corrupts an older run's numbers. ### Scope to one run View exactly the scores a single ad-hoc run produced instead of the metric's current live scores. API: `GET /scores?runId=`. ### Compare two versions Iterating on a judge's prompt? Run the old and new [versions](/guides/judges#save--and-iterate-with-versions), then compare their two runs to see, **per target**, which sessions **improved**, **regressed**, or stayed **unchanged** — with each side's score and reason and the average delta. API: `GET /judges/{id}/compare?runA=&runB=`. A run from another judge or agent is a `404`. ### Export a run with input and output Download a run's (or several runs') scores as **CSV** or **JSON**, with each target's **input** and the agent's **output** alongside the score, label, threshold, and reason — for spreadsheets, offline review, or sharing a regression. API: `GET /eval-runs/export?runs=,&format=csv|json`. **Ad-hoc runs, not continuous scorers.** Per-run scoping, version comparison, and export are built for bounded batch runs. A [continuous scorer](/guides/continuous-evaluation) re-grades traces in place and overwrites each trace's score row, so a single "run" isn't a meaningful boundary for it — compare and export are aimed at the manual runs you launch to evaluate a change. ## How it works Scorer lifecycle is entirely additive on top of judges and scores: - **Stage** lives on the judge as `lifecycle_stage`. Pre-existing judges (and any judge whose name can't be resolved to a stage) are treated as **Finalized**, so nothing disappears when the feature first appears. - **The Scores filter** resolves which judge *names* in your agent are experimental or archived and filters score rows by that set — a purely read-side filter that works the same on SQLite and ClickHouse deployments. - **Per-run scoping, comparison, and export** read the run's per-target snapshot (not the live, overwrite-in-place score table), so the numbers you compare or export always reflect what that run actually scored. ## Related - [Judges](/guides/judges) — create, version, enable, and run scorers. - [Scores](/guides/scores) — the score catalogue, the lifecycle filter, and per-run views. - [Continuous evaluation](/guides/continuous-evaluation) — the always-on scoring these ad-hoc tools contrast with. - [Pre-prod evaluations](/guides/preprod-evals) — run your trusted (finalized) judges against a candidate release before you ship. ================================================================================ # Run Targets Source: /docs/guides/run-targets/ ================================================================================ # Run Targets A **Run Target** is a named, reusable population for your scorers to run on — a golden [dataset](/guides/datasets), a live sample of production traffic, or a filtered sweep of history. It separates *what a [judge](/guides/judges) measures* (the rubric) from *what it measures it on* (the traffic), so one scorer definition can grade several populations at once and every scorer can share the same populations. ## At a glance | | | | --- | --- | | **Where** | The **Run Targets** tab on the **Judges** page (the library, and each target's **Used by** scorers); also the **Run Targets** tab on any scorer's detail page (that scorer's attachments) | | **Key API routes** | `GET`/`POST /run-targets`, `GET`/`PATCH`/`DELETE /run-targets/{id}`; attach via `POST /judges/{id}/deployments`; results via `GET /judges/{id}/targets` | | **Three kinds** | **Dataset** (static, versioned), **Live sample** (continuous, on-ingest), **Filtered batch** (historical / scheduled sweep) | | **Scope** | Agent-scoped — a Run Target and every scorer bound to it live in one agent | | **Reused by** | Any number of scorers; the library shows each target's **Used by** scorers | ## Why Run Targets exist A scorer answers one question — *is this answer faithful? is this reply coherent?* — but you rarely want that question answered on just one slice of traffic. Take **Primary Score**, the built-in composite of **Faithfulness**, **Answer Relevancy**, and **Coherence**. You want it on: - your **Golden Q&A** dataset, so a regression can't sneak past the benchmark you trust; - a **live sample** of what `support-bot` is actually saying customers today; - last night's **failures**, swept nightly, to watch a known problem area. That's *one* rubric over *three* populations. Before Run Targets, "what to score" was baked into each scorer's deployment, so answering that meant three near-duplicate scorers to keep in sync. A Run Target lifts the population out into its own object: define **Primary Score** once, then attach it to all three targets. Results come back **split per target**, so you can see at a glance that the same scorer reads `0.95` on the golden set, `0.82` on live traffic, and `0.61` on the error sweep — three honest numbers instead of one blurred average. ## The three kinds Every Run Target is one of three kinds. The kind decides how its population is resolved and which [trigger](#set-the-per-attachment-settings) fits it. | Kind | Population | Typical use | Fits trigger | | --- | --- | --- | --- | | **Dataset** | A [dataset](/guides/datasets) — either a **pinned version** (frozen) or **follow latest** (the live members) | The regression benchmark: your **Golden Q&A** set | **Manual** or **Scheduled** | | **Live sample** | A percentage of **new traces** as they arrive, matching an optional filter | Continuous quality signal on production traffic | **On new trace** (continuous) | | **Filtered batch** | Every trace matching a filter over history — run once or on a schedule | A **nightly error sweep**, or a one-off backfill | **Manual** or **Scheduled** | **Dataset targets pin a version by default.** A pinned version is immutable, so every run measures exactly the same items and a benchmark can't quietly drift under you (see [golden versions](/guides/datasets#golden-datasets-and-versions)). Turn on **Follow latest** only when you *want* the target to track the dataset's current members as it keeps syncing. ## Create a Run Target ### Open the library and choose a kind Go to **Judges** and open the **Run Targets** tab, then click **New Run Target** and pick **Dataset**, **Live sample**, or **Filtered batch**. The kind you pick decides which of the fields below you fill in. **Run Targets moved.** They used to have their own entry in the sidebar; the library now lives as a tab on the **Judges** page, next to **Judges**, **Alignment**, and **Comparison**. ### Fill in the shared fields | Field | What it does | | --- | --- | | **Name** | The label you'll see everywhere it's used — e.g. `Golden Q&A`, `Live production`, `Nightly support-bot errors`. | | **Grain** | Whether the target is a set of **traces** (one span tree each) or **sessions** (conversation rollups). This must match the scorers you attach — see [grain must match](#grain-must-match). | | **Filter** | *(Live sample and Filtered batch)* Which traffic qualifies, using the same vocabulary as the [Traces](/guides/traces-and-sessions) page — for example only `support-bot`, or only failed sessions. | | **Sampling** | **All** matches, a fixed **Count**, or a **Percentage**. A live estimate shows how many traces are **eligible** and how many the target **will run on**. | ### Point a dataset target at its items For a **Dataset** target, choose the dataset and how it resolves: - **Pin a version** *(default)* — select a frozen [version](/guides/datasets#golden-datasets-and-versions) (typically the **golden** one). The items never change until you re-point the target. - **Follow latest** — the target resolves the dataset's *current* members every run, so it grows as the dataset syncs. A dataset from another agent can't be selected — Run Targets are agent-scoped. ### Save The new target appears on the **Run Targets** tab with its kind, grain, a one-line population summary, and an eligible-traffic estimate. It scores nothing on its own — a Run Target is just a population until a scorer is [attached](#attach-a-scorer) to it. **A Run Target carries a suggested trigger, not a binding one.** Each kind offers a sensible default (a **Live sample** suggests **On new trace**, a **Filtered batch** suggests **Scheduled**), but the trigger that actually runs is set on the attachment, so two scorers can run the *same* target on different schedules. See [per-attachment settings](#set-the-per-attachment-settings). ## Attach a scorer You attach a Run Target from a scorer, not the other way round: open a [judge](/guides/judges), go to its **Run Targets** tab, and click **Attach Run Target**. Each attachment is one binding of *this scorer* to *one Run Target* with its own run-time settings. ### Reuse an existing target or define a new one The drawer lets you **Reuse existing target** (pick one from the library — this is how a target ends up shared across scorers) or **Define new** (create one inline, same fields as above). ### Set the per-attachment settings Three settings live on the attachment, not on the Run Target, so each scorer can run the same population its own way: - **Trigger** — **Manual** (run on demand), **On new trace** (score a sample continuously as traffic arrives), or **Scheduled**. - **Pass cutoff** — the score at or above which a target counts as passing for *this* scorer on *this* population (blank → the scorer's default; see [run outcomes](/guides/judges#run-outcomes)). - **Model connection** — which LLM connection(s) score this population. Leave empty to use the agent's default; pick a pool to spread load (see [LLM connections](/administration/llm-connections)). ### Save the binding The attachment now shows as a row on the scorer's **Run Targets** tab, and the scorer shows up under **Used by** on that target in the library. ### Attach one scorer to several targets Repeat **Attach Run Target** for each population. Attaching **Primary Score** to **Golden Q&A**, **Live production**, and **Nightly support-bot errors** gives it three bindings — one rubric, three populations — each with its own trigger, cutoff, and connection. The scorer's **Run Targets** tab lists all three, each with its latest metric and a sparkline. ### Grain must match **Grain must match.** A trace-grain scorer can only attach to a trace-grain Run Target, and a session-grain scorer only to a session-grain target; a mismatch is rejected at attach time with a `grain_mismatch` error. Check the scorer's **Target type** (trace or session) and the target's **Grain** agree before attaching. Grain is fixed on a Run Target once it has bindings. ## Read results split per target Because a scorer can run on several populations, its results are reported **per Run Target** rather than pooled. On the scorer's **Run Targets** tab, each attached target carries its own **scored** count, **average**, **pass rate**, and trend — so the same rubric reads independently on each: | Run Target | Kind | Scored | Avg | Pass rate | | --- | --- | --- | --- | --- | | **Golden Q&A** | Dataset (v7, pinned) | 120 | `0.95` | `98%` | | **Live production** | Live sample (10%) | 1,412 | `0.82` | `86%` | | **Nightly support-bot errors** | Filtered batch | 240 | `0.61` | `54%` | That spread is the point: a benchmark near the top, live traffic a step below, and the error sweep lowest of all. A single blended number would hide exactly the signal you're watching for — a golden set holding steady while live quality slips, or an error sweep that stops improving after a fix. A target with no runs yet reads as blank (an em dash), never a misleading `0` — an unscored population and a zero-scoring one are different things. ### Which target a run evaluated Because one scorer can be attached to several Run Targets, its **run history** and the **Judges** list both label each row with the Run Target it evaluated, so two runs of the same scorer against different populations never read as duplicates: - On the scorer's **run history** (click a scorer to open its runs), the **Target** column names the Run Target each eval run scored — e.g. a `Live production` run and a `Nightly support-bot errors` run sit side by side, each tagged with its own target. - On the **Judges** list, each enabled scorer row shows the Run Target its deployment is bound to, so the same scorer attached to two targets appears as two clearly labelled rows. A run that wasn't tied to a named Run Target — an ad-hoc **Run now** over an inline filter — shows a neutral em dash in the **Target** column rather than an invented label. ## Reuse: editing a target updates every bound scorer A Run Target is shared state. **Editing it changes the population for every scorer bound to it** — that reuse *is* the feature. Widen the **Nightly support-bot errors** filter to cover a second agent, or bump **Live production** from 10% to 25%, and every attached scorer picks up the change on its next run. There's no per-scorer copy to update, and no drift between scorers that are meant to watch the same slice. **A Run Target in use can't be deleted out from under its scorers.** Deleting one that still has attachments is refused and lists the scorers **using** it; detach it from each scorer first (remove the binding on the scorer's **Run Targets** tab), then delete. This keeps a shared population from vanishing mid-run. ## How it works - **Resolution at run time.** When a scorer runs a binding, Neens resolves the Run Target's population *then* — a **Dataset** target reads its pinned version (or the live members, if **Follow latest**), a **Live sample** samples the just-arrived traces, and a **Filtered batch** runs the filter over history — applies the target's sampling, and scores exactly that set. Editing the target changes what the next run resolves. - **Settings split by owner.** The *population* (filter, sampling, dataset, version) lives on the Run Target and is shared; the *run-time* settings (trigger, pass cutoff, model connection) live on each attachment and are private to that scorer. The two never fight — a run reads the population from the target and the run-time settings from its own binding. - **Nothing is retroactive.** A **Live sample** only scores traffic arriving after it's attached and running, exactly like [continuous evaluation](/guides/continuous-evaluation); use a **Filtered batch** (or **Run now**) to score history. - **Every LLM-scored population needs a connection.** Like every judge, a Run Target run uses the agent's configured LLM connection (**Settings → Connections**). With none configured the run finishes **Skipped** rather than borrowing another agent's credentials — see [continuous evaluation](/guides/continuous-evaluation#when-a-run-is-skipped). API reference | Route | Purpose | | --- | --- | | `GET /run-targets` | List the agent's Run Targets, each with its kind, grain, population summary, eligible estimate, and the scorers using it. | | `POST /run-targets` | Create one (`{name, kind, grain, filter?, sampling?, datasetId?, datasetVersionId?, followLatest?}`). For `kind: "dataset"`, supply a dataset and either a pinned `datasetVersionId` or `followLatest`. | | `GET /run-targets/{id}` | Detail, including which scorers use it and its eligible-traffic estimate. | | `PATCH /run-targets/{id}` | Edit it — the change applies to every bound scorer. | | `DELETE /run-targets/{id}` | Delete it; refused with the list of scorers still using it (detach first). | | `POST /judges/{id}/deployments` | Attach a scorer: include the Run Target to bind, plus the per-attachment trigger, pass cutoff, and connection pool. Grain must match (`422` otherwise). | | `GET /judges/{id}/targets` | Per-target results for a scorer — scored count, average, pass rate, and trend for each attached Run Target. | Run Targets are addressed by id and scoped to the agent of the caller; a target or dataset in another agent is a `404`. ## Troubleshooting | Symptom | Likely cause | Fix | | --- | --- | --- | | Can't attach a scorer — `grain_mismatch` | The scorer's target type (trace/session) and the Run Target's grain differ | Attach a target whose **Grain** matches the scorer's **Target type**, or create one at the right grain. | | A Run Target won't delete | Scorers are still bound to it | Detach it from each scorer on their **Run Targets** tab, then delete — the error lists the scorers using it. | | A live-sample target scored nothing for old traffic | Live samples are forward-only | Attach a **Filtered batch** over history (or use **Run now**) to score existing traffic once. | | Editing a target changed another scorer's numbers | The target is shared — that's by design | Reuse is intentional; if two scorers need different populations, give each its own Run Target. | | A dataset target's items never change | It's pinned to a frozen version | Turn on **Follow latest** to track the dataset's current members, or re-point it to a newer version. | | Runs finish **Skipped** with `no_llm_connection` | No LLM connection is visible to the agent | Add one in **Settings → Connections**; see [continuous evaluation](/guides/continuous-evaluation#when-a-run-is-skipped). | ## Related - [Judges](/guides/judges) — define, version, and enable the scorers you attach to Run Targets. - [Datasets](/guides/datasets) — the versioned collections a **Dataset** target points at. - [Continuous evaluation](/guides/continuous-evaluation) — how a **Live sample** scores new traffic. - [Scores](/guides/scores) — the catalogue every Run Target's verdicts land in. - [Pre-prod evaluations](/guides/preprod-evals) — replay a golden dataset version against a candidate release before you ship. ================================================================================ # Continuous evaluation Source: /docs/guides/continuous-evaluation/ ================================================================================ # Continuous evaluation Continuous evaluation scores your traffic **as it arrives**: every time new traces are ingested, Neens samples them and runs your continuous [judges](/guides/judges) against the sample — no manual runs, no schedules to manage. It's how an agent accumulates a live quality signal (and a failure set for [clustering](/guides/clustering)) without anyone pressing a button. ## At a glance | | | | --- | --- | | **Where** | The **Judges** page — a scorer with **Apply for** set to **Continuous (score new traces)** | | **What runs by default** | **Primary Score**, the out-of-the-box composite every agent ships with | | **Needs** | An LLM connection (**Settings → Connections**) — see [LLM connections](/administration/llm-connections) | | **Produces** | [Score](/guides/scores) rows, written moments after each trace arrives | | **Cost controls** | Percentage sampling per scorer + a per-scorer daily cap (default `500` traces/day) | | **When nothing scores** | The run ends **Skipped** with a [reason code](#when-a-run-is-skipped) — transient reasons are retried, permanent ones are not | ## How it works After a batch of traces is ingested and persisted, Neens checks the agent's enabled scorers whose trigger is **Continuous (score new traces)** and, for each one: 1. **Samples** the just-arrived traces at the scorer's effective sampling rate (a coin flip per trace at the configured percentage). 2. **Applies the daily budget.** Each continuous scorer has its own daily cap (`max_scored_per_day`); once a scorer has dispatched its cap for the day, further traces are skipped until the next UTC day. Budgets are per scorer, so one continuous judge can never starve another's. 3. **Creates a small eval run** scoped to exactly the sampled traces and queues it for the eval worker — the same scoring path, retries, and [run outcomes](/guides/judges#run-outcomes) as a manual run. These runs are attributed as **Automated** in run history and the [Activity feed](/guides/activity). 4. **Writes scores.** Verdicts land in the [score catalogue](/guides/scores) with source `llm_judge` (or `composite` for composites). A composite like Primary Score inline-scores any component that has no score yet, so its component metrics fill in as byproducts. Continuous evaluation is **best-effort by design**: it runs after ingestion completes and can never block, slow down, or fail a trace write. If scoring hiccups, the traces are still safely stored. **Scores are asynchronous.** A trace appears on the Traces page immediately; its scores land shortly after — typically seconds, longer under queue depth or a slow LLM provider. Expect a brief window where a new trace is visible but not yet scored. ### Primary Score, cold start, and steady state Every new agent ships with one continuous scorer enabled: the **Primary Score** composite (Faithfulness + Answer Relevancy + Coherence). Its defaults balance signal against cost: | Phase | Sampling rate | Why | | --- | --- | --- | | **Cold start** — until 200 traces have been dispatched | `100%` | A brand-new agent builds a scoreable failure set fast. | | **Steady state** — after that | `5%` | A representative ongoing sample. | | **Always** | Capped at `500` traces/day | Percentage sampling alone isn't safe on a firehose. | The cold-start boost graduates automatically once the threshold is crossed — if your scoring rate drops from "everything" to "a sample" after the first couple hundred traces, that's expected, not a failure. ### What continuous evaluation is *not* - **It is not retroactive.** A continuous scorer only scores traces that arrive **after** it's enabled (or resumed). To score existing traces, use **Run now** on the scorer — a one-off run over eligible historical traffic. - **It is not the Issue Classification path.** The built-in **Issue Classification** judge also shows up as a continuous scorer, but it's dispatched by a background sweep (every 30 minutes) over the agent's *failure set only* — not on ingest, and never over healthy traffic. See [Issues & failure modes](/guides/issues-and-failure-modes). ## Enable, pause, and resume ### Make a scorer continuous On the [Judges](/guides/judges) page, **Configure** the judge and set **Apply for** to **Continuous (score new traces)**. The **What traces to score** percentage becomes the continuous sampling rate. Saving configuration never runs anything by itself — new arrivals start being scored from that point on, and the hint reminds you: *"Use 'Run now' to also score existing traces."* A user-created continuous scorer that ships without a daily cap automatically gets the default (`500` traces/day) so an unbounded on-ingest scorer can't become a cost surprise. ### Pause when needed In the **Enabled scorers** table, a continuous scorer's toggle reads **Pause** / **Resume** (manual scorers say **Disable** / **Enable**). **Pause** stops automatic scoring of new traces, keeps every existing score, and cancels any in-flight run. ### Resume — forward only **Resume** turns automatic scoring back on for traces arriving from that moment. Traffic that came in while paused is not backfilled; use **Run now** if you want it scored. ## Sampling and cost Every trace a continuous scorer grades costs **at least one LLM call** on the agent's configured LLM connection — and a composite costs one call per component that doesn't already have a score (up to three for Primary Score). Your levers: - **Sampling percentage** (per scorer, in **Configure → What traces to score**) — the fraction of arriving traces that get scored. - **Daily cap** (per scorer) — the hard ceiling regardless of traffic volume. Resets at midnight UTC. - **Scoring connections (pool)** — spread continuous load across several connections to dodge provider rate limits. If the agent has **no LLM connection**, continuous runs still fire, but they score nothing: the run finishes **Skipped** and every target records the reason `no_llm_connection`. Neens never borrows another agent's credentials, and ingestion is unaffected. Configure one under **Settings → Connections** ([LLM connections](/administration/llm-connections)). ## When a run is Skipped A run finishes **Skipped** when it reached the end having scored **nothing** — no target succeeded and none failed either. It is deliberately neither green nor red: skipping can be perfectly legitimate (there is genuinely nothing to score yet) or it can mean something is misconfigured. The **reason** tells you which. Every skipped target carries a **reason code** plus a plain sentence, visible on the [Activity feed](/guides/activity) row and in its **Details** drawer. A run can carry more than one code; the drawer lists each with how many targets it accounts for. **"Unknown" is a real answer.** If Neens cannot classify a skip it says `unknown` — *"Neens could not determine why this target was skipped"* — rather than showing a blank. An unknown skip is a gap in Neens, not in your configuration: please report the run id. Runs that predate skip reasons are also shown as `unknown`; their history was never rewritten to guess a cause. ### Skip reason codes | Code | What it means | Retried? | What to do | | --- | --- | --- | --- | | `no_llm_connection` | No LLM connection is configured or visible to this agent, so the judge could not run. | No | Add a connection in **Settings → Connections**. | | `llm_connection_error` | The connection could not be resolved on *this* attempt (a credential-store or database blip). | **Yes** | Nothing — Neens retries. If it persists, check the connection's credential. | | `component_judge_missing` | A judge the composite aggregates no longer exists. | No | Re-point or remove that component in the composite's definition. | | `component_not_scoreable` | A component isn't an LLM-prompt judge and has no existing score for the target. | No | Score that component separately, or drop it from the composite. | | `component_llm_error` | A component's LLM call kept failing (timeout / rate limit / provider error) after the per-target retries. | **Yes** | Nothing immediately. A persistent failure points at the provider or your quota. | | `component_scoring_failed` | A component returned output Neens could not read as a score. | No | Check that component judge's prompt and output spec. | | `insufficient_components` | Fewer component scores were available than the composite's `min_components` requires. | No | Lower `min_components`, or make the missing components scoreable. | | `expression_component_missing` | The composite's expression references a component that had no value for that target. | No | Give the component a score, or set `missing_policy`. | | `no_taxonomy` | Issue Classification has no failure modes to classify against yet. | No | Add failure modes under **Taxonomy**. This one is benign. | | `target_not_found` | The target trace wasn't visible yet when the run resolved its targets (ingest was still landing). | **Yes** | Nothing — Neens re-dispatches it. | | `unknown` | Neens could not classify the skip. | No | Report the run id. | ### Are skipped targets retried? **Transient causes are; permanent ones are not.** A background sweep (every 30 minutes) looks for terminal runs holding targets skipped for a *retryable* reason and opens a **new run scoped to exactly those targets** — never a re-drive of the finished run, so the original's history and counts stay intact. Both runs appear in the Activity feed, and the drawer links them ("Retried as …" on the original, "Retry 1 of 2 — re-attempt of …" on the new one). Every retry is bounded twice: - **Attempts.** The retry chain is capped, so a target is attempted at most three times, ever. - **Rate.** The wait before the first retry is 10 minutes and **doubles** each attempt (10 → 20 → 40 minutes), so a provider that is already struggling isn't hammered. At most 20 runs per agent are re-dispatched per sweep. A **permanent** reason is never retried — it would produce the identical outcome every time and cost real money doing it. Neither is an `unknown` skip: Neens cannot argue the condition will clear, so it surfaces it loudly instead of looping on it. ### Debugging a skipped run, concretely ### Open the run Go to **Activity**, filter **Status → Skipped**, and click **Details** on the run. ### Read the header The drawer's header lists each reason with a count, whether it is retryable, and what to do — for example: ```text 3× no_llm_connection not retryable No LLM connection is configured for this agent, so the judge could not run. Add an LLM connection in Settings → Connections, then re-run the scorer. No retry: none of these reasons clear on their own. ``` ### Check an individual target Each target card carries the same code plus the specific detail the scorer recorded — e.g. *"only 1 of 3 components available (min_components=2)"* — so you can tell an agent-wide misconfiguration from one odd trace. ### Fix and re-run Permanent reasons need an action from you (a connection, a taxonomy, a component). After fixing it, click **Run now** on the scorer to score the targets that were skipped — continuous scoring itself is forward-only. ## Troubleshooting | Symptom | Likely cause | Fix | | --- | --- | --- | | New traces aren't being scored at all | The scorer is paused, or **Apply for** is **Manual (run on demand)** | Check the **Enabled scorers** table: **Resume** it, or **Configure** and switch to **Continuous (score new traces)**. | | New traces aren't scored, runs show **Skipped** with `no_llm_connection` | No LLM connection visible to the agent | Add a connection in **Settings → Connections**, then **Run now** to score what was skipped. | | Runs show **Skipped** with `insufficient_components` | The composite could not gather `min_components` scores for those traces | Lower `min_components` on the composite, or make the missing component scoreable. | | A run is **Skipped** with reason `unknown` | Neens could not classify the cause (or the run predates skip reasons) | Report the run id — an unclassified skip is a gap in Neens, not in your setup. | | Only *some* new traces get scored | Percentage sampling — that's the design | Raise the sampling percentage in **Configure** if you want more coverage (mind the cost). | | Scoring stops partway through the day, resumes tomorrow | The scorer's daily cap was reached | Raise the cap deliberately, or accept it as your cost ceiling; the budget resets at midnight UTC. | | Scoring rate suddenly dropped after the first ~200 traces | Cold-start graduation (100% → steady rate) | Expected behavior for Primary Score in a new agent — not an outage. | | Scores trail traces by minutes | Eval queue depth or a slow LLM provider | Normal under load; check the run's status on the Judges page. Transient provider errors are retried automatically. | | Old traces never got scored after enabling | Continuous scoring is forward-only | Click **Run now** on the scorer to score existing eligible traces once. | Related: [Judges](/guides/judges) · [Scores](/guides/scores) · [Pre-prod evaluations](/guides/preprod-evals) · [Metrics catalogue](/guides/metrics) ================================================================================ # Enrichments Source: /docs/guides/enrichments/ ================================================================================ # Enrichments An **enrichment** runs an LLM prompt over your traces or sessions and extracts **structured fields** from each one — a category, a flag, a number, a list — that you define up front. Where a [judge](/guides/judges) answers "how good was this?", an enrichment answers "what *is* this?": intent, language, product area, escalation risk, anything you can describe in a prompt. ## At a glance | | | | --- | --- | | **Where** | **Enrichments** in the sidebar; run history under **Activity** | | **Targets** | A **trace** or a **session** (conversation rollup — see [Traces & sessions](/guides/traces-and-sessions)) | | **Output** | Typed fields stored per target; filterable columns on the **Traces** page | | **Runs** | Manual (up to 500 targets per run) or continuous on ingest (up to 500 targets per enrichment per UTC day) | | **Needs** | An LLM connection (**Settings → Connections**) — a run without one fails with a clear error | ## Create an enrichment ### Define what to extract Open **Enrichments** and create a new one. Give it a **name**, pick the **target type** (trace or session), and write the **prompt** — plain instructions telling the model what to look at and what to extract. ### Declare the output fields Each output field is a typed column the model must fill in: | Field property | Meaning | | --- | --- | | **Key** | The field's identifier — how it appears as a column/filter and in the API. | | **Label** | Display name in the UI. | | **Value type** | `string`, `number`, `boolean`, `enum`, or `array`. | | **Enum values** | For `enum` (and `array`) fields: the allowed values the model must choose from. | Values are validated and coerced on write — a `boolean` field stores `true`/`false`, a `number` field stores a numeric value, an `array` field stores a list. ### Narrow the targets (optional) **Prerequisites** restrict which traces/sessions are eligible at all: - **Created after** a date. - Matching **topics**. - **Score conditions** — e.g. only targets whose score on a given metric is `gte`/`lte`/`eq`/ `gt`/`lt` a value, with a **missing-score policy** of `skip` (default) or `include` for targets that haven't been scored yet. ### Save New enrichments start as **drafts** — defined but not running anything, so creating one never spends tokens. **LLM prompt and external-API are the executable kinds.** Custom-Python enrichments can be authored but don't run — a run over one fails immediately with a clear "not executable" error rather than doing nothing silently. ## Run it Click **Run now** on an enrichment. You can point the run at: - everything eligible (prerequisites applied), - a **filter** — the same filter vocabulary as the Traces page, or - a specific **dataset** — only its member sessions are enriched. A manual run resolves at most **500 targets**; it enqueues, then progresses `queued → running → completed` (or `completed_with_failures` / `failed`). Each target is one LLM call, so a 500-target run is 500 calls — mind your provider costs. Enable **continuous** mode to enrich new traffic as it arrives: every ingested batch triggers a run over the just-ingested targets for each enabled on-ingest enrichment. Cost governance is built in: each enrichment has a per-day budget of **500 targets per UTC day**. Once an enrichment has dispatched 500 targets in a day, further ingest batches skip it until the next day. Disabling an enrichment (back to draft) also switches it back to manual triggering. Every run — manual or continuous — appears in the **Activity** feed with live progress counts and attribution (who started it, or **Automated** for on-ingest runs). ## Where the outputs go - **Traces page** — each enrichment field becomes a discoverable column and filter, so you can slice traffic by `intent = refund_request` or `escalation_risk = true` the same way you filter by status or score. - **Datasets** — from an enrichment you can create a [dataset](/guides/datasets) of all sessions where a field has a given value (e.g. every `category = billing` session). Filter-based datasets can also carry an enrichment condition directly, and stay in sync as new outputs land. - **Judges** — enrichment outputs reach [judges](/guides/judges) *through datasets*: build a dataset from an enrichment field, then target a judge's eval run at that dataset. The enrichment decides *which* sessions get scored; the judge's own evidence settings decide what it reads. ## How it works When a run starts, a worker resolves the run's LLM provider from the agent's configured connections (an enrichment can pin a specific **connection**, otherwise the agent default is used), then calls the model once per target with your prompt and the target's content. Extracted values are validated against your declared field types and stored per target; the same target re-enriched later updates in place rather than duplicating. Outputs are kept alongside your other trace signal, so filtering and dataset membership stay fast even at high volume. **No LLM connection, no run.** Enrichments always use the agent's LLM connection (**Settings → Connections**). If none is configured, a run fails with an explicit error — nothing is silently skipped or fabricated. Configure a connection before enabling continuous mode. ## Managing enrichments | Action | Effect | | --- | --- | | **Edit** | Updates the definition; prompt changes create a new immutable **version** (the new one becomes current, history is kept). | | **Disable** | Returns the enrichment to **draft**: no manual or continuous runs, and the on-ingest trigger is reset to manual. Existing outputs remain. | | **Delete** | Hard-deletes the enrichment, all versions, runs, and outputs — irreversible. | API reference | Route | Purpose | | --- | --- | | `GET /user-enrichments` | List enrichments (filter by name, mode, status, target type). | | `POST /user-enrichments` | Create (starts as `draft`). | | `PATCH /user-enrichments/{id}` | Edit, or set `status` to `enabled` / `draft` (disable). | | `DELETE /user-enrichments/{id}` | Hard-delete the enrichment, its versions, runs, and outputs. | | `POST /user-enrichments/{id}/versions` | Publish a new version (older versions stay immutable). | | `POST /user-enrichments/{id}/deployments` | Enable continuous (on-ingest) triggering. | | `POST /user-enrichment-runs` | Start a manual run (optional `filter` or `dataset_id`). | | `GET /user-enrichment-runs/{run_id}` | Run status plus a sample of outputs (`?sample=N`, default 20, max 200). | | `GET /enrichment-fields` | Output fields that have computed values (drives Traces filters and the dataset picker). | | `POST /user-enrichments/{id}/dataset` | Create a dataset from an output field/value. | ## Troubleshooting | Symptom | Cause → fix | | --- | --- | | Run immediately `failed` with a connection error | No LLM connection is visible to the agent → add one under **Settings → Connections**. | | Run `failed` with "not executable" | The enrichment is a custom-Python definition — only **LLM prompt** and **external-API** enrichments execute today. | | Continuous enrichment stopped mid-day | The 500-targets-per-day budget for that enrichment is spent; it resumes the next UTC day. Run manually if you need more today. | | New traffic isn't being enriched | The enrichment is in **draft**, or continuous mode isn't enabled → **Configure** it and turn on **Continuous (on ingest)**. | | Fields missing on the Traces page | Fields only appear once at least one output value exists — run the enrichment first. | See also the [FAQ](/faq). ================================================================================ # External API scoring Source: /docs/guides/external-api-scoring/ ================================================================================ # External API scoring An **external API** judge or enrichment does not call an LLM. Instead it makes an **HTTP request** to a third-party API — a toxicity, sentiment, moderation, grammar, or PII service, or your own scoring microservice — templates the trace's evidence into the request, and maps a value out of the JSON response. A [judge](/guides/judges) turns that value into a normalized 0–1 score or a category label; an [enrichment](/guides/enrichments) turns it into typed output fields. Reach for this whenever a signal you want already exists behind an API and doesn't need a model prompt: a moderation score, a language detector, a readability grade, a regex-service verdict, or a scoring endpoint your own team owns. ## At a glance | | | | --- | --- | | **Where** | **Judges** or **Enrichments** in the sidebar — pick the **External API** mode | | **Request** | A URL + method + content type + body template, with `{{vars}}` filled from the trace | | **Response** | A JSON path (dot/bracket) that selects the value to score or store | | **Auth** | Optional — an **HTTP API** connection (**Settings → Connections**) supplies a bearer token, header, query key, or basic credential | | **Needs no LLM** | The tenant's LLM connection is irrelevant — the third-party API does the work | | **Egress safety** | Every call is SSRF-gated; a public API works out of the box, private/internal hosts aren't callable | A public third-party API (a public IP) works with no extra configuration. A **private**, loopback, or link-local endpoint — an internal scoring service, a Docker service, `localhost` — is not callable; point the judge at a publicly reachable endpoint. ## Create an external API judge ### Choose the External API mode Create a new judge and select the **External API** type. You'll define an HTTP request and a response mapping instead of a prompt. ### Define the request | Field | Meaning | | --- | --- | | **Request URL** | The endpoint to call. May embed `{{vars}}` (percent-encoded when substituted into the URL). | | **Method** | `GET`, `POST`, `PUT`, `PATCH`, or `DELETE` (default `POST`). | | **Content type** | `json` (body sent as JSON), `form` (URL-encoded), or `none` (no body). | | **Headers** | Optional static headers as a JSON object; values may embed `{{vars}}`. | | **Body template** | A JSON object. Only **string** leaves are templated, so a numeric literal stays a number and a substituted value can never break the request envelope. | | **Auth connection** | Optional — an HTTP API connection for authenticated calls. Leave as **None (no auth)** for a keyless public API. | ### Map the response to a score or label A judge reads **one** value from the JSON response, selected by a dot/bracket [path](#response-paths): | Response setting | Used when | Meaning | | --- | --- | --- | | `score_path` | The judge's output is numeric (a 0–1 score or a numeric range). | Path to a number in the response. | | `label_path` | The judge's output is categorical (a set of labels). | Path to a string in the response. | | `reason_path` | Optional. | Path to an explanation string stored as the score's reason. | | `label_map` | Optional. | In **score** mode, map a returned string to a number, e.g. `{ "pos": 1.0, "neg": 0.0 }`. In **label** mode, rename the raw value to a friendly label (case-insensitive), e.g. `{ "true": "contains_profanity", "false": "clean" }` — an unmapped value is kept verbatim, so a partial map never drops a target. | | `invert` | Optional (score mode). | Store `1 − value` — for a "lower is better" API like a toxicity score. | Whether the judge reads `score_path` or `label_path` follows the judge's [output spec](/guides/judges) — a categorical spec reads `label_path`, a numeric spec reads `score_path`. ### Preview and test-run before you save The judge form shows a **Live preview** panel that renders the **exact HTTP request** this judge would send — method, URL, headers, and body — hydrated on a real sample trace, with a **Sample trace** picker to switch samples. The auth credential is never shown, even when a connection is attached. Click **Test run** to call the API once against the sample, map the response through your `response` config and output spec, and see the resulting score or label, the pass/fail verdict, and the raw response — all without saving the judge or writing a score. A slow, blocked, or unmappable response reports an honest error instead of a fabricated verdict. The preview posts to `POST /judges/preview` and the test call to `POST /judges/test-run`; both are read-only and persist nothing. ### Deploy it Deploy the judge and pick a trigger, the same as any judge. On each run Neens calls the endpoint once per target and writes a [score](/guides/scores) per response. ### Worked example — sentiment (no auth) This judge calls the public [text-processing.com](https://text-processing.com) sentiment API — no key required — and grades the agent's answer as a category. It is seeded, enabled, in the demo as **Answer Sentiment (external API)**, so you can open it and click **Run**. ```json { "url": "https://text-processing.com/api/sentiment/", "method": "POST", "content_type": "form", "body_template": { "text": "{{output}}" }, "response": { "label_path": "label" }, "output_spec": { "type": "categorical", "labels": ["neg", "neutral", "pos"], "passing": ["neutral", "pos"], "higher_is_better": true }, "target_type": "session" } ``` The API replies with `{"label": "pos", "probability": {...}}`; `label_path: "label"` selects `"pos"`, and the categorical [output spec](/guides/judges) marks `neutral`/`pos` as passing. ### Worked example — toxicity (with auth) This judge calls Google's [Perspective API](https://developers.perspectiveapi.com/), which returns a 0–1 toxicity score and needs an API key. It is seeded, enabled, in the demo as **Response Toxicity (Perspective)**, wired to the **Perspective API (toxicity)** HTTP API connection — add a key to that connection and it runs live. ```json { "url": "https://commentanalyzer.googleapis.com/v1alpha1/comments:analyze", "method": "POST", "content_type": "json", "connection_id": "conn-httpapi-perspective-…", "body_template": { "comment": { "text": "{{output}}" }, "requestedAttributes": { "TOXICITY": {} } }, "response": { "score_path": "attributeScores.TOXICITY.summaryScore.value", "invert": true }, "target_type": "session" } ``` `score_path` plucks the nested toxicity value; `invert: true` stores `1 − toxicity`, so a **high** score means a **clean** answer. ## Create an external API enrichment An [enrichment](/guides/enrichments) works the same way but writes **typed output fields** instead of a score. In place of `response`, an enrichment maps each output field to a response path via **`field_paths`**: ```json { "url": "https://text-processing.com/api/sentiment/", "method": "POST", "content_type": "form", "body_template": { "text": "{{output}}" }, "field_paths": { "sentiment": "label", "positive_score": "probability.pos" } } ``` Each key in `field_paths` is an output-field **key** you declared on the enrichment; its value is the [response path](#response-paths). A field whose path resolves to `null`/absent is simply not written for that target (an honest miss, never a fabricated value). This example is seeded, enabled, in the demo as **Response Sentiment (external API)** with a `sentiment` enum field and a `positive_score` number field. ## Template variables String values in the **URL**, **headers**, and **body template** may embed these `{{vars}}`, resolved from the target trace/session at run time: | Variable | Value | | --- | --- | | `{{input}}` | The user's request, as plain text — the [parsed exchange](/guides/judges#what-actually-gets-injected), not a raw span body. Empty when the trace carries no recoverable user turn. | | `{{output}}` | The agent's user-facing reply, as plain text. Empty when the agent never replied (a guardrail block, a tool-only run) — a routing or planning payload is never promoted to "the answer". | | `{{messages}}` | The full message list, JSON-encoded — every span with its raw body, for an endpoint that wants the whole trajectory. | | `{{metadata}}` | The trace/session metadata, JSON-encoded. | | `{{system_prompt}}` | The system prompt, when captured. | | `{{target_id}}` | The id of the target being scored. | An unknown variable renders as an empty string. Non-string evidence (`messages`, `metadata`) is JSON-encoded so it drops cleanly into a body. Each variable is truncated to 50,000 characters before templating, so an enormous transcript can't blow past an endpoint's input limit. ## Response paths A response path selects a value out of the parsed JSON body using dots for object keys and brackets for list indices: | Path | Selects | | --- | --- | | `$` | The **whole response body** — for an API that replies with a bare scalar at the root (`true`, `0.87`, `"toxic"`). | | `label` | The top-level `label` key. | | `$.label` | The same top-level `label` key — `$.` is an explicit-root alias. | | `probability.pos` | `body["probability"]["pos"]`. | | `attributeScores.TOXICITY.summaryScore.value` | A deeply nested value. | | `detected[0].language` | The first list element's `language`. | | `results[-1].score` | The last list element's `score`. | A missing path resolves to nothing — the judge records that one target as an honest failure and the enrichment simply omits that field, rather than inventing a value. ## Authenticate with an HTTP API connection For an endpoint that needs a credential, create an **HTTP API** connection under **Settings → Connections** and reference it from the definition by `connection_id`. The connection holds the encrypted credential and the auth style; the definition never contains the secret. Supported auth types (set on the connection): | Auth type | Where the credential goes | | --- | --- | | `bearer` | `Authorization: Bearer ` header. | | `header` | A header you name (`header_name`), e.g. `X-API-Key: `. | | `query` | A query parameter you name (`query_param`), e.g. `?key=` (Perspective-style). | | `basic` | HTTP Basic — `username` on the connection, credential as the password. | | `none` | No auth (a keyless public API). | A referenced connection must be **visible to the judge/enrichment's agent** — the same company/org/agent scoping the Settings UI applies. A missing, out-of-scope, or wrong-type connection makes the run fail with a clear error instead of silently falling back to an unauthenticated call, so a stored definition can never pull in another agent's credential. ## How it works - On each run Neens builds the request per target, applies the connection's auth, and makes **one HTTP call** per target. `follow_redirects` is off, so a redirect can't bounce the call to an internal host after the safety check. - The response body is buffered up to 4 MiB and parsed as JSON; a non-JSON success is handed back as raw text so a path against a string still works. - Each call is bounded to 30 seconds. A slow, oversized, blocked, or erroring endpoint fails **that one target** with a classified error — never a fabricated score or output. - Judge results flow through the same persistence path as any judge, so scores appear in [Scores](/guides/scores), on the **Traces** page, and in [dashboards](/guides/dashboards). Enrichment values become filterable columns like any [enrichment](/guides/enrichments). ## Troubleshooting | Symptom | Likely cause | Fix | | --- | --- | --- | | Every target errors with "url is not allowed" | The endpoint resolves to a private/loopback IP. | Point the judge at a publicly reachable endpoint — private, loopback, and internal hosts aren't callable. | | "connection … is not configured or not visible" | The `connection_id` points at a missing or out-of-scope connection. | Recreate the HTTP API connection in the same agent's scope. | | Credential could not be resolved | The connection references a `secret://` key with no matching environment variable. | Set the expected `NEENS_SECRET_` variable (e.g. `NEENS_SECRET_PERSPECTIVE_API_KEY`). | | Targets fail with a missing value | The `score_path`/`label_path`/`field_paths` path doesn't match the response shape. | Inspect the API's real response and correct the path. | See also the [Judges](/guides/judges) and [Enrichments](/guides/enrichments) guides for the shared run, trigger, and scoring mechanics. ================================================================================ # Datasets Source: /docs/guides/datasets/ ================================================================================ # Datasets A **dataset** is a curated collection of trace/session examples — the failures you want to fix, the golden interactions you want to protect, the slice of traffic you want a [judge](/guides/judges) to score. Datasets can fill themselves from a filter or a cluster, keep syncing as new traffic matches, and be frozen into immutable **versions** that anchor [pre-prod evaluations](/guides/preprod-evals). ## At a glance | | | | --- | --- | | **Where** | **Datasets** in the sidebar; **New dataset** to create | | **Sources** | `manual`, `filter` (Traces-style filters), `cluster` (a discovered failure cluster) | | **Items** | Sessions, traces, or imported JSON rows; captured fields: **Input**, **Output**, **All raw data** | | **Sync** | On demand (**Sync now**), or hourly and automatic for **streaming** datasets | | **Versions** | Immutable snapshots; one can be marked **golden** per dataset | ## Create a dataset ### Pick a source In **New dataset**, choose where items come from: - **Manual** — start empty (optionally upload entries as CSV or JSON), then add sessions or traces by hand or in bulk. You can also create a manual dataset directly from a selection of sessions in one step. - **From filter** — define a filter with the same vocabulary as the Traces page (status, agent, tags, scores, topics, even [enrichment](/guides/enrichments) field values). Every matching session becomes an item. - **From cluster** — pick a discovered failure cluster; its member sessions become the items. ### Scope and sample For filter/cluster sources: - **Sampling** — **All** matches (default), a fixed **Count**, or a **Percentage**. - **Add entries from** — **All history**, **Only new (from now)**, or **Since date** (a backfill lower bound on session start time). - **Fields to capture** — which parts of each session are stored on the item: **Input**, **Output** (both by default), and/or **All raw data**. A live preview shows how many sessions match and how many will be sampled before you create anything. ### Keep it fresh (optional) Turn on **Keep syncing new matches (streaming)** and Neens re-runs the source **every hour on the hour**, adding newly matching sessions (never duplicating existing items, and never removing any). Non-streaming filter/cluster datasets can still be synced on demand with **Sync now**. ## Work with items Inside a dataset you can search items, filter by kind and captured fields, and sort. Each item carries its captured **input**, **output**, and an optional **expected output**: - **Add** individual traces/sessions, bulk-add a selection, or import JSON rows. - **Edit** an item's input and expected output — the expected output is what correctness-style judges and [pre-prod evaluations](/guides/preprod-evals) compare against. The captured output is read-only (it's what actually happened). **The captured output is the agent's *derived* final answer** — the last assistant turn of the trace's reconstructed transcript, not "the last span that produced output". That keeps a trailing guardrail verdict or tool result from being frozen into a golden set, but it also means a trace whose **Conversation** tab is empty or wrong captures an empty or wrong output, with no error. Check the Conversation tab on a few traces before you cut a golden version — see [Conversation transcript](/guides/conversation-transcript). - **Export** the live dataset — or any version — as JSON or CSV. ## Golden datasets and versions The live item list of a filter/cluster dataset is a moving target — syncs keep adding to it. When you need a fixed reference set, cut a **version**: - A version snapshots the current items into a frozen list with an auto-incrementing number (v1, v2, …), an optional name and notes, and the item count at snapshot time. - **Version items are immutable.** Only the metadata (name, notes, golden flag) can change later; the frozen items never do. - Mark a version as **golden** to designate it the reference set — at most one version per dataset is golden at a time (marking a new one automatically unmarks the previous). Golden versions are what [pre-prod evaluations](/guides/preprod-evals) replay: a pre-prod run takes a dataset and a version (defaulting to the golden one if you don't pick), freezes those items as its work list, and scores the candidate agent against them. Because the version is immutable, every candidate is measured against exactly the same examples. **Judges target the live dataset; pre-prod runs target a version.** When you point a judge's eval run at a dataset, it scores the dataset's *current* members — useful for continuously scoring a curated slice. Pre-prod evaluations pin a frozen version so comparisons stay apples-to-apples. ## The review → annotation → golden loop Datasets are the middle of the Neens improvement loop: 1. **Find failures** — clusters, low scores, and issues surface what's going wrong (see [Traces & sessions](/guides/traces-and-sessions)). 2. **Label them** — humans confirm pass/fail with rationale in the [Review queue](/guides/annotations-and-review), building ground truth. 3. **Curate** — build a dataset from the confirmed failure cluster or a filter capturing the pattern, fill in expected outputs, and cut a **golden version**. 4. **Guard** — judges score the live dataset continuously, and every candidate release replays the golden version in a [pre-prod evaluation](/guides/preprod-evals) before it ships. Curation is a deliberate human step: labels tell you *what* belongs in the golden set, but you choose what to include — there's no automatic promotion from label to dataset item. ## How it works Creating a filter/cluster dataset materializes it immediately: the source is evaluated once and matching sessions are captured as items. **Sync** re-evaluates the same source and adds only new matches. Streaming datasets use a high-water mark — each hourly sync picks up from where the last one ended, so a busy agent isn't rescanned from scratch every hour. Deleting a dataset removes it and its items; sessions and traces themselves are never touched. API reference | Route | Purpose | | --- | --- | | `GET /datasets` / `POST /datasets` | List / create (`source_kind` ∈ `manual`|`filter`|`cluster`, `sampling`, `fields`, `streaming`, `historical_from`). | | `POST /datasets/preview` | Preview match + sample counts without creating. | | `PATCH /datasets/{id}` / `DELETE /datasets/{id}` | Rename/retag / delete. | | `POST /datasets/{id}/sync` | Re-run the source; returns `{added, total}`. | | `GET /datasets/{id}/items` | List/search items; `POST …/items`, `POST …/items/bulk`, `PATCH …/items/{item_id}`, `DELETE …/items` manage them. | | `POST /datasets/from-selection` | Create a manual dataset from selected sessions in one call (optionally snapshotting golden v1). | | `POST /datasets/{id}/versions` | Cut an immutable version (`{name?, notes?, golden?}`). | | `GET /datasets/{id}/versions` / `GET …/versions/{version}/items` | List versions / a version's frozen items. | | `PATCH /datasets/{id}/versions/{version}` | Edit version metadata only (name, notes, golden flag). | | `GET /datasets/{id}/export`, `GET …/versions/{version}/export` | Export as `?format=json` or `csv`. | Datasets are addressed by id or by name in these routes; there is no single-dataset detail endpoint — the list plus the items/versions routes cover it. ## Troubleshooting | Symptom | Cause → fix | | --- | --- | | **Sync now** isn't available | The dataset is `manual` — only filter/cluster datasets have a source to re-run. | | Streaming dataset isn't growing | Nothing new matches the source since the last sync (check the filter), or traffic stopped. Syncs run hourly on the hour. | | Pre-prod run can't find a version | The dataset has no **golden** version and none was specified — cut a version and mark it golden. | | Items look stale after editing the filter | The stored source config is fixed at creation; create a new dataset for a different filter. | See also the [FAQ](/faq). ================================================================================ # Annotations & review Source: /docs/guides/annotations-and-review/ ================================================================================ # Annotations & review Human labels are the ground truth for Neens: a reviewer looks at a session and records a **pass** or **fail** verdict with a critique. The **Review** page queues the sessions most worth a human look, the **Annotations** tab is the browsable history of every label, and the **Alignment** view measures how well your [judges](/guides/judges) agree with those humans — closing the loop that keeps automated scoring honest. ## At a glance | | | | --- | --- | | **Where** | **Review** page — **Queue** and **Annotations** tabs; **Alignment** tab on the **Judges** page | | **Label store** | One shared ground-truth store: a label written in the queue is the same record the Annotations tab and alignment metrics read | | **Verdicts** | `pass` / `fail`, plus a free-text critique and an optional failure-mode or cluster tag | | **Roles** | **Reviewer** (any annotator) and **principal** (expert — their labels and *gold* labels define truth) | | **Alignment** | Precision, recall, specificity, and Cohen's κ per judge, recomputed nightly (3:00 UTC) or on demand | ## The Review queue The **Queue** tab is a prioritized worklist, not a raw session list. Each candidate session is ranked by five signals — **Impact**, **Uncertainty**, **Novelty**, **Disagreement**, and **Redundancy** — so reviewer time goes where a human verdict changes the most: high-impact sessions, scores sitting near their pass/fail threshold, traffic unlike anything labeled before, and places where judges contradict each other, while clusters that already have several labels are de-prioritized. ### Configure the queue The queue configuration is saved per agent (open the config drawer on the Queue tab): | Setting | Default | Meaning | | --- | --- | --- | | **Statuses** | `error`, `failed`, `failure` | Which session statuses are eligible. | | **Score filters** | none | e.g. only sessions with a metric `lt` some value. | | **Failure modes / clusters** | all | Restrict the queue to specific failure modes or clusters. | | **Only unlabeled** | off | Drop sessions that already have a label. | | **Novelty budget** | 5 | Max novelty-ranked items per queue. | | **Queue size** | 20 (max 200) | How many items the queue returns. | ### Label a session ### Pick an item Select a queue item to open the full session — every span, tool call, and message — so the verdict is grounded in what actually happened (see [Traces & sessions](/guides/traces-and-sessions)). ### Record the verdict Choose **Pass** or **Fail** (keyboard: `P` / `F`), write a **critique** explaining why, and optionally tag a taxonomy **failure mode** or a discovered **cluster**. ### Submit The label lands in the shared ground-truth store with your identity attached. Relabeling the same session supersedes your earlier label rather than deleting it — the audit trail is kept. **Gold labels and principals.** Labels from **principal** reviewers (experts) and labels explicitly marked **gold** define the expert truth used for alignment and for grading other annotators. A principal can also **adjudicate** a disputed session, superseding all conflicting labels with one authoritative verdict. ## The Annotations tab The **Annotations** tab on the Review page browses the same label store: every label with its target, verdict, critique, failure-mode/cluster tag, gold flag, timestamp, and **who** wrote it — each label permanently carries its reviewer's name, and superseded labels remain inspectable. It also hosts the **Annotator leaderboard**: per reviewer, label volume, pass/fail split, gold labels authored, and — where a reviewer has labeled targets that also have expert truth — their **agreement** rate and Cohen's κ against it. Volume gets recognition; agreement tells you whose labels to trust. **One store, two views.** The Review queue and the Annotations tab read and write the same ground-truth labels — there is no separate "annotations" dataset to keep in sync. Those same labels are what the alignment metrics below are computed against. ## Judge ↔ human alignment Once you have expert labels, the **Alignment** tab on the **Judges** page measures each judge against them. A judge's most recent verdict per session is paired with the expert truth for that session, treating **failure as the positive class**: | Metric | Question it answers | | --- | --- | | **Precision** | When the judge flags a failure, how often is it really one? | | **Recall** (TPR) | Of the real failures, how many does the judge catch? | | **Specificity** (TNR) | How well does it avoid false alarms on good sessions? | | **Cohen's κ** | Chance-corrected overall agreement (0 ≈ random, 1 = perfect). | ### Convergence chart Each recompute persists a measurement, so the chart plots every judge as a line over time — toggle between **κ**, **Precision**, and **Recall**, and narrow the window with the time-range picker (Today / 24h / 7d / 30d / All / Custom). A rising line means your judge edits are converging on human judgment; a flat low line means the rubric still doesn't match what your experts consider a failure. Alignment is recomputed automatically **every night at 3:00 UTC**, and on demand with the refresh action. With no gold/expert labels yet, it degrades gracefully — you'll see an empty trend with the reason, never fabricated numbers. ### Disagreement drill-down Select a judge to see exactly where it diverged from the experts, split into: - **Missed failures** — the judge passed a session an expert failed (the costly kind). - **False alarms** — the judge failed a session an expert passed. Each row shows the judge's score and reasoning next to the expert's critique. If the *human* was wrong (or the case is genuinely ambiguous), click **Relabel** to write a corrected gold label in place — then hit **Recompute alignment** to refresh the metrics immediately. ## Closing the loop: iterate the judge Alignment turns judge editing from guesswork into a measured cycle: 1. **Measure** — check κ/precision/recall for the judge on the Alignment tab. 2. **Diagnose** — read the disagreements: is the judge missing a failure type your experts catch, or nitpicking things they don't care about? 3. **Edit** — update the judge's instructions or criteria; saving publishes a new immutable **version** (score history from older versions is preserved). See [Judges](/guides/judges). 4. **Re-run** — run the new version over labeled sessions and recompute alignment. 5. **Repeat** until the trend converges — then trust the judge at scale, including as a gate in [pre-prod evaluations](/guides/preprod-evals). API reference | Route | Purpose | | --- | --- | | `GET /review-queue` | The ranked review queue. | | `GET` / `PUT /review/queue-config` | Read / save the per-agent queue configuration. | | `POST /review` | Submit a label (`target_id`, `verdict` `pass`|`fail`, `critique?`, `failure_mode_id?`, `cluster_id?`, `is_gold?`). | | `GET /review/labels` | The unified label history (the single ground-truth store). | | `GET /review/gold` | Active gold-standard labels. | | `POST /review/adjudicate` | Principal override — supersede conflicting labels with one authoritative verdict. | | `GET /review/leaderboard` | Per-annotator volume + gold-agreement + κ. | | `GET /review/alignment` | Persisted alignment trend (`?range=`, `?judge_id=`). | | `POST /review/alignment/refresh` | Recompute + persist alignment for every judge with scores. | | `GET /review/alignment/disagreements?judge_id=…` | A judge's disagreements, split missed-failures / false-alarms. | ## Troubleshooting | Symptom | Cause → fix | | --- | --- | | Alignment tab is empty | No gold/principal labels yet — label sessions in the Review queue (mark decisive ones gold), then refresh alignment. | | Queue is empty | No sessions match the configured statuses/filters, or **Only unlabeled** is hiding everything already labeled — loosen the queue config. | | A judge's κ is high but it still feels wrong | κ is computed only on sessions with expert truth; label more (and more varied) sessions so the sample represents real traffic. | | Leaderboard shows no agreement for an annotator | They haven't labeled any targets that also have expert truth — agreement needs overlap with gold/principal labels. | See also the [FAQ](/faq). ================================================================================ # Remediations Source: /docs/guides/remediations/ ================================================================================ # Remediations A **remediation** is a typed, tracked fix proposal for one recurring failure — drafted from the *real evidence* in your failing traces (system prompts, tool schemas, tool-call errors), not from generic advice. The detail reads as the fix **loop** it actually is: **Root cause → Fix → Proof → Ship**. You simulate a remediation against real failing sessions before you ship it, apply it, and then measure whether the failure actually went away. Remediations are the action half of the Neens Fix pillar: [Issues and failure modes](/guides/issues-and-failure-modes) tell you *what* keeps going wrong; a remediation is the concrete change that stops it. ## At a glance | | | | --- | --- | | **Where** | **Fix → Remediations** in the sidebar (a sortable table; click any row to open that fix's full page), or an Issue's **Generate fix** action. Every remediation has a shareable page at `/remediations/{id}` | | **Key API** | `POST /remediations/generate`, `GET /remediations/items`, `PATCH /remediations/items/{id}`, `POST /remediations/items/{id}/simulate`, `GET /remediations/items/{id}/efficacy` | | **Needs** | Failure evidence (a cluster or a failure mode with classified sessions). An LLM connection (Settings → Connections) for richer generation and for simulation | | **Scope** | Agent-scoped — a remediation belongs to the agent whose failures it fixes | ## Generate a fix ### Pick a failure Start from a failure **cluster** (Diagnose) or an **Issue** (a failure mode). Either grounds the fix in a concrete set of failing sessions. ### Generate Use **Generate fix** on the Issue or cluster, or call the API — at least one of `clusterId` / `failureModeId` is required (the request is rejected with `422` otherwise): ```bash curl -X POST "https:///remediations/generate" \ -H "Authorization: Bearer $NEENS_API_KEY" \ -H "Content-Type: application/json" \ -d '{"failureModeId": "fm-abc123"}' ``` ### Review the draft The new remediation lands in **Fix → Remediations** with status **Proposed**, a typed artifact (the actual before/after change, with a diff you can open and copy — see [The Fix stage](#fix)), an inline proof eval, and a priority score. ### How generation works Neens reads the failure's real trace evidence — captured **system prompts**, **tool schemas**, **tool-call arguments and errors**, HTTP **status codes**, **span kinds and statuses**, **model parameters**, and representative failing input/output examples — then: 1. **Derives the root cause from the evidence, not the label.** Neens reads the dominant error signature across the failing sessions — the HTTP status codes in the errors, the error class names, the span kinds and their pass/fail status — and writes a **likely cause** grounded in what it found, e.g. *"≈94% of sampled sessions fail with HTTP 503 'connection pool exhausted' from the ticket service."* The likely cause is never a restatement of the cluster label. 2. **Classifies where the failure lives** — its **[failure locus](#where-the-failure-lives-the-failure-locus)**: is it your agent's reasoning, a tool contract, a downstream or upstream service, a genuine quality problem, or a control working exactly as intended? This is decided from the evidence (status codes, error classes, guardrail messages), so a real 503 becomes an infra advisory and a correctly-firing refund limit becomes *working-as-intended* — organically, not from a keyword in the label. 3. **Routes to a locus-appropriate fix type** (see [Remediation types](#remediation-types)) and drafts a grounded artifact with the agent's LLM connection: for an agent-fixable failure the artifact's `before` quotes the *real* captured text (the actual system prompt, tool schema, or error) and `after` is the proposed replacement, with a unified line diff. For a service failure the artifact is an **advisory** that names the owning component and the recommended resilience fix instead of a prompt edit. 4. **Self-critiques** the draft in a second LLM pass — checking that `before` is a genuine quote, that the proof eval is specific enough to catch this exact failure, and that the fix type and target are consistent — and keeps the improved version. 5. **Attaches a proof eval** — an LLM rubric defining what "fixed" means for this failure. It is reused as the default judge when you simulate. 6. **Applies the grounding gate** — a proposal with no concrete change (or too little evidence to classify) is held as **[needs grounding](#needs-grounding-held-out-of-the-backlog)** and kept out of the actionable backlog. 7. **Computes a priority** (see below), ranked by *fixability* — an advisory or a working-as-intended item ranks below a genuinely agent-fixable change. **No LLM connection? Generation still works.** Without a resolvable connection (or if the LLM call fails), Neens falls back to a deterministic draft derived from the failure locus and the captured evidence — the locus, likely cause, owning component, and advisory recommendations are computed from the evidence *without* an LLM, so an infra advisory or a working-as-intended verdict is still correct. What the deterministic path can't do is write a *specific* prompt rewrite: if it can only produce boilerplate ("clarify the agent's instructions…") for an agent-fixable failure, that proposal is held as [needs grounding](#needs-grounding-held-out-of-the-backlog) rather than shown as an actionable fix. Configure a connection to get fixes that quote and rewrite your actual prompts. **Sharpen your fixes with Agent context.** The **Agent context** drawer on the Remediations page lets you paste your agent's canonical system prompt, repo URL, and intended-behavior notes (`PUT /remediations/agent-context`). Neens folds this into future generated fixes so they get more specific to your agent. All fields are optional. ### Where the failure lives (the failure locus) Before it proposes anything, Neens decides **where** the failure actually lives — because the honest fix for "your agent chose the wrong tool" is nothing like the honest fix for "the ticket service was down." That classification is the **failure locus**, and it's read from the evidence, not the cluster label: | Failure locus | What the evidence shows | This is… | | --- | --- | --- | | **Agent reasoning** | A bad plan, wrong tool choice, hallucination, or an ignored instruction | Your agent — fixable in your repo | | **Tool contract** | Malformed arguments, a schema mismatch, a `400`/`422` from bad input | Your agent's call — fixable in your repo | | **Downstream unavailable** | A tool's *own* backend returns `5xx`/`503`/`504`, times out, or reports a pool exhausted / maintenance | A service, not your agent | | **Upstream unavailable** | The tool succeeded but a service *it* called returned `5xx`/timeout (named in the error) | A dependency, not your agent | | **Guardrail (correct)** | A guardrail or business rule fired correctly — the request genuinely violated a rule | A control working as intended | | **Quality** | A low judge score (relevancy / faithfulness) with **no** infra error present | Your agent — fixable in your repo | | **Incoherent** | The grouped sessions are heterogeneous — different tools, different errors, or no error at all | Not one failure — needs re-clustering | | **Unknown** | Not enough evidence to classify | Undetermined | From the locus, Neens sets an **actionability** — how you're meant to act on the fix — and it drives both the backlog and the sort order: | Actionability | Which loci | What it means for you | | --- | --- | --- | | **Actionable** | agent reasoning, tool contract, quality | Your agent team can fix it in-repo. Shown in the backlog by default. | | **Advisory** | downstream / upstream unavailable | Not agent-fixable — route it to the service or infra owner. Hidden from the actionable backlog unless you ask for advisories. | | **Non-actionable** | guardrail (correct), incoherent, unknown | Nothing to ship — a control worked, or there isn't a single fixable cause yet. Hidden from the actionable backlog. | **The whole point is to stop proposing prompt edits for things a prompt can't fix.** If your agent is getting `503`s from a ticket service, no rewording of its system prompt makes the service come back. Neens says so plainly, names the owner, and recommends the resilience change that *would* help — see the advisory examples below. ### Remediation types Every remediation carries a **type** — *what kind of change* the fix is — so it can carry a concrete artifact (or a concrete recommendation) rather than free-text advice. The agent-fixable types stay as before; the grounded flow adds four types for the service and control-plane loci: | Type | The remediation is… | Typical locus | | --- | --- | --- | | `prompt_change` | A change to the system prompt | agent reasoning | | `tool_schema` | A change to a tool's description or argument schema | tool contract | | `param_change` | A change to a model parameter (e.g. temperature) | tool contract | | `guardrail` | A change to an input/output guardrail rule | agent reasoning | | `retrieval` | A change to retrieval / context-assembly configuration | agent reasoning | | `routing` | A change to routing / escalation rules (e.g. hand off to a human) | agent reasoning | | `add_eval` | Encoding the failure as a permanent regression eval | quality | | `kb_fix` | A fix to stale knowledge-base content | agent reasoning | | `infra_advisory` | **New.** An advisory that a *service* is failing — names the owning component and the recommended fix; not an agent edit | downstream / upstream unavailable | | `tool_resilience` | **New.** A resilience change to how *your agent calls* a flaky tool — bounded backoff with jitter, a circuit breaker, an idempotency key, or a queue | downstream unavailable | | `escalate_to_owner` | **New.** Route to a named external owner when your agent has no viable mitigation and the fault is clearly someone else's service | downstream / upstream unavailable | | `working_as_intended` | **New.** No action — a guardrail or business rule fired correctly | guardrail (correct) | #### Worked examples Your `create_ticket` tool starts returning **HTTP 503 "connection pool exhausted"** across a burst of sessions. The failing span is the tool's own call, so the locus is **downstream unavailable** and the actionability is **advisory**. Neens does **not** propose a prompt edit. It generates an **`infra_advisory`**: > **This is a downstream service failure, not an agent defect.** Owner: **ticket-service**. > ≈94% of sampled sessions fail with `HTTP 503 "connection pool exhausted"`. The agent already > retries once with no backoff. **Recommended:** bounded exponential backoff with jitter, a circuit > breaker so the agent degrades gracefully during an outage, and an idempotency key so a retried > ticket isn't created twice. The resilience half of that recommendation is exactly a **`tool_resilience`** change — something *your* agent code can own even though the outage isn't your fault. Your `recommend_products` tool returns `200`, but its error body names a service *it* called — **`inventory-api` returned 503**. The tool itself is fine; a dependency behind it is down, so the locus is **upstream unavailable**, `owningComponent` is set to the named upstream service, and Neens generates an **`escalate_to_owner`** advisory pointing at `inventory-api`'s owner with the same resilience recommendation. There is no in-repo change that fixes someone else's outage. A cluster of "refund failed" sessions turns out to be a guardrail firing correctly: a request for a **$500 refund on a $59.50 order** was blocked because it exceeds the order total. The evidence is a guardrail span whose message states a genuine rule violation, so the locus is **guardrail (correct)** and the type is **`working_as_intended`**: > **No action recommended — a control fired correctly.** The refund guardrail blocked a request for > $500 against an order total of $59.50. Evidence: guardrail span `refund_limit`, message "requested > $500 exceeds order total $59.50." This never enters the actionable backlog, because "fixing" it would mean removing a working control. A cluster of sessions shows the agent calling `search_web` when the task needed the internal `lookup_order` tool — no infra error anywhere, a bad plan. The locus is **agent reasoning**, the actionability is **actionable**, and Neens generates a normal **`prompt_change`** with a real before/after quoting your captured system prompt. This is the classic in-repo fix, and it takes the full [simulate → apply → verify](#the-remediation-detail-the-fix-loop) loop. Failure categories Alongside the locus, Neens keeps a finer-grained **failure category** for the agent-fixable loci; a known category can refine the default fix type and the shape of the generated artifact: | Category | Typical signal | Default fix type | | --- | --- | --- | | `malformed_output` | Invalid JSON, schema violations, parse errors | `tool_schema` | | `wrong_tool` | The agent called an inappropriate or unknown tool | `tool_schema` | | `missing_context` | Retrieval/truncation left out the needed evidence | `retrieval` | | `loop_nontermination` | The agent spun on the same step, never stopped | `param_change` | | `over_refusal` | Benign, in-policy requests were refused | `prompt_change` | | `hallucination` | Ungrounded or invented claims | `prompt_change` | | `planning` | Skipped or reordered required steps | `prompt_change` | | `multi_agent` | Dropped context or misrouted hand-offs between agents | `routing` | | `other` | Nothing else matched | `prompt_change` | ### Priority Every remediation carries a **RICE-style priority score** (0–100) so you can work the highest-leverage fixes first. In the detail it folds into a single **priority chip** (its band); the full breakdown lives in the chip's tooltip. It combines: | Factor | Meaning | | --- | --- | | **Reach** | How many sessions the failure affects (log-damped, so a giant cluster doesn't drown everything else) | | **Impact** | How bad the failure is (per-category default, 0–1) | | **Confidence** | How sure we are the fix works — starts low (0.3 unsimulated) and is *earned* by simulation | | **Effort** | How much work the change is (per-type default; higher effort lowers priority) | | **Fixability** | How much *you* can actually do about it. An actionable, agent-fixable failure counts full; an advisory (someone else's service) and a *needs-grounding* item are damped down; a working-as-intended item is zero — a working control is never "high priority to fix" | The score is bucketed into a band: **critical** (≥ 60), **high** (≥ 35), **medium** (≥ 15), else **low**. Sort the table by **Priority** (or newest-first), and filter by work state, status, type, or label. Because fixability is a factor, a genuinely fixable prompt change out-ranks a higher-volume infra advisory — the backlog is sorted by *what you can fix*, not just by how loud a failure is. ## Needs grounding: held out of the backlog A remediation is only useful if it carries a **concrete change**. Neens will not put a fix in your actionable backlog just because it exists — an empty or templated proposal is worse than none, because it looks like work that isn't there. A proposal is held as **needs grounding** (and kept out of the actionable/open backlog) when any of the following is true: - it's an agent-fixable type but the artifact has **no concrete change** — `before` equals `after`, or the change is the generic filler *"clarify the agent's instructions…"* that the deterministic fallback emits when it has nothing specific to say; - there's **no usable error signature** and no confidence to stand on; - the locus is **incoherent** (the grouped sessions aren't one failure) or **unknown** (not enough evidence to classify) — here the honest next step is to re-cluster or gather more evidence, not to ship a guess. A needs-grounding item isn't deleted — it's parked. In the UI it lives in a distinct **Needs grounding** section, separate from the actionable backlog, so a real fix is never buried under placeholders. **Surfacing held items anyway.** The list and the [MCP](/guides/mcp) `list_open_remediations` tool default to showing only *grounded, actionable* items. To see what's being held, add `include_ungrounded=true` (needs-grounding items) or `include_advisories=true` (service advisories) to `GET /remediations/items`, or use the corresponding view filter in the UI. You can also filter directly by `actionability=` (`actionable` / `advisory` / `non_actionable`) or `failure_locus=`. **Advisories are complete, just not agent-fixable.** An `infra_advisory` or a `working_as_intended` item that names its cause and owner is *grounded* — it's a finished, honest answer. It's excluded from the **actionable** backlog because there's nothing for your agent team to ship, not because it's unfinished. Ask for advisories explicitly (above) to work them with the right owner. ## One root cause, one remediation (deduplication) The same underlying failure often fragments across several agents or clusters — three different agents all timing out against the same ticket service look like three problems but are one. Neens collapses them: each remediation carries a **root-cause signature** (a normalized `locus:component:error` key, e.g. `downstream_unavailable:ticket_service:503_pool_exhausted`), and when a new proposal matches the signature of an **open** remediation already in the agent, Neens **merges** it instead of inserting a duplicate — it adds the new cluster's reach to the existing row, records the additional affected agent/cluster in the evidence, and recomputes priority against the **combined** impact. The result is one remediation that says *"this affects N agents"* with the total reach behind it — so a widespread root cause rises in the backlog on its true blast radius, and you fix it once rather than triaging the same thing three times. ## Applying a fix requires a bound proof You can't move an **actionable** remediation to **applied** or **verified** on a hunch. Neens requires a **bound proof** first — evidence that the fix actually works: - a completed **verification run** (the `pass^k` result from an [eval-verified PR](/guides/eval-verified-fix)) bound to the remediation, **or** - a **proof-eval gate** — the failure mode's dataset + judge (an [eval gate](/guides/eval-gates)) — that a fix can be run against. Without one, the transition is refused with a clear message telling you what's missing (an HTTP `409`, mirrored in the UI as a disabled **Apply / Verify** control with an explanatory tooltip) — never a silent success and never a crash. The fastest way to bind a proof is to [simulate the fix](#proof) and then let the [eval-verified PR](/guides/eval-verified-fix) flow open a verified pull request; both leave a bound proof on the remediation. **Overriding the gate.** If you've verified a fix by other means, you can still record it as applied by confirming an override and recording a **reason** (the API accepts `?force=true` with a reason; the reason is stored on the remediation). Use it deliberately — the whole point of the gate is that "we shipped it" and "we proved it" stop being the same claim. **Advisories settle differently.** An `infra_advisory`, `escalate_to_owner`, or `working_as_intended` item can't be proven by an agent eval — there's no agent change to test. These close with an **acknowledgement note** (who's handling it, what was done) and settle as **closed** with an outcome note rather than **verified**. ## The remediation detail: the fix loop Click any row in the list to open that remediation's own page, `/remediations/{id}` — and the detail is laid out as the four-stage loop the fix moves through: **Root cause → Fix → Proof → Ship**. A persistent header sits on top; the four stages sit below it. Each stage toggles **independently** (it's not an accordion), so you can have several open at once. The live stage opens by default and a completed stage collapses to a one-line receipt. Use **Expand all** / **Collapse all** (just above the stages) to open or fold every section — including the trailing **Affected sessions** disclosure — in one click. ### The header (always visible) The header never collapses, so the three things you most need are on screen no matter which stages you have open: - **Title** and a **one-line root cause** — what's being fixed, in a sentence. - A **proof-state pill** (the [proof status](#triage-status-vs-work-state)) and a **work-state chip** (your team's triage lane) — the two independent tracks, side by side. - **Four stat tiles** — **Sessions** (how many failing sessions ground the fix), **Confidence** (earned by simulation), **Sim Δ** (the before→after pass-rate lift from the latest conclusive simulation), and **Efficacy** (the post-deploy verdict). A tile with nothing to report yet reads as a muted **"—"** — never `0`, never a green number (see [Reading "unknown" honestly](#reading-unknown-honestly)). - **One primary action** — the single most useful next step (below), a **View diff** button (shown unless View diff is already the primary action, so it never appears twice), and an overflow **⋯** menu (move to another work state, archive/unarchive, delete). The **primary action** is whatever the loop is waiting on: | Where the fix is | Primary action | | --- | --- | | `proposed` | **Accept** | | `accepted`, not yet simulated (or the last sim failed) | **Simulate fix** | | `accepted`, simulation passed, [fix engine](/guides/eval-verified-fix) available | **Auto-fix (eval-verified PR)** | | A pull request is open | **View PR** | | `applied`, efficacy not yet measured | **Record merge** | | Anything else | **View diff** | While a simulation or a fix run is actually in progress, the header shows that it's running rather than offering an action to start another. ### Root cause The first stage is always complete — it's *why* the remediation exists. It shows the failure summary, the **evidence-derived likely cause** (the dominant error signature across the failing sessions, e.g. *"≈94% fail with HTTP 503 'connection pool exhausted' from the ticket service"* — not a restatement of the cluster label), a **failure-locus** badge and an **actionability** badge (see [Where the failure lives](#where-the-failure-lives-the-failure-locus)), a **failure-category chip**, and the evidence it's grounded in (the failing sessions and cluster it was drafted from). For an advisory, this stage also names the **owning component** — the service responsible — so you can route it. Once you've read it, collapse it to its one-line receipt: e.g. *"Grounded in 14 `hallucination` sessions."* An **ungrounded** receipt (a fix with no attached failing sessions) is itself a signal — the fix is advice, not evidence, until it's linked to a failure. ### Fix This stage holds the grounded **fix artifact**: the **target** (what to change), the change instruction and rationale, and the concrete change itself. For a `prompt_change` you see the prompt **before → after**; for other types you see the raw unified **diff** — one representation, never both. ### View the full diff Click **View diff** in the header. A **Fix diff** dialog shows the whole unified diff — added lines in green with a leading `+`, removed lines in red with a leading `-` — with the line count and size in the footer. ### Copy it Click **Copy to clipboard** in the dialog. It confirms with **Copied to clipboard**. If your browser blocks clipboard access you'll see **Couldn't copy automatically — select the text and copy it manually** — select the diff text in the dialog and copy it with your keyboard. ### Apply it in your codebase Paste the diff into your editor or coding agent and make the change. Applying it by hand is one of three routes: the highlighted **View fix bundle** card in the Fix stage gives your coding agent a complete pack — root cause, the fix, anonymized failing examples, and the proof-eval gate command — see [Fix bundles](/guides/fix-bundles); and, where it is enabled, the [eval-verified fix](/guides/eval-verified-fix) opens a *proven* PR against your repo for you to review and merge. When the fix engine is enabled for your agent, its controls live in this stage too — launch an [eval-verified PR](/guides/eval-verified-fix), and the stage's receipt reflects the result (*"PR opened, fixes 12 of 14 evals"*, a **draft** that needs a human, or a **failed** run to read). The highlighted **View fix bundle** card opens the coding-agent-ready pack — root cause, grounded fix, proof evals — in a dialog with a one-click **Copy to clipboard**. See [Fix bundles](/guides/fix-bundles). ### Proof Before shipping, **simulate** the fix: Neens replays it against a sample of the *real* failing sessions and shows a **before → after** pass-rate comparison, so you apply fixes that actually help and skip the ones that don't. This stage shows the proof-eval rubric, the simulation plan, and the results; its **Configure** controls open in a **slide-over drawer** so the stage stays readable. Use **Simulate fix** (`POST /remediations/items/{id}/simulate`). On production deployments the simulation runs **in the background** on a worker — the stage shows a pending banner and polls for the result, so you can keep working. Simulation requires an LLM connection; without one the request is rejected with a clear `400` rather than failing mid-run. #### Editable criteria Open the **Configure** drawer to steer exactly how the simulation runs; a live plan preview shows what *would* run before you commit: | Criterion | Behavior | Default | | --- | --- | --- | | **Judge** | Which evaluator decides pass/fail | The failure mode's judge, else the fix's own proof eval, else a generic correctness judge | | **Threshold** | Pass mark: pass = score ≥ threshold | The judge's threshold (0.5 for the built-in fallbacks) | | **Model** | Which LLM connection runs the replay and the judging | The agent's default connection | | **Max traces** | How many sessions to replay | 6 (cap 20) | | **Sampling** | `first` or most `recent` failing sessions | `first` | | **Cohort** | Tick exact sessions from a per-trace list; only genuine members of the failure's cluster are accepted | Auto-sampled | | **Strategy** | `auto`, `rewrite`, or `replay` (see below) | `auto` | The resolved choices — including the exact session ids — are pinned to the run, so the background worker replays *exactly* the cohort you previewed even if new failures arrive in between. #### Two strategies | Strategy | What it does | | --- | --- | | **Answer-rewrite** | For each failing session, asks the model to produce the output the agent *would* return with the fix in place (given the original request and any retrieved context), then re-judges that corrected output. Works for any fix type. | | **Record-and-replay** | Actually **re-runs the agent's turn with the fixed system prompt installed**, letting the model drive tool calls — and serving each requested tool's result from what was *recorded* in the original trace. Nothing touches production systems. Higher fidelity: a prompt fix that changes *which tools the agent calls* is exercised for real, not paraphrased. | **Auto** picks record-and-replay when the fix is a system-prompt change and the model connection supports tool calling; otherwise it falls back to answer-rewrite. If record-and-replay turns out to be impossible at run time (e.g. the model rejects tool calling), the run degrades to answer-rewrite automatically and the result says so. The results always show which strategy actually ran. The rewrite model is deliberately **never shown the judge's rubric** — the judge grades the fixed output independently. Otherwise the simulation would degenerate into "write something that passes" and every fix would look perfect. #### Reading the results Results show pass-rate **before** and **after**, plus a **per-trace breakdown**: which traces the fix fixed, which it broke, and which were unchanged. A delta-keyed next step appears — if the fix improved things you get an **Apply** call to action; if it's flat or regressed you're prompted to adjust the criteria (with a warning when any trace got worse). When it's done, the stage collapses to a one-line receipt — e.g. *"Passed: 21% → 76%."* **Honest inconclusives.** Not every result can be trusted, and Neens says so instead of claiming a lift: - **Inconclusive** — the fixed output couldn't be synthesized for a trace (or its re-score was unparseable). Its "after" score is pinned to the unchanged baseline — *never* a guessed improvement — and the result reports how many traces could actually be simulated. The stage receipt reads **Inconclusive**, not a pass-rate. - **Diverged** (record-and-replay only) — the fixed prompt drove the agent to a tool call that was never recorded in the original trace. The trajectory has genuinely left the captured run, so Neens stops that trace and reports it honestly rather than fabricating a tool result. Diverged traces are also pinned to their baseline. Treat an inconclusive or heavily-diverged simulation as "needs more evidence," not as a pass. A completed simulation updates the remediation's **confidence** — it is *earned* from the replay: the after pass-rate dominates (weight 0.6), blended with the before→after improvement (weight 0.25). The basis is stored with the fix so you can see where the number came from. **Apply changes the lifecycle, not your agent.** The **Apply** action marks the remediation `applied` so Neens starts tracking efficacy — it does **not** modify your agent. Ship the change yourself (**View diff** → **Copy to clipboard**, or a [fix bundle](/guides/fix-bundles) / [eval-verified PR](/guides/eval-verified-fix)), then record a deploy event so the change is correlated with the outcome — see [What changed](/guides/what-changed). Note that **Apply** is only available once a proof is bound to the fix — see [Applying a fix requires a bound proof](#applying-a-fix-requires-a-bound-proof). ### Ship The last stage answers *did shipping it actually work?* — one **"After merge"** card that folds together the deploy/proof status, the PR / commit / verification links, and the **efficacy verdict**. Once a fix is applied, Neens measures whether the failure *actually* shrank in live traffic. `GET /remediations/items/{id}/efficacy` compares: - **Baseline volume** — how many sessions the failure's cluster had when the fix was proposed (snapshotted at generation time), against - **Current volume** — how many sessions that cluster has now, and returns the reduction, the reduction percentage, and a verdict: **improved** (volume fell), **flat** (no change), **regressed** (volume grew), or **unknown** (not enough data to compare). The card sits next to the **What changed** list of deploy events recorded in the 7 days before the fix was proposed, so you can correlate the failure (and the fix) with real changes — see [What changed](/guides/what-changed). Use the verdict to move the proof status forward: mark the fix `verified` when the volume drop holds, or `regressed` if the failure returns. To make sure it *stays* fixed, turn the failure mode into a standing [eval gate](/guides/eval-gates). ### Reading "unknown" honestly Everywhere in the detail, a value that hasn't been earned yet is shown as exactly that — and never dressed up as a success: - A stat tile with no result reads a muted **"—"**, not `0`. - The Proof stage reads **Not run** until you simulate, and **Inconclusive** when the replay couldn't be trusted — neither is a pass. - The Ship stage reads **unknown** when there isn't enough post-deploy data to compare — not "no improvement." Each of these is a **receipt of what hasn't happened yet**, so you can tell "this fix is proven" apart from "we haven't checked." An `unknown` is never a green tick and never a zero — it's your cue to run the next step (simulate, apply, or wait for more traffic). ### Affected sessions At the bottom of the detail, an **Affected sessions** disclosure shows the count of failing sessions grounding the remediation. Expand it to browse the actual sessions; it loads only when you open it, so a remediation with a large cluster stays fast to view. ## Triage: status vs. work state Each remediation has **two independent tracks** — keep them straight, they answer different questions. **Proof status** — *is this fix proven and shipped?* | Status | Meaning | | --- | --- | | `proposed` | Drafted from evidence; not yet reviewed | | `accepted` | Reviewed and agreed it's the right fix | | `applied` | The change has been shipped to your agent | | `verified` | Post-deploy data confirms the failure is actually reduced | | `closed` | Done and put to rest | | `regressed` | A previously-fixed failure has come back | **Work state** — *where is this in my team's queue?* A simple triage lane, orthogonal to proof status: | Work state | Meaning | | --- | --- | | `todo` | Not started (the default) | | `in_progress` | Someone is on it | | `done` | Worked through | | `archived` | Soft-hidden (reversible; shown again with **Include archived**) | A fix can be `applied` (proof) while still `in_progress` (work), or `proposed` while parked in `todo`. Both are updated with `PATCH /remediations/items/{id}` (`status`, `workState`, `labels`) — in the UI, from the header's overflow **⋯** menu — and an unknown value is rejected with `422`. Delete is a permanent hard-delete that also removes the fix's simulation history. Moving a remediation to `accepted` is normally a human decision, and by default it stays one. An agent can hand part of that over with an [autonomy level](/guides/autonomy); a human still merges every resulting PR. ## The Remediations list **Fix → Remediations** is a dense, sortable **table** — one row per remediation — so you can scan and triage many fixes at once: | Column | | | --- | --- | | **Title** | What's being fixed | | **Type** | The [remediation type](#remediation-types) | | **Priority** | The [priority band](#priority) chip | | **Status** | Proof status | | **Work state** | Your team's triage lane | | **Confidence** | Earned by simulation (a muted **"—"** until it is) | | **Sessions** | How many failing sessions ground it | | **Updated** | Last change | Click a sortable column header (every column except **Work state**) to sort by it. **Search** and the **filters** (sort, work state, status, type, label, and **Include archived**) sit right beside the rows, and label chips filter with one click. By default the table shows the **actionable backlog** — grounded, agent-fixable fixes. Service **Advisories** and **Needs grounding** items live in their own views so they don't crowd the work you can actually ship; switch to them (or add `include_advisories=true` / `include_ungrounded=true` to the API call) when you want to route advisories to a service owner or review what's being held. You can also filter by **actionability** and **failure locus** directly. **Open a remediation.** Click any row to go straight to that remediation's full page — the [fix-loop detail](#the-remediation-detail-the-fix-loop) at its own shareable URL, `/remediations/{id}`, the link to paste into a ticket or hand to a teammate. A thin coloured rail on the left of each row marks its proof status and the **Confidence** column shows a small meter, so the table scans at a glance. - Press **J** / **K** to move the highlighted row down / up without reaching for the mouse. - Press **Enter** to open the highlighted remediation. **Overview.** The failure-volume trend and the summary stats are tucked into a collapsed **Overview** disclosure at the top, so the table and filters get the space by default. Expand it when you want the fleet-level picture; it remembers your choice. A deep link from a failure mode (`?failure_mode_id=…`) opens the list already filtered to that failure's remediations, with a banner telling you so. Clear the banner to see everything again. API reference | Endpoint | Purpose | | --- | --- | | `POST /remediations/generate` | Draft a fix from `clusterId` or `failureModeId` (`422` when neither is given) | | `GET /remediations/items` | List; filters `clusterId`, `failureModeId`, `status`, `workState`, `type`, `label`, `q`, `sort=priority`, `includeArchived`, `actionability`, `failure_locus`, `include_advisories`, `include_ungrounded` (default: grounded + actionable only) | | `GET /remediations/items/{id}` | Full detail incl. simulation history and correlated deploy events | | `PATCH /remediations/items/{id}` | Update `status`, `workState`, `labels`; moving to `applied`/`verified` requires a bound proof (or `?force=true` with a `reason`) | | `DELETE /remediations/items/{id}` | Permanent hard-delete (incl. simulation history) | | `GET /remediations/items/{id}/simulation-plan` | Preview what a simulation would run (no LLM calls) | | `POST /remediations/items/{id}/simulate` | Run the counterfactual simulation (background on production deployments) | | `GET /remediations/items/{id}/efficacy` | Baseline-vs-current failure volume verdict | | `GET /remediations/stats` | Counts by status/work state/category — plus by `failureLocus`, `actionability`, and `groundingStatus` — with average confidence and priority | | `GET /remediations/agent-context` / `PUT /remediations/agent-context` | Read / set the agent context (one per agent) | ## Related - [Issues and failure modes](/guides/issues-and-failure-modes) — where the failures come from - [Fix bundles](/guides/fix-bundles) — export a remediation as a coding-agent-ready pack - [Eval-verified PR](/guides/eval-verified-fix) — let Neens open a *proven* PR for the fix - [Eval gates](/guides/eval-gates) — keep a fixed failure from silently coming back - [What changed](/guides/what-changed) — correlate regressions with deploys - [Judges](/guides/judges) — the evaluators simulations use to score before/after - [Insights](/guides/insights) — regression and anomaly detection across your fleet ================================================================================ # Regenerate remediations Source: /docs/guides/regenerate-remediations/ ================================================================================ # Regenerate remediations **Regenerating** a [remediation](/guides/remediations) re-runs today's fix engine on the *same failure* the remediation was drafted from, and produces a **fresh proposal**. The old one isn't thrown away — it's archived and marked **superseded**, linked to its replacement, and fully restorable. You get the new engine's thinking without losing any history. A remediation is only ever as good as the engine and the evidence it was drafted from. When either improves, the proposals already sitting in your backlog are stale — they reflect an older read of the same failure. Regenerating is how you pull them forward. ## At a glance | | | | --- | --- | | **Where** | The **Regenerate** action on a remediation's detail page (the **⋯** menu), and on an Issue / failure-mode card next to **View fix**. A **Regenerate all** button sits in the **Failure Modes** page header | | **What it does** | Drafts a new remediation from the same cluster / failure mode; archives the previous one as **superseded** (never deletes it) | | **Needs** | The same evidence the original used (a cluster or failure mode). An LLM connection (**Settings → Connections**) for a richer, specific proposal — it degrades gracefully without one, exactly like the first [generation](/guides/remediations#how-generation-works) | | **Scope** | Project-scoped — you can only regenerate fixes in the project you're in | ## When to regenerate Regenerate when the *proposal* is stale even though the *failure* hasn't changed. The common cases: - **The fix engine improved.** Neens ships better grounding, classification, and artifact drafting over time. An old proposal may have mislabeled where a failure lives — calling a downstream outage an agent prompt problem, say — and the current engine gets it right. Regenerating lets you *see the new logic take action* on a failure you already have on file. - **Detection / classification changed.** If your [failure-mode taxonomy](/guides/issues-and-failure-modes) or clustering has been re-curated, the evidence behind a remediation may now classify differently. A fresh draft reflects the corrected picture. - **New failing traces arrived.** The original was drafted from the sessions available *then*. When more examples of the same failure have since landed, regenerating re-reads the enlarged evidence — often sharpening the likely cause, the reach, and the proposed change. Regenerating is **not** for editing a fix by hand or for moving it through the loop — use the normal [simulate → apply → verify](/guides/remediations#the-remediation-detail-the-fix-loop) flow for that. Reach for **Regenerate** only when you want the engine to *re-draft the proposal from scratch*. ## Regenerate a single remediation You can regenerate one fix from two places: its own detail page, or the Issue / failure-mode card it came from. ### Open the remediation Go to **Fix → Remediations** and open the fix you want to refresh (`/remediations/{id}`). ### Choose Regenerate Open the header's overflow **⋯** menu and click **Regenerate**. Because this archives the current proposal, Neens asks you to confirm first. ### Review the fresh proposal Neens drafts a new remediation from the same evidence and takes you straight to it. It opens with a **Regenerated from a previous proposal** banner and a [before/after diff](#the-before-after-diff) against the one it replaced. On the **Failure Modes** page (**Diagnose → Failure Modes**), an Issue that already has a fix shows a **View fix** link. Next to it is a **Regenerate** link — click it to re-draft that Issue's fix in place. As with the detail-page action, you confirm first (it archives the current proposal), and you land on the new remediation. **Nothing is generated silently.** A single regenerate runs right away and returns the new fix. The old one is archived in the same step — you'll always end up looking at the replacement, with a link back to what it superseded. ## Regenerate all open fixes When an engine change is broad, refreshing fixes one by one is tedious. The **Failure Modes** page header has a **Regenerate all** button that re-drafts every **open** remediation in the project in one go — the proposals still in play (`proposed` / `accepted`), not ones you've already archived. ### Click **Regenerate all** It's in the **Failure Modes** page header. Neens tells you how many open fixes will be regenerated and asks you to confirm — this archives each current proposal and replaces it. ### Let it run On a production workspace the batch runs in the background so you can keep working; a large backlog doesn't block the page. A per-run cap keeps a single bulk request bounded, so if your backlog is very large you may run it more than once. ### Review the results When it finishes you get a toast with the count regenerated. Each refreshed Issue now links to its new fix, and every new fix carries the [before/after diff](#the-before-after-diff) against its predecessor, so you can scan what actually changed. **Safe to re-run.** A bulk regenerate skips anything that was already archived or superseded earlier in the same pass, so a fix is never regenerated twice in one click — and re-running it after more traces arrive simply supersedes the current proposals with fresher ones. ## The before/after diff A regenerated remediation knows which proposal it replaced, so it can show you exactly what the new engine decided differently. The **Regenerated from a previous proposal** banner sits at the top of the fresh fix, and below it a **before/after diff** compares the old proposal against the new one: | Compared | Why it matters | | --- | --- | | **Type** | The [remediation type](/guides/remediations#remediation-types) — e.g. a `prompt_change` that became an `infra_advisory` | | **Root cause / failure locus** | [Where the failure lives](/guides/remediations#where-the-failure-lives-the-failure-locus) — the single biggest thing a better engine tends to correct | | **Actionability** | Whether it's **actionable**, **advisory**, or **non-actionable** — i.e. whether *your* team ships it or routes it to a service owner | | **Summary** | The one-line description of the failure and the fix | | **Artifact** | The concrete change — the prompt before/after, the diff, or the advisory text | Read it as a receipt of *what the new engine changed its mind about*. If the locus flipped from "your agent" to "a downstream service", that's the headline — and it usually flips the actionability and the type with it. ## What happens to the old proposal This is the important part: **nothing is deleted.** When you regenerate, the previous remediation is: - **Archived** — moved to the `archived` work state so it drops out of your active backlog, exactly like any other archived fix. - **Marked superseded** — stamped with a link to its replacement and the time it was superseded. Open an archived, superseded remediation and you'll see a **Superseded by a newer proposal** link to its successor. - **Kept intact** — its proof status and everything it earned are preserved. An already-**applied** or **verified** fix keeps its status *and* its history — the pull request, the commit, and the verification run stay attached. Regenerating never detaches the proof from a fix you already shipped; it simply drafts a *new* proposal alongside it. Because the old proposal is preserved and restorable, regenerating is safe on a fix in **any** state — proposed, accepted, applied, verified, closed, or regressed. If the fresh proposal turns out to be worse, you haven't lost the original. **Superseding doesn't un-ship anything.** Archiving the old proposal is a bookkeeping move in Neens — it does not touch your codebase or roll back a change you already deployed. If you'd already applied the old fix, that change is still live in your agent; the archived record just stops competing for attention in your backlog. Work the new proposal through [simulate → apply → verify](/guides/remediations#the-remediation-detail-the-fix-loop) on its own merits. ## Walkthrough: a ticket failure that was never a prompt problem Here's the case regeneration is built for. ### The original proposal Weeks ago, a cluster of failing `create_ticket` sessions produced a remediation. The engine of the day read "ticket creation keeps failing" and drafted a **`prompt_change`** — a rewrite of the agent's system prompt telling it to be more careful when filing tickets. It's been sitting in your backlog as **Proposed** ever since, and no amount of rewording has made the failure go away. ### Regenerate it You open the remediation, click **⋯ → Regenerate**, and confirm. ### The new engine re-reads the evidence The current engine looks at the *actual* evidence in the failing sessions — and finds that ~94% of them fail with `HTTP 503 "connection pool exhausted"` returned by the `create_ticket` tool's **own backend**. That's a downstream service outage, not an agent defect. No prompt rewrite can bring a service back up. So instead of another prompt edit, it drafts an **`infra_advisory`**: the failure locus is **downstream unavailable**, the actionability is **advisory**, and the proposal names the owning service and recommends the resilience change that would actually help (bounded backoff with jitter, a circuit breaker, an idempotency key). ### Read the diff The before/after diff makes the correction obvious at a glance: | | Old proposal | New proposal | | --- | --- | --- | | **Type** | `prompt_change` | `infra_advisory` | | **Failure locus** | Agent reasoning | Downstream unavailable | | **Actionability** | Actionable | Advisory | | **Artifact** | A system-prompt rewrite | An advisory naming the ticket service owner + a resilience recommendation | ### Route it correctly The old `prompt_change` is archived and marked superseded — still there if you want it, one click away via the **Superseded by a newer proposal** link. You route the advisory to whoever owns the ticket service, which is where the fix actually lives. You stopped polishing a prompt for a problem a prompt was never going to solve. ## Related - [Remediations](/guides/remediations) — the full fix loop and how proposals are generated in the first place - [Issues and failure modes](/guides/issues-and-failure-modes) — where the evidence behind a fix comes from - [Fix outcomes](/guides/post-merge-efficacy) — measuring whether a shipped fix actually reduced the failure - [Eval-verified PR](/guides/eval-verified-fix) — let Neens open a *proven* pull request for an actionable fix ================================================================================ # Fix bundles Source: /docs/guides/fix-bundles/ ================================================================================ # Fix bundles A **fix bundle** is a [remediation](/guides/remediations) exported as a single, self-contained document you hand to a coding agent. It packages everything Neens already worked out about one recurring failure — the root cause, the concrete fix, a few anonymized failing examples, and a ready-to-run proof-eval command — into one block of Markdown you paste into **your own** Claude Code / Cursor / Codex session to implement the change and gate the pull request. Neens does the diagnosis; your coding agent, in your repo, with your credentials, does the edit. The bundle carries no keys and needs no repo access — see [Zero new trust surface](#zero-new-trust-surface). ## At a glance | | | | --- | --- | | **Where** | The **View fix bundle** button in the **Fix** stage of a remediation's detail (**Fix → Remediations**) | | **Key API** | `GET /remediations/items/{id}/fix-bundle` | | **What it contains** | Root cause · a typed fix (before/after) · anonymized failing examples · a proof-eval gate command · acceptance criteria | | **Needs** | A remediation. It is richest when the remediation is linked to a failure cluster, a grounded fix artifact, and a [proof eval gate](/guides/eval-gates) | | **Scope** | Agent-scoped — read-only. Building a bundle never calls an LLM, touches the network, or reads your repo | ## Review and copy a fix bundle Nothing is copied behind your back: you open the bundle, read the exact Markdown that will be copied, and then put it on your clipboard yourself. ### Open a remediation Go to **Fix → Remediations** and click the remediation you want to ship — its full detail page opens at `/remediations/{id}`. A remediation is a typed, evidence-grounded fix proposal — see [Remediations](/guides/remediations) for how they're created. ### Open the bundle In the remediation's **Fix** stage, click the highlighted **View fix bundle** card. Neens assembles the pack server-side (you'll briefly see **Assembling fix bundle…**) and opens a **Fix bundle** dialog showing the raw, paste-ready Markdown — exactly what will land on your clipboard, so you can review it before it leaves Neens. The footer shows the document's line count and size. ### Copy it Read through the bundle, then click **Copy to clipboard**. The button confirms with **Copied to clipboard**. If your browser blocks clipboard access, Neens says **Couldn't copy automatically — select the text and copy it manually** — select the Markdown in the dialog and copy it with your keyboard instead. ### Paste it into your coding agent Paste the whole document into your coding agent (Claude Code / Cursor / Codex) and let it implement the change in your repository. The bundle is written *for* the agent: it leads with the problem and root cause, then the proposed fix, then the failing examples, then how to verify. ### Gate the pull request Run the included `neens eval run` command in CI so the pull request only merges if the fix actually holds (see [Gate the pull request in CI](#gate-the-pull-request-in-ci)). Then move the remediation to **Applied** in Neens to record that the fix shipped. ## What's in the bundle The bundle is a Markdown document with five sections: 1. **Problem & root cause** — the failure cluster's label and description, the root-cause hypothesis and where to look, and how many failing sessions it was observed in. 2. **Proposed fix** — the remediation's typed artifact: the **target** (what to change, e.g. a system prompt or a tool schema), the change instruction and rationale, and the concrete **before**/**after** (or a unified diff). 3. **Failing examples (anonymized)** — two to three real production traces that exhibit the failure, with input/output/error. Every free-text field is redacted before it leaves your tenant (see the callout below). 4. **How to verify (proof evals)** — a ready-to-run `neens eval run` command plus a `neens-gate.json` gate-as-code policy, so CI can replay the failure against your fixed agent and block the PR on a regression. 5. **Acceptance criteria** — a short "done" checklist grounded in the actual gate, judge, dataset, and failing examples. ### The bundle matches the fix's locus A bundle is only paste-into-your-agent-shaped when the fix actually *is* an agent change. For a remediation whose [failure locus](/guides/remediations#where-the-failure-lives-the-failure-locus) isn't your agent, the bundle is rendered honestly so you don't hand a coding agent a prompt edit for a problem a prompt can't fix: - **Infra advisory / escalate to owner** — the bundle leads with *"This is a downstream/upstream service failure, not an agent defect,"* names the **owning component**, states whether the agent already handles the error (retry + backoff), and gives the recommended resilience change. Its "how to verify" is a **resilience / fault-injection check**, not a judge eval. - **Working as intended** — the bundle says *"No action recommended — a control fired correctly,"* with the evidence (the guardrail and the rule it enforced). There's no diff to apply. - **Needs grounding** — the bundle says *"Not yet actionable,"* gives the reason, and suggests the next step (re-cluster or gather more evidence) rather than a code change. For an actionable prompt or tool fix, the bundle keeps the full paste-ready format described above. The failing examples are run through the Neens redaction pass before they're included — PII and secrets (emails, keys, tokens, card and SSN-shaped values, and more) are replaced with `[REDACTED:…]` markers. The bundle is designed to be pasted into an external tool, so it redacts conservatively. If your agent has **egress sanitization** enabled (`mask` or `tokenize`), those same excerpts and the root-cause text additionally pass through your agent's egress redaction policy — producing `[PII:email]` or stable `[PII:email:…]` tokens — before the bundle leaves Neens by *any* channel (copy, webhook, or CLI handoff). The proposed code fix itself is left intact so it applies cleanly. See [What egress sanitization covers](/administration/pii-redaction#what-egress-sanitization-covers). If the policy can't be resolved, Neens refuses to emit the bundle rather than send raw content. ## Gate the pull request in CI Section 4 of the bundle contains a pre-generated command for the `neens eval` CI runner. When the remediation is linked to a [proof-eval gate](/guides/eval-gates) — a failure mode bound to a dataset and a judge — the command already references the real dataset, so it's ready to run: ```bash neens eval run --create --dataset \ --version-label "$GIT_SHA" \ --min-pass-rate 0.90 \ --gate-policy neens-gate.json \ -- ``` Replace `` with how CI invokes your agent, and save the `neens-gate.json` policy from the bundle alongside it. The process exit code **is** the gate verdict — `0` means the fix held — so your pipeline gates on it directly. The default `--min-pass-rate` is `0.90`, matching how Neens scores production. See [Pre-prod evaluations](/guides/preprod-evals) for the full CI runner. If the remediation isn't yet bound to a proof dataset, the command carries a `` placeholder instead. Run **Generate eval** on the linked failure mode (see [Eval gates](/guides/eval-gates)) to materialize the dataset, then open the bundle again — the command will resolve to the real id. ## Zero new trust surface A fix bundle is the Neens diagnosis rendered into context for *your* tools. Building it: - **never calls an LLM** — it's a deterministic assembly of data Neens already computed from your traces; - **never touches the network** and **never reads your repository**; - **carries no credentials** — the coding agent runs in your environment, on your repo, under your review. You paste it into the coding agent you already trust, so applying a fix adds no new integration, access grant, or outbound connection. ## How it works The bundle is assembled read-only from data the remediation already carries: - The **root cause** comes from the linked failure cluster's stored analysis. - The **fix** is the remediation's typed artifact (the same before/after you review in the panel). - The **failing examples** come from the evidence snapshot captured when the remediation was generated (Neens falls back to a fresh read of the cluster's member sessions if that snapshot is empty), each field redacted. - The **proof gate** resolves the flywheel eval gate bound to the remediation's failure mode — its dataset and judge — to fill in the `neens eval run` command and acceptance criteria. - The **repo link**, when present, comes from your agent's optional [agent context](/guides/remediations) (Settings the remediation reads for grounding). Because it's pure assembly, the same remediation always produces the same bundle. Prefer to automate this? A coding agent can pull the same bundle without copy-paste using the [MCP fix loop](/guides/mcp#fix-loop-over-mcp) — `get_fix_bundle` returns this pack, and `run_verification` proves the fix against your preview deploy before you merge. ## Reference Response shape `GET /remediations/items/{id}/fix-bundle` returns a JSON object with: | Field | Meaning | | --- | --- | | `markdown` | The paste-ready document — what the **Fix bundle** dialog shows and **Copy to clipboard** puts on your clipboard | | `bundle` | The structured pack (remediation, cluster, exemplars, proof, acceptance criteria) | | `evalCommand` | The pre-generated `neens eval run` CI one-liner | | `gatePolicy` | The gate-as-code policy to save as `neens-gate.json` | | `proof` | The resolved dataset / judge / gate ids and names | | `bundleVersion`, `generatedAt`, `format` | Bundle metadata | ## Troubleshooting - **"Couldn't build the fix bundle. Try again in a moment."** — the bundle request failed. It's usually transient: close the dialog and click **View fix bundle** again. If it keeps failing, the remediation may have no usable content to assemble (e.g. no fix artifact and no linked cluster) — generate or complete the remediation first ([Remediations](/guides/remediations)). - **The dialog says "Nothing to show."** — the bundle was built but came back empty. Complete the remediation (a fix artifact or a linked failure cluster) and open it again. - **"Couldn't copy automatically — select the text and copy it manually."** — your browser denied clipboard access (common in embedded or non-HTTPS contexts). Select the Markdown in the dialog and copy it with your keyboard; the text is the complete bundle. - **There is no View fix bundle button** — fix bundles are turned off for this deployment; ask whoever runs your Neens instance. - **The command shows ``** — the remediation isn't bound to a proof dataset yet. Run **Generate eval** on its failure mode ([Eval gates](/guides/eval-gates)) and open the bundle again. ================================================================================ # Eval-verified PR Source: /docs/guides/eval-verified-fix/ ================================================================================ # The eval-verified PR The **fix engine** closes the Neens failure→fix loop into a single, trustworthy action: from a confirmed [remediation](/guides/remediations), Neens proposes a code/prompt change, applies it on a branch, **verifies it with pre-prod evals**, and opens a pull request — but only when the fix actually holds. It never merges: a human always reviews and merges the PR. This is the difference between "here's a suggested fix" and "here's a fix we *proved* works." Every PR the fix engine opens carries its proof: the pass^k eval results, the held regression set, and the human-aligned judges that verified it. The fix engine builds on three things you already have: a **remediation** with a typed fix (see [Remediations](/guides/remediations)), a **proof-eval gate** derived from the failure (see [Eval gates](/guides/eval-gates)), and a **pre-prod eval** runner (see [Pre-prod evaluations](/guides/preprod-evals)). If you can already export a [fix bundle](/guides/fix-bundles), you have everything a fix run needs. **This is how you bind a proof.** A remediation can't move to **Applied** or **Verified** without a bound proof — a completed verification run or a proof-eval gate (see [Applying a fix requires a bound proof](/guides/remediations#applying-a-fix-requires-a-bound-proof)). An eval-verified PR run is the straight path to that proof: a green run stamps its verification back onto the remediation, so the fix is *provably* apply-able rather than merely marked. This path is for **agent-fixable** (actionable) remediations only — a service **advisory** or a **working-as-intended** item has no agent change to verify and settles with an acknowledgement note instead (see [Remediation types](/guides/remediations#remediation-types)). ## At a glance | | | | --- | --- | | **Where** | The **Auto-fix (eval-verified PR)** action on a remediation's detail panel (**Fix → Remediations**) | | **Key API** | `POST /fix-engine/runs` · `GET /fix-engine/runs/{id}` · `GET/POST /fix-engine/vcs-installations` | | **MCP tools** | `start_fix_run` (launch) · `get_fix_run` (poll) | | **What it needs** | A remediation with a resolvable proof-gate dataset, and a VCS installation (GitHub App or local git) | | **What it opens** | A pull request whose body is the proof — pass^k results, root cause, eval report, verifying judges | | **Merges?** | **Never automatically.** A human always merges | ## The loop 1. **Propose** — a driver turns the remediation's fix bundle into a patch. The default `artifact` driver replays the remediation's own typed before/after fix deterministically (no LLM, no repo access). A `webhook` driver hands the bundle to *your* coding agent; a `cli` driver runs a headless coding-agent CLI in a clone. 2. **Apply** — the patch is applied on a new branch through a **VCS driver** (a real GitHub App, or a local git remote for self-hosting/demos). The PR is **not** opened yet. 3. **Verify (pass^k)** — Neens runs the candidate through **k independent pre-prod eval runs** against the failure's own failure-derived evals **and** your accumulated regression set. 4. **Gate** — the fix passes only when *every* run is green (zero regressions) **and** a human-aligned judge that the fix did **not** optimize against verified it. 5. **Open the PR** — only a passing candidate opens a pull request; Neens stamps the PR URL and the verification run back onto the remediation. A failing candidate retries up to a bounded budget, then lands a **draft** with the failure report instead of a clean PR. At no point does Neens merge, hold your repository credentials beyond the VCS installation you configured, or modify a running system. And to run this loop unattended — Neens starting the verified run itself, or accepting an eligible remediation and starting it with no human in between — set a per-agent [autonomy level](/guides/autonomy). The human merge gate stays exactly where it is at every level. ## Start a fix run ### Pick a remediation Open a remediation under **Fix → Remediations** that has a proof-eval gate (its **Eval gate ✓** state — see [Eval gates](/guides/eval-gates)). Without a gate there is nothing to verify a fix against, and the run is rejected. Remediations written by the Neens [prompt optimizer](/guides/prompt-optimization) appear here like any other and take exactly this path — there is no separate, faster gate for a machine-written prompt. ### Choose where the PR goes Point the run at a **VCS installation** (below) or pass a repository URL directly. For a real PR use a GitHub App installation; for a self-hosted or demo loop use a local git remote. ### Launch it Click **Auto-fix (eval-verified PR)**, or call the API / MCP tool directly. ```bash curl -X POST https://your-neens/api/fix-engine/runs \ -H "Authorization: Bearer nk_live_…" \ -H "Content-Type: application/json" \ -d '{ "remediationId": "rem_…", "vcsInstallationId": "vcs_…", "versionLabel": "fix/order-status-grounding" }' ``` A `nk_live_…` agent key can launch a run — the same credential your CI already uses. From a coding agent connected to the Neens [MCP server](/guides/mcp): - `start_fix_run` — launch the Neens-orchestrated fix loop for a remediation (needs `remediation_id` with a resolvable proof-gate dataset). - `get_fix_run` — poll the run's status, attempt, pass^k eval report, verifying judges, and PR URL. Under the async (Celery) queue the run returns `{ "runId": …, "status": "queued" }` immediately — poll `GET /fix-engine/runs/{id}` until the status is terminal: - `pr_opened` — a green PR was opened; `prUrl` and `commitSha` are set. - `drafted` / `failed` — the attempt budget was exhausted; read `failureReport`. - `cancelled` — cancelled via `POST /fix-engine/runs/{id}/cancel`. You can tune a run per call: `passK` (how many verification runs must be green), `maxAttempts` (retry budget), `driver` (`artifact` / `webhook` / `cli`), and `baseBranch`. Omit them to use the agent/server defaults. ## Configure a VCS installation The fix engine opens PRs through a **VCS installation** you register once under **Fix → Remediations** (or via `POST /fix-engine/vcs-installations`, admin-only). The credential is encrypted at rest and never returned — reads show only whether one is present. A **GitHub App** is the least-privilege way to let Neens open PRs: it acts as an installation, not as a person, and you scope it to exactly the repositories you choose. | Field | Value | | --- | --- | | **provider** | `github_app` | | **config.app_id** | your GitHub App's App ID | | **config.installation_id** | the installation ID for the org/repo you installed it on | | **credential** | the App's PEM private key (stored encrypted) | | **config.api_base** | optional — set for GitHub Enterprise | | **repoUrl** | `https://github.com/{owner}/{repo}` | Grant it only **Contents: read & write** and **Pull requests: read & write** on the repos you want Neens to open PRs against. Use **Test** to mint an installation token and confirm reachability. Public `github.com` is reached directly; a self-hosted Enterprise host on a private network must be allowlisted (see [Reachability](#reachability--self-hosting)). The **local git** driver needs no GitHub App, OAuth, or network identity — it clones a bare git remote, pushes a branch, and commits a PR-body record. It is the zero-credential path for self-hosting and for the seeded demo. | Field | Value | | --- | --- | | **provider** | `local_git` | | **config.remote_path** *(or* **remote_url***)* | a bare git repo (a local path, `file://`, or any git URL) | | **config.base_branch** | default base branch (e.g. `main`) | Because there is no PR server, the "PR" is the pushed branch plus a committed `.neens/pr/.md` body record; the run's PR URL is a `#` reference. The Neens demo seeds exactly this — a scratch bare repo and a green fix run — so the whole loop is visible with no external credentials. ## pass^k — why one green run isn't enough A single eval run can pass by luck: a non-deterministic agent, a flaky tool, or a judge that happened to grade generously. **pass^k** requires the candidate fix to pass **k independent verification runs** — all green, zero regressions — before a PR is opened. `k ≥ 3` is what defeats a *flaky green*: a fix that only passes sometimes will fail at least one of the k runs and never opens a PR. The default is **3**. Raise it for higher-stakes changes; the run reports `greens/k` and refuses to open a PR until every run is green. ## Anti-gaming — who is allowed to verify A fix that is graded by the *same* signal it was tuned against isn't verified — it's overfit. Neens enforces two non-negotiable rules when it picks the judges that VERIFY a fix: 1. **Separation from the authoring signal.** Any judge the fix optimized against is **excluded** from verification. The verifier must be a *different* judge than the one the change was aimed at. 2. **Human anchoring.** A verifying judge must be **human-aligned** — its agreement with expert labels (Cohen's κ) must clear a floor (default **0.4**). A judge with no measurement, or one below the floor, cannot anchor a verification. See [Annotations & review](/guides/annotations-and-review) for how judge↔expert alignment is measured. If pass^k is green but *no* human-anchored judge is available to verify, Neens lands a **draft** PR (lower confidence) rather than a clean one — an unanchored verification is never presented as trusted. This is also why a fix that the Neens [prompt optimizer](/guides/prompt-optimization) wrote needs a **second** human-aligned judge: the optimizer records the judges it optimized the prompt against, and those judges are excluded from verifying it. If they are the only ones available, the run lands a draft by design. **Judge-alignment drift is a P0 alert.** A judge that verified fixes yesterday can drift out of agreement with your human reviewers. Neens watches each verifying judge's κ and raises a **critical** insight when it falls below the floor or drops sharply from its prior measurement — because a fix "verified" by a judge that no longer agrees with humans is not verified at all. Keep labelling gold items so alignment stays measured. ## The living regression set Every fix run verifies against **two** things: the failure's own failure-derived evals *and* your **accumulated regression set** — a single, growing golden dataset of everything the agent must keep getting right. A candidate that fixes the target failure but breaks a past correction fails the gate. The regression set **compounds**: when a human corrects a judge during [review](/guides/annotations-and-review), that corrected example is captured into the set. So every human correction makes the gate stronger, and each new fix must hold the entire accumulated history — not just its own narrow test. ## Two ways to verify: preview URL vs. CI gate The verification step reuses the Neens [pre-prod evaluation](/guides/preprod-evals) engine, so you can run it in whichever lane fits your deploy: - **Preview URL — Neens calls your agent.** Point the run at a preview deployment's endpoint URL (or an existing agent connection) at a version label. Neens calls it per golden prompt, captures the traces, and scores them — no test harness on your side. This is the one-command "verify this candidate" lane. - **Customer-CI gate.** Run the [`neens eval run`](/guides/eval-gates) command the fix bundle pre-generates in *your* CI, with your gate policy. The pull request only merges if the gate passes. Neens provides the dataset, the judges, and the gate-as-code; your pipeline runs it against your build. Both lanes verify against the same failure-derived evals plus the regression set, and both are inspectable in Neens — every verification eval is a real run you can open and read. ## Read the eval-verified PR The PR the fix engine opens is written to be reviewed. Its body leads with the **proof**, because that's what a human merges on: - A one-line **proof headline** — e.g. *"Fixes 12 of 14 new evals, regresses 0 of 210."* - **pass^k** — how many verification runs were green out of k. - **Root cause** — the diagnosed reason the agent was failing, and where to look. - **The change** — the fix summary and the target it touches. - **Verified by** — the human-κ-anchored judges that verified it (with their κ), and an explicit note that they are *distinct* from any signal the fix optimized against. - **Regression set** — how many accumulated regression checks the fix held. On your side you review the diff and the proof, then **merge** (Neens never does) and move the remediation to **Applied** in Neens to record that the fix shipped. ## Reachability & self-hosting The fix engine's outbound calls — a self-hosted GitHub Enterprise API, a preview-deploy endpoint, or a webhook coding-agent — are SSRF-gated: a host that resolves to a private, loopback, or link-local address is refused **unless it is allowlisted**. Public hosts (like `github.com`) are never gated. To reach an internal or local target, ask your operator to allowlist its host or CIDR. ## Troubleshooting - **The run is rejected with "no proof-gate dataset."** The remediation has no failure-derived eval gate to verify against. Generate one for its failure mode first — see [Eval gates](/guides/eval-gates). - **pass^k never turns green.** The fix isn't reliably fixing the failure. Read the per-run results in the fix run detail; a candidate that's green sometimes but not always is exactly what pass^k is meant to catch. - **The run lands a draft, not a clean PR.** pass^k passed but no human-anchored judge was available to verify. Label more gold items so a judge's alignment is measured (see [Annotations & review](/guides/annotations-and-review)). - **A verification call is blocked.** The target host resolves private/loopback and isn't allowlisted — ask your operator to allowlist its host. ================================================================================ # Autonomy levels Source: /docs/guides/autonomy/ ================================================================================ # Autonomy levels The [eval-verified PR](/guides/eval-verified-fix) proves a fix works before it ships. **Autonomy levels** decide how much of the path to that PR runs *without you clicking anything*: at the highest level, a failure can go from ingested trace to an open, proof-carrying pull request with no human action in between. **A human still merges every pull request, at every level.** Autonomy removes human *gates* — who accepts a remediation, who starts the verified run — and never the human *merge*. Neens does not merge, approve, or deploy anything, and nothing on this page changes that. ## At a glance | | | | --- | --- | | **Where** | **Settings → Autonomy** (admin-only) | | **Key API** | `GET /autonomy/status` · `PATCH /autonomy/settings` · `POST /autonomy/suspend` · `POST /autonomy/resume` · `GET /autonomy/decisions` · `PUT /spend-budgets/fix_engine` | | **Scope** | Per **agent**. Two agents in the same company can sit at different levels | | **Default level** | `propose_only` — today's behaviour: a human accepts, a human starts the run | | **What it needs** | The [fix engine](/guides/eval-verified-fix) working end to end: remediations with proof-eval gates, a VCS installation, and at least one human-aligned verifying judge | | **Merges?** | **Never.** At every level a human reviews and merges the PR | | **Cadence** | A sweep runs every **15** minutes | ## The four levels | Level | Who accepts the remediation | Who starts the eval-verified run | Who merges | | --- | --- | --- | --- | | `off` | *(nothing autonomous happens)* | — | — | | `propose_only` **(default)** | A human | A human | A human | | `auto_verify` | **A human** | Neens | A human | | `auto_pr` | The auto-accept policy | Neens | A human | In the **Autonomy level** card these appear as **Off**, **Propose only (default)**, **Auto-verify** and **Auto-PR**. - **Off** — no autonomous activity at all. Nothing is accepted and no fix run is started for you. - **Propose only (default)** — exactly today's behaviour. The clustering run pre-populates [remediations](/guides/remediations); a human accepts one, and a human starts the eval-verified fix run. - **Auto-verify** — **a human still accepts the remediation.** That is the whole difference from `auto_pr`, and it is easy to oversell: at this level Neens only removes the *second* click. Once a person has moved a remediation to `accepted`, the next sweep starts the verified fix run for it. A PR opens only if pass^k, the accumulated regression set and the κ-anchored verifying judges all hold — unchanged fix-engine behaviour. - **Auto-PR** — the [auto-accept policy](#the-auto-accept-policy) accepts an eligible remediation without a human, then the same verified run happens. No human action between ingest and PR. Raising the level never weakens the proof. `auto_verify` and `auto_pr` start the *same* fix run a human would have started, through the same validation: the remediation must be in scope, it must have a resolvable proof-gate dataset, and the same pass^k / regression-set / anti-gaming rules decide whether a PR opens at all. There is no faster path for a machine-started run. ## Turn it on and pick a level ### Get the prerequisites in place At `auto_verify` and `auto_pr` an agent needs everything a manual [eval-verified PR](/guides/eval-verified-fix) needs, plus one thing it can live without manually: a **human-aligned verifying judge**. If no judge in the agent has a measured Cohen's κ at or above the effective floor (see [The κ floor](#the-verifying-judge-floor)), autonomy suspends itself rather than running unanchored — label gold items under [Annotations & review](/guides/annotations-and-review) so alignment is measured. ### Choose a level Open **Settings → Autonomy**, pick a radio card in the **Autonomy level** card, and press **Save**. Every card restates what stays human, including the merge. Start at **Auto-verify**. It is the level where you keep the judgement call ("is this the right fix?") and hand over only the mechanical one ("go prove it"). Move to **Auto-PR** once the decision log shows you agree with what the policy would have admitted. ### Set the auto-accept policy The **Auto-accept policy** card is visible at every level but only editable at **Auto-PR**, so you can see what *would* apply before you switch. Its defaults are deliberately strict — see [The auto-accept policy](#the-auto-accept-policy). ### Set a spend budget Fill in **Limit (USD per period)** and **Period** in the **Autonomous spend budget** card and press **Save budget**. A human clicking "start fix run" is, incidentally, a rate limiter; once that human is gone the budget is what bounds the bill. See [The spend budget](#the-spend-budget). ### Read the decision log The **Autonomous decision log** at the bottom of the tab records every admission, refusal, suspension, resume and cancellation with its reason and cost. It is the surface you check after the first day, not the PR list. ### API-first Every control has an API. Reads (`GET`) accept a `nk_live_…` agent key; **writes require an admin credential** (an operator/company-admin key, or a signed-in user with the admin role). Autonomy has its **own** admin-only authority, deliberately separate from the one that registers a VCS installation: holding a repo credential is not the same privilege as authorising Neens to spend your LLM budget and open pull requests unattended. ```bash curl -s https://your-neens/api/autonomy/status \ -H "Authorization: Bearer nk_live_…" ``` Returns the level, the suspension state and reason, the resolved policy, `effectiveMinKappa`, the `platformCeilings`, live `health` signals, the current-period `budget`, `inFlightRuns`, `runsToday`, `lastSweepAt` and the five most recent decisions — everything the tab renders, in one call. ```bash curl -X PATCH https://your-neens/api/autonomy/settings \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{ "level": "auto_pr", "policy": { "minClusterSize": 8, "minConfidence": 0.75, "minVerifyKappa": 0.55, "maxRunsPerDay": 3, "allowedRemediationTypes": ["prompt_change"] } }' ``` At least one of `level` / `policy` is required (`422` otherwise), an unknown level is rejected with the valid set, and unknown body fields are refused outright. A `policy` patch is **merged** onto the current policy, so omitting a field leaves it alone. ```bash curl -X PUT https://your-neens/api/spend-budgets/fix_engine \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{"limitUsd": 40, "period": "month", "enabled": true}' # what the current period has actually spent, plus the ledger behind it curl -s https://your-neens/api/spend-budgets/fix_engine/usage \ -H "Authorization: Bearer nk_live_…" ``` `"limitUsd": null` clears the limit (unlimited). `period` is `day`, `week` (ISO) or `month`. ```bash # the human kill switch — also cancels every in-flight autonomous run curl -X POST https://your-neens/api/autonomy/suspend \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{"detail": "pausing while we rotate the staging repo"}' curl -X POST https://your-neens/api/autonomy/resume \ -H "Authorization: Bearer " ``` `resume` returns `409` when the agent is not suspended — resuming what is already running would write a misleading entry into the decision log. ## The auto-accept policy At **Auto-PR** — and only there — this policy decides whether a `proposed` remediation may become `accepted` with no human. **Every check must pass.** A missing or unknown signal *fails* its check; nothing passes quietly. Three fields in this table are **not** Auto-PR-only. `minVerifyKappa` sets the κ floor that the health gate uses to suspend the agent, and `maxConcurrentRuns` / `maxRunsPerDay` cap how many unattended runs may start — all three are read at **Auto-verify** as well, before any admission decision is reached. They stay editable at that level in the UI. The remaining six fields are admission checks and apply at Auto-PR only. | Field | UI label | Default | What it requires | | --- | --- | --- | --- | | `minClusterSize` | **Min cluster size** | `5` | The remediation's cluster has at least this many sessions of evidence | | `minVerifyKappa` | **Min verifying κ (0–1)** | `0.4` | Your κ floor; the effective floor is the **higher** of this and the platform floor | | `minConfidence` | **Min remediation confidence (0–1)** | `0.6` | The remediation's own confidence score | | `maxConcurrentRuns` | **Max concurrent autonomous runs** | `1` | Autonomous runs in flight for this agent | | `maxRunsPerDay` | **Max autonomous runs per day** | `5` | Autonomous runs started today (UTC) | | `requireContainedBlastRadius` | **Require a contained blast radius** | `true` | Off, a `broad` change may also be accepted. `protected` and `unknown` never pass either way | | `protectedGlobs` | **Protected paths (one glob per line)** | *(see below)* | Any touched path matching one of these makes the change `protected` | | `blockedCategories` | **Blocked failure categories (one per line)** | `auth`, `authentication`, `authorization`, `authn`, `authz`, `login`, `logout`, `password`, `credential`, `oauth`, `saml`, `sso`, `permission`, `rbac`, `encryption`, `pii`, `security`, `payments`, `billing`, `compliance`, `privacy` | A failure in one of these subjects is `protected` | | `allowedRemediationTypes` | **Allowed remediation types (one per line; blank = all)** | *(blank — all types)* | Restrict auto-accept to specific [remediation types](/guides/remediations#remediation-types) | The six checks are named in the decision record: `cluster_size`, `confidence`, `verify_kappa`, `blast_radius`, `remediation_type`, `status`. Every one is evaluated even after an earlier one has failed, so the log shows the whole card rather than making you fix thresholds one at a time. `status` requires the remediation to still be `proposed` — anything else was already acted on. ### Worked example — one admitted, one refused Take an agent at **Auto-PR** with the default policy, an effective κ floor of `0.4`, and a best verifying judge at κ `0.61`. **Admitted.** A remediation for a grounding failure in the order-status agent: cluster of 41 sessions, confidence 0.82, type `prompt_change`, status `proposed`, fix artifact touching one file, `app/agents/order_status.py`. | Check | Result | Detail recorded | | --- | --- | --- | | `cluster_size` | ✅ | Cluster has 41 sessions; policy requires at least 5. | | `confidence` | ✅ | Confidence 0.82; policy requires at least 0.6. | | `verify_kappa` | ✅ | Best verifying judge κ is 0.61; the effective floor is 0.4. | | `blast_radius` | ✅ | Blast radius is 'contained'; policy allows contained. 1 file would change, none of them protected. | | `remediation_type` | ✅ | Policy allows every remediation type. | | `status` | ✅ | Remediation is `proposed`. | The remediation transitions to `accepted` (stamped as a scheduler action), a fix run is created with origin `autonomous`, and an `admitted` decision is written carrying the full check array, the exact policy that admitted it, the blast radius, κ and the cost estimate. **Refused.** A remediation for a token-refresh failure: cluster of 63 sessions, confidence 0.91 — *stronger* evidence than the first one — whose artifact touches `services/auth/token_refresh.py` and `services/api/session.py`. | Check | Result | Detail recorded | | --- | --- | --- | | `cluster_size` | ✅ | Cluster has 63 sessions; policy requires at least 5. | | `confidence` | ✅ | Confidence 0.91; policy requires at least 0.6. | | `verify_kappa` | ✅ | Best verifying judge κ is 0.61; the effective floor is 0.4. | | `blast_radius` | ❌ | Blast radius is 'protected'; policy allows contained. Protected path `services/auth/token_refresh.py` matches `**/auth/**`. | | `remediation_type` | ✅ | Policy allows every remediation type. | | `status` | ✅ | Remediation is `proposed`. | A `refused` decision is recorded with reason `blast_radius`, and the remediation stays `proposed`, waiting for a human exactly as it does at `propose_only`. **No amount of κ, confidence or cluster size buys a way past this** — see below. A refusal is re-derived on every sweep tick, so an identical `(remediation, reason)` refusal is written to the log at most once per **24 hours**. A 15-minute cadence would otherwise record the same refusal 96 times a day and bury the entries that matter. ## Blast radius — what a fix is allowed to touch Before any admission, Neens classifies how much of your system the remediation would move. | Radius | When | Can auto-accept? | | --- | --- | --- | | `contained` | Up to **20** distinct touched files, none protected | Yes | | `broad` | More than **20** distinct touched files | Only with **Require a contained blast radius** unchecked | | `protected` | The failure's category/label is a blocked category, **or** a touched path matches a protected glob | **Never** | | `unknown` | No touched paths could be derived for the remediation | **Never** | The category check runs **before** the path check, because what a fix is *about* is known earlier and more reliably than its file list. It matches the failure's category **and every label on the remediation**, on word boundaries and ignoring plurals — so a remediation labelled `payment_flow` or categorised `auth_failure` is `protected` even if its artifact happens to name only `docs/`. Word boundaries are why the default list spells out `authentication`, `authorization`, `authn` and `authz` separately: the token `auth` matches `auth_failure` and `authFailure`, but **not** `authorization` — that is one word, not two, and a substring rule would also catch `author`. If you edit this list, add the spellings your cluster labels actually use rather than assuming a stem covers them. ### The default protected paths These are the `protectedGlobs` defaults, verbatim. `**` spans directories, and matching is case-insensitive; a leading `**/` also matches a top-level file, so `**/*.tf` matches `main.tf` as well as `infra/main.tf`. ```text **/auth/** **/auth*.py **/access.py **/access_control* **/access_control*/** **/login* **/login*/** **/logout* **/password* **/password*/** **/credential* **/credential*/** **/oauth* **/oauth*/** **/saml* **/sso* **/*jwt* **/rbac* **/rbac*/** **/permission* **/permission*/** **/payment*/** **/billing/** **/secrets/** **/security/** **/migrations/** infra/** **/*.tf **/Dockerfile* **/.github/workflows/** **/iam/** **/crypto/** ``` The list is deliberately over-broad. A false `protected` costs one human click; a false `contained` costs an unreviewed change to your auth path. Edit it per agent — but understand that an auth, payments, billing, crypto, migration or CI-workflow change is refused **regardless of κ**, because κ measures how well a judge agrees with your reviewers about *quality*, not whether an unreviewed credential change is acceptable. Those are different questions, and no score answers the second one. ### Why `unknown` fails instead of passing An empty file list is the absence of evidence, never evidence of a small change. A remediation whose artifact has not been generated yet, or whose diff could not be parsed, is exactly the case where guessing "contained" opens an unreviewed PR. So `unknown` fails the `blast_radius` check and the refusal is logged with that reason — the log says *"we could not tell what this touches"* rather than going quiet. ## The verifying-judge floor A verifying judge must be human-anchored. The floor in force is the **higher** of your policy's **Min verifying κ (0–1)** and the platform floor (default **0.4**) — an agent policy can raise the bar, never lower it. The **Auto-accept policy** card shows the resulting **Effective κ floor in force**. The κ that counts is the κ of a judge that will actually be *allowed* to verify the fix: judges the fix was written against are excluded by the same anti-gaming rule the [eval-verified PR](/guides/eval-verified-fix#anti-gaming--who-is-allowed-to-verify) applies. If no eligible judge clears the floor, the answer is `None` — not `0` — and autonomy suspends. ## Fail safe, not fast Before it considers a single candidate, each sweep asks whether the agent is still fit to act on its own. If it is not, **autonomy suspends and says so** — it never quietly drops to a weaker level, which would look like the product working while the guarantee it advertises is gone. The level you set is preserved through a suspension, so resuming restores exactly the autonomy you configured. | Reason | What triggers it | Clears | | --- | --- | --- | | `manual` | A human pressed the kill switch (`POST /autonomy/suspend`) | A human resumes | | `kappa_drift` | An open judge-alignment drift alert: a verifying judge no longer agrees with human labels, so nothing it "verifies" can be trusted | A human resumes | | `kappa_below_floor` | No eligible verifying judge has a κ at or above the effective floor (including "no κ measured at all") | A human resumes | | `regression_failure` | An autonomous run failed the accumulated regression set inside the look-back window (**24** hours) | A human resumes | | `budget_exhausted` | The period spend limit is reached, **or** the next run's cost cannot be priced | **Clears itself** when the budget period rolls over — or a human resumes sooner | When several apply, the reported reason follows a fixed precedence: `kappa_drift` → `kappa_below_floor` → `regression_failure` → `budget_exhausted`. That order runs from "the measuring instrument is broken" to "the instrument says stop": a drifting judge invalidates the κ figure the next check would read, and an invalid κ invalidates the "verified" claim on the regression runs after it. Reporting the shallowest symptom first would send you to fix the wrong thing. **What you get told.** A suspension raised by the fail-safe (any reason except `manual`) creates a **critical** insight — *"Autonomy suspended (…)"* — which reaches the in-app feed and any Slack or webhook sink you have configured under [Insights & notifications](/guides/insights). It also writes a `suspended` row into the decision log and an `autonomy.suspend` entry into the [audit trail](/administration/audit-log). A suspension you perform yourself is recorded in the decision log and the audit trail, but raises no insight — you already know. **How to resume.** Press **Resume autonomy** in the red banner at the top of the **Autonomy** tab, or `POST /autonomy/resume`. Fix the underlying signal first: resuming with a drifted judge simply suspends again on the next sweep. The one exception is `budget_exhausted`, which the sweep lifts by itself at the period roll-over and records as its own decision and audit event — the money is back and nothing about the agent's health changed. ## When one remediation keeps failing A suspension stops the whole agent. This is the narrower case: **one remediation the fix engine can never apply**. After **3** consecutive failed autonomous runs, the sweep stops offering that remediation and moves on to the next candidate. Nothing else changes — the agent stays healthy, the loop keeps running, and every other remediation is still eligible. It is a **stop**, not a slower retry, and that is deliberate. A remediation whose fix driver produces no patch, whose artifact names no file, or whose target no longer exists will fail exactly the same way on the hundredth attempt as on the third. Retrying it more slowly would keep spending the daily run cap on a known-bad candidate and keep it ahead of the fixable remediations behind it — candidates are selected highest-priority-first, so one unfixable `priority: 99` remediation is candidate #1 forever. A fix that cannot be applied is a signal a person should see, so the loop hands it back to you instead of grinding on it. **Held back is not suspended.** Autonomy is still on, the sweep still runs, and no one has to press **Resume**. A held-back remediation is offered again the moment a human touches it — there is nothing to clear and no flag to reset. ### What counts toward the streak The count walks that remediation's fix runs newest-first and stops at the first thing that is not a failure. Only **autonomous** runs are read at all: a run you started by hand neither adds to the streak nor clears it. | Run status | Effect on the streak | | --- | --- | | `failed` | **+1** | | `cancelled` | **Neutral** — skipped, and the walk continues past it. The loop halted that run (a suspension, or you lowering the level); that is not the remediation's fault | | `pr_opened`, `drafted`, `queued`, `applying`, `verifying` | **Stops the walk.** The streak ends there — a run that got as far as a draft or a PR is not a consecutive failure | | Any run that started **before** the remediation was last edited | **Stops the walk.** History from before a human last touched it is not held against it | That last row is the reset. Editing, re-accepting or re-prioritising a remediation updates its timestamp, and only autonomous runs created at or after that moment are counted. The loop cannot clear its own streak this way: at **Auto-PR** the auto-accept write happens *before* the run it authorises, so it never post-dates the run it would need to hide. If the streak cannot be read at all — a database error mid-sweep — the candidate is skipped for that tick and **no** decision row is written. "We could not check" must never be recorded as "this failed three times". ### How to clear it Touch the remediation. Any change to it counts; you do not have to fix it first, though a remediation put back unchanged will simply fail three more times and be held back again. ### Open it Go to **Remediations** and select the remediation named on the **Held back** card. ### Change something about it Advance its **Lifecycle** (for example `proposed → accepted`), move its work state, or edit its labels. Each of those is a write, and a write is the touch. ### Wait for the next sweep Within **15** minutes (the next sweep) it is a candidate again, with a streak of zero. ```bash # re-accept the remediation — any write to it resets the streak curl -X PATCH https://your-neens/api/remediations/items/rem_abc \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{"status": "accepted"}' ``` `PATCH /remediations/items/{id}` takes at least one of `status`, `workState`, `labels`, `prUrl` or `commitSha` (`422` otherwise) and stamps the remediation's updated timestamp. The next sweep counts only autonomous runs created from that moment on, so the streak starts again at zero. ### How to see it **The Held back card** on **Settings → Autonomy** lists each parked remediation with its title, how many consecutive autonomous runs failed, when the last one failed, and that run's failure text. It appears only while something is actually held back, and it is styled as a note rather than an alarm — nothing is suspended. **The decision trail** records the hold as a `refused` decision with the reason `consecutive_failures`. It is deduplicated for 24 hours per remediation, so a 15-minute sweep writes one row a day, not ninety-six: ```bash # exactly what the loop is holding back, newest first curl -s "https://your-neens/api/autonomy/decisions?reason=consecutive_failures&limit=50" \ -H "Authorization: Bearer nk_live_…" # or scope it to the one remediation curl -s "https://your-neens/api/autonomy/decisions?decision=refused&remediationId=rem_abc" \ -H "Authorization: Bearer nk_live_…" ``` The endpoint filters on `decision`, `remediationId` and `reason`, in any combination. Filtering by `decision=refused` alone is a wider question — it returns every refusal the auto-accept policy has ever made, most of which are ordinary "this one did not meet the bar" decisions rather than holds. **An insight** — `autonomy_remediation_quarantined` — is raised in the [Insights](/guides/insights) feed, deduplicated per remediation, so it reaches whatever Slack or webhook sink you have configured without repeating every sweep. ### Worked example An agent at **Auto-PR** with `maxRunsPerDay: 5`, a spend budget of $40/month, and one remediation `rem_abc` (`priority: 99`) whose artifact the fix driver cannot apply. Each run is estimated at **$2.00** and fails in about two seconds, before the driver produces anything. **Without the quarantine.** Every 15-minute tick re-admits `rem_abc` — the run failed, and a failed run does not block re-admission. Five runs a day, five reservations settled at $2.00 each: **$10/day of spend against a run that never called a model**, and none of the five daily slots ever reaches a remediation that could have been fixed. Four days in, the $40 month budget is full and the agent suspends with `budget_exhausted` — for money nobody spent. **With the defaults.** Runs 1, 2 and 3 fail the same way, and because none of them created a verification run each reservation is **released** rather than billed: the period total is still $0.00. On the fourth sweep the streak is 3, so `rem_abc` is not a candidate. One `refused` decision with `consecutive_failures` is written, one `autonomy_remediation_quarantined` insight is raised, and the sweep admits the next remediation down the priority list. The daily cap of five now belongs to the remediations that can actually be fixed. When someone repairs the artifact and re-accepts `rem_abc`, the next sweep sees no autonomous failures since that edit and offers it again. ## The kill switch, mid-flight A fix run takes minutes of work. Switching autonomy off has to affect work *already in flight*, not just work not yet started — otherwise "I turned it off" still opens a PR ten minutes later. Three actions trip the kill switch: **suspending** the agent, **lowering the level** out of Auto-verify/Auto-PR, and any **automatic suspension** from the table above. Each one: 1. Cancels every in-flight fix run whose origin is `autonomous`, writing status `cancelled` with the reason on the run. A run that already reached a terminal state (`pr_opened`, `drafted`, `failed`, `cancelled`) is left exactly as it is — nothing already produced is rewritten. 2. **Releases** the run's spend reservation, so cancelled work does not hold budget. 3. Records a `cancelled` decision per run, so the log explains why the run stopped. The worker enforces the same thing from its side: an autonomous run re-reads its agent's autonomy state at the top of every attempt **and immediately before it would open the pull request**. An agent whose autonomy was switched off never wakes up to a PR nobody sanctioned. **Runs a human started are never touched.** Switching the unattended loop off is not an instruction to cancel a person's work, so `manual`-origin runs keep going. ## When a run's worker dies A fix run is executed by a worker process, and a process can disappear: an out-of-memory kill, a pod eviction, a lost node. When that happens between the moment the worker picks the run up and the moment it reaches a result, the run is left mid-flight with **nothing alive to finish it**. Its own error handling cannot help — that code died with the process. Left alone, one such run does three kinds of damage, in increasing order of how long it takes you to notice: 1. The run shows as in progress in the UI. Forever. 2. It holds its **spend reservation**, so the period's budget is permanently smaller. 3. It counts against the **concurrent-run cap**. At the default `maxConcurrentRuns` of 1, that means every later sweep hits the cap and starts nothing — the unattended loop is stopped, indefinitely. Neens handles all three automatically. This section is about how to recognise it and what you will see. ### "It says 1 in flight and nothing has happened for days" That is the symptom, and it used to be genuinely ambiguous: a run started thirty seconds ago and a run whose worker died last Tuesday both read as **In-flight autonomous runs: 1**. **Settings → Autonomy** now puts an age next to the count: ``` In-flight autonomous runs: 1 · oldest seen 6 hours ago Autonomous runs today: 0 Last sweep: 12 minutes ago ``` Read it like this: | What you see | What it means | | --- | --- | | A count with a **recent** "oldest seen" (minutes) | Normal. A fix run takes minutes; verification is the slow part | | A count with an "oldest seen" of **hours or days**, in warning colour, plus the panel **A fix run's worker has stopped checking in** | The worker is gone. The reaper will cancel the run on its next pass and give the slot back | | **Autonomous runs today: 0** with a non-zero in-flight count | Nothing new is being started. Check the decision log for a `concurrency_cap` refusal | | **Last sweep** stuck far in the past | Different problem — the sweep itself is not running. Check that the scheduler process is up | The same numbers are on `GET /autonomy/status`, as `inFlightRuns`, `staleInFlightRuns`, `oldestInFlightRunId`, `oldestInFlightRunLastSeenAt` and `oldestInFlightRunAgeSeconds`. A stale run does **not** make `health.healthy` false and does **not** suspend the agent. Health is about your fix *quality* gates (κ, drift, regressions, budget) and every one of its reasons needs a human to resume. A stale run needs no human — it heals on the next pass, usually within minutes. ### What the reaper does A background job — the **stale-run reaper** — looks for in-flight fix runs whose worker has stopped reporting, and cancels them. It runs every **5** minutes, independently of whether autonomy is enabled, and it covers runs of **every** origin: a run *you* started by hand is just as dead when its worker vanishes; it simply is not holding the autonomous cap. The autonomy sweep also runs the same check immediately before it counts the caps, so the cap heals on the very path that enforces it. While a run is executing, its worker stamps a liveness heartbeat every **30** seconds. A run is stale when: | Situation | Stale after | | --- | --- | | The run **has** a heartbeat and it has gone quiet | **15 minutes** | | The run has **no** heartbeat at all — it was never picked up, or it was started before this release | **180 minutes** | The two bounds are deliberately far apart, because the two signals mean different things. A missed heartbeat is **positive evidence of death**: nothing but a live worker writes that timestamp. General silence is only an **absence of evidence** — a healthy run spends most of its wall time inside verification, replaying your golden dataset *k* times without writing anything — so it gets a much longer rope. ### What a reaped run looks like ### The run Status **`cancelled`**, and its **error** says what was observed: > the worker running this fix run stopped reporting: no sign of life since > 2026-08-01T04:12:55+00:00 (372 minute(s) silent). Neens cancelled the run and released its budget > reservation. Nothing is known to be wrong with the fix itself — the process executing it > disappeared (an OOM kill, a pod eviction or a lost node). Start a new run when you are ready. ### The budget The run's reservation is **released**. It disappears from the period's reserved total and the headroom comes back. A settled cost is never unwound — only a reservation that was never spent. ### The decision log For an autonomous run, one `cancelled` row with reason `worker_lost`, rendered as **The run's worker stopped checking in, so the run was cancelled**, carrying the same sentence as the run's error. ### The insight feed A `fix_failed` insight, **Fix run abandoned (worker lost)** — the feed you already read, rather than a new surface nobody has learned to look at. ### The next sweep The slot is free, so the loop starts a run again on its next tick. Nothing needs a human. **Why `cancelled` and not `failed`:** because the fix did not fail — the process running it vanished, and calling that a failed fix would be a claim about your patch that Neens has no evidence for. (There is a second, quieter reason: `cancelled` is the one status a straggler worker can never overwrite, so the reaper's verdict is final by construction.) Nothing about a reaped run marks the remediation as bad. Start a new run whenever you like — the remediation, its proof gate and its regression set are untouched. ### When the loop is at a cap, it now says so Both cap skips write a `refused` row into the decision log instead of returning silently. They are **deduped over 24 hours**, so a 15-minute sweep records one row per reason per day rather than 96. | Reason | What it means | What to do | | --- | --- | --- | | `concurrency_cap` | Enough autonomous runs are already in flight. The detail names the counts, the effective cap, **the oldest in-flight run's id and how long since it was last seen**, and whether any of them are stale | If the oldest run is stale, nothing — the reaper takes it and the loop resumes. If the runs are genuinely working, this is the cap doing its job; raise **Max concurrent runs** if you want more parallelism | | `daily_cap` | This agent has already started its allowance of autonomous runs today (UTC) | Usually nothing — the count resets at 00:00 UTC. Raise **Max autonomous runs per day** if the loop should get further each day. Both caps are still bounded by the platform ceilings | Because the row is deduped, its detail is the state **at the first refusal of the window**, not a live figure. The live one is always on **Settings → Autonomy** / `GET /autonomy/status`. ### Worked example An agent at **Auto-verify** with `maxConcurrentRuns: 1`. Times are relative. ### 00:00 — a run is admitted The sweep finds an accepted remediation, estimates the run at $0.18, reserves it, creates the run and records an `admitted` decision. **In-flight: 1 · oldest seen just now.** ### 00:03 — the worker is OOM-killed The run is mid-verification. Its last heartbeat was seconds ago; the row still says it is running. ### 00:03–00:18 — inside the stale bound The run is silent but not yet stale — this is indistinguishable from a slow verification, on purpose. **In-flight: 1 · oldest seen 15 minutes ago.** The sweep at 00:15 finds the cap consumed and records one `refused` / `concurrency_cap` row naming this run and its age. ### 00:18 — it crosses the bound The Autonomy tab turns the count to its warning colour and shows **A fix run's worker has stopped checking in — the oldest is `fxr_…`, last seen 18 minutes ago.** ### ≤00:23 — the reaper takes it Status `cancelled` with the "no sign of life since…" error. The $0.18 reservation is released. A `cancelled` / `worker_lost` decision and a **Fix run abandoned (worker lost)** insight are recorded. ### ≤00:38 — the loop resumes The next sweep sees zero in-flight runs, and admits the next eligible candidate. **In-flight: 1 · oldest seen just now.** Total unattended downtime: under 40 minutes, with a written explanation for every minute of it. Before this existed, step 5 never happened, and steps 3 and 6 lasted until somebody noticed by hand. ## The spend budget The **Autonomous spend budget** card caps what the fix engine may spend on this agent per period. Set **Limit (USD per period)**, a **Period** (**Per day**, **Per week (ISO)**, **Per month**) and tick **Enforce this budget**. ### How the estimate is computed Before an autonomous run starts, Neens prices it and reserves that amount against the period: ```text verification runs = passK + (1 if the accumulated regression set is non-empty else 0) total items = passK × (proof-gate dataset items) + (regression set items) input tokens = total items × 1200 (assumed per item) output tokens = total items × 512 (assumed per item) ``` The item counts are **read**, never assumed — the proof-gate dataset's frozen golden version and the real regression set. With `passK` at its default of 3, a 14-item gate dataset and a 210-item regression set: 4 verification runs, 3 × 14 + 210 = **252 items**, 302,400 input and 129,024 output tokens, priced at the rate in force for the model on the agent's default LLM connection. Prices resolve through **your tenant's** price table — the shipped catalogue **plus any rate you set yourself** under **Settings → Model pricing** — so a self-hosted or open-weight model you have priced yourself is priced here too, and a negotiated rate beats the list price. See [Cost & model pricing](/guides/cost-and-model-pricing). ### The honesty rule: unpriced is never $0 If the model has no rate, the estimate is **not zero** — it is *absent*. The card shows **Cost could not be priced** and the decision log shows `— could not be priced` in the **Cost** column, never `$0.00`. This matters more here than anywhere else in the product. A free run fits inside *every* budget, so treating an unpriced model as $0 would turn it into an unlimited autonomous spend allowance — the exact inversion of what a budget is for. So the verdict becomes **Cost unknowable**, and an unknowable cost **refuses** the autonomous run and suspends the agent with `budget_exhausted`. | Verdict | Condition | Effect | | --- | --- | --- | | **Within budget** (`ok`) | No limit configured, or spent + reserved + estimate ≤ limit | Autonomous runs may spend | | **Budget exhausted** (`exhausted`) | spent + reserved + estimate > limit | Suspends with `budget_exhausted` | | **Cost unknowable** (`unknown`) | The estimate cannot be priced, or any spend already in the period could not be priced | Suspends with `budget_exhausted`; the detail says the cost is unknowable rather than over the limit | A manual run is still allowed in every one of these states. What is refused is spending money nobody can bound without a person looking at it. To clear an `unknown`, add the missing rate under **Settings → Model pricing** — the card links straight there, and the rate you set is the one the next estimate uses. ### These are estimates, not metered actuals **Neens does not measure what an individual fix run cost.** Set your limit accordingly: budget enforcement is arithmetic over *estimates*, and the ledger says so rather than dressing them up. - Before a run, its estimated cost is **reserved** against the period. - When the run reaches a terminal state **having run at least one verification run**, that reservation is **settled at the same admission estimate** — recorded as priced when the estimate was priceable, and marked in the ledger entry as an estimate rather than a measurement (`source: "admission_estimate"`, `measured: false`). A run that never verified anything is **released** instead; see below. - A ledger row is unpriced (`priced: 0`) **only when the cost was genuinely unpriceable** — an unpriced model — never merely because it was not metered. So a successful run against a priced model does **not** turn the period `partial`, and an agent with a healthy budget is not suspended by its own first success. - On the decision row, `estimatedUsdMicros` and `costPartial` carry the admission figure, while `actualUsdMicros` stays **null** — a field named "actual" must not be filled with an estimate. `remainingMicros` is not clamped at zero, so an already-overspent period reports a negative number rather than a reassuring `0`, and the usage bar is only drawn when the total is trustworthy. ### A run that verified nothing is released, not billed The admission estimate prices **verification runs** — `passK` proof-gate runs plus a regression run — so it is only meaningful once at least one of them exists. A run that reaches a terminal state without ever creating a verification run therefore has its reservation **released**, not settled: the ledger records it as `kind: released` and the period total is unchanged. That covers every way an autonomous run can die before it costs anything: the remediation was deleted between admission and execution, the configured fix driver could not be built, the driver produced no patch, or the run drafted a change with no verification target. All of them used to settle the full estimate — a run that failed in two seconds having called no model still moved the budget, and enough of them suspended an agent on `budget_exhausted` for spend that never happened. **A run that got partway through still settles the whole estimate.** If one verification run of three completed and the run then failed, the full admission estimate is booked. Neens records no measured per-run cost, so there is no honest way to bill a fraction of it — and for a control whose job is bounding unattended spend, over-billing is the safe direction. The alternative is a budget that silently under-counts real money. You can tell the two apart in the ledger (**Autonomous spend budget → the current period's rows**, or `GET /spend-budgets/fix_engine/usage`): a `reserved` row followed by `settled` is spend that counts, and a `reserved` row followed by `released` is spend that was authorised and then given back. ## The audit trail Every autonomous action is reconstructable after the fact, in two places. **The decision log** (**Settings → Autonomy**, or `GET /autonomy/decisions`) holds one row per decision: | Decision | Written when | | --- | --- | | `admitted` | A candidate cleared everything and a fix run was created | | `refused` | A candidate failed the policy, had no proof-gate dataset, or is held back after too many consecutive failures (reason `consecutive_failures`) — **or** the sweep stopped at a cap before looking at any candidate (`concurrency_cap`, `daily_cap`). Deduped over 24 h per remediation and reason | | `suspended` | The fail-safe fired, or a human hit the kill switch | | `resumed` | A human resumed, or a budget period rolled over | | `cancelled` | An in-flight autonomous run was cancelled — by the kill switch, or by the stale-run reaper with reason `worker_lost` | Each row carries the decision, level, remediation and cluster, the fix run, a short machine reason and a human sentence, **the full checks array**, **the exact policy snapshot that decided it**, the blast radius and what matched, the verifying κ, and the **estimated** cost with a flag for whether it could be priced. `actualUsdMicros` stays null: Neens measures no per-run cost and will not present an estimate as one. Filter with `?decision=refused`, `?remediationId=rem_…` or `?reason=…` (`?reason=consecutive_failures` is the held-back set). **The [audit trail](/administration/audit-log)** records the mutations alongside every other change in the tenant: `autonomy.update` (with the before/after level and policy, and any runs cancelled), `autonomy.suspend`, `autonomy.resume`, `autonomy.start_run` (per autonomous run, with the blast radius, κ, estimate and decision id) and `spend_budget.update`. A fix run started by autonomy is also marked at the run itself: `origin` is `autonomous` and `autonomyDecisionId` points back at the decision that authorised it. ## How it works A scheduled sweep runs every **15** minutes across every agent that has opted in to a non-`off` level. An agent that has never opened the tab costs one indexed read per tick — no rows are created for it. Per agent, in this order: 1. **Lift a rolled-over budget suspension** — and only a budget suspension. A κ drift does not heal at midnight. 2. **Stop if suspended.** Nothing is recorded; the suspension was already surfaced when it fired. 3. **Gather health signals** — open judge-drift alerts, the best eligible verifying κ, recent autonomous regression failures, and the period budget. These are the same four numbers `GET /autonomy/status` shows you. 4. **Run the fail-safe gate.** If it says stop: suspend, notify, cancel in-flight autonomous runs, and return. 5. **Reap stale in-flight runs** before counting them, so a run whose worker died cannot consume the concurrency cap. See [When a run's worker dies](#when-a-runs-worker-dies). 6. **Apply the caps.** The effective cap is the **lower** of your policy value and the platform ceiling (**2** concurrent runs; **20** runs per day). Only *autonomous* runs count toward them — a human's three manual runs do not consume the unattended loop's budget, and the loop never makes room by cancelling someone's work. Either cap being hit records a deduped `refused` decision, so a tick that did nothing still says why. 7. **Select candidates**, highest priority first. `auto_verify` considers `accepted` remediations only; `auto_pr` additionally considers `proposed` ones, each of which must still clear the whole auto-accept policy. A remediation that already has a fix run `queued`, `applying`, `verifying`, `pr_opened` or `drafted` is skipped, which is what makes the sweep safe to re-run. A remediation with **3** consecutive failed autonomous runs since a human last touched it is skipped too — see [When one remediation keeps failing](#when-one-remediation-keeps-failing). 8. **Per admitted candidate**: estimate the cost → re-check the budget *with that estimate included* → reserve → create the run → record the decision → commit → enqueue. "We are under the limit" is not the same claim as "this run fits under the limit", so the second check is not redundant. One malformed candidate never aborts the sweep, and a failure after the reservation releases it. ## Reference Routes | Method + path | Who | What | | --- | --- | --- | | `GET /autonomy/status` | read | Level, suspension, policy, effective κ floor, platform ceilings, health, budget, in-flight and today's run counts (with the oldest in-flight run's id, age and last-seen time, plus a stale count), last activity, five recent decisions | | `GET /autonomy/settings` | read | Level, suspension state and the resolved policy | | `PATCH /autonomy/settings` | admin | Set `level` and/or merge a `policy` patch. Lowering out of the autonomous levels cancels in-flight autonomous runs | | `POST /autonomy/suspend` | admin | The human kill switch; optional `reason` and `detail` | | `POST /autonomy/resume` | admin | Clear a suspension; `409` when not suspended | | `GET /autonomy/decisions` | read | The decision trail; `limit`, `offset`, `decision`, `remediationId`, `reason` | | `GET /spend-budgets` | read | Every feature's budget for the agent | | `GET /spend-budgets/{feature}` | read | One feature's limit, period and enabled flag | | `PUT /spend-budgets/{feature}` | admin | Set `limitUsd` (`null` = unlimited), `period`, `enabled` | | `GET /spend-budgets/{feature}/usage` | read | The current period's spend, verdict and ledger rows | `{feature}` is `fix_engine` today; `model_sweep` is accepted as a name but nothing bills against it yet. Reads accept a `nk_live_…` agent key; writes need an admin credential. ## Troubleshooting - **The Autonomy tab is missing.** You are not an admin — the tab is admin-only. - **The level is `auto_pr` but nothing is ever admitted.** Open the decision log and read the `refused` rows — they name the failing check. The most common are `blast_radius` (`unknown`: the remediation has no derivable file list) and `verify_kappa` (no judge is anchored). A `no_proof_gate` refusal means the remediation has no [eval gate](/guides/eval-gates) to verify a fix against. - **Autonomy suspended with `kappa_below_floor` on a brand-new agent.** No judge has a measured κ yet, which reads as "nothing is anchored" — not as zero. Label gold items so alignment is measured (see [Annotations & review](/guides/annotations-and-review)), then resume. - **Autonomy suspended with `budget_exhausted` and the detail says the cost is *unknowable*.** The model on the agent's default LLM connection has no rate. Add it under **Settings → Model pricing** ([Cost & model pricing](/guides/cost-and-model-pricing)), then resume. - **The budget shows "Cost could not be priced" even though runs are finishing.** That state means a model genuinely has no rate — not that a run went unmeasured. Find the unpriced model and give it a rate in **Settings → Model pricing**; the estimate then prices, the period stops being partial, and the verdict leaves `unknown`. A *priced* run settling does not make the period partial. - **One remediation stopped being picked up, but nothing is suspended.** It is held back after **3** consecutive failed autonomous runs. Look at the **Held back** card on the **Autonomy** tab for the last failure text, fix whatever it names, and touch the remediation — any edit puts it back in the queue. See [When one remediation keeps failing](#when-one-remediation-keeps-failing). - **A remediation is held back and I do not think it should be.** Read the last failure report on the card. If the failures were caused by something agent-wide that you have since fixed, just re-accept the remediation: the streak only counts autonomous runs started after your edit. - **The period total looks lower than my provider invoice.** It is a sum of pre-run **estimates**, not metered spend — Neens records no per-run actual. Treat the limit as a governor on how much unattended work may be authorised, and reconcile real spend with your provider. - **A run vanished into `cancelled` with "autonomy cancelled this run".** Someone lowered the level or suspended the agent while it was flying. That is the kill switch working; the decision log names the reason. - **`In-flight autonomous runs: 1` for hours and nothing new ever starts.** The run's worker died and is holding the concurrency cap. The count shows the oldest run's age beside it, and a warning panel appears once it is stale; the reaper cancels it within a few minutes and the loop resumes on its own. See [When a run's worker dies](#when-a-runs-worker-dies). - **A run was cancelled with "the worker running this fix run stopped reporting".** The reaper took it. Nothing is known to be wrong with the fix — start a new run. If this happens repeatedly, look at worker memory limits and evictions rather than at the remediation. - **The decision log shows a `concurrency_cap` or `daily_cap` refusal.** The loop stopped at a cap before considering any candidate. Read the detail: a `concurrency_cap` row naming an in-flight run that was last seen hours ago is a dead worker, not a busy loop. - **I resumed and it suspended again within 15 minutes.** The underlying signal is unchanged. Resume clears the flag, not the cause. ## Related - [Eval-verified PR](/guides/eval-verified-fix) — the run autonomy starts, and the proof it carries - [Remediations](/guides/remediations) — where candidates come from, and what `proposed` / `accepted` mean - [Eval gates](/guides/eval-gates) — the proof gate a candidate needs before it can be admitted - [Annotations & review](/guides/annotations-and-review) — how judge↔human alignment (κ) is measured - [Fix outcomes](/guides/post-merge-efficacy) — what happened after the PR merged - [Insights & notifications](/guides/insights) — where a suspension alert lands ================================================================================ # Prompt optimization Source: /docs/guides/prompt-optimization/ ================================================================================ # Prompt optimization The Neens **prompt optimizer** searches for a better system prompt for one confirmed failure mode. It replays that failure's real historical traces against candidate prompts, grades every replay with your judges, reflects on the graded results to write a better candidate, and keeps searching — then proves the winner on a **held-out** set of traces it never optimized against. The winner leaves the optimizer as an ordinary [remediation](/guides/remediations), so it earns a pull request the same way every other fix does. This is not "the agent rewrites its own prompt in production". Every rollout is a replay of a trace that already happened, and the result is a *proposal* a human reviews and merges. The optimizer sits on top of things you already have: a confirmed failure mode with a [remediation](/guides/remediations) rail, an [eval gate](/guides/eval-gates) derived from that failure, the [pre-prod eval](/guides/preprod-evals) runner that verifies candidates, and the [eval-verified PR](/guides/eval-verified-fix) gate the winner exits through. If you can already open an eval-verified PR for a failure mode, you can optimize its prompt. ## At a glance | | | | --- | --- | | **Where** | The **Optimizer** tab on the **Prompts** page | | **Key API** | `POST /prompt-optimization/runs` · `GET /prompt-optimization/runs/{id}` · `GET /prompt-optimization/preview` · `POST /prompt-optimization/runs/{id}/emit-remediation` | | **MCP tools** | `start_prompt_optimization` (launch) · `get_prompt_optimization` (poll) | | **What it needs** | A confirmed failure mode (or cluster) with at least **3** replayable failing traces, judges that grade them, and your agent's LLM connection | | **What it produces** | A `prompt_change` remediation carrying the winning prompt, the before/after diff, and the held-out proof — plus the full candidate lineage you can inspect | | **Merges?** | **Never.** The winner is a proposal; it goes through the same pass^k verification and human merge as every other fix | The optimizer lives on the **Optimizer** tab of the [Prompts](/guides/prompt-management) page, alongside prompt versions and deploy tags — it used to be its own page under **Fix**. An old `/prompt-optimizer` link still works: it redirects to the new tab. ## How it works The loop is deliberately small and bounded. Each step is either a replay of something that already happened or a single LLM call, and both are capped. ### Seed from a confirmed failure A run always starts from a **failure mode** (preferred) or a **cluster** — never from "all traffic". Neens takes that failure's failing traces as the *task set* and reads the **baseline prompt**: the system prompt those traces actually ran under. Grounding is mandatory, because a prompt tuned against undifferentiated traffic produces a weakly targeted repair. ### Split train vs. held-out The task set is split deterministically into a **train** split the optimizer may optimize against and a **held-out** split it may not touch until the end. Reporting a train-split gain is exactly how a prompt optimizer fools itself, so the split is not optional. ### Replay and grade Each *rollout* re-runs one historical request against a candidate prompt, with the agent's tool results served from that trace's recorded tool spans, and grades the result with your judges. A rollout that can't be replayed (the fixed prompt calls a tool the trace never recorded, the provider errors) scores **0** and is recorded as a failure — never dropped, so a candidate can't win by breaking replay on the tasks it is worst at. ### Reflect Neens shows the current prompt, the worst rollouts, and — critically — the **judges' reasoning** for each one to your LLM connection, and asks for a complete replacement prompt. Reflecting on *why* a rollout was graded down is what makes this affordable: a bare score would need orders of magnitude more rollouts to learn the same thing. ### Accept or discard, cheaply A child prompt is re-run on the *same* small batch its parent was judged on. It earns a full evaluation only if it beats its parent there; a tie is discarded. This is where the rollout budget is saved. ### Keep the Pareto frontier Surviving candidates are kept as a **Pareto frontier**: every candidate that is best on at least one task stays, not just the one with the best average. The next parent is sampled from that frontier, weighted by how many tasks it wins. A mean-score hill-climb collapses to one lineage and throws away the candidate that solved the one hard task nobody else did. ### Prove on held-out, then emit When the budget runs out, every frontier candidate *and* the baseline are rolled out on the held-out split. The winner is the best **held-out** mean — and it becomes a remediation only if it beats the baseline there by a real margin (default **0.05**) over at least **2** held-out tasks. A run that never beats the baseline finishes normally and emits nothing. That is a result, not an error: the honest answer is "prompt changes of this shape didn't fix this failure." ## Preview a run before you spend anything Every rollout is a billable LLM call on **your** connection, so check what a run would cost first. `GET /prompt-optimization/preview` resolves the task set, the baseline prompt, the graders and the estimated rollout count without calling a model. The launch form on **Prompts → Optimizer** shows the same estimate live as you change the budget. ```bash curl -G https://your-neens/api/prompt-optimization/preview \ -H "Authorization: Bearer nk_live_…" \ --data-urlencode "failureModeId=fm-2a91c4d80f13" ``` ```json { "failureModeId": "fm-2a91c4d80f13", "clusterId": null, "modeName": "Unsupported order-status claims", "rootCause": "The prompt never tells the agent to ground order claims in tool output.", "seedSource": "failure_mode", "tasks": ["ses-1a2b", "ses-3c4d", "…"], "trainTasks": ["ses-1a2b", "…"], "valTasks": ["ses-9x8y", "…"], "warnings": [], "baselinePrompt": "You are Acme Support. Answer the customer's question…", "hasBaselinePrompt": true, "judgeIds": ["jdg-7f10c2"], "judgeNames": ["Faithfulness"], "judgeSource": "deployments", "estimatedRollouts": 216, "maxRollouts": 240, "runnable": true, "provable": true } ``` `runnable` is the one field to branch on: it is `false` when the seed resolves to fewer than 3 in-scope sessions (no honest held-out split is possible) or when none of them carries a recoverable system prompt — either way `POST /prompt-optimization/runs` will refuse with a 422. `provable` is a softer warning: the run can optimize but has too few held-out tasks to prove a gain, so it will complete and emit nothing. The preview also accepts `maxTasks`, `maxIterations`, `minibatchSize` and `valFraction`, so you can see how a smaller budget changes the split and the estimate before you launch. ## Launch a run ### Pick the failure to optimize against Open **Prompts → Optimizer**, click **New optimization** in the top right to open the launch form, and choose the failure mode you want a better prompt for. Prefer a failure mode over a bare cluster: a failure mode carries the [eval gate](/guides/eval-gates) that becomes the emitted remediation's proof gate, and **only a remediation with a failure mode can start an eval-verified fix run**. ### Choose the graders and the budget The graders default to the judges already deployed for that agent; you can narrow them. The budget caps how many rollouts and reflection iterations the run may spend — the defaults (**240** rollouts, **12** iterations) are the ceiling, and a per-run value is clamped to that cap, never above it. ### Launch and let it run Click **New optimization** on **Prompts → Optimizer**, fill in the launch form, and start the run. It appears in the run list as **queued**, then **running**; the page polls until it reaches a terminal state, so you can leave it open and watch candidates appear generation by generation. ```bash curl -X POST https://your-neens/api/prompt-optimization/runs \ -H "Authorization: Bearer nk_live_…" \ -H "Content-Type: application/json" \ -d '{ "failureModeId": "fm-2a91c4d80f13", "name": "Order-status grounding", "objective": "Stop the agent asserting a delivery date the tools did not return.", "maxRollouts": 160, "maxTasks": 24 }' ``` A `nk_live_…` agent key can launch a run, so this works from CI with the credential you already have. The response comes back immediately: ```json { "id": "opt-9c41ab77e2d0", "status": "queued", "name": "Order-status grounding", "failureModeId": "fm-2a91c4d80f13", "judgeNames": ["Faithfulness"], "trainTasks": ["sess-6b1e…", "sess-9d02…"], "valTasks": ["sess-4417…", "sess-c8ab…"], "config": { "maxRollouts": 160, "maxIterations": 12, "minibatchSize": 4, "maxTasks": 24, "valFraction": 0.34, "minImprovement": 0.05, "autoEmit": true }, "createdAt": "2026-07-24T09:12:44Z" } ``` Other options on the request: `clusterId` or an explicit `sessionIds` list instead of `failureModeId`, `judgeIds` to choose the graders, `connectionId` to pin a specific LLM connection, `minibatchSize`, `valFraction`, `minImprovement`, `maxIterations`, and `autoEmit`. From a coding agent connected to the Neens [MCP server](/guides/mcp): - `start_prompt_optimization` — launch a run for a failure mode, with the same budget and grader options as the API. - `get_prompt_optimization` — poll the run: status, budget used, held-out improvement, the winning prompt, and the id of the remediation it emitted. ### Poll until it finishes ```bash curl https://your-neens/api/prompt-optimization/runs/opt-9c41ab77e2d0 \ -H "Authorization: Bearer nk_live_…" ``` ```json { "id": "opt-9c41ab77e2d0", "status": "completed", "baselineScore": 0.412, "bestScore": 0.703, "improvement": 0.291, "rolloutsUsed": 154, "iterationsUsed": 9, "winnerCandidateId": "optc-31f0aa92b7c4", "remediationId": "rem-88d1e4c07a29", "outcome": { "emit": true, "winnerId": "optc-31f0aa92b7c4", "baselineVal": 0.412, "winnerVal": 0.703, "improvement": 0.291, "improvedTasks": ["sess-4417…", "sess-c8ab…", "sess-1f70…"], "regressedTasks": ["sess-b294…"], "reason": "candidate beat baseline on held-out rollouts: 0.703 vs 0.412 (Δ +0.291) over 8 task(s); improved 5, regressed 1.", "warnings": [ "the winning candidate REGRESSES 1 held-out task(s) the baseline handled; the reviewer sees this on the remediation." ] }, "candidates": [ { "id": "optc-31f0aa92b7c4", "generation": 3, "parentId": "optc-0b7d5518e2fa", "trainMean": 0.688, "valMean": 0.703, "accepted": true, "onFrontier": true, "isWinner": true, "rolloutsUsed": 24, "rationale": "Requires the agent to quote the tool's returned status verbatim and to say it doesn't know when no status was returned." } ], "frontier": ["optc-31f0aa92b7c4", "optc-0b7d5518e2fa"] } ``` Statuses are `queued`, `running`, `completed`, `failed` and `cancelled`. `POST /prompt-optimization/runs/{id}/cancel` stops a queued or running run; the optimizer checks for cancellation between iterations, so a cancelled run stops spending budget and never writes a result. `GET /prompt-optimization/runs/{id}/candidates/{candidateId}` returns one candidate with its individual rollouts. ## Read the results The run detail view leads with the numbers that decide whether the candidate is real. ### Held-out improvement is the only number that matters The KPI row shows the baseline's held-out score, the winner's held-out score, and the **Δ** between them. That Δ — measured on traces the optimizer was never allowed to optimize against — is the claim. A candidate's train-split score will almost always look better than the baseline's, because that is what it was fitted to; a train gain that doesn't survive the held-out split is an optimizer fooling itself, and Neens will not emit it. ### The Pareto frontier and the lineage Candidates are shown by generation with their parent, their train mean, their held-out mean, and badges for **accepted**, **frontier** and **winner**. Being *on the frontier* means the candidate is the best of all candidates on at least one individual task, even if its average is unremarkable — that's why it was kept and why it could be sampled as the next parent. Dominated candidates (best at nothing) are still listed, so you can see what the search tried and rejected. ### "Regresses N held-out tasks" A candidate can raise the average while breaking a task the baseline handled. Neens reports that count explicitly and carries it onto the remediation instead of hiding it in the mean. Treat it as a review item: open the regressed tasks and decide whether the trade is acceptable for your product before you accept the remediation. A run with a large regression count is often better re-run with a narrower objective. ### The prompt diff The winner is shown as a before/after diff against the baseline prompt — the same typed `prompt_change` artifact the remediation carries, so what you review here is exactly what a fix run would apply. ## What happens to the winner When a run clears the bar, Neens writes a `prompt_change` **remediation** in the `proposed` state against the same failure mode, carrying the winning prompt, the diff, the held-out proof, and a **View remediation** link from the run. From there it is an ordinary remediation: you review it, accept it, and it can start an [eval-verified fix run](/guides/eval-verified-fix) that verifies the prompt with pass^k pre-prod runs against the failure's proof gate plus your accumulated regression set, and opens a pull request only if every run is green. You can also emit the winner by hand — the **Emit remediation** button on a finished run, or `POST /prompt-optimization/runs/{id}/emit-remediation`. Emitting twice returns `409`; emitting a run that never cleared the held-out bar returns `422`. **The judges that optimized the prompt cannot verify it.** Neens records which judges a candidate was optimized against and excludes them from verifying the resulting fix — a candidate graded by its own optimization target is overfit, not verified. The consequence is real and intended: if the failure mode's *only* human-aligned judge is the one the optimizer used, no eligible verifier remains, and the fix engine lands a **draft** PR instead of opening a clean one. A real PR for an optimizer-authored fix needs a **second**, human-κ-anchored judge. Deploy one and keep labelling gold items so its alignment stays measured — see [Annotations & review](/guides/annotations-and-review). ## Offline by design The framing matters, because it is what makes this safe to run against a production agent's prompt: - **It is offline.** Every rollout is a record-and-replay re-run of a *historical* trace. The agent's tool results are served from that trace's recorded tool spans — no tool is actually called, no customer request is affected, and no live traffic is scored or steered. - **It never deploys.** The winner does not go live. It becomes a proposal in your remediation queue. - **It has one trust mechanic.** An optimizer-authored fix goes through exactly the same gate as a hand-written one: pass^k pre-prod verification against the failure-derived proof dataset *and* the accumulated regression set, verified by judges it did not optimize against, and merged by a human. There is no separate, faster path for machine-written prompts. - **It is grounded upstream.** A run is seeded from a diagnosed failure mode and its root cause, not from aggregate traffic, so the change it proposes targets a failure you already confirmed. ## LLM connection and cost The optimizer runs entirely on **your agent's configured LLM connection** (**Settings → LLM providers**) — Neens has no model access of its own. Both kinds of call it makes are billed to that connection: each **rollout** replays and grades one trace, and each **reflection** is one call that writes the next candidate prompt. An agent with no LLM connection cannot run the optimizer: launching returns `400` and the launch form tells you to add one. Add a connection in **Settings → LLM providers** — see [Getting started](/getting-started#connect-an-llm). Because a run is expensive by construction, it is bounded on every axis: - a hard **rollout cap** (default **240** per run) that a per-run value can lower but never raise; - an **iteration cap** (default **12** reflections); - a per-call timeout on each reflection (**90s**) and each rollout (**60s**), so one hung call fails that item instead of the run; - a per-caller launch throttle (default **10** runs per hour). Use the [preview](#preview-a-run-before-you-spend-anything) endpoint to see the estimated rollout count before you commit, and lower `maxRollouts` or `maxTasks` for a cheaper first pass. ## Troubleshooting - **The run completed but emitted nothing.** The best candidate didn't beat the baseline on the held-out split by the required margin (default `0.05`). Read the run's verdict — it states the two held-out scores and the gap. This is the expected outcome when the failure isn't actually caused by the prompt; look at the failure mode's root cause and consider a tool or retrieval fix instead ([Remediations](/guides/remediations)). - **"Too few sessions to optimize."** The seed resolved to fewer than **3** replayable failing traces, so there is no honest held-out split and nothing could be proved. Widen the seed — a larger failure mode or cluster, a longer time range — or pass an explicit `sessionIds` list ([Clustering](/guides/clustering)). - **The run is rejected with "grounding required."** No `failureModeId`, `clusterId` or `sessionIds` was given. The optimizer will not optimize against undifferentiated traffic; start from a confirmed failure ([Issues & failure modes](/guides/issues-and-failure-modes)). - **No LLM connection.** Launching returns `400` because the agent has no connection to run rollouts and reflections on. Add one in **Settings → LLM providers** — see [Getting started](/getting-started#connect-an-llm). - **The fix engine drafted a PR instead of opening one.** Every eligible verifying judge was excluded because the optimizer optimized against it. Deploy a second, human-aligned judge for that failure mode and label enough gold items for its alignment to be measured ([Annotations & review](/guides/annotations-and-review), [Eval-verified PR](/guides/eval-verified-fix)). - **The run is stuck at `queued`.** Nothing is draining the scoring queue. Optimization runs on the same worker fleet as judges and eval runs, so if judge runs are also stuck, the worker is the problem — see [Judges](/guides/judges) and your deployment's worker configuration. - **The emitted remediation can't start a fix run.** It was seeded from a cluster rather than a failure mode, or its failure mode has no proof gate. Generate one for the failure mode ([Eval gates](/guides/eval-gates)) and re-open the remediation. ================================================================================ # Fix outcomes Source: /docs/guides/post-merge-efficacy/ ================================================================================ # Fix outcomes Opening an [eval-verified PR](/guides/eval-verified-fix) proves a fix works *before* it ships. **Fix outcomes** closes the loop *after* it ships: when a Neens-opened fix PR is **merged**, Neens records the deploy, watches the fixed failure in real production traffic for a window, and reports whether the failure actually went down — per agent, per merged fix. This is the difference between a dashboard number and a business outcome. "Failures shown" is a chart. "Failure volume is down 60% across 8 merged fixes, with a median cluster-to-merge time of 2 days" is a result you can take to a stakeholder — the public close-out of the Neens failure → fix → proof loop. ## At a glance | | | | --- | --- | | **Where** | The **Outcomes** page under **Fix** | | **Key API** | `GET /remediations/efficacy-report` · `POST /remediations/items/{id}/record-merge` · `POST /api/webhooks/github` | | **MCP tool** | `record_fix_merge` (report a merge from a coding agent) | | **What it measures** | Real production volume of the fixed failure mode, in the window **before** vs **after** the deploy, whether the proof-eval gate still holds, and — beside it — whether each active [business KPI](/guides/business-kpis) actually moved | | **Verdict** | Each merged fix auto-transitions to **verified**, **regressed**, or **inconclusive** | | **Needs** | A merged fix PR correlated to a remediation (via the GitHub webhook, the record-merge API, or MCP) | ## How the loop closes ### A fix PR merges Neens learns the PR was merged one of three ways ([below](#tell-neens-a-fix-merged)) and correlates the merged PR back to its remediation (by PR URL / fix run). ### Neens records the deploy and starts a watch On a merged PR, Neens writes a **deploy event** and opens a **close-out watch** for the remediation's failure mode. ### The window elapses After the measurement window (**7 days**), a scheduled sweep measures the failure mode's real production volume in the equal-length windows **before** and **after** the deploy, and re-checks that the remediation's proof-eval gate still holds. ### The remediation gets a verdict The remediation auto-transitions based on what actually happened in production — **verified**, **regressed**, or **inconclusive** (see [How the verdict is decided](#how-the-verdict-is-decided)) — and the **Outcomes** report updates. ## Tell Neens a fix merged Neens needs to know when a PR merges. Pick whichever path fits how you ship. Configure your repository or GitHub App webhook to POST **`pull_request`** events to: ``` https:///api/webhooks/github ``` Set a shared secret on the webhook; your operator configures Neens with the same value. Neens verifies GitHub's **`X-Hub-Signature-256`** HMAC header against that secret on every delivery and ignores anything that doesn't match — so an unsigned or mis-signed call is never trusted. On a **merged** PR event Neens correlates the PR to its remediation and starts the watch automatically. If you don't run the webhook — a CI step, or the self-hosted [local-git](/guides/eval-verified-fix) flow — record the merge directly. A `nk_live_…` agent key works, so your existing CI credential is enough: ```bash curl -X POST https://your-neens/api/remediations/items/rem_123/record-merge \ -H "Authorization: Bearer nk_live_…" \ -H "Content-Type: application/json" \ -d '{ "prUrl": "https://github.com/acme/agent/pull/42", "commitSha": "9f3c1ab", "mergedAt": "2026-07-10T00:00:00Z" }' ``` This records the deploy event and opens the close-out watch exactly as the webhook does. From a coding agent connected to the Neens [MCP server](/guides/mcp), the **`record_fix_merge`** tool reports the merge (PR URL, commit, merged-at) and starts the watch — the same path a human's CI step would take, driven by the agent that opened the PR. ## Read the Outcomes report Open **Fix → Outcomes** for a per-agent rollup of every merged fix and what it did in production. The same data is served by `GET /remediations/efficacy-report`. ### The KPIs The headline tiles answer "did shipping fixes make the product better?": - **Fixes shipped / verified / regressed** — how many merged fixes have a verdict, and how many held up in production vs. came back. - **Failure-volume reduction %** — total volume of the fixed failure modes **before** vs **after** their deploys, as one headline percentage across all merged fixes. - **Median MTTR** — the median time from a failure first being seen (its cluster's first-seen timestamp) to the fix merging. This is your **cluster-to-merge** time: how fast the loop actually turns. ### Before vs after, per fix Each merged fix shows its failure volume **before** the deploy next to its volume **after**, and the per-fix reduction. A fix that drove its failure mode from 40 occurrences to 6 reads as an 85% drop and a **verified** verdict; a fix whose volume didn't move (or whose gate broke) reads as **regressed**. ### What verified / regressed / inconclusive mean - **Verified** — production volume of the failure mode dropped **and** the proof-eval gate still holds. The fix did what it claimed. - **Regressed** — volume didn't drop, or a proof gate that used to pass now fails. The failure is still happening (or came back) despite the merge — worth reopening. - **Inconclusive** — too little traffic to call it. When the **before-window** volume is too low to form a baseline, there is no reduction to measure against, so Neens won't manufacture a verdict from noise. (An after-window spike is a **regression**, not an inconclusive.) The measurement is over **real production traces**, not the pre-merge eval run. Post-merge efficacy and the [pre-prod eval](/guides/preprod-evals) that gated the PR are complementary: the pre-prod gate proves the fix works on the golden dataset before merge; the outcome report confirms it held on live traffic after merge. Example GET /remediations/efficacy-report response ```json { "window": { "days": 7 }, "summary": { "fixesShipped": 8, "fixesVerified": 6, "fixesRegressed": 1, "fixesPending": 1, "fixesInconclusive": 0, "totalVolumeBefore": 240, "totalVolumeAfter": 96, "volumeReductionPct": 60.0, "mttrMedianSeconds": 172800, "mttrMedianDays": 2.0, "mttrCount": 7 }, "closeouts": [ { "id": "co-…", "remediationId": "rem-…", "remediationTitle": "Guardrail over-blocks refunds", "status": "verified", "prUrl": "https://github.com/acme/agent/pull/42", "mergedAt": "2026-07-10T00:00:00Z", "beforeVolume": 40, "afterVolume": 6, "reductionPct": 85.0, "volumeStatus": "improved", "gatesHeld": true, "mttrDays": 1.5, "windowDays": 7 } ], "trend": [ { "period": "2026-07", "fixesVerified": 6, "volumeBefore": 240, "volumeAfter": 96, "reductionPct": 60.0 } ] } ``` `summary` drives the KPI tiles, `closeouts` the per-fix before/after table, and `trend` the period-over-period chart. ## Did the KPI move? Volume answers *"is the failure happening less?"* It does not answer *"did the number we funded this fix to move actually move?"* — and those two come apart all the time: a failure can get rarer while the business metric it was hurting barely budges, or a fix can quietly lift a KPI well beyond the one cluster it targeted. So beside the [volume leg](#before-vs-after-per-fix), each close-out now carries a **KPI leg**. For every active [business KPI](/guides/business-kpis), Neens measures the KPI over the window **before** the deploy and over the equal window **after** it, and reports the before→after move. It measures each KPI at **two scopes**, so you can tell a local win from a fleet-wide one: - **Cluster** — the KPI computed over just the [failure cluster](/guides/issues-and-failure-modes) this fix targeted. *"Did containment recover for the cases this fix was about?"* - **Agent** — the KPI over the whole agent, unfiltered. *"Did the fleet number move, or only this corner of it?"* A fix can read **improved** at the cluster scope and **flat** at the agent scope — that is a real, common, honest result (the fix worked where it was aimed, and that corner is a small slice of all traffic), not a contradiction. ### The four statuses Every KPI-delta row lands in one of four statuses, and each renders differently on purpose. | Status | What it means | | --- | --- | | **improved** | The KPI moved in the good direction — per the KPI's own [direction](/guides/business-kpis#the-fields-and-what-each-one-is-for) — by more than sampling noise. | | **flat** | The KPI barely moved: the change is real arithmetic but small, or inside sampling noise. Reported as *no meaningful move*, not celebrated. | | **regressed** | The KPI moved the wrong way, past sampling noise, after the deploy. Worth a look even when the failure's volume dropped. | | **unknown** | Neens could not form an honest delta — too few decided cases either side, no before sample, or the [definition changed](/guides/business-kpis#a-definition-change-breaks-the-series) between the windows. Shown as **—**, never `0`. | **A move inside sampling noise is flagged, not celebrated.** For a rate KPI, Neens puts a confidence interval on the delta; if that interval straddles zero, the move is *not distinguishable from noise* and the row says so — even when the point estimate looks good. A 2-point containment bump over sixty cases is not a win yet; it is a number that will move again next week. Only a move that clears the noise floor reads **improved**. **`unknown` is a dash, never a zero.** A thin sample, a window with no *before* data, or a KPI whose [definition](/guides/business-kpis#a-definition-change-breaks-the-series) changed between the two windows all read `unknown` with a stated reason and render as **—**. The definition change is the subtle one: the two ends are answering different questions, so Neens draws no delta across it rather than a movement nobody made. Cost KPIs carry no confidence interval, so their move is reported without a *distinguishable-from-noise* claim; a window that was only partly priced reads `unknown` rather than a floor-minus-floor number nobody can defend. ### A worked example The remediation **"Guardrail over-blocks refunds"** was funded because an over-eager guardrail was escalating refund cases a human then had to pick up. After it merged, its close-out shows: - **Volume** — the failure mode fell from **40 occurrences to 6** (an 85% drop): the failure is happening far less. - **KPI, cluster scope** — **containment rate** on that cluster's cases rose **62% → 81%**, a distinguishable **+19-point** move: the cases this fix was about are now handled without a human. - **KPI, agent scope** — agent-wide containment moved **87% → 88%**, reported **flat**: the fleet number barely shifted, because this one cluster is a small share of all traffic. Honest, and not oversold. That is what the KPI leg adds: not just *"the failure is rarer"*, but *"and here is the business number it was costing you, measurably recovered for the cases it touched."* **Today the KPI leg reports beside the volume verdict; it does not override it.** The [verified / regressed / inconclusive](#how-the-verdict-is-decided) verdict is still decided from failure volume and the proof gate. The KPI move is shown next to it so you can see whether the business number followed the volume — a fix can cut volume without moving the KPI, and that gap is worth knowing. A KPI regression that clearly stands out from noise may in future be allowed to weigh on the verdict; for now it is reported, not enforced. Example close-out with kpiDeltas (from GET /remediations/efficacy-report) Each object in `closeouts` now carries a `kpiDeltas` array — one row per active KPI per scope. An empty array (`[]`) means no KPIs are active, or none applied to this fix. `unknown` rows are kept in the array with a reason; they are a recorded fact, never dropped and never shown as `0`. ```json { "id": "co-9f2a1c", "remediationId": "rem-4471", "remediationTitle": "Guardrail over-blocks refunds", "status": "verified", "beforeVolume": 40, "afterVolume": 6, "reductionPct": 85.0, "volumeStatus": "improved", "kpiDeltas": [ { "kpiId": "kpi-7c1a", "kpiLabel": "Containment rate", "measureKey": "containment_rate", "scope": "cluster", "kind": "rate", "before": 0.62, "after": 0.81, "delta": 0.19, "deltaPct": 30.65, "ciLow": 0.04, "ciHigh": 0.34, "distinguishable": true, "status": "improved", "unknownReason": null, "beforeN": 210, "afterN": 240, "targetStatus": "met" }, { "kpiId": "kpi-7c1a", "kpiLabel": "Containment rate", "measureKey": "containment_rate", "scope": "project", "kind": "rate", "before": 0.87, "after": 0.88, "delta": 0.01, "deltaPct": 1.15, "ciLow": -0.02, "ciHigh": 0.04, "distinguishable": false, "status": "flat", "unknownReason": null, "beforeN": 704, "afterN": 731, "targetStatus": "met" }, { "kpiId": "kpi-3d81", "kpiLabel": "Cost per case", "measureKey": "cost_per_case", "scope": "cluster", "kind": "cost", "before": 0.51, "after": 0.42, "delta": -0.09, "deltaPct": -17.65, "ciLow": null, "ciHigh": null, "distinguishable": null, "status": "improved", "unknownReason": null, "beforeN": 210, "afterN": 240, "targetStatus": "met" }, { "kpiId": "kpi-55c0", "kpiLabel": "Deflection rate", "measureKey": "custom:deflection-rate", "scope": "cluster", "kind": "rate", "before": 0.44, "after": 0.51, "delta": null, "deltaPct": null, "ciLow": null, "ciHigh": null, "distinguishable": null, "status": "unknown", "unknownReason": "definition_changed", "beforeN": 96, "afterN": 120, "targetStatus": "unknown" } ] } ``` Reading a row: | Field | What it tells you | | --- | --- | | `kpiId` · `kpiLabel` · `measureKey` | Which KPI this row is about. | | `scope` | `cluster` (just the fix's failure cluster) or `project` (the whole agent). | | `kind` | `rate` or `cost` — the two kinds of KPI a delta is defined for. | | `before` · `after` | The KPI over the window before the deploy, and over the equal window after. `null` (→ **—**) when that side had nothing to measure. | | `delta` · `deltaPct` | The move, in the KPI's own unit and as a percentage. Both `null` across a definition change — the two ends aren't subtractable. | | `ciLow` · `ciHigh` | The confidence interval on the delta, for a rate KPI. `null` for a cost KPI, which has no interval. | | `distinguishable` | `true` when the move clears sampling noise, `false` when it is inside it (the UI says so rather than celebrate). `null` for a cost KPI or when there is no interval. | | `status` | `improved` · `flat` · `regressed` · `unknown`, oriented by the KPI's own direction. | | `unknownReason` | Why a row is `unknown` — too few decided cases, no before sample, the definition changed, and so on. `null` otherwise. | | `beforeN` · `afterN` | The decided denominators either side. A +19-point move over 210 cases is a firmer fact than the same move over 12. | | `targetStatus` | The *after* value against the KPI's committed [target](/guides/business-kpis#met-missed-and-the-answer-most-dashboards-get-wrong): `met` · `missed` · `unknown`. | The same moves also feed the KPI's own page: every business KPI lists the [fixes that moved it](/guides/business-kpis#fixes-that-moved-this-kpi), so you can start from a number that recovered and see which merged fixes recovered it. ## How the verdict is decided For each merged fix, once the watch window has elapsed: 1. **Measure production volume** of the fixed failure mode in the window **before** the deploy and in the equal window **after** it. 2. **Re-check the proof gate** — does the remediation's failure-derived [eval gate](/guides/eval-gates) still pass? 3. **Decide:** - volume **dropped** and the gate **holds** → **verified**; - volume **did not drop**, or a gate **broke** → **regressed**; - **before-window** volume too low to form a baseline → **inconclusive**. The verdict is written back onto the remediation, so its lifecycle (`applied` → `verified` / `regressed`) reflects what really happened in production — not just that a PR merged. ## Troubleshooting - **A merge didn't start a watch.** Neens couldn't correlate the PR to a remediation. Confirm the PR URL matches the one Neens stamped when it opened the fix, or record the merge explicitly with [`POST /remediations/items/{id}/record-merge`](#tell-neens-a-fix-merged). - **Webhook deliveries are ignored.** The `X-Hub-Signature-256` didn't verify — the secret on the GitHub side doesn't match the one Neens was configured with. - **A fix stays inconclusive.** Not enough of the failure mode's traffic landed in the window. Wait for more production traffic to accumulate before the verdict can be called. ================================================================================ # Eval gates Source: /docs/guides/eval-gates/ ================================================================================ # Eval gates An **eval gate** is a standing evaluation **built from one of your own confirmed failures** and used as a release guard. This is the evals-from-failures flywheel: instead of writing evals from scratch, you turn a real recurring failure into a regression test — so once you fix it, it can't silently come back. One action materializes the whole chain from a failure mode: a **dataset** of its real evidence sessions, a **judge** that scores the failure as *absent*, and the **gate** binding them together with a tracked baseline. ## At a glance | | | | --- | --- | | **Where** | **Fix → Eval Gates** in the sidebar; created from an Issue card's **Generate eval** in Diagnose | | **Key API** | `POST /flywheel/failure-modes/{id}/generate-eval`, `GET /flywheel/gates`, `GET /flywheel/gates/{id}`, `PATCH /flywheel/gates/{id}`, `POST /flywheel/gates/{id}/run` | | **Needs** | A failure mode with evidence (classified sessions, a linked cluster, or exemplars). An LLM connection to actually run the drafted judge | | **Scope** | Agent-scoped | ## Create a gate ### Confirm a failure mode Gates come from failure modes in [Issues and failure modes](/guides/issues-and-failure-modes) — a named, defined pattern of failure with real evidence sessions behind it. ### Generate the eval On the mode's Issue card use **Generate eval**, or call `POST /flywheel/failure-modes/{id}/generate-eval`. In one step Neens: - builds a **dataset** from the mode's evidence — its exemplar sessions, every session the classifier tagged with the mode, and the members of any linked failure cluster (de-duplicated); - drafts an **LLM judge** whose rubric scores the failure as **absent**: a *high* score means the agent did *not* exhibit it (the mode's definition is folded into the criteria); - parks a **human-gated deployment** for that judge — created *disabled*, so nothing runs until an expert has reviewed the draft and enabled it; - registers the **gate** binding mode ↔ judge ↔ dataset ↔ deployment, active with an empty baseline. The response returns all the created ids (`gateId`, `judgeId`, `datasetId`, `deploymentId`, `datasetItemCount`). ### Review and enable the judge The drafted judge is a starting point, not gospel. Review its rubric on the [Judges](/guides/judges) page and enable its deployment when you're satisfied — until then the gate's detail shows the deployment as disabled. ### Run it Use **Run gate** to fire the first eval run. The first completed run **seeds the baseline pass rate**; every later run updates the latest pass rate and is compared against that baseline. **A gate needs evidence.** If the failure mode has no classified sessions, linked cluster, or exemplars yet, generation is refused with `422` — Neens won't create a gate bound to an empty dataset that would pass forever. Classify sessions into the mode or link a cluster first. ## What a gate tracks Each gate row carries: | Field | Meaning | | --- | --- | | `name` | Derived from the failure mode (`FM: `) | | `status` | `active` or `paused` | | `baselinePassRate` | Seeded by the first completed run; the reference every later run is compared to | | `lastPassRate` | Pass rate of the most recent completed run | | `lastRunId` / `lastRunAt` | The most recent run and when it fired | | `failureModeId` / `judgeId` / `datasetId` / `deploymentId` | The provenance chain the gate binds together | A run's **pass rate** is computed from its scored items: an item passes when its score meets the judge's threshold (0.7 when the judge doesn't set one). Since the judge scores the failure as *absent*, a **drop in pass rate means the failure is coming back**. The **Eval Gates** page lists every gate with its source failure mode, status, baseline, and latest pass rate; you can filter by status or failure mode, search by name, and sort any column. Per-row actions: **Run gate**, and **Pause** / **Activate**. ## The gate detail drawer Click a gate to open its detail drawer (`GET /flywheel/gates/{id}`), which explains the gate on one screen: - **Provenance** — the **What this gate checks** section shows the source failure mode (name, definition, severity), the drafted judge (name, type), the dataset (name, item count), and the deployment's status and trigger — so you always know *why this gate exists* and whether its judge has been enabled yet. - **Baseline vs. last** — the baseline pass rate side by side with the most recent run's pass rate, plus a green/red delta badge. This is the at-a-glance regression signal. - **Recent runs** — the last 10 eval runs, newest first, each with its status, progress, computed pass rate, and who (or what) triggered it. From the drawer you can **Run gate**, **Pause**/**Activate**, and re-baseline: `PATCH` the gate with a new `baselinePassRate` when you've intentionally changed the bar (for example after a big fix landed and you want future runs compared to the new normal). ## Using gates to guard releases A practical rhythm: 1. **After shipping a fix** — run the gate for that failure to confirm the fix holds against the original failing evidence. A remediation reaching `verified` and its gate staying green are the two halves of "this failure is handled" (see [Remediations](/guides/remediations)). 2. **As part of release checks** — run your active gates when a new agent version is about to ship. A pass-rate drop below baseline shows up as a failing gate instead of a silent production incident. 3. **When a gate drops** — check [What changed](/guides/what-changed): recording deploy events lets you correlate the drop with the prompt/model/tool change that landed just before it, turning a failing gate into "this deploy did it" rather than a mystery. Pause gates you're not ready to enforce; they keep their history and can be re-activated any time. ### Gates vs. pre-prod evals Both guard releases, at different granularities: - An **eval gate** guards *one known failure* — a focused regression test over that failure's own evidence dataset, with a per-gate baseline. - A [pre-prod evaluation](/guides/preprod-evals) scores a *whole candidate version* against a golden dataset and surfaces regressions versus a baseline run or your production window. Use gates for "this specific bug must never return," and pre-prod evals for "this release, as a whole, is not worse." Both are powered by the same [judges](/guides/judges) machinery, so a judge you refine in one place improves the other. API reference | Endpoint | Purpose | | --- | --- | | `POST /flywheel/failure-modes/{id}/generate-eval` | Materialize dataset + judge + disabled deployment + gate from a failure mode (`404` unknown mode, `422` no evidence) | | `GET /flywheel/gates` | List gates; filters `status`, `failureModeId`, `name`; sortable by name, failure mode, status, baseline, last pass rate, last run, created | | `GET /flywheel/gates/{id}` | Rich detail: provenance, deployment status, and the 10 most recent runs with pass rates | | `PATCH /flywheel/gates/{id}` | Update `status` (`active` \| `paused`) and/or `baselinePassRate` | | `POST /flywheel/gates/{id}/run` | Fire an eval run against the gate's dataset; the first completed run seeds the baseline | ## Related - [Issues and failure modes](/guides/issues-and-failure-modes) — the confirmed failures gates are built from - [Remediations](/guides/remediations) — fix the failure the gate guards - [Judges](/guides/judges) — review and tune the drafted judge - [Pre-prod evaluations](/guides/preprod-evals) — whole-version release gating - [What changed](/guides/what-changed) — correlate a gate drop with a deploy - [Insights](/guides/insights) — automatic regression detection across metrics ================================================================================ # What changed Source: /docs/guides/what-changed/ ================================================================================ # What changed When quality drops, the first question is always *"what did we ship just before this?"* Neens answers it with a **deploy-event ledger**: record every prompt, model, tool, or config change your agent goes through, and Neens correlates score regressions and failure spikes to the events that landed in the window just before — closest change first. It's a small habit with a big payoff: a failing [eval gate](/guides/eval-gates) or a regression [insight](/guides/insights) stops being a mystery and becomes "pass rate dropped right after prompt v7." ## At a glance | | | | --- | --- | | **Where** | The **What changed** panel on a remediation's detail, plus the correlation API for any timestamp | | **Key API** | `POST /deploy-events`, `GET /deploy-events`, `GET /deploy-events/what-changed` | | **Needs** | Nothing but the events themselves — no LLM involved; correlation is a pure time-window computation | | **Scope** | Agent-scoped; events can additionally be tagged with an agent name | ## Record deploy events Create an event whenever something about your agent changes — ideally automatically from CI, so the ledger is complete without anyone remembering to write it. ### The event shape `POST /deploy-events` takes: | Field | Required | Meaning | | --- | --- | --- | | `kind` | yes | One of `prompt`, `model`, `tool`, `param`, `config`, `judge` — rejected with `422` otherwise | | `title` | yes | Human-readable summary, e.g. `"Prompt v7: stricter grounding rules"` | | `agentName` | no | Which agent this change applies to (used to filter correlations) | | `oldValue` / `newValue` | no | The before/after value — a model id, a prompt version, a parameter | | `deployedAt` | no | ISO-8601 timestamp of the deploy; defaults to the time of the API call | | `deployedBy` | no | Who or what shipped it, e.g. `"ci"` or a teammate's name | ```bash curl -X POST "https:///deploy-events" \ -H "Authorization: Bearer $NEENS_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "kind": "model", "title": "Switched support agent to new model version", "agentName": "support-agent", "oldValue": "model-v1", "newValue": "model-v2", "deployedBy": "ci" }' ``` Add a step to your deploy workflow so every release is tagged with the commit that shipped it: ```yaml - name: Record deploy in Neens env: NEENS_BASE_URL: https:// NEENS_API_KEY: ${{ secrets.NEENS_API_KEY }} run: | curl -sf -X POST "$NEENS_BASE_URL/deploy-events" \ -H "Authorization: Bearer $NEENS_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "kind": "config", "title": "Deploy '"$GITHUB_SHA"'", "agentName": "support-agent", "newValue": "'"$GITHUB_SHA"'", "deployedBy": "ci" }' ``` Record *every* kind of change, not just code deploys — a prompt tweak, a model swap, a new tool, a temperature change, or an updated judge can each move your metrics. The `kind` field is what makes the correlation readable later. Browse the ledger with `GET /deploy-events` (newest first; filter by `agentName` and `since`). ## The correlation `GET /deploy-events/what-changed` answers "what changed before this moment?" for any timestamp: ```bash curl "https:///deploy-events/what-changed?at=2026-07-15T09:00:00Z&agentName=support-agent" \ -H "Authorization: Bearer $NEENS_API_KEY" ``` | Parameter | Required | Meaning | | --- | --- | --- | | `at` | yes | The ISO-8601 reference instant — when the regression / anomaly surfaced | | `agentName` | no | Restrict to one agent's events | | `windowHours` | no | Look-back window; default `168` (7 days) | The response lists the events deployed within the window *before* `at`, **closest first**, each annotated with `hoursBefore` — how many hours before the reference instant it shipped. Events after `at` are excluded (they can't have caused it), and events with missing timestamps are skipped rather than erroring. ```json { "at": "2026-07-15T09:00:00Z", "windowHours": 168, "events": [ {"kind": "prompt", "title": "Prompt v7: stricter grounding rules", "hoursBefore": 6.5, "...": "..."}, {"kind": "model", "title": "Switched support agent to new model version", "hoursBefore": 30.25, "...": "..."} ] } ``` The correlation is deliberately honest about what it is: a *time-window* correlation, not proof of causation. Its job is to shrink "anything could have caused this" down to one or two candidate changes you can actually investigate. ## Where it surfaces in the product - **Remediations** — every fix's detail panel includes a **What changed** section listing the deploy events recorded for the failure's agent in the 7 days before the fix was proposed, each with its "N h before" distance. If a failure appeared right after a prompt change, you'll see it next to the proposed fix — and if a *fixed* failure regresses, the same panel points at the change that likely broke it again. See [Remediations](/guides/remediations). - **Eval gates** — when a gate's pass rate drops below its baseline, query the correlation at the run's timestamp to see what shipped just before. See [Eval gates](/guides/eval-gates). - **Release flow** — pre-prod evaluations compare candidate versions explicitly; the deploy ledger complements them by covering everything that ships *outside* that gated path. See [Pre-prod evaluations](/guides/preprod-evals). ## Practical workflow ### Tag deploys from CI Add the `POST /deploy-events` call to every pipeline that changes your agent — code deploys, prompt updates, model swaps, config pushes. Use a consistent `agentName` (matching the agent name your traces report) so correlations stay tight. ### Watch your quality signals Let [judges](/guides/judges) score production continuously, keep your [eval gates](/guides/eval-gates) active, and let [insight detectors](/guides/insights) watch for score regressions and failure-volume anomalies. ### When something drops, ask what changed Open the affected remediation's **What changed** panel, or hit `GET /deploy-events/what-changed?at=`. The closest-first ordering usually puts the culprit at the top of the list. ### Close the loop Fix forward (or roll back), record *that* change as a deploy event too, and confirm recovery with a gate run. The failure, the fix, and both deploys now sit on one timeline. ## Related - [Eval gates](/guides/eval-gates) — the regression guards that make a drop visible - [Remediations](/guides/remediations) — fixes with deploy correlation built into their detail - [Insights](/guides/insights) — automatic anomaly and regression detection - [Issues and failure modes](/guides/issues-and-failure-modes) — the failure taxonomy behind it all - [Judges](/guides/judges) — the continuous scoring that produces the signals worth correlating ================================================================================ # Pre-prod evaluations Source: /docs/guides/preprod-evals/ ================================================================================ # Pre-prod evaluations A **pre-prod evaluation** runs a *candidate* version of your agent against a frozen [golden dataset](/guides/datasets) **before** it ships, scores the results with the same [judges](/guides/judges) you run in production, and compares them to a baseline — a previous candidate or your live production quality. Items that **passed the baseline but fail the candidate** are regressions, and the run's gate turns them into a pass/fail verdict your CI can block on. Where [eval gates](/guides/eval-gates) guard a *single known failure*, a pre-prod eval sweeps a whole candidate version across your curated set — so a quality drop is caught before users see it, not after. ## At a glance | | | | --- | --- | | **Where** | The **Pre-prod Evals** page (comparison view under **Compare Versions**) | | **Key API routes** | `POST /preprod-evals`, `GET /preprod-evals/{id}`, `POST /preprod-evals/{id}/run`, `GET /preprod-evals/{id}/comparison`, `GET /preprod-evals/{id}/metrics`, `GET /preprod-evals/{id}/gate` | | **Auth** | The same **`nk_live_` agent API key** you use to send traces authenticates the whole flow — no separate login. | | **Needs** | A dataset with a **golden** version ([Datasets](/guides/datasets)); enabled judges ([Judges](/guides/judges)); to let Neens call your agent, an **Agent endpoint** connection | | **Scope** | Agent-scoped, like all your data; runs execute on a dedicated worker fleet so a big sweep never slows live scoring | | **Produces** | Per-item scores tagged `source: preprod` (their own score type — filterable out of production widgets by **Score source**) and attributed to the model frozen on the run, a candidate-vs-baseline comparison, a gate verdict, and a **Pre-prod regression** [insight](/guides/insights) when something regressed | ## Two ways to run Both ways produce the same run, comparison, and gate — they differ only in **who invokes your agent**: - **Neens calls my agent.** You register your agent's HTTP endpoint once; Neens calls it with every golden prompt, records each response as a session, and scores it. No harness, no exporter wiring — the lowest-effort path. - **I run my agent.** You drive your agent over the golden prompts yourself — **by hand or wired into your CI pipeline** — and the traces it emits carry correlation attributes that link them back to the run. Use this when your agent already runs in a harness, or isn't reachable from Neens. The `neens eval run` CLI does the busywork. ## Authentication — one key, set once Every pre-prod route accepts the **`nk_live_` agent API key** you already mint to send traces (**Settings → API keys**), sent as `Authorization: Bearer `. It resolves to a company + agent on its own, so: - No `nk_sess_` session/login token. - No `--project-id` — the key already scopes the run to an agent (pass one only to override). For both the CLI and CI, set two environment variables and you're done: ```bash ``` The **same** key drives create, list, start, run, cancel, items, gate, and the comparison endpoints. In CI, store it once as a secret (e.g. `NEENS_API_KEY`) and reuse it for every run. ## Prerequisites - **A golden dataset.** A dataset version marked **golden** is the reference set — each item carries the frozen prompt (and optional expected answer) the candidate is asked to run. Build one from human-labeled failures or a filter — see [Datasets](/guides/datasets) and [Annotations & review](/guides/annotations-and-review). - **Judges.** A run defaults to your agent's **Primary Score** — the overall quality gate — and you can pick any other enabled judge(s) to score the replay. LLM-prompt, managed, **and composite** judges (like Primary Score) all participate. A classifier judge only assigns a category label rather than a pass/fail score, so it isn't offered here. - **If you run your agent:** an agent that exports OpenTelemetry traces to Neens ([Send traces](/guides/send-traces)). **If Neens calls your agent:** an HTTP endpoint Neens can reach. ## Create a run In **Pre-prod Evals → New pre-prod eval**, everything except the version label has a smart default: | Field | Purpose | | --- | --- | | **Run name** | A human name for the run (e.g. `Nightly regression check`). | | **Version label** | Free-form label for the *candidate agent version* (a git SHA, a branch, `v2.1`). Recorded on every captured trace so candidates stay distinguishable. | | **How does the run happen?** | **Neens calls my agent** or **I run my agent**. | | **Agent endpoint** | Only when Neens calls your agent — the registered **Agent endpoint** connection it calls. | | **Golden dataset** | The dataset whose frozen prompts get replayed. | | **Version** | **Auto — golden** uses the dataset's golden version; pin a specific version to replay an older frozen snapshot. | | **Baseline** | **Compare to production** (a production time window, default last 7 days) or **Compare to a previous run**. | | **Max regressions** | Gate threshold: fail when more than this many items regress. Blank = not enforced. | | **Min pass rate (%)** | Gate threshold: fail when the candidate's pass rate falls below this floor. Blank = not enforced. | | **Judges** | Which judges score the replay. Defaults to **Primary Score**; pick any enabled scoreable judge(s). | The API accepts an explicit `judge_deployment_ids` list; omit it to default to Primary Score. Only scoreable judge types (LLM-prompt, managed, composite) are accepted — a classifier-only selection is rejected. **Version label ≠ dataset version.** The **version label** names your *candidate agent*; the **dataset version** selects the *golden prompts* it runs against. They are independent knobs. Creating the run **snapshots the chosen version's items immediately**, so the exact prompt set is frozen for the run even if the dataset is edited later. A dataset with no golden version is rejected unless you pin a version explicitly. Via the API: ```bash curl -sf -X POST "$NEENS_BASE_URL/preprod-evals" \ -H "Authorization: Bearer $NEENS_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "name": "release-candidate", "dataset_id": "ds_golden", "version_label": "'"$GIT_SHA"'", "baseline": {"kind": "prod_window", "range": "7d"}, "gate": {"max_regressions": 0, "min_pass_rate": 0.9} }' ``` Optional fields: `dataset_version_id` (pin a snapshot), `runner_mode` (`"runner"` = you run the agent, the default, or `"push"` = Neens calls your agent, with `agent_connection_id`), `judge_deployment_ids`, and `baseline: {"kind": "preprod_run", "value": ""}`. The `gate` object can also carry a **cost budget** — see [the server-side cost gate](#the-server-side-cost-gate). ### The candidate's model is frozen at create time When Neens calls your agent, the run's **Agent endpoint** connection declares a model. Neens copies that model (and the connection's provider) onto the run when the run is created, and never re-reads it — so *"Haiku passed at 94% on the June release"* keeps meaning the same thing after somebody edits the connection in July. Every score the run produces is attributed to that frozen model, which is what makes a pre-prod run the cleanest [model comparison](/guides/model-comparison) available: same golden prompts, same code, one model changed. The run header shows it as a **Model** badge — or an explicit *Model not recorded* rather than a blank that would read as a model name. The run detail carries the same values: ```json { "id": "ppr_…", "versionLabel": "pr-482", "runnerMode": "push", "model": "claude-haiku-4-5", "modelProvider": "agent_http" } ``` `model` is `null` when there is nothing honest to record — most often a run you drive yourself, where Neens does not know what served the replay, or a connection that declares no model. Those scores fall back to the model recorded on the captured trace's own spans, and are reported as **Unknown** if the trace recorded none. Nothing is guessed at. `modelProvider` is the *connection's* provider — for an agent endpoint that is `agent_http`, the transport Neens dialled, not an LLM vendor (which an HTTP endpoint does not reveal). ## I run my agent: `neens eval run` The **`neens-eval`** SDK (dependency-free) drives the whole flow for you: it fetches the frozen prompts, invokes **your** agent command once per item with the correlation attributes injected (via `OTEL_RESOURCE_ATTRIBUTES`), starts the run, polls it to a verdict, and **exits with the gate result**. Run it by hand on your laptop or wire it into your CI pipeline — the command is the same. It ships in **two flavors** with identical flags and exit-code contract — pick the one that matches your stack: ```bash pip install neens-eval # installs the `neens` console script # or, from a Neens source checkout: pip install ./sdk ``` Python 3.10+, stdlib only. Invocation is `neens eval run …`. ```bash npm install --save-dev @neens/eval # installs the `neens-eval` bin ``` Node 18+ (uses the platform `fetch`), zero runtime dependencies. Invocation is `npx neens-eval run …` — the same flags as the Python CLI. ### Ad-hoc: create and run in one command With `NEENS_BASE_URL` and `NEENS_API_KEY` exported (see [Authentication](#authentication--one-key-set-once)), `--create` makes a run against your dataset's golden version and drives it in a single command: ```bash neens eval run --create \ --dataset ds_golden \ --version-label local-test \ -- python -m myagent ``` ```bash npx neens-eval run --create \ --dataset ds_golden \ --version-label local-test \ -- node dist/myagent.js ``` Everything after `--` is **your** agent command; it's invoked once per frozen prompt with the prompt piped to stdin (and available as `$NEENS_ITEM_INPUT`). No `--project-id`, no session token — the `nk_live_` key scopes and authenticates the whole run. To drive a run you already created, swap `--create --dataset …` for `--run-id `. Flags for neens eval run | Flag | Purpose | | --- | --- | | `--run-id ID` | Drive an existing run. Mutually exclusive with `--create`. | | `--create` | Create a run first (requires `--dataset` and `--version-label`). | | `--dataset ID` | Golden dataset to snapshot (with `--create`). | | `--dataset-version-id ID` | Pin an explicit dataset version (defaults to the golden version). | | `--version-label LABEL` | Candidate version label; also tags every emitted trace. | | `--name NAME` | Run name (defaults to the version label). | | `--baseline SPEC` | `prod`, `prod:` (e.g. `prod:30d`), or `run:`. | | `--max-regressions N` | Gate: fail if regressions exceed `N`. | | `--min-pass-rate R` | Gate: fail if the pass rate is below `R` (0..1). | | `--gate-policy PATH` | A [gate-as-code](#gate-as-code-a-richer-policy-file) policy file; env `NEENS_GATE_POLICY`. The Python SDK reads JSON (or YAML with PyYAML); the TypeScript SDK reads **JSON only**. | | `--max-cost-delta-pct PCT` | Inject a **block** rule: fail if candidate avg cost rises more than `PCT`% vs baseline. | | `--max-latency-delta-pct PCT` | Inject a **block** rule: fail if candidate avg latency rises more than `PCT`% vs baseline. | | `--min-avg-score R` | Inject a **block** rule: fail if the run's aggregate avg score is below `R` (0..1). | | `--max-abs-failures N` | Inject a **block** rule: fail if absolute candidate failures exceed `N`. | | `--base-url URL` | Neens origin (env `NEENS_BASE_URL`). | | `--api-key KEY` | `nk_live_…` agent key (env `NEENS_API_KEY`). | | `--project-id ID` | Optional `X-Neens-Project-Id` override (env `NEENS_PROJECT_ID`; usually unneeded). | | `--timeout SECONDS` | Per-item command timeout. | | `--concurrency N` | Parallel item invocations (default 1). | | `--poll-interval SECONDS` | Status poll cadence (default 5). | | `--poll-timeout SECONDS` | Overall scoring budget (default 1800). | | `--no-gate` | Report the gate but always exit 0 (don't fail CI). | | `--json` | Emit a machine-readable result to stdout (human logs go to stderr). | ### In CI: a GitHub Actions gate Store your Neens origin and agent key as repo secrets, then add one step. The exit code is the gate verdict, so the step fails the build automatically when the candidate regresses. Using `${{ github.sha }}` as the version label ties the run to the commit under test: ```yaml - name: Pre-prod eval gate env: NEENS_BASE_URL: ${{ secrets.NEENS_BASE_URL }} NEENS_API_KEY: ${{ secrets.NEENS_API_KEY }} # a nk_live_… agent key # Point YOUR agent's OTel exporter at the Neens receiver: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT: ${{ secrets.NEENS_BASE_URL }}/v1/traces run: | neens eval run --create \ --dataset ds_golden \ --version-label "${{ github.sha }}" \ --baseline prod:7d \ --min-pass-rate 0.9 --max-regressions 0 \ -- python -m myagent ``` Because `--create` self-creates the run each time, the only sticky identifiers in CI are the **dataset id** and the **`NEENS_API_KEY`** secret — there's no run id to copy between builds. ### The exit-code gate contract The process exit code **is** the gate verdict, so a plain CI step blocks a bad release: | Exit code | Meaning | | --- | --- | | `0` | Gate passed (or `--no-gate` was set) | | `1` | Gate failed — regressions or pass rate breached a threshold | | `2` | Run error — no items, backend error, poll timeout, or bad arguments | `--no-gate` still fetches and prints the gate but always exits `0` (report without blocking CI). `--json` emits a machine-readable result to **stdout** (human logs go to stderr) for a step to parse. ### Gate-as-code: a richer policy file `--min-pass-rate` and `--max-regressions` are the thresholds the **server** gate owns from the command line — and the run's stored gate can carry a **[cost budget](#the-server-side-cost-gate)** alongside them. On top of those you can commit a **gate policy** — a JSON (or YAML, if PyYAML is installed) file that declares budgets and floors the CLI evaluates **client-side** over the comparison, per-metric rollup, and run-detail data Neens already returns. Point the CLI at it with `--gate-policy PATH` (or set `NEENS_GATE_POLICY`); it works with both `neens eval run` and `neens eval push`. ```json { "gate": { "max_regressions": 0, "min_pass_rate": 0.9 }, "rules": [ { "cost": { "max_avg_usd": 0.05, "max_delta_pct": 20 }, "severity": "warn" }, { "latency": { "max_avg_ms": 3000, "max_delta_pct": 25 }, "severity": "warn" }, { "steps": { "max_avg": 8, "max_delta_pct": 50 }, "severity": "block" }, { "max_abs_failures": 3, "severity": "block" }, { "min_avg_score": 0.8, "severity": "block" }, { "metric": "faithfulness", "min_avg": 0.8, "severity": "block" }, { "metric": "toxicity_safety", "min_avg": 0.9, "severity": "warn" } ], "on_missing_baseline": "warn" } ``` The `gate` block is handed to the server at run-create time, so Neens stays authoritative for `max_regressions`/`min_pass_rate`. The `rules` list is what the CLI evaluates. Each rule carries **exactly one** dimension key plus an optional `severity`: | Rule | Fails when | | --- | --- | | `cost` | `max_avg_usd` — candidate avg cost exceeds the budget; `max_delta` / `max_delta_pct` — cost rose more than that vs the baseline. Costs come from the [model price table](/guides/cost-and-model-pricing); if no session on a side has a priced model the average is `null`, and the rule is reported as **skipped (unpriced)** rather than silently passing. | | `latency` | `max_avg_ms`; `max_delta_ms` / `max_delta_pct` — same, for average latency. | | `steps` | `max_avg`; `max_delta` / `max_delta_pct` — same, for average step count. | | `max_abs_failures` | Absolute candidate failures exceed the cap. | | `max_regressions` | Regressions vs the baseline exceed the cap. | | `min_pass_rate` | The run's pass rate falls below the floor. | | `min_avg_score` | The run's aggregate average score falls below the floor. | | `metric` | The named metric's average score (from the [rollup](#per-metric-rollup)) is below `min_avg` or above `max_avg`. Scores are normalized 0–1 (higher is better), so use `min_avg` even for a safety floor. | **Two severity tiers.** `severity: "block"` (the default) fails the build — a breached block rule sets the CLI exit code to `1` on top of the server gate. `severity: "warn"` is reported in the summary but **never** changes the exit code. **Missing baseline.** `on_missing_baseline` governs baseline-relative rules (any `*_delta` / `*_delta_pct` budget, and `max_regressions`) when the run has no baseline cohort to compare against — `warn` (the default) reports without blocking, `block` honors each rule's own severity, and `pass` skips them entirely. Absolute rules (`max_avg_*`, `max_abs_failures`, `min_pass_rate`, `min_avg_score`, and `metric` floors) always evaluate — they need no baseline. For a quick one-off gate you don't need a file: four convenience flags inject the equivalent **block** rules straight from the command line, overriding any same-dimension rule in the file — `--max-cost-delta-pct`, `--max-latency-delta-pct`, `--min-avg-score`, and `--max-abs-failures`. The `metric` rule reads `GET /preprod-evals/{id}/metrics` — a per-metric rollup of the run's captured, scored sessions. It returns one row per judge metric with its normalized average score (`avgScore`), pass rate (`passRate`), and the number of sessions scored, letting the gate enforce a floor the run's single aggregate score can't (e.g. *faithfulness must stay ≥ 0.8*). A run with nothing scored yet returns an empty list rather than erroring. ### The server-side cost gate A run's **stored gate** — the `gate` object you pass to `POST /preprod-evals`, and the one a [model sweep](/guides/model-sweeps) pins onto every child run — accepts a **`cost` family** alongside `max_regressions` and `min_pass_rate`, evaluated by Neens rather than by the CLI: ```json { "max_regressions": 0, "min_pass_rate": 0.9, "cost": { "max_avg_usd": 0.004, "max_delta_pct": -50 } } ``` The three sub-keys are spelled **exactly** as the SDK's client-side `cost` rule spells them, so one vocabulary covers both sides: | Sub-key | Fails when | | --- | --- | | `max_avg_usd` | The candidate's average cost per captured session exceeds this budget | | `max_delta` | Cost rose more than this many dollars versus the baseline | | `max_delta_pct` | Cost rose more than this percentage versus the baseline. A **negative** threshold is a *required saving* — `-50` means "must come in at least 50% cheaper" | `GET /preprod-evals/{run_id}/gate` returns a `costRules` array whenever the gate declares the family — one row per declared sub-key, each with `rule`, `status`, `observed`, `threshold` and a plain-English `reason`. The key is **absent** (not an empty array) when there is no cost family, so every gate written before this existed returns exactly what it always did. ```json { "passed": false, "reasons": ["cost avg 0.0062 USD exceeds budget 0.004 USD"], "regressions": 0, "passRate": 0.95, "gate": { "max_regressions": 0, "min_pass_rate": 0.9, "cost": { "max_avg_usd": 0.004 } }, "costRules": [ { "rule": "max_avg_usd", "status": "fail", "observed": 0.0062, "threshold": 0.004, "reason": "cost avg 0.0062 USD exceeds budget 0.004 USD" } ] } ``` **An unpriced run is `skipped`, never `pass`.** If no session in the run ran on a model with a price, the average cost is `null` — and reading `null` as *"$0, therefore under budget"* is exactly the bug per-model pricing exists to kill. The rule comes back `status: "skipped"` with a reason naming the unpriced models and pointing at **Settings → [Model pricing](/guides/cost-and-model-pricing)**, and the skip is repeated in `reasons`. Only a `fail` blocks. A gate whose conditions all held but whose cost rule could not be measured says *"all **evaluated** gate conditions satisfied"* — a deliberately weaker sentence than the usual one. A rule needing a baseline (`max_delta`, `max_delta_pct`) is also `skipped` when the run has no baseline cohort, rather than being scored against nothing. **Two places, two evaluators — same vocabulary.** `cost` is spelled `{max_avg_usd, max_delta, max_delta_pct}` wherever you write it, but *where* you write it decides *who* checks it: | Where | Evaluated by | When it runs | | --- | --- | --- | | the policy file's top-level `gate` block (or the run's / sweep's `gate` over the API) | Neens, server-side | `GET /preprod-evals/{id}/gate`, reported under `costRules`; stored with the run | | the policy file's `rules` list | the SDK, client-side | at the end of `neens eval run`, against the comparison payload; supports `severity` and `on_missing_baseline` | Put it in `gate` when the budget should hold for **everyone** who reads the run — including the UI and anyone re-checking it later. Put it in `rules` when it is **this pipeline's** opinion, or when you want `severity: warn`. Writing it in both is fine and simply means both sides check it. ```json filename="neens-gate.json" { "gate": { "min_pass_rate": 0.9, "cost": { "max_delta_pct": -50 } } } ``` That policy is literally *"cheaper and still hits the number"* — a candidate must pass at least 90% of items **and** cost at least 50% less than the baseline — enforced in CI and visible in the product, from one file. ### The correlation contract Neens links each trace your agent emits to a pre-prod item by three attributes, readable from either **span attributes or resource attributes** (both OpenTelemetry and OpenInference conventions work — resource attributes are flattened onto spans at ingest): | Attribute | Required | What it does | | --- | --- | --- | | `neens.eval_run_id` | Yes | Links the trace to the pre-prod run (the run's `id`, e.g. `ppr_…`). | | `neens.dataset_item_id` | Strongly recommended | Pins the trace to the exact frozen prompt. Use the `item_id` returned by `GET /preprod-evals/{id}/items`. | | `neens.version_label` | Optional | Records the candidate version on the trace. | These three keys are a stable wire contract, held fixed across releases by a CI guard, and they are matched at exactly this spelling — an attribute under any other name is stored as ordinary trace metadata and the run stays in `awaiting_traces` with no error. The `neens eval run` CLI injects all three for you via `OTEL_RESOURCE_ATTRIBUTES` — you don't touch your agent code. If you drive the run without the CLI, tag them yourself: The simplest way to tag every span an agent process emits is a resource attribute: ```bash python my_agent.py # exporter pointed at the Neens /v1/traces as usual ``` Or set them as span attributes in code: ```python with tracer.start_as_current_span("agent.run") as span: span.set_attribute("neens.eval_run_id", "ppr_1234") span.set_attribute("neens.dataset_item_id", item_id) span.set_attribute("neens.version_label", "release-2.1") ... ``` With `POST /ingest/raw`, put the attributes on a span's `attributes` object: ```json { "session": { "id": "trace-eval-001", "agent_name": "my-agent" }, "spans": [ { "id": "span-1", "name": "agent.run", "kind": "agent", "input": "", "output": "", "attributes": { "neens.eval_run_id": "ppr_1234", "neens.dataset_item_id": "", "neens.version_label": "release-2.1" } } ] } ``` `GET /preprod-evals/{run_id}/items` returns the frozen prompt list to drive your agent with: ```json { "items": [ { "item_id": "…", "input": "", "expected_output": "…" } ] } ``` How linking behaves, exactly: - **With `neens.dataset_item_id`:** the trace links *only* to that item. If the id matches no still-pending item (a duplicate replay of an already-captured prompt, or a typo), the trace is **not** linked to some other prompt — mislinking would corrupt that prompt's score. - **Without it:** the trace links to the first still-captured-less item in frozen order. Fine for a strictly sequential harness; tag the item id if you run prompts concurrently or retry. - A trace whose run id belongs to another agent, or to a cancelled run, is ignored. The first captured trace moves the run from `awaiting_traces` to `running`; once **every** item has a captured trace, scoring is dispatched automatically (exactly once, even if the last traces land simultaneously). ## Neens calls my agent When Neens calls your agent, it HTTP-calls your agent's endpoint once per golden prompt, records each response as a session, links it to its item, and scores the run — no harness on your side. ### Register an agent endpoint An agent endpoint is a connection whose provider is **Agent endpoint** (`agent_http`). Create it under **Settings → Connections** like any [LLM connection](/administration/llm-connections) — its credential is encrypted at rest and it's visible only to the agents the connection is scoped to, so a run in one agent can never invoke another agent's endpoint. | Field | Purpose | | --- | --- | | **Base URL** | The origin Neens calls, e.g. `https://my-agent.internal`. Required. | | **Request shape** | `openai_chat` or `input_json` (see the contract below). | | **Invoke path** | Appended to the base URL. Defaults to `/chat/completions` for `openai_chat`, `/` for `input_json`. | | **Auth** | `bearer` sends the stored credential as `Authorization: Bearer …`; `none` sends no auth header. | | **Output JSONPath** | Optional dotted/bracketed path (e.g. `data.reply`) to pluck the answer when it isn't at the shape's default location. | | **Model** | `openai_chat` only — the model id sent in the request body. | ### Know the request/response contract Any OpenAI-compatible chat-completions endpoint works unmodified. Neens `POST`s: ```json { "model": "your-model", "messages": [{ "role": "user", "content": "" }] } ``` and reads the answer from `choices[0].message.content` (falling back to `choices[0].message.reasoning_content` for gateways that route answers there, or your **Output JSONPath** if set). A minimal JSON endpoint. Neens `POST`s the prompt plus the run's correlation ids: ```json { "input": "", "eval_run_id": "ppr_…", "dataset_item_id": "…", "version_label": "agent-v2.3" } ``` and reads the answer from the top-level `"output"` key (aliases `response`, `answer`, and `result` are also accepted, or your **Output JSONPath** if set): ```json { "output": "" } ``` Either way, if your endpoint responds with a full **OpenTelemetry** (`resourceSpans`) or **OpenInference** (`spans`) trace body instead of a plain answer, Neens parses and records the real multi-span run faithfully. Otherwise it synthesizes a minimal single-span session from the prompt and answer. ### Create and run Create the run with **Neens calls my agent** and your endpoint selected, then click **Run now**. Via the API, `POST /preprod-evals/{id}/run` kicks it off (this trigger only applies when Neens calls your agent — when *you* run the agent there's nothing to trigger; the run captures traces on ingest). Neens invokes the endpoint per frozen prompt, captures and links each response, then scores automatically. Or drive the whole thing from the CLI with `neens eval push` (create + trigger + gate, same exit codes), pointing `--agent-connection` at the registered endpoint: ```bash neens eval push \ --dataset ds_golden --version-label "$GIT_SHA" \ --agent-connection "$AGENT_CONNECTION_ID" \ --min-pass-rate 0.9 --max-regressions 0 ``` ### Guardrails on the outbound call When Neens calls an endpoint you own, every call is bounded and every failure is honest — Neens **never fabricates an output or a score**: - **Per-call timeout** — **60 s**. A slow or hung endpoint errors *that one item*; the rest of the run continues. - **Prompt truncation** — the prompt sent to your endpoint is truncated at **200,000** characters, so a pathological golden item can't blow up the request body. - **Response cap** — **8 MiB**; an oversized body errors the item instead of exhausting worker memory. - **Per-item failures stay per-item** — a timeout, connection failure, non-2xx status, empty body, or a reply with no answer at the expected location marks that item `error` with the reason recorded. If *every* item fails (e.g. the endpoint was down), the run is marked `failed` — never "completed" with nothing scored. **The endpoint must be reachable from the public internet.** Neens calls the URL you register from its own servers, guarded against SSRF: only `http`/`https` URLs are allowed, redirects are never followed, and any host that resolves to a private, loopback, or link-local address (cloud metadata IPs, `localhost`, VPC ranges, …) is **refused**. Register a publicly reachable endpoint — an agent that only lives on an internal address won't be callable this way; drive the run yourself instead (**I run my agent**) and emit traces to Neens. ## How scoring works When a run's items are captured (either way of running), the dedicated pre-prod worker scores each captured session with every judge on the run: 1. Each judge scores the session and its verdict is persisted as a normal score row — but tagged **`source: preprod`**, so pre-prod results appear in the [score catalogue](/guides/scores) as their own score type (a separate card, never merged into the production one) and on the captured trace. The tag is also what lets any dashboard widget keep them out: slice or filter by **Score source** (`score_source`) on the score-grain [measures](/guides/metrics). A widget that doesn't will count pre-prod scores alongside production ones. 2. The item's score is the **mean** of its judge scores; the item **passes** when that mean is at or above **0.7**. 3. An item whose judges all errored is marked `error` (with the reason) — never silently scored. Then the run's baseline is resolved and each item's `baseline_passed` is stamped: - **Previous pre-prod run** (`preprod_run`) — the prior run's verdict for the *same frozen prompt* (both runs snapshot the same golden version, so the prompt id is a stable join key). - **Production window** (`prod_window`) — production scores (any source except `preprod`) over the window, matched to each golden item by **input text**: a prod session whose input equals the item's frozen prompt supplies the verdict (its newest score wins; pass = a `pass` label, or score ≥ 0.7). Items whose prompt never ran in production get no baseline verdict — they can't regress, but they still count toward the pass rate. A **regression** is an item where the baseline **passed** and the candidate **failed**. Any regressions put the run in `completed_with_regressions` and emit a **Pre-prod regression** [insight](/guides/insights) — so a blocked release shows up in your feeds (and notification sinks) with the run attached, not just as a red CI check. ## Reviewing a run Open a run on **Pre-prod Evals** to see: - **Per-item results** — each frozen prompt's status, score, pass/fail, baseline verdict, and captured trace (click through to the full trace), plus the recorded error for failed items. - **The comparison** (`GET /preprod-evals/{id}/comparison`) — a per-item pass/fail matrix, the **regression set**, **new passes** (items the candidate fixed), and average **cost / latency / step-count deltas** between the candidate's captured sessions and the baseline cohort. Each row also carries the candidate's **agent output** and the item's **expected output**. Average cost covers the **priced** sessions only; if a side has no priced session the cost delta is `null` rather than a comparison against a made-up rate — see [Cost & model pricing](/guides/cost-and-model-pricing). - **The gate** (`GET /preprod-evals/{id}/gate`) — pass/fail with reasons. The gate never reports green before the candidate has actually been evaluated: a run that is `failed` or `cancelled`, still in progress, or finished with **zero scored items** fails the gate outright. Only then are the thresholds checked — **max regressions** and **min pass rate**. ### Comparing multiple versions **Compare Versions** lines up N runs that snapshot the *same* golden version side by side — one row per prompt, a cell per version, with per-version pass rate, cost, latency, and step count. Pick any run as the baseline to highlight which versions regressed against it. Via the API: `GET /preprod-compare?runs=ppr_a,ppr_b,ppr_c&baseline=ppr_a` (mixing golden versions returns `422`). Expand a prompt row to read each run's **agent output** (the real final answer that version produced) alongside the item's **expected output**, so a red cell is "here's what it actually said" — not just a failed check. A run with nothing captured for that prompt shows *no output captured*. For any single prompt, the **trajectory diff** (`GET /preprod-compare/trajectory?run=…&baseline=…&item=…`) aligns the two versions' step sequences and marks each step **added / removed / changed / same** — turning "v3 regressed on this prompt" into "v3 stopped calling `lookup_order`." ## Run lifecycle | Status | Meaning | | --- | --- | | `awaiting_traces` | Created; items snapshotted; waiting for captures (or for **Run now** when Neens calls your agent). | | `running` | At least one trace captured (you run the agent), or the endpoint sweep is in flight (Neens calls it). | | `scoring` | All items captured; judges are scoring. | | `completed` | Scored, no regressions vs the baseline. Terminal. | | `completed_with_regressions` | Scored, one or more regressions. Terminal. | | `failed` | Nothing could be scored (e.g. every endpoint call errored, or scoring crashed). Terminal. | | `cancelled` | Cancelled via `POST /preprod-evals/{id}/cancel`. Terminal — an in-flight endpoint sweep stops calling out, and a cancelled run is never resurrected by late traces or scoring. | Item statuses: `pending` → `captured` → `scored`, or `error` (invocation or scoring failed — the reason is recorded on the item). ## Troubleshooting | Symptom | Cause | Fix | | --- | --- | --- | | `403` from `neens eval run` / the API | The key isn't a `nk_live_` agent key, or it's scoped to a different agent | Use the agent's **`nk_live_` API key** (Settings → API keys); set it as `NEENS_API_KEY`. No session token or `--project-id` is needed | | Run stuck in `awaiting_traces` (you run the agent) | Not every item has a matching trace — scoring starts only when the run is *fully* captured | Check the tagged `neens.eval_run_id` / `neens.dataset_item_id` values against `GET /preprod-evals/{id}/items`; a mistagged item id is deliberately **not** linked to another prompt | | An item errors with "url is not allowed" (Neens calls your agent) | The endpoint resolves to a private/loopback address Neens can't call | Register a publicly reachable endpoint, or drive the run yourself (**I run my agent**) and emit traces to Neens | | Run marked `failed` (Neens calls your agent) | Every item's endpoint call failed (down, timing out, or returning an unreadable body) | Fix the endpoint or its **Request shape** / **Output JSONPath**, then create a new run | | Some items have no baseline verdict (`prod_window`) | Those golden prompts never ran (or were never scored) in production during the window | Expected — widen the window, or baseline against a previous pre-prod run | | Gate fails with "no items were scored" | Judges couldn't run — e.g. no enabled LLM-prompt judges, or an unusable [LLM connection](/administration/llm-connections) | Enable at least one LLM-prompt judge and verify its connection, then re-run | | Pre-prod scores showing up in a production dashboard | Pre-prod scores carry `source: preprod`, but a score-grain widget counts every source unless you say otherwise | Filter or slice the widget by **Score source** (`score_source`) — see [Metrics catalogue](/guides/metrics) and [Model comparison](/guides/model-comparison) | | A run's `model` is `null` | The run has no agent connection (you ran the agent yourself), or the connection declares no model | Expected — Neens never invents one. Its scores fall back to the captured trace's own spans; see [Model comparison](/guides/model-comparison) | ================================================================================ # Model sweeps Source: /docs/guides/model-sweeps/ ================================================================================ # Model sweeps A **model sweep** replays **one frozen golden dataset** against **several models at once** — each model behind its own agent endpoint — under **identical conditions**, and requires every model to pass **k independent runs** before it is called a pass. It answers the question a single [pre-prod evaluation](/guides/preprod-evals) cannot: *which of these models is good enough for this job, and what does each one cost?* Where a pre-prod eval asks "is this candidate version safe to ship", a sweep asks "which model should this agent run on" — and it refuses to answer at all if the comparison it would draw is not a fair one. ## At a glance | | | | --- | --- | | **Where** | The **Agent model** tab on the **Cost & Quality** page | | **Key API routes** | `POST /model-sweeps/preview`, `POST /model-sweeps`, `GET /model-sweeps`, `GET /model-sweeps/{id}`, `POST /model-sweeps/{id}/cancel`, `GET /model-sweeps/{id}/comparison` | | **Auth** | The same **`nk_live_` agent API key** that drives pre-prod evals — reads need `READ`, launching a sweep needs the same authority as scheduling a pre-prod run | | **Needs** | A dataset with a **golden** version, one **Agent endpoint** connection per model, and enabled [judges](/guides/judges) | | **Produces** | Per-arm pass^k (`greens/k`), per-arm run detail, a pre-flight cost estimate, a [decidable comparison](/guides/sweep-decisions), and — when the arms did not run under identical conditions — a **void** sweep that renders no comparison | | **Runs on** | The dedicated pre-prod worker fleet, so an N×k sweep never starves live scoring | A sweep **launches ordinary pre-prod runs**. Every child run is a normal pre-prod evaluation you can open, inspect, and re-read item by item — the sweep adds the fair-comparison rules and the roll-up on top, it does not invent a second kind of evaluation. **Model sweeps moved twice.** It used to have its own sidebar entry, then the **Sweeps** tab on Model Bench. It is now the evidence behind the **Agent model** tab on [**Cost & Quality**](/guides/cost-and-quality), which reads the newest finished sweep per agent version and states the verdict. An old `/model-sweeps` link still works: it opens that tab, and an individual sweep (`/model-sweeps/{id}` and its comparison) is unchanged. ## When to reach for one - **Model selection.** You are deciding between a frontier model, a cheaper hosted model and a self-hosted one for the same agent. Run all three against the same golden set and read the pass rates side by side. - **A vendor price or model change.** A new model version lands, or a price moves. Re-run last month's sweep and see whether the answer changed. - **Monthly in CI.** A sweep is cheap enough to schedule and expensive enough to want a budget ceiling on. The CLI's preview-then-launch flow is built for exactly this (see [Run a sweep from CI](#run-a-sweep-from-ci)). - **Against your own failures, not a public benchmark.** The strongest golden set for this decision is a [stress-test suite](/guides/stress-testing) generated from *your* confirmed failure modes. A public benchmark tells you how a model does on somebody else's problems; a suite built from last quarter's incidents tells you whether it survives yours. Pass the suite id when you create the sweep and Neens uses its frozen synthetic dataset version. ## Arms, k, and what stays fixed A sweep is made of **arms**. One arm = one model, reached through one **Agent endpoint** connection. Arms differ in **exactly one thing** — the endpoint (and therefore the model). Every other input is pinned once on the sweep and reused verbatim for every child run: | Pinned for the whole sweep | Why | | --- | --- | | **Dataset version** | The same frozen prompts, in the same frozen state, for every arm | | **Judge set** | The same graders, at the same deployed versions | | **k** | The same number of repeats per arm | | **Version label** | The same candidate label recorded on every captured trace | | **Gate policy** | The same thresholds applied to every child run | Two arms may not point at the **same** endpoint connection: that is not a comparison, it is the same model twice, and it would render as two independent results that look like corroboration. ## pass^k is per arm — not per sweep A single green run can be luck: a non-deterministic model, a flaky tool call, a judge that graded generously on the day. **pass^k** requires an arm to pass **k independent runs** — all green, zero regressions — before that model is reported as passing. The critical detail is that **k is evaluated per arm, not across the sweep**: > A cheap model that passes once and fails twice **has not hit the number.** It reports > `1/3 green`, `passed: false` — not "passed, with variance". This is the same rule the [eval-verified fix engine](/guides/eval-verified-fix) uses to decide whether a fix may open a pull request, applied to a model choice instead of a patch. `k = 3` is the default and is what defeats a flaky green; a run-to-run difference that a single run would have hidden shows up as an arm that cannot get to `3/3`. An arm with **fewer than k finished runs** has **no verdict at all** — it stays in progress with `passed` unset rather than reporting the greens it has so far as a result. A partially finished arm never borrows the benefit of the doubt. ## Identical conditions, and what `void` means The whole value of a sweep is that the only difference between the arms is the model. If that stops being true, the comparison is worthless — and a worthless comparison that still renders a chart is worse than no chart, because somebody will screenshot it into a decision doc. So Neens pins the controlled variables at create time as a **conditions fingerprint** (the dataset version, the sorted judge set, k, the version label, and the gate policy — deliberately *not* the model, which is the independent variable), and re-derives that fingerprint from each child run's **actual persisted values** when it rolls the sweep up. If any arm's real conditions diverge from the pinned ones, the sweep becomes **`void`**: - The status is **terminal** — a void sweep never recovers into a comparable one. - The detail page shows a prominent banner naming **which arms diverged**, and **suppresses every cross-arm comparison affordance**. No chart, no ranking, no "winner". - A child run that vanished, or was never created, counts as divergence — not as a pass. **A void sweep is not a failure of the models — it is a failure of the experiment.** Read the void reason, fix the thing that drifted (usually a judge redeployed mid-sweep or a dataset version pinned differently on one arm), and run a fresh sweep. Neens deliberately gives you no way to "just show it anyway". ## Cost: estimate first, then decide Every arm multiplies the bill: an arm costs `k × item_count` model calls, and a sweep costs that again for every arm. So the cost pre-flight is a first-class step, not a nicety. ### Preview `POST /model-sweeps/preview` (the **Estimate** step in the launch form) computes the estimate and **writes nothing and spends nothing**. In the UI the launch button stays disabled until the estimate has been rendered. ### Read the basis, not just the total The estimate reports **where each number came from** in a `basis` block — input tokens are *measured* from the frozen items' own text, and output tokens are either *observed* from your agent's recent history or, when there is no history, a stated default assumption. An assumption is never presented as a measurement. ### Check the unpriced arms An arm whose model has **no price** in your [price catalogue](/guides/cost-and-model-pricing) contributes **nothing** to the total — not zero. The estimate comes back with `partial: true` and names the unpriced models and arms, so the total is explicitly a **floor**, not the bill. Set a rate for that model (Settings → **Model pricing**) and re-run the preview to get a complete number. ### Launch `POST /model-sweeps` re-runs the same estimate and refuses the launch if it breaks a limit. ### The budget ceiling and the run caps | Refusal | When | | --- | --- | | **Over budget** | The estimate's priced total already exceeds the sweep's `budgetUsd` (or the **$50** platform ceiling, whichever is lower). A *partial* estimate whose priced portion alone is over budget still refuses — the real cost is higher than the number shown, never lower. | | **Too many runs** | `arms × k` exceeds the platform ceiling of **40** child runs. | | **Too many arms / too high k** | Above the per-sweep ceiling of **8 arms** or **k = 5** — a higher `passK` is clamped down to 5. | A **fully unpriced** estimate does **not** refuse on budget — Neens will not block on a number it does not have. It surfaces `partial` and leaves the decision with you. ## Per-arm failure isolation A dead endpoint fails **its own arm** and nothing else. If one model's endpoint is unreachable, times out, or returns nothing usable: - That arm goes **`failed`** with the error recorded on it. - Every **other** arm keeps running and is reported normally. - The sweep finishes as **`completed_with_failures`** — every arm terminal, at least one failed, at least one completed — which is a different, and honest, status from `completed`. Partial results are the normal case, not an error state: each arm carries its own status and its own progress, so a sweep detail page shows three arms in three different states rather than one spinner over the whole page. Statuses **Sweep:** `queued` · `running` · `completed` · `completed_with_failures` · `failed` · `cancelled` · `void` - `completed` — every arm terminal and no arm failed. - `completed_with_failures` — every arm terminal, at least one failed, at least one completed. - `failed` — every arm failed, or the launch itself failed. - `void` — condition divergence. Terminal, and never comparable. **Arm:** `pending` · `running` · `completed` · `failed` · `cancelled` A sweep never reports a terminal status while an arm is still non-terminal. ## Create a sweep On **Cost & Quality → Agent model**, press **New sweep** in the top right to open the launch dialog. In it, fill in **Sweep name** and **Version label**, choose the **Source** (**Golden dataset** or **Stress-test suite**), add one entry per model under **Model arms** — each an **Agent endpoint** connection you have already registered — set **Runs per arm (k)** and an optional **Budget (USD)**, then press **Estimate cost**. The estimate renders per arm under **Before you spend anything**, with **Some arms have no price** called out when an arm is unpriced and a **Where these numbers come from** block for the basis. **Launch sweep** stays disabled until you have run an estimate — and if you change the sweep after estimating, it asks you to estimate again. Preview first — this spends nothing: ```bash curl -sf -X POST "$NEENS_BASE_URL/model-sweeps/preview" \ -H "Authorization: Bearer $NEENS_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "datasetId": "ds_golden", "passK": 3, "arms": [ {"label": "sonnet", "agentConnectionId": "conn_sonnet"}, {"label": "mini", "agentConnectionId": "conn_mini"}, {"label": "self-host", "agentConnectionId": "conn_selfhost"} ] }' ``` Then launch: ```bash curl -sf -X POST "$NEENS_BASE_URL/model-sweeps" \ -H "Authorization: Bearer $NEENS_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "name": "Q3 model selection", "versionLabel": "2026-07-candidate", "datasetId": "ds_golden", "passK": 3, "budgetUsd": 10, "arms": [ {"label": "sonnet", "agentConnectionId": "conn_sonnet"}, {"label": "mini", "agentConnectionId": "conn_mini"}, {"label": "self-host", "agentConnectionId": "conn_selfhost"} ] }' ``` Read it back — the detail response reconciles live, so per-arm state is current without waiting for the next background pass: ```bash curl -sf "$NEENS_BASE_URL/model-sweeps/{sweep_id}" \ -H "Authorization: Bearer $NEENS_API_KEY" ``` Stop one early (cancels the sweep and every child run that has not finished): ```bash curl -sf -X POST "$NEENS_BASE_URL/model-sweeps/{sweep_id}/cancel" \ -H "Authorization: Bearer $NEENS_API_KEY" ``` See [Run a sweep from CI](#run-a-sweep-from-ci) below for the `neens sweep` verbs. Request fields | Field | Required | Meaning | | --- | --- | --- | | `name` | yes (create) | Human name for the sweep | | `versionLabel` | yes (create) | Candidate label recorded on every captured trace, identical across arms | | `arms` | yes | `[{label, agentConnectionId}]` — one per model. Labels must be unique; two arms may not share a connection | | `datasetId` | one of | The golden dataset to replay | | `datasetVersionId` | one of | Pin an explicit frozen version instead of the golden one | | `scenarioSuiteId` | one of | Replay a [stress-test suite](/guides/stress-testing)'s frozen synthetic version | | `passK` | no | Repeats per arm (default 3, clamped to the platform ceiling) | | `judgeDeploymentIds` | no | Judges that score every arm; defaults to your agent's Primary Score | | `budgetUsd` | no (create) | Refuse the launch above this estimated spend | | `gate` / `baseline` | no (create) | Applied identically to every child run | A dataset with **no golden version** is rejected: a sweep whose arms could see different prompts cannot guarantee identical inputs, which is the entire point. An arm whose endpoint declares **no model** is allowed — Neens never invents a model name — but it is unpriced in the estimate and named in `unpricedArms`. ## Then read the answer A finished sweep is evidence, not a decision. Press **Compare arms** on the sweep detail page to get the verdict — *which* model to ship, at what bar, at what cost per case, **and per agent**, with the sample size, confidence interval and unpriced arms shown rather than rounded away: ```bash curl -sf "$NEENS_BASE_URL/model-sweeps/{sweep_id}/comparison" \ -H "Authorization: Bearer $NEENS_API_KEY" ``` That surface has its own guide: **[Sweep decisions](/guides/sweep-decisions)**. It also covers the regression drill-down (*where* a cheaper arm breaks), the `neens sweep decide` CI verb, and the redacted share link you can paste into a decision doc. The Sweeps tab also renders the finished sweep as a **[cost–quality frontier](/guides/cost-quality-frontier)** — each arm plotted by cost per case against its pass rate, with the cheapest arm that clears your bar highlighted as the recommended move. It's the same chart the Cost tab uses for judge scorer models, and a fast way to see which cheaper model still clears the quality bar. A [void](#identical-conditions-and-what-void-means) sweep draws no frontier, for the same reason it renders no comparison. ## Run a sweep from CI The `neens-eval` SDK (Python and TypeScript, same flags, same exit codes) has a `sweep` group beside `eval`. The CI story is **estimate, then decide**: ```bash # 1. What will this cost? Writes nothing, spends nothing. neens sweep preview \ --dataset-id ds_golden \ --arm "sonnet=conn_sonnet" \ --arm "mini=conn_mini" \ --arm "self-host=conn_selfhost" \ --pass-k 3 --json # 2. Launch it and wait for every arm. neens sweep start \ --name "monthly model sweep" \ --dataset-id ds_golden \ --version-label "$GIT_SHA" \ --arm "sonnet=conn_sonnet" \ --arm "mini=conn_mini" \ --arm "self-host=conn_selfhost" \ --pass-k 3 --budget-usd 10 --wait # 3. Or poll one later. neens sweep get msw-… --json # 4. Print the verdict: which arm to ship, at what bar and cost, per agent. neens sweep decide --sweep-id msw-… --bar 0.9 ``` ```bash npx neens-eval sweep-preview \ --dataset-id ds_golden \ --arm "sonnet=conn_sonnet" \ --arm "mini=conn_mini" \ --pass-k 3 --json npx neens-eval sweep-start \ --name "monthly model sweep" \ --dataset-id ds_golden \ --version-label "$GIT_SHA" \ --arm "sonnet=conn_sonnet" \ --arm "mini=conn_mini" \ --pass-k 3 --budget-usd 10 --wait npx neens-eval sweep-get msw-… --json npx neens-eval sweep-decide --sweep-id msw-… --bar 0.9 ``` ### Exit codes - **A sweep is informational by default.** Arms that did not all pass do **not** fail your build — a sweep is a model-*selection* decision, not a release gate. Use [eval gates](/guides/eval-gates) and [pre-prod evals](/guides/preprod-evals) to block a release. - Pass **`--require-all-arms`** when you *do* want a non-zero exit unless every arm passed pass^k. - **A `void` sweep always exits non-zero**, with the void reason printed — a comparison that cannot be trusted must never look like a pass. - `neens sweep decide` follows the same shape: informational by default, non-zero only for a void or failed sweep — or with `--require-winner`. See [its exit codes](/guides/sweep-decisions#exit-codes). ## Drive it from a coding agent (MCP) Three [MCP](/guides/mcp) tools expose the same flow to an agent: | Tool | Does | | --- | --- | | `start_model_sweep` | Launches a sweep (`name`, `version_label`, `arms`, plus the optional `dataset_id` / `dataset_version_id` / `scenario_suite_id`, `pass_k`, `judge_deployment_ids`, `budget_usd`) | | `get_model_sweep` | Reads one sweep back: header plus per-arm `label`, `model`, `status`, `passed`, `greens`, `k`, and run counts — compacted, never the full child-run list | | `get_model_sweep_comparison` | Reads the [verdict](/guides/sweep-decisions): which arm to ship, the per-arm leaderboard with sample sizes and confidence intervals, and a verdict per agent | ## How it works ### Create pins the conditions Neens resolves the frozen dataset version (from the suite, the explicit version, or the dataset's golden version), snapshots the judge set, freezes each arm's model off its endpoint connection, computes the estimate, checks it against the caps and the budget, and stores the conditions fingerprint. ### The launch fans out N × k child runs One background task creates `arms × k` ordinary pre-prod runs — each pointed at its arm's endpoint, all sharing the pinned conditions — and hands them to the **dedicated pre-prod fleet**. It does not wait for them: a parent task that blocked on N×k children would hold a worker slot for hours and starve live scoring. If creating an arm's runs fails, **that arm** is marked failed and the launch continues with the next one. ### Aggregation is lazy and idempotent Per-arm state is recomputed from the child runs — never cached as a claim. That happens on every read of a sweep, and again on a background pass every few minutes, so a sweep converges even if nobody is watching. Reconciliation never overwrites a cancelled sweep and never moves a sweep out of `void`. ### A run is green only if it passed AND had zero regressions Each child run contributes one outcome: its pass rate over scored items against the platform success threshold (**0.90** by default), plus its regression count. `greens` counts the runs that cleared both. pass^k is green only when `greens == k`. An arm whose every run finished having scored **nothing** — a dead endpoint, an unreachable model — is `failed`, not "0%". We do not know how that model performs, and reporting 0% would claim we do. ## Troubleshooting | Symptom | Cause | Fix | | --- | --- | --- | | The launch is refused with a budget message | The estimate's **priced** total is over the sweep's budget or the platform ceiling | Lower `passK`, drop an arm, use a smaller golden version, or raise the budget | | The estimate says `partial` and the total looks too low | At least one arm's model has no price | Set a rate in **Settings → Model pricing**, then preview again. The shown total is a floor | | An arm is `failed` while the others completed | That arm's endpoint was unreachable or returned nothing usable | Check the arm's error and the endpoint connection; the other arms' results are still valid | | The sweep is `void` and shows no comparison | The arms did not run under identical conditions (e.g. a judge was redeployed mid-sweep) | Read the void reason, fix the drift, run a fresh sweep. A void sweep is intentionally not renderable | | An arm shows `2/3` and `passed: false` | The model passed some runs and failed others | That *is* the result: an unreliable pass is not a pass. See [pass^k](#passk-is-per-arm--not-per-sweep) | | The sweep finished but you still can't say which model to ship | You're reading the evidence, not the verdict | Press **Compare arms** — see [Sweep decisions](/guides/sweep-decisions) | ================================================================================ # Sweep decisions Source: /docs/guides/sweep-decisions/ ================================================================================ # Sweep decisions A [model sweep](/guides/model-sweeps) produces evidence: N models × k runs over one frozen golden dataset. The **comparison** turns that evidence into the answer — *"for the refund agent the cheap model clears your 90% bar at a fraction of the cost; for the escalation agent it does not."* That second half is the point. "A cheaper model still hits the number" is almost never true of a whole workspace; it is true of one agent and false of another, and a decision made on the workspace average quietly ships the wrong model to the work that matters most. ## At a glance | | | | --- | --- | | **Where** | **Cost & Quality → Agent model** → open a sweep → **Compare arms** | | **Key API routes** | `GET /model-sweeps/{id}/comparison`, `GET /model-sweeps/{id}/arms/{arm_id}/regressions` | | **Auth** | Read-only. The same **`nk_live_` agent API key** that drives the sweep | | **CLI** | `neens sweep decide --sweep-id …` (TypeScript: `neens-eval sweep-decide --sweep-id …`) | | **Needs** | A sweep whose arms have finished. Nothing else — no LLM call, no configuration | | **Produces** | One verdict sentence, an arm leaderboard, a verdict **per agent**, a regression drill-down, and an optional redacted share link | **Nothing here is stored, and nothing is spent.** Every number is computed on read from the sweep's own child runs, the traces they captured and your [price table](/guides/cost-and-model-pricing). Opening the page reconciles the sweep first, so a still-running sweep is current — and re-reading it after an admin corrects a model's price gives you the corrected figure immediately. ## The whole thing in one worked example `make demo-stack` seeds a real sweep you can open and follow along with: **Support agent — model selection**, six arms over **30 frozen golden items** spanning **two agents** — a `refund-agent` (22 items: lookups, policy answers, multi-step resolutions) and an `escalation-agent` (8 items: fraud, chargebacks, safety, over-limit money — cases the agent must *not* settle alone), each arm run **k = 3** times. ### Open the comparison **Cost & Quality → Agent model** → **Support agent — model selection** → **Compare arms**. ### Read the verdict, first and largest > **cost-optimized** clears the 90% bar at $0.0002 per case (93.3% pass rate, n=30). That is 68.5% > cheaper than **incumbent**. 2 other arms also clear the bar and are not distinguishable from it > at n=30. At n=30 the 95% confidence interval reaches below the bar, so this sample does not > prove it. Four separate claims, and the last two are the ones a chart would have hidden: two other arms are just as good as far as this sample can tell, and the sample is not yet large enough to *prove* the winner clears the bar. ### Audit it against the leaderboard | Arm | Model | Pass rate (n=30) | pass^k | Cost per case | Regressions | | --- | --- | --- | --- | --- | --- | | **incumbent** *(baseline)* | `claude-sonnet-4-6` | 96.7% · CI 83.3–99.4% | 3/3 | $0.00067 | — | | **cost-optimized** *(winner)* | `claude-haiku-4-5` | 93.3% · CI 78.7–98.2% | 3/3 | $0.00021 | 2 | | **challenger** | `gpt-4o-mini` | 93.3% · CI 78.7–98.2% | 3/3 | **—** *(unpriced)* | 2 | | **previous-gen** | `claude-sonnet-4-5` | 86.7% · CI 70.3–94.7% | **2/3** | $0.00068 | 4 *(2 flaky)* | | **budget** | `gpt-oss:120b` | 33.3% · CI 19.2–51.2% | 0/3 | $0.00 | 20 | | **self-hosted** | `gpt-oss:20b` | **no data** | 0/3 · failed | — | — | Read the last three rows before the first three, because each is a trap this page exists to avoid: - **`challenger` has no price**, so it shows an em dash and is *excluded from the cost ranking* — even though it clears the bar. Treating its missing rate as `$0` would have handed it the win. - **`budget` is the cheapest arm there is** (a real, catalogue-priced `$0` self-hosted model) and loses anyway, because the ranking is quality-first. - **`self-hosted` reports *no data*, not 0%.** Its endpoint was unreachable, so nothing was scored. We do not know how that model performs, and 0% would claim we do. ### Then read the per-agent section — this is where the answer changes | Agent | Items | Verdict | | --- | --- | --- | | `refund-agent` | 22 | **cost-optimized** clears the 90% bar at $0.0002 per case (100% pass rate, n=22) — 68.3% cheaper than incumbent | | `escalation-agent` | 8 | **incumbent** clears the bar at $0.0010 per case (100% pass rate, n=8). cost-optimized drops to **75%** and does not clear | The workspace-level winner is the cheap model. The decision is **not** "switch to the cheap model" — it is *"you can drop to the cheap model on the refund agent and not on the escalation agent."* That sentence is only available because the sweep's golden set covers both agents. ### Ask where it breaks Click **cost-optimized**'s regression count. The drill-down groups the 2 regressed items by a failure signal and names it: both land in the failure mode **Escalation policy not applied**, and each failing item shows the frozen input, the expected output, **what this arm answered** and the judge's reason. The seeded cheap arm states the refund limit and then approves the refund anyway — a genuine small-model instruction-following failure you can read in its own words. **A verdict is a decision, not a gate.** Neens names the cheapest arm that clears *your* bar; it never switches a model, opens a PR, or changes a deployment. Shipping stays a human action. ## Set the quality bar A bar you did not choose is a bar you should not trust, so the comparison reports where its bar came from — `bar.source` on the wire, and a line under the verdict in the UI: | Precedence | `bar.source` | Where it comes from | | --- | --- | --- | | 1 | `query` | `?bar=` on the request, or the **Quality bar (pass rate %)** control on the page. In the UI it is a percentage; on the wire it is a `0`–`1` rate, and anything outside that range is a `422` | | 2 | `gate` | The sweep's own pinned gate `min_pass_rate` — the bar you committed to when you launched | | 3 | `default` | The deployment's success threshold, **0.90** by default | Pinning `{"min_pass_rate": 0.9}` on the sweep's gate at launch is the durable option: everyone who opens the comparison later reads it against the same number, with nothing to configure and no query parameter to remember. That is what the seeded demo does. ### The baseline arm Savings, regressions and *new passes* are all measured against **one** arm — the **declared incumbent**, which is the arm at position 0 (the first one you listed when you created the sweep). Override it with `?baselineArm=` or the **Baseline arm** control. Neens never picks the best-performing arm as the baseline. A baseline chosen after seeing the results is not a baseline, it is a flattering comparison. ## How to read the honesty signals Every one of these exists because the alternative is a number that reads like a fact and isn't. ### Sample size and confidence interval Every rate carries its `n` and a 95% **Wilson** confidence interval, and an arm that this sample cannot separate from the winner is annotated *"not distinguishable at n=…"* rather than silently ranked above or below it. > 94% and 91% over 40 items are not different. At those sample sizes the intervals overlap > heavily, and the ordering you see is sampling noise wearing a ranking's clothes. Two bounds are reported, and they answer different questions: | Field | Question it answers | | --- | --- | | `clearsBar.clears` | Does the **point estimate** reach the bar? This is what the leaderboard shows | | `clearsBar.clearsLowerBound` | Does the interval's **lower bound** reach the bar? This is the claim that survives scrutiny | In the worked example the winner clears on the point estimate and *not* on the lower bound — which is why the verdict says so out loud. The fix is more golden items, not a rounder number. ### pass^k, and why 2 of 3 is not a pass Two independent rules, both of which have to hold: - **Per arm:** the arm must be green on **all k** of its runs. An arm at `2/3` reports `passed: false` and is excluded from the ranking — see [pass^k](/guides/model-sweeps#passk-is-per-arm--not-per-sweep). - **Per item:** an item counts as passed only when **every scored repeat of it passed**. An item that passed 2 of 3 identical attempts has not passed; it is counted in `flakyItems` instead. Averaging the flaky items into the rate would re-introduce exactly the lucky green that k repeats exist to catch. In the worked example `previous-gen` carries 2 flaky items — the same prompts, a different answer on one repeat. ### Unpriced arms — the one to read twice **An unpriced arm shows `—`, never `$0.00`, and can never win "cheapest."** If a model has no rate in your [price catalogue](/guides/cost-and-model-pricing), Neens does not know what it costs. Treating that as free would make it the cheapest arm in every comparison it ever appears in — the single most dangerous thing this page could get wrong. So an unpriced arm is still reported, still shown as clearing (or not clearing) the quality bar, still listed among the arms that clear — with `costPerCaseUsd: null` — and **also** listed under *excluded* with the reason `unpriced`, because it cannot be ranked on cost. The page's pricing block comes back `partial: true` and names the models. **Fix it in one place:** set a rate for that model under **Settings → Model pricing**, then reload the comparison. A self-hosted model you pay GPU time for is a real `$0` — enter it once and it becomes a legitimate winner, which is a different thing entirely from having no rate at all. A partly-priced arm is reported the same way: `pricedCases` versus `cases` tells you how much of the arm the figure covers, and unpriced sessions are excluded from the mean rather than counted as zero. ### Unattributed agents The per-agent split groups items by the **agent name** on the captured trace. Items whose sessions carry no agent name group under a row labelled **Unattributed** — never folded into a named agent, because *"we do not know which agent this was"* and *"it was the refund agent"* are different facts, and merging them moves items (and a verdict) onto an agent that never ran them. If your whole workspace lands under **Unattributed**, your agent isn't setting an agent name on its traces — see [Send traces](/guides/send-traces). A genuinely single-agent workspace legitimately renders one row; that is honest, not a bug. ### Unclassified regressions The drill-down picks **one** failure signal for the whole response and says which: | `signal` | Meaning | | --- | --- | | `issue_class` | The regressed sessions carry classifier issue labels — the failure modes shown are those | | `cluster` | No issue labels, but the sessions belong to [failure clusters](/guides/clustering) — the cluster labels are used | | `none` | Neither. The sessions are unclassified | One signal, never a blend, so the same regression can never be counted twice under two names. And regressed sessions that carry no classification are reported in an explicit `unclassified` count — never dropped, because hiding them shrinks the denominator until the modes that *are* shown look like the whole story. **`signal: "none"` is the normal answer for a fresh sweep.** Pre-prod sessions are frequently unclassified — nothing has labelled them yet. The panel says so and lists the failing items directly, rather than showing an empty taxonomy or inventing modes. ### A void sweep refuses to name a winner If a sweep's arms did not run under identical conditions it is **[void](/guides/model-sweeps#identical-conditions-and-what-void-means)**, and the comparison declines: `comparable: false`, `verdict.outcome: "unavailable"`, no ranking, no winner. The arms' own numbers are still shown — they are facts about each run — but no cross-arm claim is drawn from them, in the UI, in the CLI, or in the export. There is deliberately no way to "just show it anyway". A comparison of arms that measured different things is the most convincing wrong answer this feature could produce. ### "No data" is never 0% An arm with nothing scored reports `rate: null` and renders **no data**. An arm still running is excluded as `not_terminal` rather than judged on the runs it happens to have finished. ## Where the cheap model breaks Clicking an arm's regression count opens the drill-down, or call it directly: ```bash curl -sf "$NEENS_BASE_URL/model-sweeps/{sweep_id}/arms/{arm_id}/regressions?limit=50" \ -H "Authorization: Bearer $NEENS_API_KEY" ``` A **regression** is precisely: *the baseline arm passed this item on every one of its k runs, and this arm — having been scored at least once — did not.* An item this arm never captured is missing evidence, not a regression, and is not counted. The inverse is reported too, as `newPasses`. The response gives you three levels, in the order you want them: 1. **`failureModes`** — the pinned signal's labels with counts, each linking to the [failure mode](/guides/issues-and-failure-modes) when the signal resolved to real taxonomy rows, plus `unclassified` for the rest. 2. **`items`** — up to `limit` (default **50**, max **200**) failing golden prompts: the frozen input and expected output, the baseline's and candidate's `passedRuns`/`scoredRuns`, the candidate's score, **the answer it actually gave**, and the judge's reason. 3. **`sessionIds`** on each item — the captured traces themselves, for a full [trace view](/guides/traces-and-sessions). `totalRegressions` is the true total and `truncated` says when the item list was cut to `limit`; the per-mode counts always describe the whole regression set, not just the page. ## Decide from CI `neens sweep decide` reads the comparison and prints the **server's** verdict — the same sentence, the same leaderboard, the same per-agent verdicts the UI renders. The ranking is never re-derived client-side: two rankings that can disagree is worse than one. ```bash # Decide a finished sweep against its own pinned bar. neens sweep decide --sweep-id msw-… # Or against a bar you set here, comparing to a specific incumbent, and wait for it to finish. neens sweep decide --sweep-id msw-… --bar 0.9 --baseline-arm msa-… --wait # Fail the build unless some arm clears the bar. neens sweep decide --sweep-id msw-… --bar 0.9 --require-winner --json ``` ```bash npx neens-eval sweep-decide --sweep-id msw-… npx neens-eval sweep-decide --sweep-id msw-… --bar 0.9 --baseline-arm msa-… --wait npx neens-eval sweep-decide --sweep-id msw-… --bar 0.9 --require-winner --json ``` ### Flags | Flag | Meaning | | --- | --- | | `--sweep-id ID` | Required. Unlike `sweep get`, the id is a flag, not a positional | | `--bar 0.9` | The pass rate an arm must clear, `0`–`1`. Anything outside that range is rejected before a request is made. Omit it to use the sweep's pinned gate, then the deployment default | | `--baseline-arm ID` | The arm to measure savings and regressions against. Default: position 0 | | `--require-winner` | Exit non-zero unless an arm clears the bar. Off by default. **Implies `--wait`** | | `--wait` | Poll until the sweep is terminal before comparing | | `--poll-interval` / `--poll-timeout` | Poll cadence (default `10`s) and total budget (default `3600`s) | | `--json` | Emit the full machine-readable result on stdout; the human table still goes to stderr | ### Exit codes | Code | When | | --- | --- | | **2** | The sweep is **void** / `comparable: false`, or the sweep `failed`, or a transport/API error. Checked **first**, before any verdict — a void sweep still has per-arm numbers, and reading those first is exactly how an untrustworthy comparison gets to look like a pass | | **0** | Anything else, by default — including *no arm clears the bar*. **Naming a winner is a decision, not a gate** | | **1** | With `--require-winner`: no arm cleared the bar, or the sweep has not finished (you cannot require a winner from a sweep you have not watched finish) | Use [pre-prod evals](/guides/preprod-evals) and [eval gates](/guides/eval-gates) to *block* a release. Use `sweep decide` to *choose a model*. `--require-winner` exists for the narrow case where "at least one model in this list is shippable" really is a build condition. ### Gate on cost, not just quality A sweep's pinned gate — and any pre-prod run's gate — accepts a **`cost` family**, evaluated server-side, spelled exactly as the SDK's client-side cost rule spells it: ```json { "max_regressions": 0, "min_pass_rate": 0.9, "cost": { "max_delta_pct": -50 } } ``` That gate is literally *"cheaper and still hits the number"*: every child run must hold a 90% pass rate **and** come in at least 50% below the baseline's average cost per session. See [the server-side cost gate](/guides/preprod-evals#the-server-side-cost-gate) for the full rule set, and for the rule that matters most: an **unpriced** run is reported **skipped**, never passed. ## Share the decision The comparison exports as a **read-only, redacted, expiring public link** — the artifact you paste into the doc that justifies a model switch, without granting anyone access to Neens. Press **Share (redacted)** on the comparison page, or: ```bash curl -sf -X POST "$NEENS_BASE_URL/share-links" \ -H "Authorization: Bearer $NEENS_API_KEY" \ -H "Content-Type: application/json" \ -d '{"resourceType": "sweep_decision", "resourceId": "msw-…", "ttlHours": 168}' ``` **It must be enabled for the agent before it shows anything:** turn on shareable links for the **agent** — `PUT /share-links/settings` with `{"enabled": true}`, which requires an admin. Until then the button is hidden rather than broken, and minting a link returns `403`. | | | | --- | --- | | **What is exported** | The whole comparison envelope: the verdict, every arm's rate, CI, pass^k, latency, cost and pricing provenance, the per-agent verdicts, and the pricing/warnings blocks | | **What is redacted** | Free text — the verdict sentence, arm labels, agent names, per-run failure reasons — passes through the same PII/secret redaction the shared trace and cluster links use. Numbers, model ids and provenance pass through intact, because a redacted rate is a useless artifact | | **What is never exported** | The regression drill-down. Golden inputs, model answers, judge reasons and session ids are not part of the shared payload at all | | **Lifetime** | Default **168 hours** (7 days), capped at **720 hours** (30 days). Revocable at any time, and served `no-store` + `noindex` | | **Freshness** | Minting reconciles the sweep once, so the link starts current. The public page then renders it as of that state and never triggers a write — for the terminal sweep anyone actually shares, that is the final state | ## Drive it from a coding agent (MCP) | Tool | Does | | --- | --- | | `get_model_sweep_comparison` | Reads the comparison back: the verdict, the per-arm leaderboard, the per-agent verdicts, the pricing block and any warnings. Optional `bar` and `baseline_arm`. Read-only — computes nothing new and spends nothing | The tool description hands the agent the same four rules a human reads: `comparable: false` means void so never rank; `costPerCaseUsd: null` means unpriced, never free; `rate: null` means no data, not 0%; and an item counts as passed only when it passed **every** repeat. See [MCP server](/guides/mcp). ## Reference Comparison response `GET /model-sweeps/{sweep_id}/comparison?bar=&baselineArm=` | Field | Meaning | | --- | --- | | `comparable` / `voidReason` | `false` ⇒ the sweep is void and no ranking is drawn | | `bar` | `{passRate, source}` — `source` ∈ `query` · `gate` · `default` | | `baseline` | `{armId, label, kind}` — the declared incumbent | | `passK` / `itemCount` | Repeats per arm, and frozen golden prompts | | `arms[]` | Per arm, below | | `agents[]` | `{agent, label, itemCount, arms[], verdict}` — one row per agent name, plus **Unattributed** last | | `verdict` | The decision, below | | `pricing` | `{partial, unpricedModels, version}` — `partial: true` means at least one figure is incomplete | | `warnings[]` | Anything that limits the read (a capped session scan, a reconcile that could not run) | Per arm: | Field | Meaning | | --- | --- | | `passK` | `{passed, k, greens, runs[], reason}` — `runs` are the sweep's real child pre-prod runs | | `quality` | `{n, successes, rate, ciLow, ciHigh, ciLevel, marginOfError, scoredRuns, flakyItems, unscoredItems}`. `n` counts items with at least one scored repeat; `successes` counts items that passed **every** repeat | | `clearsBar` | `{bar, clears, clearsLowerBound, margin}` — all `null` when the rate or bar is missing | | `distinguishableFromBaseline` | `true` · `false` · `null`. `false` = this sample cannot tell the two apart | | `latency` | `{n, avgMs, p95Ms}` over the captured sessions | | `cost` | `{cases, pricedCases, costPerCaseUsd, totalUsd, priced, partial, unpricedModels, pricing}`. `costPerCaseUsd` is the mean over **priced** sessions — what serving one case costs, not k× it. `totalUsd` is what the sweep actually spent on the arm | | `regressions` | `{count, flaky, newPasses, vsArmId}`. The baseline arm reports zeros with `vsArmId: null` | Verdict outcomes and reason codes | `outcome` | `reasonCode` | Meaning | | --- | --- | --- | | `winner` | `cheapest_clearing` | The cheapest **priced** arm that clears the bar | | `none_clear` | `no_arm_clears` | Arms were measured; none reached the bar. Names the best observed rate | | `none_clear` | `no_priced_arm` | Arms clear the bar, but none has a resolved price, so none can be named cheapest | | `unavailable` | `no_bar` | No quality bar was supplied | | `unavailable` | `no_data` | Nothing comparable — including a **void** sweep | `clearing[]` is sorted cheapest-first with unpriced arms last; `tiedWith[]` lists arms that clear the bar and are statistically indistinguishable from the winner; `savingsVsBaselinePct` is `null` when either side is unpriced. Among priced clearing arms the order is fully deterministic: cheapest cost per case, then higher pass rate, then lower arm position, then arm id. The same evidence always produces the same winner — a verdict that flips between refreshes is not a decision. Why an arm was excluded Evaluated and reported in this order — the first applicable reason wins: | Code | Meaning | UI wording | | --- | --- | --- | | `not_terminal` | Still running, or pass^k has no verdict yet | *still running* | | `no_data` | Nothing scored | *no scored items* | | `failed_pass_k` | Did not pass every one of its k runs | *did not pass pass^k* | | `below_bar` | Measured, terminal, green — but under the bar | *below the quality bar* | | `unpriced` | No resolvable price, so it cannot be ranked on cost | *its model has no price, so it cannot be ranked on cost* | An arm that is both below the bar **and** short of pass^k reports `failed_pass_k`, because that disqualifies it regardless of its rate. Both facts stay visible on the arm's own row (`passK` and `clearsBar`), so the exclusion code is never the only thing you have. Limits | Limit | Default | Notes | | --- | --- | --- | | Captured sessions folded into one comparison | 20000 | Over the cap, cost, latency and the per-agent breakdown cover the newest 20000 sessions and a `warnings` entry says so. Pass rates and regressions still cover every item | | Regression items per request | 50, max 200 (`limit`) | `totalRegressions` is always the true total | ## Troubleshooting | Symptom | Cause | Fix | | --- | --- | --- | | An arm shows `—` in the cost column | Its model has no price in your catalogue | Set a rate under **Settings → [Model pricing](/guides/cost-and-model-pricing)**, then reload. It is *unknown*, not free | | The verdict says arms are "not distinguishable" | The confidence intervals overlap at this sample size | Add golden items. There is no threshold that makes a small sample decisive | | The winner clears the bar but "this sample does not prove it" | The point estimate clears; the interval's lower bound does not | Same fix — more items. The claim is real, it is just not yet proven | | Every agent row says **Unattributed** | The captured traces carry no agent name | Set an agent name on your traces — see [Send traces](/guides/send-traces) | | The drill-down shows no failure modes | The regressed sessions are unclassified — the normal state for a fresh sweep | Read the failing items directly; they are listed. Classification arrives with clustering and issue labels | | No verdict, `comparable: false` | The sweep is [void](/guides/model-sweeps#identical-conditions-and-what-void-means) | Fix what drifted and run a fresh sweep. A void sweep is intentionally not rankable | | `sweep decide` exits 2 on a finished sweep | Void, failed, or the API call errored | The reason is printed above the exit — void prints its void reason | | The **Share (redacted)** button is missing | Share links are off for this agent | Turn them on for the agent with `PUT /share-links/settings` (requires an admin) | ## Related - [Model sweeps](/guides/model-sweeps) — how the evidence is produced, and what `void` means - [Cost & model pricing](/guides/cost-and-model-pricing) — where a cost per case comes from, and how to price a model - [Model comparison](/guides/model-comparison) — the same question asked of *production* traffic - [Pre-prod evaluations](/guides/preprod-evals) — the child runs a sweep launches, and the cost gate - [Issues & failure modes](/guides/issues-and-failure-modes) — the taxonomy the drill-down groups by - [MCP server](/guides/mcp) — reading the verdict from a coding agent ================================================================================ # Stress tests Source: /docs/guides/stress-testing/ ================================================================================ # Stress tests A **stress test** turns your [failure-mode taxonomy](/guides/issues-and-failure-modes) from a backward-looking record into a forward-looking asset: pick the failure modes your agent already got caught on, and Neens **generates fresh synthetic test scenarios grounded in those real failures** — then hands them straight to the [pre-prod evaluation](/guides/preprod-evals) gate so you can *attack your next candidate with last quarter's failures before it ships*. Where a [pre-prod eval](/guides/preprod-evals) replays a golden dataset you curated by hand, a stress test **builds that golden dataset for you** from the sessions where the agent actually failed — so the set that guards a release grows automatically as your taxonomy does. ## At a glance | | | | --- | --- | | **Where** | The **Stress tests** tab on the **Pre-prod Evals** page | | **Key API routes** | `POST /scenario-suites`, `GET /scenario-suites`, `GET /scenario-suites/{id}`, `POST /scenario-suites/{id}/regenerate`, `POST /scenario-suites/{id}/launch`, `DELETE /scenario-suites/{id}` | | **Auth** | The same **`nk_live_` agent API key** that drives [pre-prod evals](/guides/preprod-evals) — so CI can run the whole *generate → launch → gate* flow with one credential. | | **Needs** | One or more **confirmed failure modes** with exemplar traces ([Issues & failure modes](/guides/issues-and-failure-modes)). An [LLM connection](/administration/llm-connections) makes the generated scenarios richer; **without one, a suite still builds** from the deterministic replay floor. | | **Scope** | Agent-scoped, like all your data; generation runs on the eval worker fleet. | | **Produces** | A **scenario suite** whose scenarios become an immutable **golden dataset version**, which a one-click launch replays as a normal pre-prod run (scores tagged `source: preprod` — their own score type, filterable out of production widgets by **Score source**). | ## How it fits together A stress test is three moving parts, and the last one is just a pre-prod eval: 1. **A scenario suite** is generated from failure modes you pick. Each scenario is a user `input` plus an `expected_output` — a *behavioral assertion* of what a correct response must do or avoid (a synthetic prompt has no ground-truth answer, so the judges grade behavior, not a fixed string). 2. The suite's scenarios are materialized as an **immutable golden dataset version** — the same frozen-golden contract a hand-curated [dataset](/guides/datasets) uses. 3. **Launching** the suite creates a [pre-prod eval run](/guides/preprod-evals) over that golden version. From there everything is a normal pre-prod eval — Neens calls your agent, or your own harness emits the traces — scored by your judges, compared to a baseline, and gated on regressions and pass rate. So a stress test adds **no new scoring, comparison, or gate machinery** — it feeds the pre-prod gate you already know. ## Generate a suite ### Confirm the failure modes you want to attack A stress test grounds every scenario in **real exemplar traces** — the sessions where the agent actually exhibited that failure mode. Curate and confirm the modes you care about first in [Issues & failure modes](/guides/issues-and-failure-modes); **confirmed** modes carry the strongest exemplar evidence and float to the top of the picker. ### Open the Generate stress test wizard On **Pre-prod Evals → Stress tests**, start a new suite. The wizard asks for: | Field | Purpose | | --- | --- | | **Name** | A human name for the suite (e.g. `Refund-policy regressions`). | | **Failure modes** | One or more of your agent's non-archived modes. Each row shows its severity and how many exemplar traces it carries — more exemplars means more grounding. | | **Scenarios per mode** | How many **new** (LLM-authored) scenarios to request for each mode. Default **5**; capped at **25**. | | **Strategies** | Which kinds of synthetic scenario to ask for — **paraphrase**, **escalate**, **boundary** (explained [below](#the-strategies)). The deterministic **replay** floor is always available and isn't picked here. | | **Include replay** | On by default — always add the deterministic replay scenarios (the literal past-failing inputs) alongside the LLM-authored ones. | ### Let it generate Submitting kicks off generation on the eval worker fleet. For each selected mode, Neens gathers its real failing sessions, reduces each to the user's input, the agent's (wrong) output, and any error, and asks the tenant's LLM to author fresh scenarios that probe the same weakness from new angles — never copying the exemplars verbatim. It **always** adds the deterministic replay floor so the suite is runnable regardless. The suite moves `pending → generating → ready` (or `failed`). While it generates you can inspect it via `GET /scenario-suites/{id}`. Via the API — the whole body except `name` and `failure_mode_ids` has a default: ```bash curl -sf -X POST "$NEENS_BASE_URL/scenario-suites" \ -H "Authorization: Bearer $NEENS_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "name": "Refund-policy regressions", "failure_mode_ids": ["fm_abc", "fm_def"], "per_mode": 5, "strategies": ["paraphrase", "escalate", "boundary"], "include_replay": true }' ``` Optional fields: `per_mode` (LLM scenarios per mode, 1–25), `strategies` (subset of `paraphrase`/`escalate`/`boundary`), `include_replay` (default `true`), and `connection_id` to pin a specific [LLM connection](/administration/llm-connections) for generation. Only failure modes that belong to **your** agent are accepted — a request that names none of your agent's modes is rejected (`422`), so a suite can never be seeded from another agent's taxonomy. ## Review the scenarios A ready suite lists its generated scenarios. Each one carries: - **Input** — the synthetic end-user prompt. - **Expected output** — the behavioral assertion the judges grade against ("a correct response must not exhibit the *'…'* failure mode…"). - **Strategy** — which kind of scenario it is (`replay`, `paraphrase`, `escalate`, or `boundary`). - **Source failure mode** — which mode it was generated from, so every scenario is traceable back to the real failure that inspired it. Not happy with the set — added more exemplars, or just configured an LLM connection? **Regenerate** the suite (`POST /scenario-suites/{id}/regenerate`) to rebuild it, optionally overriding `per_mode`, `strategies`, `include_replay`, or `connection_id`. Regeneration replaces the suite's scenarios and snapshots a fresh golden version. ## The strategies Each scenario is generated with one **strategy**. Three are LLM-authored variations grounded in the mode's real exemplars; the fourth is a deterministic floor that needs no model: | Strategy | What it produces | Needs an LLM | | --- | --- | --- | | `replay` | Re-uses the **literal past-failing inputs** verbatim — the exact prompts that triggered the mode last time. Deduplicated across the suite. | **No** — deterministic. | | `paraphrase` | Rewords a real failing input: same intent, fresh surface wording. | Yes | | `escalate` | A harder, more adversarial variant that pushes the failure mode further. | Yes | | `boundary` | An edge-case input that sits right on the boundary of the failure. | Yes | **No LLM connection? A suite still builds.** With no [LLM connection](/administration/llm-connections) configured, Neens degrades to the **replay floor alone** — re-attacking the candidate with the real prompts that failed before. It never fabricates a scenario it can't ground. A mode with no exemplar traces *and* no model contributes nothing; a suite that ends up with zero scenarios is marked `failed` with a reason, never left wedged in `generating`. ## Launch a stress test A ready suite launches as a pre-prod eval run in one step. ### Launch the suite In the **Stress tests** tab, launch a ready suite. You supply: - **Version label** — required; names the *candidate agent version* under test (a git SHA, a branch, `v2.1`). Recorded on every captured trace. - **Agent endpoint** *(optional)* — an [agent endpoint](/guides/preprod-evals#neens-calls-my-agent) connection. Leave it on **Auto** and Neens decides how the run executes. ### Neens picks the execution mode - If the agent has an **agent endpoint** (or you selected one), **Neens calls your agent** — once per scenario, recording each response and scoring it. No harness. - Otherwise **you run your agent** — you drive it over the scenarios (by hand or in CI) and the traces it emits are correlated back to the run. Either way, launching lands you on the created pre-prod run, where scoring, the candidate-vs-baseline comparison, and the gate work exactly as in [Pre-prod evaluations](/guides/preprod-evals). Via the API, `POST /scenario-suites/{id}/launch` delegates to the pre-prod eval create endpoint and returns the created run: ```bash curl -sf -X POST "$NEENS_BASE_URL/scenario-suites/$SUITE_ID/launch" \ -H "Authorization: Bearer $NEENS_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "version_label": "'"$GIT_SHA"'", "baseline": {"kind": "prod_window", "range": "7d"}, "gate": {"max_regressions": 0, "min_pass_rate": 0.9} }' ``` The launch body accepts the same knobs a pre-prod run does — `version_label` (required), `runner_mode` (`"push"`/`"runner"`, defaulted for you), `agent_connection_id`, `judge_deployment_ids`, `baseline`, and `gate`. Omit `judge_deployment_ids` to score with your agent's **Primary Score**, like any pre-prod run. A suite can only be launched once it's **ready** — launching one that's still generating (or that failed) returns `409`. Deleting a suite removes it and its synthetic dataset, but a run you already launched snapshotted its scenarios and is unaffected. ## Read the gate Because a launched stress test **is** a pre-prod eval run, everything in [Pre-prod evaluations](/guides/preprod-evals) applies unchanged: - Each scenario's captured trace is scored by your judges; scores are tagged **`source: preprod`** so they stay out of production metrics and dashboards. - A scenario **passes** when the mean of its judge scores is at or above **0.7**. - The run is compared to a baseline (a previous run or a production window), and any item that **passed the baseline but fails the candidate** is a **regression**. - The [gate](/guides/preprod-evals#reviewing-a-run) turns the run into a pass/fail verdict on your **max regressions** and **min pass rate** thresholds — and emits a **Pre-prod regression** [insight](/guides/insights) when something regresses. Review it on the run's page, or drive and read the gate from CI (below). ## In CI: generate → launch → gate Because every route accepts the **`nk_live_` agent API key** you already use to send traces, a CI job can build a stress-test suite from your live taxonomy and gate a release on it with a single credential — no login, no session token. Set the shared pre-prod environment variables once (see [Pre-prod evals → Authentication](/guides/preprod-evals#authentication--one-key-set-once)): ```bash ``` Then generate a suite, wait for it to become `ready`, and launch it against the commit under test: ```bash # 1. Generate a suite from the confirmed failure modes you want to guard against. SUITE=$(curl -sf -X POST "$NEENS_BASE_URL/scenario-suites" \ -H "Authorization: Bearer $NEENS_API_KEY" -H "Content-Type: application/json" \ -d '{"name":"CI stress test","failure_mode_ids":["fm_abc","fm_def"]}' | jq -r .id) # 2. Poll until the suite is ready (generation runs on the worker fleet). until [ "$(curl -sf -H "Authorization: Bearer $NEENS_API_KEY" \ "$NEENS_BASE_URL/scenario-suites/$SUITE" | jq -r .status)" = "ready" ]; do sleep 5; done # 3. Launch it as a pre-prod run against the commit under test, gating on regressions. curl -sf -X POST "$NEENS_BASE_URL/scenario-suites/$SUITE/launch" \ -H "Authorization: Bearer $NEENS_API_KEY" -H "Content-Type: application/json" \ -d '{"version_label":"'"$GITHUB_SHA"'","gate":{"max_regressions":0,"min_pass_rate":0.9}}' ``` The launch returns a pre-prod run; from there the same **exit-code gate contract** and `neens eval` CLI described in [Pre-prod evaluations](/guides/preprod-evals#the-exit-code-gate-contract) turn the verdict into a passing or failing build step. On the inline queue (local `make demo` / tests) generation runs synchronously, so the suite is already `ready` when the create call returns. On a Celery deployment, generation runs on the worker fleet and you poll `GET /scenario-suites/{id}` (step 2 above). ## Run lifecycle | Suite status | Meaning | | --- | --- | | `pending` | Created; generation is queued. | | `generating` | The worker is gathering exemplars and authoring scenarios. | | `ready` | Scenarios generated and snapshotted as a golden version — launchable. Terminal until you regenerate. | | `failed` | Generation produced no scenarios (e.g. the selected modes had no exemplar traces and no LLM was configured). The reason is recorded on the suite. | ## Troubleshooting | Symptom | Cause | Fix | | --- | --- | --- | | Create returns `422` "none of the supplied failure_mode_ids belong to this project" | The mode ids are from another agent, are archived, or don't exist | Pass ids from **this** agent's non-archived modes ([Issues & failure modes](/guides/issues-and-failure-modes)) | | Suite marked `failed` with "no scenarios generated" | The selected modes have no exemplar traces **and** no LLM connection is configured | Pick modes that carry real exemplar traces, or configure an [LLM connection](/administration/llm-connections), then **regenerate** | | Scenarios are all `replay` (no paraphrase/escalate/boundary) | No LLM connection resolved | Configure an [LLM connection](/administration/llm-connections), then regenerate | | Launch returns `409` "suite is not ready yet" | The suite is still generating or failed | Wait for `ready` (poll `GET /scenario-suites/{id}`), or regenerate a failed suite | | The launched run behaves unexpectedly (scoring, baseline, gate) | It's a normal pre-prod run under the hood | See [Pre-prod evaluations → Troubleshooting](/guides/preprod-evals#troubleshooting) | ================================================================================ # Dashboards Source: /docs/guides/dashboards/ ================================================================================ # Dashboards When the built-in Overview and Agent home don't show exactly what your team watches, build your own dashboard: pick measures from the [metrics catalogue](/guides/metrics), slice them by dimension, and share the result with exactly the audience you intend. Neens also ships four ready-made **persona dashboards** — executive, product-quality, finance, and org-scorecard views — that work on every plan. ## At a glance | | | |---|---| | **Where** | The **Dashboards** page (your boards as cards, with a **Favorites** strip on top) | | **Key API routes** | `GET/POST /dashboards`, `POST /dashboards/{id}/widgets`, `GET /dashboards/{id}/widgets/{wid}/data`, `POST /dashboards/{id}/clone`, `GET /dashboards/by-slug/{slug}` | | **Data source** | Every widget resolves live through the [metrics catalogue](/guides/metrics) — no widget can query outside it | | **Scope** | A dashboard is pinned to an agent set, an org, or the whole workspace; viewers only ever see data from agents they can access | | **Plan** | Creating and cloning dashboards requires the Silver plan or above; *viewing* any dashboard (including the platform persona dashboards) works on every plan | ## Create a dashboard ### New dashboard On the **Dashboards** page, click **New dashboard**, give it a name, and pick a **scope & sharing** setting (see [Scope and visibility](#scope-and-visibility)). If you create it while an agent is selected, it defaults to that agent's scope and agent visibility. ### Add widgets Click **Add widget**. Either pick a ready-made **preset** from the gallery (one click drops in a fully configured chart, organized under **Volume · Quality · Latency · Errors · Cost · Topics · Business** tabs — the [Business](/guides/business-kpis) ones are measured per case and need a KPI definition first), or use the **Custom** tab to choose: - a **measure** — what to chart (traces, error rate, latency, spend, scores, …), - optional **dimensions** — what to break it down by (time, agent, model, your own metadata fields, …), - a **visualization** — one of `kpi`, `timeseries`, `bar`, `table`, `heatmap`, `donut`. The builder only offers valid measure × dimension combinations — the full menu is documented in the [metrics catalogue](/guides/metrics). Invalid combinations are rejected by the API (`422`), so a broken widget can never be saved. ### Arrange Widgets flow in order across a grid: drag a widget's edges to resize, drag its grip handle to reorder, or duplicate/remove it from its menu. ### Per-widget accent color In the **Custom** tab, pick a **Color** for the widget: the theme **Default**, a curated swatch, or your own hex value (`#RGB` or `#RRGGBB` — anything else is rejected). Single-series charts use the accent directly; a donut derives a harmonized multi-color palette from it. Tables and heatmaps ignore color. ### Per-widget scope By default a widget inherits the dashboard's scope. You can override an individual widget to its own scope — for example, one tile pinned to a single agent on an org-wide dashboard. A badge on the widget shows when it has its own scope. Widget scope overrides are access-checked the same way dashboard scopes are. ## Time ranges — the precedence Every dashboard has a shared time picker in its header with the standard presets (**Today · Last 24h · 7 days · 30 days · All time · Custom**). Widgets resolve their window in this order, most specific first: 1. **The widget's own pinned range** — an authoring choice that fixes one tile to a window. It beats even the header picker, so a deliberately pinned panel stays put. 2. **The header picker selection** — drives every non-pinned tile, so the whole board shares one window. 3. **The dashboard's saved default range** — the board's initial window (one of `today`, `24h`, `7d`, `30d`, `all`), used when no interactive selection is supplied. Editors set it from the dashboard header. 4. **The measure default** — all-time (no bound). When a dashboard has no saved default, the picker starts at the last 30 days. ## Scope and visibility A dashboard's **scope** pins what data it shows; its **visibility** controls who can see it. They're related but separate settings: | Scope | Data shown | |---|---| | **Agent** | One or more pinned agents | | **Org** | All agents in one org (resolved live — new agents appear automatically) | | **Tenant** | The whole workspace | | Visibility | Who sees the dashboard | |---|---| | **Private** | Only the owner | | **Agent** | Anyone with access to at least one of the pinned agents | | **Org** | Anyone with access to that org | | **Company** | Everyone in the workspace | Sharing is permission-gated: any member can create dashboards and share to agent or org visibility, but sharing **company**-wide requires an admin role. Deleting someone else's dashboard also requires admin; owners can always delete their own. **Sharing a dashboard never shares data.** Widget queries are always constrained to the *viewer's* accessible agents at render time. A company-visible dashboard scoped to agents a viewer can't access simply renders no data for them — there is no way to leak another team's numbers through a shared board. ## Favorites, cloning, deleting - **Favorite** (star) a dashboard to float it into the **Favorites** strip at the top of the Dashboards page. - **Clone** a dashboard — from the card's hover action or the header **Clone** button — to get a **private** copy you own, with every widget's configuration (including colors and scope overrides) deep-copied. A clone never inherits the original's audience: it always starts private, and any scope pins you can't access degrade to what you can see. Favorites aren't carried over. - **Clone a widget** in place to duplicate one tile within the same dashboard — it appends at the end at the default size, ready to tweak. - **Delete** a dashboard from its header (owner or admin). This can't be undone. ## Project Home **Project Home** is the landing page for the project you have selected — the built-in read-out of what your agent has been doing. Unlike a custom dashboard, its layout is fixed: you can't add or rearrange tiles, but it needs no setup and is the same for every project. Top to bottom: - **KPI tiles** — **Traces**, **Sessions**, **Total Spans**, **Session p50 / p99** and **LLM p50 / p99** latency, and **Failure Clusters**. Once you've shipped fixes, **Remediations applied** and **Remediations pending** tiles appear too. - **Top failure modes** and **Open issues** — two side-by-side lists linking into [Failure clustering](/guides/clustering) and [Issues & failure modes](/guides/issues-and-failure-modes). - **Four trend charts** — **Trace Volume**, **Latency Trend**, **Token usage over time**, and **Traces by model** (described below). Hover any chart to read exact values at a point in time. ### Time range Project Home has a time picker in its header with the standard presets — **Today · Last 24h · 7 days · 30 days · All time · Custom** — and opens on the last **30 days**. The KPI tiles, both charts, and the open-issues list all follow the selected window. The **Failure Clusters** tile and **Top failure modes** list are a snapshot of the latest clustering run rather than a windowed count, so they don't change with the picker. ### Token usage over time A **stacked-area** chart of the tokens your agent used across the window, split into two bands: - **Input tokens** — the tokens sent *to* the model (your prompts, context, and tool results). - **Output tokens** — the tokens the model generated *back*. They stack, so the **top of the shaded area is total token usage** at each point, and the two bands show how that total splits between prompt and completion. Use it to spot cost and usage spikes: a sudden jump in the total height means your agent burned far more tokens than usual, and which band grew tells you why. For example, if a prompt-template change accidentally pastes an entire document into every request, the **input** band roughly doubles while the **output** band stays flat — you'll see the stack jump in height on the day the change shipped, with all the growth in the lower band. That points you straight at the prompt rather than at the model's responses. Token usage tracks closely with spend — see [Cost & model pricing](/guides/cost-and-model-pricing) to turn tokens into dollars. ### Traces by model A **bar chart** counting traces by the LLM model each one used — one bar per model, tallest first — so you can see your model mix at a glance. Two buckets need a word of explanation: **Mixed** counts a single trace that called **two or more different models** (for example, a cheap model to route the request and a stronger one to write the final answer). Such a trace is counted **once**, in **Mixed**, instead of being added to every model it touched — so each named model's bar stays a clean count of traces that used *only* that model. **Unknown** counts traces with **no model attribution** — the trace didn't record which model was used (common when an integration doesn't set the model on its spans). It isn't an error; it just means Neens couldn't attribute the trace to a specific model. ## Platform persona dashboards Neens ships four platform-managed dashboards, one for each of the leadership-facing persona lenses. They are seeded into every workspace, render on **every plan**, and are kept up to date by the platform: | Dashboard | Audience | What it shows | |---|---|---| | **Executive digest** | Executives | Traces, error rate, eval pass rate, and spend — as headline KPIs, trends over time, and per-org breakdowns | | **Product quality board** | Product managers | Eval pass rate, average score, score volume — trends, per-judge averages, and the score distribution | | **Finance cost explorer** | Finance | Total spend, input/output tokens, average tokens per trace — sliced by model and org, and over time. Spend uses your [model price table](/guides/cost-and-model-pricing); widgets whose traffic includes an unpriced model are labelled *partial* | | **Org scorecard** | Org leads | Traces, error rate, and eval pass rate compared across orgs | All four are tenant-scoped (whole-workspace data), visible company-wide, and default to a 30-day window. **Platform dashboards are immutable to your workspace.** You can't edit, rearrange, or delete them — any attempt returns `403`. To customize one, **Clone** it: the copy is a normal private dashboard you fully own (cloning requires Silver+ like any dashboard creation). When a user picks (or is assigned) a persona lens, that persona's dashboards are automatically pinned to their Favorites, and personas whose home page *is* a dashboard resolve it by its stable slug (`GET /dashboards/by-slug/{slug}` — e.g. `exec-digest`, `pm-quality`, `finance-cost-explorer`, `org-scorecard`), so the right board loads no matter which workspace you're in. ## How it works - Dashboards and widgets are stored durably per workspace; **no data is snapshotted**. Every tile fetches live at render time (`GET /dashboards/{id}/widgets/{wid}/data`), resolving its measure through the same query layer the Overview page uses. - The effective scope for a tile is: its own scope override if set, else the dashboard's scope — then intersected with the viewer's accessible agents. - Widget definitions are validated against the metrics catalogue at save time *and* re-validated at render, so a stale widget referencing a removed combination fails loudly (`422`) rather than showing wrong numbers. - Dashboards record who created them and who added each widget, so a shared board shows attribution. ## Troubleshooting | Symptom | Cause | Fix | |---|---|---| | **New dashboard** is rejected | Your plan doesn't include custom dashboards | Custom dashboards are Silver+; viewing shared and platform dashboards still works on every plan | | Can't set **Company** visibility | Sharing company-wide is admin-only | Ask an admin to change visibility, or share at org/agent level | | Editing a platform dashboard returns an error | Platform dashboards are immutable | **Clone** it and edit the copy | | A shared dashboard shows no data for a teammate | Its scope pins agents they can't access | Data follows the viewer's access — grant agent/org membership, or re-scope the dashboard | | A tile ignores the header time picker | The widget has its own pinned range | Edit the widget and clear its range so it follows the picker | ## Related - [Metrics catalogue](/guides/metrics) — every measure and dimension the widget builder offers. - [Cost & model pricing](/guides/cost-and-model-pricing) — the prices behind every spend widget, and how to set your own. - [Model comparison](/guides/model-comparison) — the **Quality** tab's model presets: pass rate by model, by agent × model, and by model attribution. - [Insights](/guides/insights) — automatically detected anomalies, regressions, and issue spikes. - [Dashboards](/guides/dashboards) — the built-in Overview, Agent home, and Insights surfaces. ================================================================================ # Metrics catalogue Source: /docs/guides/metrics/ ================================================================================ # Metrics catalogue Everything you can chart in Neens comes from one semantic catalogue: a registry of **measures** (the things you can compute — trace counts, error rate, latency percentiles, spend, eval pass rate, …) and **dimensions** (the columns you can slice them by — time, org, project, agent, model, …). Widgets, digest tiles, and overview stats all resolve through it, so the same number means the same thing everywhere. ## At a glance | | | |---|---| | **Model** | Measure × Dimension: a query is a registered measure, optionally grouped by dimensions valid for it | | **Key API routes** | `GET /measures/catalogue` (the full menu), `GET /measures/fields` (your agent's dynamic metadata dimensions) | | **Safety** | Only valid measure × dimension combinations can be requested — never arbitrary SQL; filter values are always parameterized | | **Scope** | Every query is constrained to the caller's accessible agents | | **Used by** | [Dashboard](/guides/dashboards) widgets, the platform persona dashboards, the daily [digest email](/guides/insights#the-digest-email) tiles, and the Overview / Insights fleet stats | **Naming:** the measure keyed `sessions` is labeled **Traces** (individual agent runs), and `conversations` is labeled **Sessions** (traces rolled up by conversation) — matching how the rest of the app names them. See [Core concepts](/concepts). ## Measures Twenty measures ship, organized into the seven categories the widget gallery uses (**Volume · Quality · Latency · Errors · Cost · Topics · Business**). The headline set: - **Volume** — `sessions` (Traces), `conversations` (Sessions), `span_count` (Spans) - **Quality** — `eval_pass_rate`, `avg_score`, `score_count`, `cluster_count` - **Latency** — `avg_latency`, `p50_latency`, `p99_latency` - **Errors** — `error_rate` - **Cost** — `spend_usd`, `tokens_in`, `tokens_out`, `avg_tokens_per_trace` - **Topics** — `topic_size` - **Business** — `containment_rate`, `resolution_time_p50`, `resolution_time_p90`, `cost_per_case` Full measure reference — key, label, unit, and what each computes | Key | Label | Unit | Grain | What it computes | |---|---|---|---|---| | `sessions` | Traces | count | traces | Number of individual traces (agent runs) | | `conversations` | Sessions | count | traces | Distinct conversations — traces rolled up by conversation id | | `span_count` | Spans | count | spans | Number of spans (steps) recorded | | `error_rate` | Error rate | ratio | traces | Share of traces whose status is `error` | | `avg_latency` | Average latency | ms | traces | Mean end-to-end trace duration | | `p50_latency` | p50 latency | ms | traces | Median trace duration | | `p99_latency` | p99 latency | ms | traces | 99th-percentile trace duration | | `spend_usd` | Spend | usd | model calls (spans) | LLM cost — tokens × the model's price from the [price table](/guides/cost-and-model-pricing), computed at query time (not a stored column). Unpriced models contribute $0 and the result is labelled *partial* | | `tokens_in` | Input tokens | count | model calls (spans) | Sum of prompt tokens sent to models | | `tokens_out` | Output tokens | count | model calls (spans) | Sum of completion tokens generated | | `avg_tokens_per_trace` | Avg tokens / trace | count | traces | Average total tokens (in + out) per trace | | `eval_pass_rate` | Eval pass rate | ratio | scores | Share of scores at or above their threshold | | `avg_score` | Average score | score (0–1) | scores | Mean score value | | `score_count` | Total scores | count | scores | Number of scores recorded | | `cluster_count` | Failure clusters | count | clusters | Number of failure clusters found | | `topic_size` | Topic size | count | topic assignments | Traces assigned to each topic | | `containment_rate` | Containment rate | ratio | cases | Share of **decided** cases the agent handled with no escalation — see [Business KPIs](/guides/business-kpis) | | `resolution_time_p50` | Resolution time (p50) | ms | cases | Median time to resolve a case, from your measured outcomes | | `resolution_time_p90` | Resolution time (p90) | ms | cases | The slow tail — the 10% of cases that take longest to resolve | | `cost_per_case` | Cost per case | usd | cases | LLM spend divided by the cases handled in the range | Notes: - **Grain matters for slicing.** Trace-grain measures slice by trace attributes (agent, status, your metadata); span-sourced cost measures slice by `model`; score-grain measures slice by judge, metric, and — because the model and agent behind each score are recorded on the score itself — by `model`, `agent`, and `model_source`; case-grain measures slice by the attributes of a case's **first** trace. - `spend_usd`, `tokens_in`, and `tokens_out` share one source and one price table, so tokens and dollars always stay mutually consistent. A model with no price contributes tokens but no dollars, and the spend result carries a **partial** flag naming the unpriced models — see [Cost & model pricing](/guides/cost-and-model-pricing). - `topic_size` has no timestamp, so the `time` dimension and time-range filters don't apply to it. - The four **Business** measures are computed per **case** (`conversation_id`, falling back to the trace id), and `containment_rate` / `resolution_time_*` need the agent to have declared what an escalation and a resolution mean. They render with the provenance (**measured** vs **inferred**) and the coverage behind the number — see [Business KPIs](/guides/business-kpis). ## Dimensions A dimension only applies to measures whose grain carries it — the builder (and the API) enforce this, so you can't ask for "spend by judge" or "error rate by metric". - `time` and the scope dimensions `org` / `project` apply to (almost) everything — `time` buckets by **hour** or **day**. - Trace-grain slices: `agent`, `status`, `source`, `conversation`, plus the derived bins `latency_bucket` and `token_bucket`. - `model` has two homes. On the span-sourced cost measures (`spend_usd`, `tokens_in`, `tokens_out`) it's **spend and token volume per model**; those three also slice by `agent`, so **spend by agent** — which agent is burning the budget — is a first-class breakdown. On the score-grain measures (`eval_pass_rate`, `avg_score`, `score_count`) it's **quality per model** — the model that produced the answer each score graded. See [Model comparison](/guides/model-comparison). - Case-grain slices (the Business measures): `agent`, `status`, `source`, `conversation` — read off the case's **first** trace — plus `queue` and `outcome_meta:…` from the matched outcome's metadata. - Score-grain slices: `judge`, `score_source`, `score_label`, `target_type`, `model`, `agent`, and `model_source` (**Model attribution** — how the score's model was determined), plus `metric_key` (pass rate only) and the derived `score_bucket` distribution. - `span_kind` / `span_status` slice `span_count`; `topic` slices `topic_size`. Full dimension reference — key, label, and which measures it applies to | Key | Label | Applies to | |---|---|---| | `time` | Time | Every measure except `topic_size` (bucketed by `hour` or `day`) | | `org` | Org | All measures | | `project` | Project | All measures | | `agent` | Agent | Trace-grain: `sessions`, `conversations`, `error_rate`, `avg_latency`, `p50_latency`, `p99_latency`, `avg_tokens_per_trace`; the cost measures `spend_usd`, `tokens_in`, `tokens_out`; the score measures `eval_pass_rate`, `avg_score`, `score_count`; and every Business measure | | `model` | Model | `spend_usd`, `tokens_in`, `tokens_out` (the model that ran) and `eval_pass_rate`, `avg_score`, `score_count` (the model that produced the graded answer) | | `model_source` | Model attribution | `eval_pass_rate`, `avg_score`, `score_count` — how the score's model was determined (`preprod_run` / `span` / `session_uniform` / `mixed` / `unknown`) | | `status` | Status | `sessions`, `error_rate`, `avg_latency`, `p50_latency`, `p99_latency`, and the Business measures | | `source` | Source | `sessions`, `error_rate`, `avg_latency`, `p50_latency`, `p99_latency`, and the Business measures (ingest source: otlp / openinference / raw) | | `conversation` | Conversation | Trace-grain measures and the Business measures | | `queue` | Queue | `containment_rate`, `resolution_time_p50`, `resolution_time_p90` — the `queue` field of the matched **outcome's** metadata | | `outcome_meta:…` | *your outcome metadata key* | `containment_rate`, `resolution_time_p50`, `resolution_time_p90` — any other key in the outcome's metadata | | `latency_bucket` | Latency range | Trace-grain measures (derived bins of duration; group-by only) | | `token_bucket` | Token range | Trace-grain measures (derived bins of in + out tokens; group-by only) | | `metric_key` | Metric | `eval_pass_rate` | | `judge` | Judge | `eval_pass_rate`, `avg_score`, `score_count` | | `score_source` | Score source | `eval_pass_rate`, `avg_score`, `score_count` | | `score_label` | Score label | `eval_pass_rate`, `avg_score`, `score_count` | | `target_type` | Target type | `eval_pass_rate`, `avg_score`, `score_count` (session vs span scores) | | `score_bucket` | Score range | `score_count` (derived bins of the score value; group-by only) | | `span_kind` | Span kind | `span_count` | | `span_status` | Span status | `span_count` | | `topic` | Topic | `topic_size` (grouped by topic name) | | `meta:…` | *your metadata key* | Trace-grain measures — see below | Derived-bin dimensions (`latency_bucket`, `token_bucket`, `score_bucket`) can only be used as group-bys, not as filters. Every other dimension works as both a group-by and a filter. ### Dynamic metadata dimensions (`meta:` keys) Any top-level scalar key you send in a trace's metadata becomes a sliceable dimension of the form `meta:` + the key — for example, traces tagged with a `region` field can be grouped or filtered by `meta:region` on any trace-grain measure. There's nothing to register: - `GET /measures/fields` discovers the metadata keys present in your accessible traces (optionally for one agent with `?projectId=`), and the widget builder offers them automatically. - Keys must look like identifiers (letters, digits, `_`, `.`, `-`; up to 64 characters) — anything else is rejected. - Metadata dimensions apply to trace-grain measures only (the metadata lives on the trace). ## Where measures are used | Surface | How | |---|---| | [Custom dashboards](/guides/dashboards) | Every widget is a measure + dimensions + filters; the builder's menu *is* this catalogue (`GET /measures/catalogue`), including ~45 ready-made presets across the seven categories | | Platform persona dashboards | The Executive digest, Product quality board, Finance cost explorer, and Org scorecard are built entirely from catalogue measures — nothing bespoke | | [Digest email](/guides/insights#the-digest-email) | The fleet-health tiles (Traces, Error rate, Eval pass rate, Spend) resolve through the catalogue, so the email can never drift from the in-app numbers | | Overview & Insights | The workspace Overview KPIs/trends/segments and the Insights fleet line resolve through the same layer | ## How it works A request names a measure, dimensions, filters, and a time range. The catalogue validates the combination and emits a neutral query description; a resolver turns that into SQL for whichever backend stores your traces, **always** constrained to the agents you can access. Two guarantees fall out of this design: - **No invention:** an unknown measure, an inapplicable dimension, or a filter on a derived bucket is rejected with a clear error — nothing is silently approximated. - **No injection:** filter values are bound as parameters, and metadata keys are strictly validated, so no widget or API call can smuggle SQL. ## Related - [Dashboards](/guides/dashboards) — build widgets from these measures. - [Business KPIs](/guides/business-kpis) — the case-grain measures (containment, resolution time, cost per case), and how to declare what they mean in your data. - [Custom measures](/guides/custom-measures) — add measures of your own to this catalogue; they behave like the ones above everywhere. - [Cost & model pricing](/guides/cost-and-model-pricing) — where the prices behind `spend_usd` come from, and how to set your own. - [Model comparison](/guides/model-comparison) — slicing the quality measures by model and by agent × model, and what the **Mixed** / **Unknown** buckets mean. - [Insights](/guides/insights) — detectors that watch some of these signals for you. - [Core concepts](/concepts) — traces vs sessions, scores, and thresholds. ================================================================================ # Business KPIs Source: /docs/guides/business-kpis/ ================================================================================ # Business KPIs Nobody outside the engineering team asks how many traces the agent emitted. They ask **what share of cases the agent handled on its own**, **how long a case takes to resolve**, and **what a case costs to run**. Those are the numbers that end up in a board deck, and they are measured per **case** — one customer conversation — not per model call. The **Business KPIs** page is where those numbers live. It sits in your agent's left nav, and it gathers everything behind a business number onto one page: the KPIs themselves, the definitions that decide what "escalated" means for *your* data, the outcome feeds that supply the evidence, and the custom measures you add in your own vocabulary. **Every business KPI arrives with two things attached: its provenance and its coverage.** Provenance says whether the number came from your system of record, from your agent's own claim, from a deterministic Neens computation, or from an LLM's opinion. Coverage says what fraction of the cases in the window actually had a signal. A containment rate over 12% of your cases is a very different fact from the same number over 95% — and the page shows you which one you're looking at. ## At a glance | | | | --- | --- | | **Where** | The **Business KPIs** page in your agent's left nav (the gauge icon). | | **Tabs** | **Overview** · **Failure impact** · **Definitions** · **Data sources** · **Custom measures** | | **The KPIs** | Tiles on this page: **Containment** · **Resolution** · **Resolution time**. `cost_per_case` is computed from your traces and lives on [Cost optimization](/guides/cost-optimization), plus dashboard widgets and alerts. | | **Grain** | **Case** — `conversation_id` when the trace has one, the trace's own id when it doesn't. | | **Key API** | `GET /kpi-definitions` · `PUT /kpi-definitions` · `GET /kpi-definitions/options` · `GET /measures/catalogue` · `GET /kpis/{id}/eroding-clusters` | | **Who can view** | Anyone with read access to the agent sees the page and its numbers. Connecting or editing an outcome feed is admin-only, so a non-admin sees the match coverage on **Data sources** but not the feed management. | | **Who can change** | An **admin** declares the definitions and connects the outcome feeds; everyone else reads them. | | **Commitments** | Any catalogued measure can be [promoted to a KPI](#promote-a-measure-to-a-kpi) — a target, a direction, an owner, a status and a review cadence — over `GET`/`POST /kpis`, `GET /kpis/options` and `GET /kpis/summary`. **Promoting and committing is API-first for now**, so those examples are `curl`. | | **History** | Every active KPI is [recorded once per completed day](#history-and-trends), so it has a trend: `GET /kpis/{id}/history`, `POST /kpis/{id}/snapshot`, `POST /kpis/snapshot`. | | **Needs an LLM?** | No for the measured, emitted and derived paths. The [inferred path](#inferred-kpis-the-ready-made-classifiers) reads transcripts with an LLM classifier on your agent's configured connection — either the [ready-made escalation/resolution/sentiment classifiers](#inferred-kpis-the-ready-made-classifiers) or the [issue classifier](/guides/issues-and-failure-modes)'s existing labels. | | **Prerequisite** | Containment and resolution time need [business outcomes](/guides/business-outcomes) (or issue-classifier labels). Cost per case needs [model prices](/guides/cost-and-model-pricing) and nothing else. | ## The Business KPIs page The page is one place with five tabs. The tab you're on is in the URL, so you can link straight to one. (The time range resets to its default when you reload.) | Tab | What it's for | | --- | --- | | **Overview** | The KPI tiles for the window, a **Data health** strip that tells you how trustworthy they are, and the status of each definition. This is the glance a PM takes in the morning. | | **Failure impact** | Pick a KPI and see the [failure clusters eroding it](#failure-impact--which-failures-are-eroding-a-kpi), worst first — how far each one's rate sits below your agent baseline, and its share of the total erosion. Clusters too thin to measure are listed apart as *not enough data yet*, never as 0 impact. | | **Definitions** | Declare what each KPI means for your data — what counts as "escalated", which outcome carries the resolution time. Admin-only to change; anyone can read. See [Define a KPI](#define-a-kpi). | | **Data sources** | Connect and inspect the outcome feeds behind the measured numbers — push an outcome, run a [helpdesk connector](/guides/connectors/outcome-inlets), and read the match coverage. See [Business outcomes](/guides/business-outcomes). | | **Custom measures** | Define KPIs of your own — your label, your source, your denominator — that behave like the ones Neens ships. See [Custom measures](/guides/custom-measures). | **Reading the Overview honestly: the tiles fill in over time.** Today the Overview shows you your real **definitions**, their **provenance**, and the **match coverage** behind them — the machinery behind the number. The current *values* on the tiles populate as Neens accrues KPI snapshots for your window, so a freshly configured agent sees `—` on a tile, with the real coverage figure in the **Data health** strip above the tiles rather than a fabricated number. A dash means "no snapshot yet", never zero — and if a value could not be *read*, the tile says that instead, so you can tell a missing number from a failed one. Cost per case does not wait on any of this: it is [derived](#the-four-provenance-tiers) straight from your traces, and you can see it today on [Cost optimization](/guides/cost-optimization). ## What a case is, and why the KPI is per case A **case** is `conversation_id` when the trace carries one, and the trace's own id when it doesn't — the same rollup the **Sessions** page shows and the same grain [business outcomes](/guides/business-outcomes#the-case-grain) are keyed to. A ten-turn conversation is **one** case, not ten. That matters more than it sounds. "Contained" is a property of a conversation, not of a model call: a case where the agent answered nine turns and then handed off to a human is one escalation, not nine successes and one failure. Measuring per trace would let a chatty conversation outvote a terse one, and it would let a single case appear on both sides of the same ratio. Two consequences worth knowing before you read a chart: - **A case is attributed to the agent and the day of its first trace inside the window.** If a conversation starts on a triage agent and is picked up by a billing agent, the case counts once, against triage — which keeps a per-agent breakdown summing to the total instead of double-counting hand-offs. - **A case's identity is the pair (agent, case key).** `conversation_id` is *your* id — a helpdesk ticket number, an upstream session id — so a staging and a production agent can carry identical ids as a matter of routine. A widget spanning both counts them as two cases, not one. ## The four provenance tiers The same tile can be backed by four very different kinds of evidence, and Neens refuses to let them look identical. The provenance is **derived from where the number comes from**, never chosen — so a badge can never claim a number is stronger than the evidence behind it. | Provenance | Where the number comes from | A concrete example | | --- | --- | --- | | **Measured** | Your **system of record** reported it — a [business outcome](/guides/business-outcomes) from your helpdesk, CRM or warehouse. The strongest claim available. | Your Zendesk feed sends a `containment` outcome per ticket. Containment rate reads those facts directly. | | **Emitted** | Your **agent asserted it about itself**, in the trace metadata. Useful, but nothing outside the agent confirmed it. | Your agent writes `resolved: true` onto each trace. You can measure it — badged **Emitted** so nobody mistakes the agent's own word for the customer's. | | **Derived** | **Neens computed it** from your traces with a deterministic rule — no LLM, no external system. Reproducible from the spans. | `cost_per_case` — token counts on your traces multiplied by your model prices, divided by the cases in the window. Pure arithmetic over data you already send. | | **Inferred** | An **LLM classifier read the transcript and decided**. An opinion, not a fact — a directional signal when you have nothing better yet, and one Neens [calibrates against reality](#calibration-does-the-inference-agree-with-reality) where it can. | You have no outcome feed, so containment reads a [ready-made classifier](#inferred-kpis-the-ready-made-classifiers)'s label — `contained` vs `escalated_to_human` — on each sampled case. | **Emitted is not measured, and inferred is not measured.** If your agent writes `resolved: true` into its own metadata, that is the opinion of the same agent whose failures you are trying to find. If an LLM reads a transcript and calls a case "escalated", that is a guess about the past, not a record of it. Both are perfectly reasonable to measure — and both are badged so nobody reads them as the customer's own confirmation. Provenance travels with the number everywhere it renders — on the tile, in a dashboard widget, and next to an alert. A screenshot of a containment rate is only interpretable with its badge, which is exactly why the badge is never optional. **The derived tier is more than `cost_per_case`.** A whole family of business numbers is computed straight from your traces with no feed and no definition — turns and traces per case, case duration, cost per turn, tool success and retry rates, span error rate, first-response latency. They have a value on day one and are promotable to KPIs just like the ones on this page. See [Derived measures](/guides/derived-measures). ## How to read a KPI tile Every business tile shows more than a figure: - **The value** — or **`—`** when there's nothing honest to show. A dash means *no snapshot yet*, *no decided cases in the window*, or *unavailable for a stated reason*. It never means zero, and you never see `$0.00` standing in for "we don't know". - **A provenance badge** — **Measured**, **Emitted**, **Derived** or **Inferred** — so you know what kind of claim the number is. - **A coverage figure** — the fraction of the window's cases the definition could actually decide. A 92% containment rate over 8% of your cases is not a 92% containment rate. On the Business KPIs Overview you read this from the **Data health** strip; on dashboard widgets it rides on the tile itself as a coverage badge. On an [inferred](#inferred-kpis-the-ready-made-classifiers) tile the coverage reads **sampled** — a deliberately partial read, not a data gap; see [reading sampled coverage](#read-the-coverage-on-an-inferred-tile-sampled-is-not-missing). - **A partial marker on `cost_per_case`** when some models in the window have no price — the value is then a **floor**, not the whole spend. See [Cost & model pricing](/guides/cost-and-model-pricing). ## The named KPIs The Overview shows a tile per KPI, headed by its short name — **Containment**, **Resolution**, **Resolution time**. The measure keys below are what you use on dashboards, in alerts and over the API; `cost_per_case` is available in those places but is not a tile on this page. | Measure | Label | Unit | What it is | | --- | --- | --- | --- | | `containment_rate` | Containment rate | ratio | Share of **decided** cases the agent handled end-to-end, with no escalation. | | `resolution_time_p50` | Resolution time (p50) | ms | Median time to resolve a case. | | `resolution_time_p90` | Resolution time (p90) | ms | The slow tail — the 10% of cases that take longest. | | `cost_per_case` | Cost per case | usd | LLM spend in the window ÷ cases handled in the window. | ### Containment rate — the denominator is the whole point ``` containment_rate = contained cases / DECIDED cases ``` A case is **decided** when it carries a real signal — an outcome (or a classifier label) that says what happened. A case with no signal is **undecided**: it is excluded from *both* sides of the ratio, and reported as missing coverage. - **Only containment and escalation signals decide containment.** A *resolution* fact does not: "the case reached a resolution" is a different question from "the agent handled it", and a case escalated to a human and then resolved by that human is resolved and *not* contained. A feed that only reports resolutions therefore reports **no containment coverage** — the truth about that feed. - **A signal whose value can't be read decides nothing.** A `containment` fact carrying text instead of a yes/no is missing coverage, not a pass. **An unmeasured case is never counted as contained.** This is not configurable. If it were, a agent with 5,000 cases and 40 outcomes would report a 99% containment rate, and everything downstream of that number would be wrong in a way nothing else in the product would catch. A case counts as **escalated** if either of these is true: 1. it has an escalation signal with the value you declared as "escalated" (by default: an `escalation` outcome valued `true`); **or** 2. it has a containment signal whose value is **not** the value you declared as "contained" (by default: a `containment` outcome valued anything other than `true`). The second arm is load-bearing. Without it, a case your own backend explicitly recorded as *not contained* would read as "no escalation seen, therefore contained" — silently and systematically generous. `contained = decided AND NOT escalated`. ### Resolution time Per-case duration, reported as a p50 and a p90. There are three ways to obtain it, chosen per agent on the **Definitions** tab: | Mode | How the duration is obtained | | --- | --- | | **From the outcome's duration** | The outcome **carries** the elapsed time (a resolution-time or handle-time fact your helpdesk reports). Durations are normalized to milliseconds on the way in, so this is read directly — the most accurate option when you have it. | | **From the outcome's timestamp** | Neens computes it: the earliest resolving outcome's time minus the case's first trace start. | | **Unavailable** | There is no measured resolution instant. The measure reports **unavailable** — and this is forced for every classifier (inferred) definition. | **A classifier can't produce a resolution time, and Neens won't pretend otherwise.** A classifier label is a judgement about a trace; it records no moment at which the customer's problem was settled. Resolution time therefore reports *unavailable* with that reason, rather than falling back to the trace's own duration — which measures how long the *agent* ran, not how long the *case* took, and would usually be off by hours. An outcome dated **before** its case started produces a negative duration. Neens **excludes** it (and counts it as invalid) rather than clamping it to zero — a silently-clamped negative drags a median down and looks like an improvement. The percentile is **nearest rank**: p90 is the value at or below which at least 90% of cases resolved. ### Cost per case `cost_per_case` divides the LLM spend of the window by the cases in the same window, both computed from **one** pass over the same set of cases so they can never disagree. It is the **derived** KPI: pure arithmetic over your traces and your prices, no feed and no model required. It inherits everything from [cost & model pricing](/guides/cost-and-model-pricing): a model with no price contributes tokens but no dollars, so the ratio is a **floor** and carries a *partial* badge. A window with zero cases returns nothing (`—`), never `$0.00`. Unlike containment and resolution time, it needs no definition. Cost per case is not one of the Business KPIs tiles: you read it on [Cost optimization](/guides/cost-optimization), and it is available as a dashboard widget and an alert metric. ## Coverage Coverage is the honesty number under every business KPI: how many of the window's cases the definition could actually decide. The **Data health** strip at the top of the Overview is where you read it. It carries two stats and an as-of stamp: | The strip shows | Meaning | | --- | --- | | **KPI definitions** | How many KPIs this agent has, and whether they are **Configured** (you declared them) or **Platform defaults** (you haven't yet). | | **Outcome match rate** | The share of the window's cases that have a matched outcome, with the raw *X of Y matched* underneath. With no cases it reads *No outcomes yet*; if the figure could not be read it says so instead — an unread number is never reported as an empty one. | | **As of** | When the numbers were computed — see [late-arriving outcomes](#late-arriving-outcomes). Until snapshots exist it reads *No snapshot yet*. | The fuller breakdown — decided counts, invalid signals thrown out (e.g. an outcome dated before its case), and the per-kind split — is on the **Data sources** tab's match-coverage panel and in the coverage dashboard widgets. Coverage on a business KPI *is* your [outcome match rate](/guides/business-outcomes#read-the-coverage-panel) carried forward to the case grain. If it's low, the fix is almost always upstream on the **Data sources** tab: read the match coverage, find the correlation type that's failing to match, and fix it there. An entirely separate reason coverage can be low is that your feed simply only covers *some* cases (only closed tickets, only one queue) — a real gap that coverage is showing rather than hiding. ## Failure impact — which failures are eroding a KPI Coverage tells you how *trustworthy* a KPI is. The **Failure impact** tab tells you what to *do* about it: pick a KPI, and Neens ranks the [failure clusters](/guides/issues-and-failure-modes) dragging it down — the recurring failures that are costing you the most containment, or the most money per case — worst first. For each cluster, Neens computes the KPI two ways over the same window: once across that cluster's cases, and once across the whole agent (the **baseline**). The gap between them is the cluster's **erosion** — how far its rate sits below the number the rest of your traffic hits. Ranking the clusters by that gap turns "containment is at 87%, and I wish it were higher" into "these three recurring failures are where the missing points went, in this order". ### At a glance | | | | --- | --- | | **Where** | The **Failure impact** tab on the Business KPIs page; a chip on each [Failure modes](/guides/issues-and-failure-modes) card. | | **What it answers** | *Which failures are eroding this KPI, and by how much each?* | | **Defined for** | **Rate** KPIs (containment, and any rate custom measure) and the **cost per case** KPI. Not a percentile — see below. | | **Key API** | `GET /kpis/{id}/eroding-clusters` | | **Freshness** | Recomputed after each analysis run over your recent window (30 days by default). | | **Who can view** | Anyone with read access to the agent. | ### Reading the ranked list Every cluster in the ranked list carries three numbers that matter: - **Δ vs baseline** (`erosionRate`) — how far this cluster's rate sits below (for containment) or above (for a cost or error rate) the agent baseline. A containment cluster at 58% against an 87% baseline is **29 points** of erosion. - **Impact share** (`impactShare`) — this cluster's fraction of *all* the KPI's measured erosion, so the shares of the ranked clusters add toward 100%. This is the number that answers "if I could fix one thing, which?" — and it is deliberately weighted by size, not just by gap. A 29-point miss on 96 decided cases outranks a 40-point miss on 5, because it is responsible for more lost cases. - **Decided cases** (`clusterDecided`) — the denominator the verdict rests on. A big gap over a handful of cases is a smaller, less certain problem than a moderate gap over hundreds. **Impact share is weighted by lost cases, not by the raw gap.** Erosion is measured in *cases* — the count of contained (or cheap) cases the cluster would have delivered at the baseline rate. That is the quantity that adds up across clusters, which is what lets one cluster's share be compared to another's. A dramatic percentage gap on a tiny cluster is a small share; a modest gap on a large one can be the biggest share on the page. ### Reading the "not enough data yet" list Clusters with too few decided cases to tell a real regression apart from sampling noise are listed **separately**, as *not enough data yet* — never mixed into the ranked list, and never shown as *0 impact*. **"Not enough data yet" is not "harmless".** A cluster we could not measure is not a cluster that erodes nothing — those are different statements, and collapsing the first into the second is exactly how a real problem hides behind a green `0`. So a thin cluster gets its own list with a stated reason, its erosion and impact share both read `—` (never `0`), and it is never ranked as if it were fine. It moves into the ranked list on its own once it has enough decided cases. The reason a cluster is in this list is one of: | Reason | What it means | | --- | --- | | Not enough decided cases | The cluster has fewer decided cases than the minimum needed to separate signal from noise. Give it time, or fix your [coverage](#coverage) so more of its cases are decided. | | No baseline to compare against | The agent-wide number could not be computed for the window — usually no decided cases at all yet. | | No measurable cases in the cluster | The cluster carried no decided cases in the window. | | Cost only partly priced | *(cost per case only)* Some models in the window have no price, so the cost is a **floor** — subtracting two floors would produce a number nobody can defend, so Neens declines rather than guess. Fill the gaps under **Settings → Model pricing**. | ### Rate KPIs and the cost KPI — and the one type it can't attribute Erosion is defined for the two kinds of KPI where "worse than baseline" has an unambiguous meaning: - **Rate KPIs** — containment, and any rate [custom measure](/guides/custom-measures). Erosion is the proportion gap, always oriented by the KPI's own [direction](#the-fields-and-what-each-one-is-for), so a *lower* containment cluster and a *higher* error-rate cluster both rank as erosion. - **Cost per case** — erosion is the **extra spend** a cluster's cases cost above the baseline cost per case, in dollars. It refuses (lands in the *not enough data yet* list) whenever the window is only partly priced, for the same reason [cost per case](#cost-per-case) itself carries a *partial* badge. **A percentile KPI has no erosion to attribute.** Resolution time (p50/p90) is a percentile, not a proportion — there is no baseline rate for a cluster to sit below — so the tab says *erosion isn't defined for this KPI* rather than inventing a ranking. The API says the same thing with `attributable: false` and two empty lists, so a client can tell "this KPI can't be attributed" apart from "nothing is eroding it". ### The chip on a failure mode You don't have to open this tab to see the connection. On the [Failure modes](/guides/issues-and-failure-modes) page, each failure-mode card carries a small chip naming the **one KPI it erodes most** and its share of that KPI's total erosion — *"Eroding Containment · 34%"* — linking the failure you're triaging straight to the business number it's costing. A cluster with no measurable erosion carries **no chip** rather than a *0%* one, for the same reason it sits in the *not enough data yet* list: an empty chip would read as "measured and harmless". ### Measured vs estimated remediation impact The ranking feeds the [remediations](/guides/remediations) queue. When a cluster has a measured erosion of a KPI, the remediation Neens proposes for that cluster takes its **impact** — the *I* in its priority score — from that real erosion share, and marks it **measured**. When there is no measured erosion to lean on, the impact stays a model's estimate, marked **estimated**. **A measured priority and a guessed one never render identically.** A remediation whose priority is grounded in a KPI's real erosion by that cluster shows **measured**; one resting on the model's own guess shows **estimated**. Same score, different confidence in it — so you can tell "this is ranked high because it demonstrably costs us containment" apart from "the model thinks this looks important". ### Over the API `GET /kpis/{id}/eroding-clusters` returns the same two lists the tab draws: `ranked` (worst first, each with its `impactShare`) and `unknown` (the thin clusters, with a reason). Read access is enough. ```bash curl -sf "$NEENS_BASE_URL/api/kpis/$KPI_ID/eroding-clusters" \ -H "Authorization: Bearer $NEENS_API_KEY" ``` ```json { "kpiId": "a68e0cbb059f4b0ba38f409ee2145b19", "measureKey": "containment_rate", "unit": "ratio", "direction": "higher_is_better", "windowDays": 30, "attributable": true, "ranked": [ { "clusterId": "cl_7f21a3", "clusterLabel": "Refund policy mis-quoted, customer escalates", "clusterSessionCount": 118, "clusterStatus": "active", "status": "known", "unknownReason": null, "clusterCases": 118, "clusterDecided": 96, "clusterValue": 0.5833, "baselineCases": 812, "baselineDecided": 704, "baselineValue": 0.8731, "erosionUnits": 27.82, "erosionRate": 0.2898, "ci": [0.485, 0.676], "distinguishable": true, "impactShare": 0.624, "computedAt": "2026-08-19T02:14:07Z" }, { "clusterId": "cl_3b90e2", "clusterLabel": "Order-lookup tool times out mid-conversation", "clusterSessionCount": 160, "clusterStatus": "active", "status": "known", "unknownReason": null, "clusterCases": 160, "clusterDecided": 140, "clusterValue": 0.7536, "baselineCases": 812, "baselineDecided": 704, "baselineValue": 0.8731, "erosionUnits": 16.73, "erosionRate": 0.1195, "ci": [0.676, 0.820], "distinguishable": true, "impactShare": 0.376, "computedAt": "2026-08-19T02:14:07Z" } ], "unknown": [ { "clusterId": "cl_c14af8", "clusterLabel": "Non-English greeting not recognised", "clusterSessionCount": 11, "clusterStatus": "active", "status": "unknown", "unknownReason": "insufficient_decided", "clusterCases": 11, "clusterDecided": 7, "clusterValue": 0.5714, "baselineCases": 812, "baselineDecided": 704, "baselineValue": 0.8731, "erosionUnits": null, "erosionRate": null, "ci": [0.25, 0.84], "distinguishable": false, "impactShare": null, "computedAt": "2026-08-19T02:14:07Z" } ] } ``` Reading it: | Field | What it tells you | | --- | --- | | `attributable` | `true` for a rate or cost KPI; `false` for a percentile, where both lists are empty and the tab says erosion isn't defined for it. | | `ranked` | The clusters with a defensible erosion, **worst first** (largest `erosionUnits`). | | `unknown` | The thin clusters, listed apart with an `unknownReason`. Their `erosionUnits`, `erosionRate` and `impactShare` are `null` — rendered as **—**, never `0`. | | `clusterValue` · `baselineValue` | The KPI over this cluster's cases, and over the whole agent — the two numbers whose gap is the erosion. | | `erosionRate` | The per-case gap vs baseline, oriented by the KPI's `direction`. For containment above, `0.2898` is a 29-point drop. | | `erosionUnits` | The gap expressed in **cases** — the "lost" contained (or cheap) cases the cluster is responsible for. This is what the ranking and the shares are computed from. | | `impactShare` | This cluster's `0..1` fraction of the KPI's total measured erosion. The shares of the `ranked` clusters sum toward 1. `null` when it couldn't be formed — never `0`. | | `distinguishable` | Whether the sample can tell this cluster's rate apart from the baseline. `false` means the gap is inside sampling noise; a thin cluster reads `false` and sits in `unknown`. | | `clusterDecided` · `baselineDecided` | The denominators behind each side. A 90% rate over 7 decided cases is not the fact a 90% rate over 700 is. | | `computedAt` | When this verdict was last recomputed. | **A `null` renders as "—", and it always means "we could not measure this", never "zero".** A cluster in the `unknown` list, a KPI that can't be attributed, an impact share that couldn't be formed — all read as a dash, the same honesty rule the [KPI tiles](#how-to-read-a-kpi-tile) follow. ## Fixes that moved this KPI [Erosion](#failure-impact--which-failures-are-eroding-a-kpi) tells you what is dragging a KPI down. The other half of the story is what has pushed it back up: the shipped fixes that actually moved this number. Every KPI now lists them, so *"containment recovered"* arrives with *"…and here are the merged fixes that recovered it."* The list comes from the same [Fix outcomes](/guides/post-merge-efficacy) close-out that measures whether a merged fix reduced its failure's volume. Alongside that volume leg, each close-out measures every active KPI before vs after the deploy; the ones that moved **this** KPI are gathered here, newest merge first — each with the before→after Neens measured around the deploy and whether the move stood out from sampling noise. **Unmeasurable movers are listed apart, never ranked as `0`.** A fix whose KPI move could not be formed honestly — too thin a sample, no *before* data, or the [definition changed](#a-definition-change-breaks-the-series) between the windows — is listed separately in `movedByFixesUnknown`, with its delta shown as **—**. It is never folded into the ranked movers as a zero, the same rule the [eroding-clusters](#failure-impact--which-failures-are-eroding-a-kpi) list follows. ### Read it over the API `movedByFixes` and `movedByFixesUnknown` ride on `GET /kpis/{id}`, [beside the KPI's value](#read-the-current-value). Read access is enough. ```bash curl -sf "$NEENS_BASE_URL/api/kpis/$KPI_ID" \ -H "Authorization: Bearer $NEENS_API_KEY" ``` ```json { "kpi": {"id": "a68e0cbb059f4b0ba38f409ee2145b19", "measureKey": "containment_rate", "label": "Self-serve containment", "…": "…"}, "value": 0.8731, "…": "…", "movedByFixes": [ { "closeoutId": "co-9f2a1c", "remediationId": "rem-4471", "remediationTitle": "Guardrail over-blocks refunds", "prUrl": "https://github.com/acme/agent/pull/42", "status": "improved", "delta": 0.19, "deltaPct": 30.65, "distinguishable": true, "before": 0.62, "after": 0.81, "mergedAt": "2026-07-10T00:00:00Z", "verdictAt": "2026-07-17T02:11:00Z", "clusterId": "cl_7f21a3" }, { "closeoutId": "co-71bd08", "remediationId": "rem-4390", "remediationTitle": "Order-lookup tool retries on timeout", "prUrl": "https://github.com/acme/agent/pull/38", "status": "flat", "delta": 0.01, "deltaPct": 1.16, "distinguishable": false, "before": 0.86, "after": 0.87, "mergedAt": "2026-07-02T00:00:00Z", "verdictAt": "2026-07-09T02:07:00Z", "clusterId": "cl_3b90e2" } ], "movedByFixesUnknown": [ { "closeoutId": "co-55c0a2", "remediationId": "rem-4210", "remediationTitle": "Escalation-label taxonomy revised", "prUrl": "https://github.com/acme/agent/pull/31", "status": "unknown", "delta": null, "deltaPct": null, "distinguishable": null, "before": null, "after": null, "mergedAt": "2026-06-20T00:00:00Z", "verdictAt": "2026-06-27T02:04:00Z", "clusterId": "cl_c14af8" } ] } ``` | Field | What it tells you | | --- | --- | | `closeoutId` · `remediationId` · `remediationTitle` · `prUrl` | Which fix — and a link straight to the merged PR. | | `status` | `improved` · `flat` · `regressed` · `unknown`, oriented by the KPI's [direction](#the-fields-and-what-each-one-is-for). Every `unknown` sits in `movedByFixesUnknown`, not here. | | `before` · `after` | The KPI before the deploy and after it. `null` (→ **—**) for an unmeasurable mover. | | `delta` · `deltaPct` | The move, in the KPI's own unit and as a percentage. `null` when there is no honest delta to draw. | | `distinguishable` | Whether the move cleared sampling noise. `false` means it moved, but inside the noise — the list says so rather than celebrate it. `null` for a cost KPI, which carries no interval. | | `mergedAt` · `verdictAt` | When the fix merged, and when its close-out settled. The list is ordered by newest `mergedAt` first. | | `clusterId` | The failure cluster the fix targeted — the corner of your traffic whose recovery this move reflects. | **A `null` here is a dash, and means "we could not measure this move", never "zero".** A mover in `movedByFixesUnknown`, a `delta` that couldn't be formed across a definition change — both render as **—**, the same honesty rule the [KPI tiles](#how-to-read-a-kpi-tile) and the [eroding-clusters](#failure-impact--which-failures-are-eroding-a-kpi) list follow. The full **two-scope** breakdown for any of these fixes — the KPI over just the fix's cluster vs over the whole agent — lives on its [Fix outcomes close-out](/guides/post-merge-efficacy#did-the-kpi-move), where the same before→after is reported for both scopes. ## Define a KPI The **Definitions** tab is where an admin tells Neens what each KPI means for your data. Members can read the effective definition; only an admin can change it. ### Open the Definitions tab It shows the **effective** definition for each KPI — your declarations merged over sensible platform defaults — with a marker on anything you haven't set yourself, and a *"Not configured yet"* banner until you save for the first time. ### Pick the signal, then declare what "escalated" means Choose whether the KPI reads a **measured outcome** or an **inferred classifier label**, then pick the kinds and values (or the label set) that mean *escalated* and *contained*. The dropdowns are populated from **what this agent has actually recorded** — the outcome kinds you've received and the exact labels the classifier has really emitted, each marked *seen in your data* or *not seen yet*. Free-typing a kind no exporter sends is the fastest way to a permanent 0%, so the picker doesn't offer one. ### Save, then check the coverage Save, then look at the Overview. The coverage badge under each tile tells you how many of the window's cases your definition actually decided. If it's low, the fix is on the **Data sources** tab. **Choosing labels is a real modelling decision, not a checkbox.** "Escalated" means *the customer had to go somewhere else to get this done* — not *the agent did something wrong*. A hallucination is a serious quality failure, but the agent still handled the case; folding every failure mode into the escalation set quietly redefines containment as "the classifier found nothing". Watch for a label whose name reads backwards, like *"Human escalation not triggered"* — that describes a case that was **not** escalated. ### Do it over the API Everything the tab does is available over REST, so CI or your infrastructure-as-code can set it. Read the effective definition with an agent API key; change it with an admin credential. ```bash # Read the effective definition (any read access) curl -sf "$NEENS_BASE_URL/api/kpi-definitions" -H "Authorization: Bearer $NEENS_API_KEY" # Discover what you can pick from — the catalogue annotated with what THIS agent has observed curl -sf "$NEENS_BASE_URL/api/kpi-definitions/options" -H "Authorization: Bearer $NEENS_API_KEY" ``` Declaring a definition is a **full replacement**, sent by an admin session — omit a KPI to revert it to the platform default. A partial merge is how half a definition produces a number nobody intended, so Neens doesn't do one. ```bash curl -sf -X PUT "$NEENS_BASE_URL/api/kpi-definitions" \ -H "Authorization: Bearer $SESSION_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "definitions": { "containment": { "mode": "outcome", "escalationKind": "escalation", "escalationValue": true, "containmentKind": "containment", "containmentValue": true }, "resolution_time": { "mode": "outcome", "resolutionKind": "resolution_time", "resolutionTimeMode": "measured_duration" } } }' ``` `resolution_time.resolutionKind` names the kind that carries the **duration**, which is usually not the same kind as the boolean that says a case was resolved. Point it at a duration kind (`resolution_time`, `handle_time`); pointed at a boolean, the measure reports *unavailable* and names the mistake rather than reading `true` as a one-millisecond case. ```bash curl -sf -X PUT "$NEENS_BASE_URL/api/kpi-definitions" \ -H "Authorization: Bearer $SESSION_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "definitions": { "containment": { "mode": "classifier", "classifierMetricKey": "issue_class", "escalationLabels": ["Unwarranted Refusal", "Unrecovered Tool Error", "Incomplete Answer"] } } }' ``` Resolution time is forced to *unavailable* for any classifier definition — a classifier records no resolution instant. A classifier containment definition with **no** escalation labels is rejected: it would express no opinion and read 100% forever. A definition is validated **at write time**, so a malformed one is a `422` on the `PUT` and never a silently-wrong number on a dashboard later. ## Connect the data behind a measured KPI The **Data sources** tab is the measured path's home. There are two ways to get outcomes in, and both land on this tab: - **Push** — your ETL or backend `POST`s outcomes to Neens. Full control over the mapping. - **Pull** — a scheduled [helpdesk connector](/guides/connectors/outcome-inlets) (Zendesk, Intercom, Salesforce, Jira Service Management) that an admin sets up once. Either way, the tab shows the **match coverage** — the single most important number on it — so you can see what fraction of your outcomes actually matched a case before you trust any KPI built on them. Connecting a feed is an admin action; every member can read the coverage. The full model is in [Business outcomes](/guides/business-outcomes). ## Inferred KPIs: the ready-made classifiers Some cases arrive with a fact about what happened — a helpdesk disposition, a CRM outcome. Many don't. When you have no [outcome feed](#connect-the-data-behind-a-measured-kpi) for a question, Neens can still put a number on the board: it **reads the transcript with an LLM classifier** and labels each case itself. That number is the [**inferred**](#the-four-provenance-tiers) tier — it wears the purple **Inferred** badge everywhere it renders, because it is Neens' opinion about the past, not a record of it. You don't have to build a classifier to use this path. Neens ships three ready-to-use ones, each labelling a case on a single axis: | Classifier | The label it writes on each case | What it powers | | --- | --- | --- | | **Escalation** | `escalated_to_human` · `handoff_requested` · `contained` | An inferred **containment** rate — the share of decided cases it labels `contained`. | | **Resolution** | `resolved` · `unresolved` · `unclear`, judged from the closing turns | An inferred **resolution** rate. | | **Sentiment** | `frustrated` · `neutral` · `satisfied` | A directional read on how a case *felt* — back it with a [custom measure](/guides/custom-measures). | **An inferred number is a starting point, not a destination.** It exists so a brand-new agent isn't stuck on *"no data"* while it wires up a real feed — a directional read you can act on today, honestly badged so nobody mistakes it for the customer's own word. As soon as the [measured](#the-four-provenance-tiers) signal lands, point the KPI at *that*; the inferred one was scaffolding, and Neens tells you [how far it agreed with reality](#calibration-does-the-inference-agree-with-reality) so you know when it's safe to lean on. ### Turn a classifier on ### Enable the classifier The three classifiers are **opt-in**. Turn one on from your agent's classifiers area, alongside your [judges](/guides/judges) and scorers. Until you do, it writes nothing and costs nothing — an inferred containment tile has no labels to read and stays blank. ### Let it label your cases Once it's on, Neens starts reading conversations and writing that classifier's label onto each case it [samples](#the-classifiers-sample-your-whole-population). Because it uses an LLM, it runs on your agent's configured [LLM connection](/guides/connectors) (Settings → Connections); an agent with no connection can't label, so its inferred tiles stay blank until one is set. ### Point a KPI at the labels On the [**Definitions**](#define-a-kpi) tab, set the KPI's signal to **Inferred classifier label** and choose the labels that mean *escalated* — for the Escalation classifier, `escalated_to_human` and `handoff_requested`, leaving `contained` as the settled side. The picker only offers labels the classifier has **actually emitted**, each marked *seen in your data* or *not seen yet*, so you can't typo your way to a permanent 0%. Save, and an **Inferred** containment tile appears. **An inferred KPI can never produce a resolution *time*.** A classifier label is a judgement about a transcript; it records no moment at which the customer's problem was settled. So the Resolution classifier can back a resolution *rate*, but [resolution time](#resolution-time) stays *unavailable* with that reason on any classifier definition — Neens won't fall back to the trace's own duration, which measures how long the *agent* ran, not how long the *case* took. ### The classifiers sample your whole population [Failure clustering](/guides/issues-and-failure-modes) looks only at your *failing* sessions — that's the whole point of it. The inferred classifiers do the opposite. They sample across your **entire population**: a share of your traffic each day, spread across your agents and your outcomes, so the resulting rate reflects the **fleet**, not just the cases that went wrong. A containment rate built from failures alone would describe nothing; one built from a representative slice of everything is a real estimate. Reading a slice rather than every case is deliberate — scoring every conversation with an LLM would be costly and unnecessary. A well-spread daily sample estimates a fleet-wide rate without touching every conversation, which is exactly why the coverage on an inferred tile is **expected to be a fraction**, and says so. ### Read the coverage on an inferred tile: "sampled" is not "missing" Every KPI tile carries a [coverage figure](#how-to-read-a-kpi-tile). On a **measured** KPI, a low coverage is usually a *problem* — a feed that isn't matching, a queue you forgot to send. On an **inferred** KPI it is the design working: Neens scored a representative sample, not the whole population, so the tile labels its coverage **sampled** rather than leaving you to read `12%` as a data gap. So on an inferred containment tile you'll see something like **sampled · 12%/day across agents and outcomes**. Read that as *"Neens measured roughly 12% of this window's cases each day, balanced across your agents and outcomes"* — a healthy sample, not 88% missing data. Hover the coverage for the raw *measured X of Y cases so far*. **A `12%` on a measured tile and a `12%` on an inferred tile mean opposite things.** On the measured tile, 88% of your cases had no outcome — go fix the feed. On the inferred tile, Neens deliberately sampled 12% to keep the read cheap and representative — nothing is wrong. The **sampled** label is how you tell the two apart at a glance, so you never chase a coverage gap that was the plan all along. ### Calibration: does the inference agree with reality? An inferred number is only worth trusting if it agrees with what really happened. An LLM classifier can be systematically wrong — too eager to call a case escalated, too generous about resolution — and a containment rate printed to two decimals hides exactly that uncertainty. So whenever an inferred KPI *also* has a measured counterpart — inferred containment on the cases where your helpdesk **also** recorded an escalation, say — Neens computes a **calibration**: how well the classifier's labels agree with the measured outcome on the cases where you have **both**. It appears on the tile as a plain sentence: > **inferred containment agrees with measured at κ=0.71 over 312 cases** **κ is Cohen's kappa — chance-corrected agreement.** Read it on a simple scale: **1.0** is perfect agreement, **~0** is no better than a coin flip, and a **negative** κ means the classifier disagrees with reality more often than random guessing would. So κ=0.71 over 312 cases is a strong signal that this classifier is tracking your real escalations; you can lean on the inferred number where the measured one runs out. **When there aren't enough matched cases yet, Neens says so — it never prints a number it can't stand behind.** A κ computed from a dozen labels is a coin flip wearing a decimal point, so below a minimum count of matched pairs the calibration reads *"Not enough measured cases to calibrate yet"* instead of a figure. A KPI that is **not** inferred (an outcome-backed one) has no classifier to check, so it shows no calibration at all — there's nothing to calibrate against itself. **Reading a low κ.** A κ near 0 (or below it) is telling you the inference has drifted from reality — the classifier and your system of record are describing different worlds. When that happens, trust the **measured** KPI over the inferred one, and either retune what you count as *escalated* on the [Definitions](#define-a-kpi) tab or, better, move the KPI onto the measured feed entirely. Calibration is the dial that tells you *when* the scaffolding has done its job and can come down. The full calibration block — κ plus the classifier's precision and recall against the human labels, over a trailing window — is available over the API at [`GET /kpis/{id}/calibration`](/api-reference/more#get-kpiskpi_idcalibration). ### End-to-end: Acme turns on Escalation Acme's support agent has traces but no helpdesk feed wired up yet, so its containment tile is blank. ### Enable the Escalation classifier An admin turns on the **Escalation** classifier. Neens begins sampling Acme's conversations and labelling each `escalated_to_human`, `handoff_requested` or `contained`. ### Declare inferred containment On **Definitions**, Acme sets containment to **Inferred classifier label** and picks `escalated_to_human` and `handoff_requested` as the escalated labels. An **Inferred** containment tile appears reading **78%**, with coverage shown as **sampled · 14%/day** — the sampler working as intended, not a missing 86%. ### Wire up the helpdesk, and watch the calibration A week later Acme connects its helpdesk [outcome feed](#connect-the-data-behind-a-measured-kpi). Now some cases carry *both* the classifier's label and a recorded escalation, so the tile grows a calibration line: **inferred containment agrees with measured at κ=0.71 over 312 cases**. κ=0.71 is strong, so Acme trusts the inferred 78% for the corners its feed doesn't yet cover — and knows exactly when to switch the KPI over to the measured signal. ## Add a measure of your own Three named KPIs prove the idea; they don't describe *your* business, because "handled it" means something different for support, sales, IT and collections. The **Custom measures** tab lets an admin declare a measure in your own vocabulary — your label, your source, your denominator — that then behaves exactly like one Neens ships: it charts on a dashboard, backs an alert rule, and carries its own provenance and coverage. A custom measure's source decides its badge, the same four tiers as above. See [Custom measures](/guides/custom-measures) for the walkthrough. ## Starting from nothing A brand-new agent isn't stuck. The Overview offers three real ways to a first number, in increasing order of trust: 1. **Cost per case, today.** It's [derived](#the-four-provenance-tiers) from your traces and prices — no feed, no definition. As soon as you have priced traffic it has a value, which you read on [Cost optimization](/guides/cost-optimization) (the empty state links you straight there). 2. **Connect a feed, for a measured number.** Wire up an [outcome feed](/guides/business-outcomes) on the Data sources tab and declare containment against it — the strongest evidence there is. 3. **Attach a classifier, for a directional number.** No feed yet? Turn on a [ready-made classifier](#inferred-kpis-the-ready-made-classifiers) — escalation, resolution or sentiment — or point containment at your [issue-classifier](/guides/issues-and-failure-modes) labels, for an **inferred** rate while you build the real thing. Neens [calibrates it against your measured feed](#calibration-does-the-inference-agree-with-reality) once both exist, so you know when to trust it. ## Promote a measure to a KPI A definition says what a number **means**. A **KPI** says what you **promised** about it: the figure you committed to, which direction is good, who owns it, whether the commitment is live, and how often somebody is supposed to look at it. Those are different facts with different lifetimes — a definition changes when your data changes, a commitment changes when your goals do — so Neens keeps them apart. | Thing | The question it answers | Scope | | --- | --- | --- | | [KPI definition](#define-a-kpi) | *What does "escalated" mean in our data?* | Agent · the three business measures | | [Custom measure](/guides/custom-measures) | *What number do we care about that isn't in the catalogue?* | Company | | **KPI** | *What did we commit to on this number, and who owns that promise?* | Agent | A KPI is deliberately thin. It **references** a measure key and carries only the promise — it never redefines a measure, never recomputes anything, and holds no number of its own. Every value a KPI displays is resolved through the same path a dashboard widget uses, which is what guarantees that a KPI and a widget pointed at the same measure can never disagree: not about the value, not about the provenance, not about the coverage, and not about the blank. Anything in the catalogue can be promoted — the three business measures on this page, an operational measure like error rate, or one of your own [custom measures](/guides/custom-measures) (`custom:`). ### The fields, and what each one is for | Field | Required | What it's for | | --- | --- | --- | | `measureKey` | ✅ | Which measure this is a commitment about. **Immutable** — see [Changing one](#changing-one). | | `label` | ✅ | The name your organisation uses for it, which is rarely the catalogue's. Max 120 characters. | | `description` | — | Why this is a commitment and what a reader should do about it. Max 500 characters. | | `direction` | defaulted | `higher_is_better` or `lower_is_better`. This is the field that decides whether a value **above** the target is a success or a failure — nothing else does. | | `target` | — | The committed figure, in the measure's own unit. `null` is a real state, not zero. | | `owner` | — | Free text: a person, a team, a rota. Max 200 characters. Neens does not resolve it to an account — the point is that a name appears next to a promise. | | `status` | `draft` | `draft` (authored, not committed) → `active` (live) → `archived` (retired). | | `priority` | `100` | Display rank, 1–999, **lower sorts first**. The default is mid-range so a new KPI lands after the ones somebody bothered to rank, without renumbering the list. | | `reviewCadence` | `none` | `none` · `weekly` · `monthly` · `quarterly`. Declarative: it records how often this is supposed to be looked at. `none` is explicit rather than a null, so "deliberately unscheduled" and "nobody decided" stay distinguishable. | **Direction is proposed, never inferred.** Neens pre-fills it from the measure's semantics — containment is `higher_is_better`, cost per case and resolution time are `lower_is_better`, a custom measure reuses the direction its author already declared — and you can always override it. What is stored is what every later read uses; nothing looks at the data and guesses which way is good at render time. **A ratio target is a fraction, not a percentage.** `containment_rate` has unit `ratio`, so an 85% commitment is `0.85`. A target of `85` is rejected with a `422` naming the unit rather than stored — otherwise the KPI would read *missed* forever while the agent was doing fine. Targets on `count`, `ratio`, `ms`, `seconds` and `usd` measures must also be `>= 0`. ### Find out what you can promote `GET /kpis/options` returns the measures this agent may promote, each annotated with two things you want to know **before** you commit: whether the agent can actually get a number out of it today (`ready`), and whether it is already promoted. ```bash curl -sf "$NEENS_BASE_URL/api/kpis/options" \ -H "Authorization: Bearer $NEENS_API_KEY" ``` ```json { "projectId": "proj_x", "measures": [ { "key": "containment_rate", "label": "Containment rate", "unit": "ratio", "agg": "rate", "category": "business", "grain": "case", "isCustomMeasure": false, "provenance": null, "defaultDirection": "higher_is_better", "promoted": false, "promotedKpiId": null, "ready": true, "unavailable": null }, { "key": "cost_per_case", "label": "Cost per case", "unit": "usd", "agg": "ratio", "category": "business", "grain": "case", "isCustomMeasure": false, "provenance": null, "defaultDirection": "lower_is_better", "promoted": true, "promotedKpiId": "3d81f0a742be4c1e9a05d6b3f81c47ab", "ready": true, "unavailable": null }, { "key": "custom:deflection-rate", "label": "Deflection rate", "unit": "ratio", "agg": "rate", "category": "business", "grain": "case", "isCustomMeasure": true, "provenance": "measured", "defaultDirection": "higher_is_better", "promoted": false, "promotedKpiId": null, "ready": false, "unavailable": {"reason": "no_matching_outcomes"} } ], "directions": ["higher_is_better", "lower_is_better"], "statuses": ["draft", "active", "archived"], "reviewCadences": ["none", "weekly", "monthly", "quarterly"], "max": 24, "remaining": 21 } ``` The list is ordered so the top of it is something you can promote right now: ready measures first, then ones already promoted, then alphabetically. | Annotation | What it means | | --- | --- | | `ready` | Whether this agent can get a number out of the measure **today**. | | `unavailable` | `null` when `ready`, otherwise `{"reason": ""}` — `definition_not_configured` (the measure's meaning hasn't been [declared](#define-a-kpi) yet), `no_matching_outcomes` or `no_matching_scores` (the signal the measure reads has never arrived in this agent). | | `promoted` / `promotedKpiId` | Whether a live KPI already exists on this measure, and which one — what keeps you from creating a [second commitment](#one-live-kpi-per-measure-and-the-cap) on the same number. | | `defaultDirection` | The proposal you can accept or override. | | `provenance` | For a custom measure, the badge it will carry (**Measured** / **Inferred**). `null` for a platform measure, which decides its own. | | `max` / `remaining` | Your agent's [cap](#one-live-kpi-per-measure-and-the-cap) and what's left of it. `null` means no cap. Read them rather than assuming a figure. | `ready: false` does **not** stop you promoting the measure — committing to a number you are about to start collecting is perfectly reasonable. It tells you the tile will read blank today, so the blank is expected rather than alarming, and it names the page that fixes it. ### Create one ### Pick the measure From `GET /kpis/options` above. Take `defaultDirection` with it: you only need to send `direction` when you disagree with the proposal. ### POST the commitment Only `measureKey` and `label` are required. Everything else has a default, and every default is the conservative one — `draft`, no target, priority 100, no review cadence. ```bash curl -sf -X POST "$NEENS_BASE_URL/api/kpis" \ -H "Authorization: Bearer $SESSION_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "measureKey": "containment_rate", "label": "Self-serve containment", "description": "Share of support cases the assistant closes without a human. Board metric for FY27.", "direction": "higher_is_better", "target": 0.85, "owner": "Support Platform (@rota-support)", "status": "active", "priority": 10, "reviewCadence": "monthly" }' ``` ```json { "kpi": { "id": "a68e0cbb059f4b0ba38f409ee2145b19", "projectId": "proj_x", "measureKey": "containment_rate", "label": "Self-serve containment", "description": "Share of support cases the assistant closes without a human. Board metric for FY27.", "direction": "higher_is_better", "target": 0.85, "owner": "Support Platform (@rota-support)", "status": "active", "priority": 10, "reviewCadence": "monthly", "isCustomMeasure": false, "createdAt": "2026-08-14T10:22:41+00:00", "createdBy": "admin@example.com", "updatedAt": "2026-08-14T10:22:41+00:00", "updatedBy": "admin@example.com", "archivedAt": null } } ``` The response is a `201`. A KPI created without `status` stays a `draft`: it is authored, it is not yet a promise, and it does not appear on the summary. ### Commit to a figure when you're ready A KPI with `"target": null` is a legitimate, common state — *we watch this number, we have not committed to a figure*. Promote first, argue about the number later: ```bash KPI_ID=a68e0cbb059f4b0ba38f409ee2145b19 curl -sf -X PATCH "$NEENS_BASE_URL/api/kpis/$KPI_ID" \ -H "Authorization: Bearer $SESSION_TOKEN" \ -H "Content-Type: application/json" \ -d '{"target": 0.88, "status": "active", "reviewCadence": "quarterly"}' ``` ### Read the current value `GET /kpis/{id}` returns the KPI **and** the number, resolved over the range you ask for (`?range=30d`). ```bash curl -sf "$NEENS_BASE_URL/api/kpis/$KPI_ID?range=30d" \ -H "Authorization: Bearer $NEENS_API_KEY" ``` ```json { "kpi": {"id": "a68e0cbb059f4b0ba38f409ee2145b19", "measureKey": "containment_rate", "label": "Self-serve containment", "direction": "higher_is_better", "target": 0.85, "status": "active", "…": "…"}, "value": 0.8731, "unit": "ratio", "provenance": "measured", "coverage": {"cases": 812, "decided": 704, "rate": 0.867, "invalid": 0, "asOf": "2026-08-14T11:04:18Z"}, "definitionVersion": "02f3318341c0df32", "pricing": null, "unavailable": null, "targetStatus": "met", "trend": {"direction": "improved", "current": 0.8731, "previous": 0.8104, "delta": 0.0627, "pctDelta": 7.73692, "fromDay": "2026-08-06", "toDay": "2026-08-13", "lookbackDays": 7, "unavailable": null}, "trendUnavailable": null, "range": "30d", "asOf": "2026-08-14T11:04:18Z" } ``` Everything below `value` is there for the same reason the rest of this page exists: a number without its evidence is not a number you can act on. | Field | What it tells you | | --- | --- | | `value` | The measure's current value over `range`. `null` when it could not be computed — never `0`. | | `unit` | The measure's unit (`ratio`, `ms`, `usd`, `count`, …) — the unit `target` is in, too. | | `provenance` | **Measured** or **Inferred**, exactly as on a widget. Promoting a measure does not upgrade its evidence. | | `coverage` | The block documented under [Coverage](#coverage): how many cases the window held, and how many the definition could decide. `null` when the measure produced no coverage to report. | | `definitionVersion` | The fingerprint of the [KPI definition](#define-a-kpi) this number was computed under. If it changed since you last looked, the meaning of "escalated" changed — and so did the number, for reasons that have nothing to do with the agent. | | `pricing` | `null` unless the measure is priced. On a cost measure it carries the pricing detail for the window — see the callout below. | | `unavailable` | `null` when the value came out. Otherwise `{"reason": …}`, naming why **the value** could not be computed. | | `targetStatus` | `met` · `missed` · `unknown`. See below. | | `trend` | Direction of travel, read off [recorded history](#history-and-trends) — `improved`, `flat` or `regressed`, plus the two days and values behind it. `null` when there is no drawable line. | | `trendUnavailable` | Why there is no `trend`: `no_snapshots`, `insufficient_history` or `definition_changed`. Separate from `unavailable`, because "we have no history" and "we have no value" are different facts and a tile has to be able to show one without the other. | | `range` · `asOf` | The window, and when this was computed. Non-negotiable on a business number — see [late arrival](#late-arriving-outcomes-and-what-the-window-selects). | | `movedByFixes` · `movedByFixesUnknown` | The shipped [fixes that moved this KPI](#fixes-that-moved-this-kpi) — merged remediations whose close-out registered a move on this number, with the unmeasurable ones listed apart. | **A non-null `pricing` block means the figure is a floor, not a value.** When some of the models in the window have no price, their tokens are counted and their dollars are not, so a `cost_per_case` KPI reads *lower* than the truth. The block is what lets the number be badged **partial** instead of quietly under-reporting spend — and a target judged `met` against an under-reported cost is exactly the wrong answer to get silently. Fill the gaps under **Settings → Model pricing**; see [Cost & model pricing](/guides/cost-and-model-pricing). ### Met, missed, and the answer most dashboards get wrong `targetStatus` compares `value` against `target` in the KPI's declared `direction`, **inclusively** — hitting the target exactly is meeting it, in both directions. | `direction` | `met` when | `missed` when | | --- | --- | --- | | `higher_is_better` | `value >= target` | `value < target` | | `lower_is_better` | `value <= target` | `value > target` | **No value means `unknown`. It never means `missed`.** A KPI whose window produced no decided cases has not been missed — nobody knows whether it was. Rendering a red *missed* tile for absent data is how a dashboard manufactures a crisis for an agent that simply hasn't sent outcomes yet, and it is the same mistake as counting an unmeasured case as contained. When `value` is `null`, `targetStatus` is `unknown` and `unavailable.reason` says why in a sentence you can act on. **A `null` target is not a target of zero.** "We watch this number but haven't committed to a figure" is a real state — it is what every KPI looks like on day one. It also reads `unknown`: there is nothing to have met or missed. A `lower_is_better` KPI with a null target is *not* silently holding you to `0`. So a KPI with no data this window looks like this — and says why, in the same breath: ```json { "kpi": {"id": "a68e0cbb059f4b0ba38f409ee2145b19", "label": "Containment rate", "direction": "higher_is_better", "target": 0.72, "status": "active", "…": "…"}, "value": null, "unit": "ratio", "provenance": "measured", "coverage": null, "definitionVersion": "02f3318341c0df32", "pricing": null, "unavailable": {"reason": "Business outcomes are not available in this workspace yet, so containment cannot be measured."}, "targetStatus": "unknown", "trend": null, "trendUnavailable": {"reason": "no_snapshots"}, "range": "30d", "asOf": "2026-08-19T04:49:16.109253+00:00" } ``` Note the shape: `unavailable` is **flat**, and its `reason` is usually a readable sentence rather than a code — it is written to be shown to the person looking at the tile, so treat it as prose and don't switch on it. Three values are stable codes worth branching on: `measure_removed` (the measure behind the KPI was deleted), `measure_redefined` (a measure with this KPI's key exists again, but it is not the one that was promoted — see below) and `no_data_in_window` (the measure resolved, but the window you asked for selected nothing). Everything else is prose that can be reworded, so match on those three and render the rest verbatim. Three more honesty rules ride on that block: - **`trend` is `null` whenever there is no drawable line**, and `trendUnavailable.reason` says which of the three reasons it is — here `no_snapshots`, because an agent with no outcomes has no [recorded days](#history-and-trends) either. Neens does not draw a sparkline it cannot back with recorded history, and it never interpolates one across a [definition change](#a-definition-change-breaks-the-series). A missing trend never suppresses the value: the two live in separate fields precisely so a KPI can show today's number while saying it cannot yet show the direction of travel. - **If the custom measure behind a KPI is deleted, `unavailable.reason` is the literal `measure_removed`** rather than a number. The commitment survives the measure — you can see that the promise is now pointing at nothing, which is a fixable situation, instead of reading a wrong figure that looks fine. See [before you delete a shared measure](/guides/custom-measures#before-you-delete-a-shared-measure). - **`coverage` and `asOf` come with every value, always.** Outcomes arrive late by design, so re-reading the *same* window tomorrow can legitimately show higher coverage and a different value. Nothing changed retroactively; more evidence arrived about the same cases. ### Every active KPI at once `GET /kpis/summary` is the scorecard: every **active** KPI with its value, in display order (live before retired, then `priority` ascending, then label — deterministic, so two identical reads never render in two different orders). ```bash curl -sf "$NEENS_BASE_URL/api/kpis/summary?range=30d" \ -H "Authorization: Bearer $NEENS_API_KEY" ``` ```json { "projectId": "proj_x", "range": "30d", "version": "a17be0c94d2f5561", "asOf": "2026-08-14T11:04:18Z", "count": 2, "truncated": false, "kpis": [ { "kpi": {"id": "a68e0cbb059f4b0ba38f409ee2145b19", "measureKey": "containment_rate", "label": "Self-serve containment", "direction": "higher_is_better", "target": 0.85, "priority": 10, "…": "…"}, "value": 0.8731, "unit": "ratio", "provenance": "measured", "coverage": {"cases": 812, "decided": 704, "rate": 0.867, "invalid": 0, "asOf": "2026-08-14T11:04:18Z"}, "definitionVersion": "02f3318341c0df32", "pricing": null, "unavailable": null, "targetStatus": "met", "trend": {"direction": "improved", "current": 0.8731, "previous": 0.8104, "delta": 0.0627, "pctDelta": 7.73692, "fromDay": "2026-08-06", "toDay": "2026-08-13", "lookbackDays": 7, "unavailable": null}, "trendUnavailable": null, "range": "30d", "asOf": "2026-08-14T11:04:18Z" }, { "kpi": {"id": "3d81f0a742be4c1e9a05d6b3f81c47ab", "measureKey": "cost_per_case", "label": "Cost to serve", "direction": "lower_is_better", "target": 0.42, "priority": 20, "…": "…"}, "value": 0.5108, "unit": "usd", "provenance": "measured", "coverage": {"cases": 812, "decided": 812, "rate": 1.0, "invalid": 0, "asOf": "2026-08-14T11:04:18Z"}, "definitionVersion": null, "pricing": {"…": "the pricing detail behind the dollars"}, "unavailable": null, "targetStatus": "missed", "trend": null, "trendUnavailable": {"reason": "insufficient_history"}, "range": "30d", "asOf": "2026-08-14T11:04:18Z" } ] } ``` Each entry is exactly the `GET /kpis/{id}` block above — same fields, same rules — so a scorecard row and a detail view can't tell you different things about the same commitment. `version` is a content fingerprint of the KPIs themselves. It changes when a commitment changes — a new target, a flipped direction, an archive — and **not** when somebody re-saves a KPI without changing anything, so a cached scorecard is not invalidated by a no-op edit. `count` is how many rows came back, and `truncated` is the one you should never see: it is `true` when the per-agent [cap](#one-live-kpi-per-measure-and-the-cap) was lowered underneath data that already existed, so the tail of the list was dropped. It is reported rather than hidden, because a silently shortened scorecard reads as *"these are all your KPIs"* — which would be a lie about a promise somebody made. Archive down to the cap and it goes back to `false`. Drafts and archived KPIs are excluded here. To see them, list with an explicit status: ```bash curl -sf "$NEENS_BASE_URL/api/kpis?status=all" \ -H "Authorization: Bearer $NEENS_API_KEY" ``` ```json {"projectId": "proj_x", "kpis": ["…"], "version": "a17be0c94d2f5561", "count": 3, "max": 24} ``` `GET /kpis` without a `status` returns everything **except** archived. `status=active`, `draft`, `archived` and `all` narrow or widen it. ### Changing one `PATCH /kpis/{id}` is a genuine partial update: only the keys you send are touched, so `{"target": 0.9}` cannot blank the description you didn't mention. An explicit `null` **is** meaningful — it withdraws a committed target, or clears a free-text field — so it is the *absence* of a key, not its null value, that means "leave this alone". **`measureKey` cannot be changed.** Repointing a KPI at a different measure is a `422`, not a silent success. Every review note, screenshot and conversation that cites *"Self-serve containment"* would silently start describing a different number, retroactively. Promote the other measure as its own KPI and archive this one — that leaves an honest record of what was promised and when it changed. ```json {"detail": "measure_key cannot be changed on an existing KPI. History that already cites this KPI would silently change meaning. Create a new KPI instead."} ``` ### Retiring one ```bash curl -sf -X POST "$NEENS_BASE_URL/api/kpis/3d81f0a742be4c1e9a05d6b3f81c47ab/archive" \ -H "Authorization: Bearer $SESSION_TOKEN" ``` Archiving retires the commitment without deleting it. An archived KPI keeps its history — the history of what a team promised is most of the point of having promised it — is excluded from `GET /kpis/summary` and from the default `GET /kpis` listing, stops counting against the one-per-measure rule, and stamps `archivedAt`. Nothing about the underlying measure changes: the widget on your dashboard keeps working, because a KPI never owned that number in the first place. ### One live KPI per measure, and the cap Two limits, both enforced loudly: - **One live KPI per measure per agent.** Two simultaneous commitments to `containment_rate` means two targets, and the honest answer to "did we hit it?" becomes "which one?". Creating a duplicate is a `409` **naming the KPI that already exists**, so you can go and edit it: ```json {"detail": "This project already has a KPI on 'containment_rate' ('Self-serve containment', id a68e0cbb059f4b0ba38f409ee2145b19). Edit it, or archive it first."} ``` **Archived KPIs are exempt.** You can re-commit to a measure you retired last quarter — the old commitment stays in the record and the new one starts clean. - **A per-agent cap on KPIs.** Reaching it is a `409` naming the limit, never a silent drop: ```json {"detail": "This project already has 12 KPIs, the maximum is 12. Archive one you no longer commit to, then try again."} ``` Read the live figures from `max` / `remaining` on `GET /kpis/options` (or `max` on `GET /kpis`) rather than hard-coding one. The cap exists because a scorecard with sixty rows is not a scorecard — it is a table nobody reads, and every entry on it stops meaning "we promised this". KPI endpoints and vocabularies, in full | Endpoint | What it does | | --- | --- | | `GET /kpis` | List. `?status=active\|draft\|archived\|all`; the default is everything except archived. Returns `{projectId, kpis, version, count, max}`. | | `GET /kpis/options` | The promotable measures, each with `ready` / `unavailable` and `promoted` / `promotedKpiId`, plus the vocabularies and `max`/`remaining`. | | `POST /kpis` | Create. `201 {kpi}`. `409` on a duplicate measure or the cap; `422` on a bad field. | | `GET /kpis/{id}` | The KPI **plus its current value** and the [fixes that moved it](#fixes-that-moved-this-kpi) (`movedByFixes` / `movedByFixesUnknown`). `?range=30d`. | | `PATCH /kpis/{id}` | Partial update. `422` on `measureKey` or an unknown field. | | `POST /kpis/{id}/archive` | Retire the commitment; keeps the history. | | `GET /kpis/summary` | Every active KPI with its value, in display order. `?range=30d`. Returns `{projectId, range, version, asOf, count, truncated, kpis}`, where each entry is the `GET /kpis/{id}` block. | | `GET /kpis/{id}/eroding-clusters` | The [failure clusters eroding this KPI](#failure-impact--which-failures-are-eroding-a-kpi): `ranked` (worst first, each with `impactShare`) and `unknown` (thin clusters, with a reason), plus `attributable` — `false` for a percentile KPI. | | `GET /kpis/{id}/history` | The [recorded days](#history-and-trends) for one KPI, oldest first, plus `trend`, `breaks` and `definitionChangedAt`. `?days=30`, clamped; `422` on `days=0`. | | `POST /kpis/{id}/snapshot` | Record one KPI's recent days now. `?days=` to redo a specific stretch. Idempotent. | | `POST /kpis/snapshot` | The same for every active KPI in the agent. | Every route is also reachable at its bare path (`/kpis`) as well as under `/api`. | Vocabulary | Values | | --- | --- | | `direction` | `higher_is_better` · `lower_is_better` | | `status` | `draft` · `active` · `archived` | | `reviewCadence` | `none` · `weekly` · `monthly` · `quarterly` | | `targetStatus` | `met` · `missed` · `unknown` | | `priority` | Integer 1–999, lower sorts first. Default `100`. | | `unavailable.reason` (value) | Free text meant for a reader — **not** a closed enum. `measure_removed`, `measure_redefined` and `no_data_in_window` are stable codes; anything else is prose to render as-is. | | `unavailable.reason` (options) | `definition_not_configured` · `no_matching_outcomes` · `no_matching_scores` | | `trendUnavailable.reason` | `no_snapshots` · `insufficient_history` · `definition_changed` | | `trend.direction` | `improved` · `flat` · `regressed` · `unknown` | ## History and trends A KPI's value is resolved **live**, over whatever window you ask for. That answers *where are we now?* and it can never answer *are we getting better?* — a live query knows what last Tuesday looks like today, not what it looked like on the Wednesday after. Outcomes land late, prices change, definitions get sharpened; the number you'd read for a past day today is not the number that day actually had. So Neens **records** each active KPI once per completed day, and every trend is read off those recorded days rather than re-derived. You don't have to start it: once a KPI is `active`, its history accumulates on its own, and it keeps accumulating whether or not anyone opens a chart. The two `snapshot` routes below exist for when you want a day recomputed *right now* rather than whenever the next pass comes round. | Route | What it does | | --- | --- | | `GET /kpis/{id}/history` | The recorded days for one KPI, oldest first, plus the trend and any series breaks. | | `POST /kpis/{id}/snapshot` | Record (or re-record) one KPI's recent days now. | | `POST /kpis/snapshot` | The same, for every active KPI in the agent. | `GET /kpis/{id}` and `GET /kpis/summary` grow a real `trend` from the same recorded days — the `trendUnavailable: {"reason": "no_snapshots"}` placeholder now only shows up when there genuinely is no history yet. ### What a trend is, and how to read it A trend compares the **newest recorded day that has a value** against a day at least **7 days** older. A week, not a day: a business number with a weekly shape — support volume, containment, cost — gets compared against the same weekday, so what you read is a change in the business rather than the fact that Sunday is quiet. | Field | What it tells you | | --- | --- | | `direction` | `improved` · `flat` · `regressed` · `unknown`. Read through the KPI's own `direction`: a **falling** `cost_per_case` is `improved`; a falling `containment_rate` is `regressed`. | | `current` / `previous` | The two values compared — both real recorded values, never interpolated. | | `delta` | `current − previous`, in the measure's own unit. Signed arithmetically, so you can render the arrow yourself; `direction` is the one that already knows which way is good. | | `pctDelta` | The same movement as a percentage, or `null` when `previous` is `0`. A percentage change from nothing is not a number, and a tile reading `Infinity%` is worse than a blank. | | `fromDay` / `toDay` | The two days compared. Both are complete UTC days. | | `lookbackDays` | How far apart the two days had to be — `7`. | | `unavailable` | `null` on a real trend. `{"reason": …}` whenever `direction` is `unknown`. | `flat` is a genuine fourth answer, not a rounding artefact: a movement smaller than floating-point noise is reported as *unchanged* rather than as a microscopic improvement that flips sign every time the page reloads. ### The three ways a trend reads `unknown` `unknown` is a first-class answer, and it is deliberately three different answers — because whether *waiting* fixes it is exactly what you need to know: | `unavailable.reason` | What happened | What to do | | --- | --- | --- | | `no_snapshots` | Nothing has been recorded for this KPI yet — it was promoted minutes ago, or it isn't `active`. | Wait, or `POST /kpis/{id}/snapshot`. | | `insufficient_history` | There are recorded days, but not two comparable ones a week apart with values in them. | Wait. A KPI promoted on Monday has no trend until the following Monday. | | `definition_changed` | There *is* older history, and it measured something else. See [series breaks](#a-definition-change-breaks-the-series). | Nothing to fix. The trend restarts from the change. | On `GET /kpis/{id}` and `GET /kpis/summary` an unknown trend is reported as `trend: null` plus `trendUnavailable: {"reason": …}` — there is no line to draw, so no line is drawn, and the value beside it is unaffected. `GET /kpis/{id}/history` always returns the full `trend` block, `unknown` and all, because a client charting the series needs to know *why* it isn't drawing an arrow. ### Only completed days are recorded **The newest day that can appear in a history is yesterday, UTC.** Today is never a point on the line. That is not caution for its own sake. A day still filling up is a partial window recorded as if it were a whole one: containment drops off a cliff at 09:00 and heals by itself overnight, cost per case reads a third of its real figure, and somebody escalates. Every day on the line covers the same 24 hours as every other day on it, which is the only way two points on one axis are comparable at all. The *current* value on `GET /kpis/{id}` is unaffected — that is still a live read over the range you ask for, today included. The history and the tile answer two different questions on purpose. ### A day with no number is recorded as a day with no number A recorded day whose measure produced nothing is written down anyway, with `value: null` and an `unavailableReason`. It is not skipped, and it is not a zero. That matters because *"we looked on the 12th and there was nothing"* is history, and it is a different statement from *"we never looked"* and from *"the measure was deleted"*. Collapsing all three into a gap in the series throws away the only evidence that distinguishes them. Two consequences you'll see directly: - **Every point carries its own `targetStatus`, and a point with no value reads `unknown` — never `missed`.** A day with no decided cases has not broken the commitment; nobody knows whether it did. This is the same rule the live tile follows, and it is why a history chart never paints absent days red. - **A trend steps over an empty day rather than stopping at it.** The comparison uses the newest day that *has* a value, so a quiet Sunday in the middle of the window does not erase the trend of the days around it. ### Recent days get corrected as outcomes arrive A recorded day is not written once and frozen. [Outcomes arrive late](#late-arriving-outcomes-and-what-the-window-selects) — a ticket closed on Thursday decides a case that started on Monday — so what Monday's containment *was* keeps changing for a few days after Monday ends. Neens therefore re-checks the most recent days on every pass and rewrites them when the answer changed. Three fields on each point tell you exactly what happened to it: | Field | Meaning | | --- | --- | | `computedAt` | When this day was **first** recorded. It never moves again. | | `asOf` | When it was last re-checked. This moves on every pass, whether or not anything changed. | | `revisions` | How many times the recorded fact actually **changed** — the value, the reason it was absent, or the definition behind it. Re-reading a day forty times is not forty revisions, and a counter that said so would be worth nothing. | So `revisions: 0` with a recent `asOf` means *we keep checking, and the number keeps coming out the same* — which is a stronger statement than a number nobody re-examined. Days far enough in the past stop being re-checked at all; their `asOf` stops moving, and that is the number settling, not the recording stopping. **Re-recording a day rewrites it — it never appends.** Each (agent, KPI, day) is one row, so calling `POST /kpis/snapshot` five times in a row leaves the history exactly as it was after the first call, with a newer `asOf` and the same `revisions`. Re-running a window is always safe. ### A definition change breaks the series A KPI is a commitment to a *measure*, and what a measure means is something your agent [declares](#configure-it). Change the declaration — repoint `escalationKind`, edit the escalation labels, change a [custom measure](/guides/custom-measures)'s rule — and the days either side of that change are two different quantities. Subtracting one from the other produces a movement nobody made. Neens records the fingerprint of the definition behind every day, and reports the change instead of averaging across it: - `breaks` lists each change: the **first day under the new definition**, and the fingerprints it moved `from` and `to`. - `definitionChangedAt` is the most recent of those — the day a chart should annotate. - Each point carries `comparableWithPrevious`, so a client can break the line rather than draw through it. - A trend that would span a break resolves `unknown` with `reason: definition_changed`, rather than a number. This is the over-time twin of a rule you may already have met: a widget scoped across agents that define a KPI differently reports *unavailable* rather than an average. Same argument, one axis over. **A break is not an error, and it does not delete anything.** The older days stay readable and stay correct *under the definition they were measured with*. What you lose is the right to subtract across the boundary — which you never had. When a change was a correction rather than a redefinition, re-record the affected days (`POST /kpis/{id}/snapshot?days=…`) so the whole window is measured the new way and the series becomes continuous again. ### Read a KPI's history ```bash curl -sf "$NEENS_BASE_URL/api/kpis/$KPI_ID/history?days=30" \ -H "Authorization: Bearer $NEENS_API_KEY" ``` ```json { "kpiId": "a68e0cbb059f4b0ba38f409ee2145b19", "measureKey": "containment_rate", "unit": "ratio", "direction": "higher_is_better", "target": 0.72, "days": 30, "points": [ "…", { "id": "09ed0073f8232afc9ba68bc00669fccc74484dea", "projectId": "proj_x", "kpiId": "a68e0cbb059f4b0ba38f409ee2145b19", "measureKey": "containment_rate", "day": "2026-08-11", "value": 0.3333333333333333, "unit": "ratio", "provenance": "measured", "coverage": {"cases": 3, "decided": 3, "rate": 1.0, "invalid": 0, "asOf": "2026-08-18T02:14:07.540138+00:00"}, "unavailableReason": null, "unavailable": null, "definitionFingerprint": "a8bb2e3b1a48b2d8", "measureVersions": {"kpiDefs": "1:2026-08-02T09:12:44+00:00", "customMeasures": "0:", "priceTable": "3:2026-07-30T00:00:00+00:00"}, "asOf": "2026-08-18T02:14:07.540138+00:00", "computedAt": "2026-08-12T02:11:52.771904+00:00", "revisions": 1, "comparableWithPrevious": true, "targetStatus": "missed" }, { "id": "7804621f6b59d526c878a7279b9a81d611242021", "day": "2026-08-12", "value": null, "unit": "ratio", "provenance": "measured", "coverage": {"cases": 0, "decided": 0, "rate": null, "invalid": 0, "asOf": "2026-08-18T02:14:07.541940+00:00"}, "unavailableReason": "no_data_in_window", "unavailable": {"reason": "no_data_in_window"}, "definitionFingerprint": "a8bb2e3b1a48b2d8", "asOf": "2026-08-18T02:14:07.540138+00:00", "computedAt": "2026-08-13T02:10:31.118442+00:00", "revisions": 0, "comparableWithPrevious": true, "targetStatus": "unknown", "…": "…" }, "…", { "id": "fa740ee4144e8d007a887f42d1357baac7cd7d37", "day": "2026-08-18", "value": 0.75, "unit": "ratio", "provenance": "measured", "coverage": {"cases": 4, "decided": 4, "rate": 1.0, "invalid": 0, "asOf": "2026-08-19T02:12:19.985752+00:00"}, "unavailableReason": null, "unavailable": null, "definitionFingerprint": "a8bb2e3b1a48b2d8", "asOf": "2026-08-19T02:12:19.985752+00:00", "computedAt": "2026-08-19T02:12:19.985752+00:00", "revisions": 0, "comparableWithPrevious": true, "targetStatus": "met", "…": "…" } ], "breaks": [], "definitionChangedAt": null, "trend": { "direction": "improved", "current": 0.75, "previous": 0.3333333333333333, "delta": 0.4166666666666667, "pctDelta": 125.00000000000003, "fromDay": "2026-08-11", "toDay": "2026-08-18", "lookbackDays": 7, "unavailable": null }, "truncated": false } ``` | Field | What it tells you | | --- | --- | | `days` | The window actually served, in complete days. Ask for more than the workspace allows (a year, unless yours is set lower) and this is the number you got — with `truncated: true` beside it, so a clamp never passes for *"that's all the history there is"*. | | `points` | The recorded days, **oldest first**, one per calendar day. A day that was never recorded simply isn't there; a day recorded with no number is there with `value: null`. | | `unit` · `direction` · `target` | The KPI's own vocabulary, echoed so a chart can label and threshold the series without a second call. | | `breaks` · `definitionChangedAt` | Where the meaning of the number changed. Empty and `null` when it never did. | | `trend` | The block documented above — always present here, `unknown` included. | And per point: | Field | What it tells you | | --- | --- | | `day` | The complete UTC day this covers. | | `value` | What the measure came out as that day. `null` when it produced nothing — never `0`. | | `unavailableReason` · `unavailable` | Why there is no value. The same vocabulary the live tile uses, so a point and a tile never give two accounts of one absence. | | `targetStatus` | `met` · `missed` · `unknown`, judged for **that day** against the KPI's current target. `unknown` whenever the day has no value, or the KPI has no target. | | `coverage` | How many cases that day held, and how many the definition could decide — the same block as on a live read. A 90% containment over four decided cases is not the same fact as 90% over four hundred. | | `provenance` | `measured` or `inferred`, for that day. Promoting a measure never upgrades its evidence, and neither does recording it. | | `definitionFingerprint` · `comparableWithPrevious` | The definition behind the number, and whether this point may be compared with the one before it. | | `measureVersions` | Which versions of your definitions, custom measures and price table were in force. It is what makes a past number explainable a quarter later. | | `asOf` · `computedAt` · `revisions` | When it was last re-checked, when it was first recorded, and how many times the fact actually changed. | | `id` · `projectId` · `kpiId` · `measureKey` | Identity. The `id` is derived from the agent, the KPI and the day, which is why re-recording a day rewrites it instead of appending. | `?days` must be a positive integer — `days=0` is a `422`, not an empty chart: ```json {"detail": "days must be a positive integer."} ``` ### Record history now Both routes run the same recording pass Neens runs for you, so a manual catch-up and the automatic one can never disagree. Recording needs the same authority as any other change to a KPI (**admin**); reading history needs only read access. ```bash curl -sf -X POST "$NEENS_BASE_URL/api/kpis/$KPI_ID/snapshot?days=14" \ -H "Authorization: Bearer $SESSION_TOKEN" ``` ```json {"projectId": "proj_x", "kpis": 1, "days": 14, "computed": 14, "written": 14, "revised": 0, "skipped": 0, "errors": 0, "truncated": false} ``` ```bash curl -sf -X POST "$NEENS_BASE_URL/api/kpis/snapshot" \ -H "Authorization: Bearer $SESSION_TOKEN" ``` ```json {"projectId": "proj_x", "kpis": 3, "days": 4, "computed": 12, "written": 12, "revised": 2, "skipped": 0, "errors": 0, "truncated": false} ``` | Field | What it tells you | | --- | --- | | `kpis` | How many active KPIs were in scope. Drafts and archived KPIs are not recorded — a draft is not a commitment, and an archived one is over. | | `days` | How many distinct days the pass covered. | | `computed` · `written` | (KPI, day) pairs measured, and rows written. | | `revised` | Rows whose stored fact actually changed — the late-outcome corrections described above. | | `skipped` · `truncated` | What a large catch-up left for the next pass. `truncated: true` means the backlog was bigger than one pass, and the oldest, stalest days went first; call it again to keep draining. A cap that said nothing would read as *"everything is covered"*. | | `errors` | KPIs that could not be recorded. One bad KPI never aborts the rest. | Omit `days` and each KPI gets the window it needs: the last few days for one that already has history (enough to absorb late outcomes), and a one-time backfill of the recent past for one being recorded for the first time. Pass `days` explicitly when you need a *specific* stretch redone — after correcting a definition, or after a late bulk import of outcomes. It must be between `1` and the workspace's ceiling: ```json {"detail": "days must be between 1 and 365."} ``` **Backfill only reaches as far as your data does.** Recording a day is a measurement of that day, not a reconstruction of it: days before your outcomes started arriving are recorded honestly as days with no number and an `unavailableReason`, which is the correct answer and a much better one than a flat line at zero. ### How long history is kept Recorded days are kept for as long as your workspace's [data-retention window](/administration/data-retention) keeps anything else, and are aged out with it. On a workspace set to keep data forever (the default), history is kept forever. Two things worth planning around: - **A shorter retention window shortens your trends.** If you keep 30 days, a year-long KPI chart is not available — not because Neens forgot to record it, but because the workspace asked for it to be removed. Export what you need to keep beyond the window. - **Archiving a KPI does not delete its history.** The recorded days stay readable, which is most of the point of having promised something: *"we committed to 85%, here is what actually happened, here is when we retired the commitment"* is the story a KPI exists to be able to tell. ## Where your KPIs show up A KPI is a commitment, so Neens carries it to the places people already look — not just the Business KPIs page. Every surface below reads the **same** value, target status and trend as the page, through the same [request-free resolution path](#promote-a-measure-to-a-kpi), so a KPI can never say one thing in an email and another on the page — including *why* a number is missing. - **Dashboards.** The widget gallery grows a **Business** group — containment rate, containment by agent, resolution time (p50/p90), cost per case, and cost per case over time. See [Dashboards](/guides/dashboards). You can slice by agent, status, source, and the metadata on the outcome (its queue, region, plan tier) — so "is containment worse in billing than in shipping?" is a slice, not a new instrumentation agent. - **The Executive digest dashboard.** The shipped **Executive** persona dashboard now leads with **Containment rate** and **Cost per case** — the two numbers an exec reads first. An agent that hasn't configured those definitions yet sees them render **unavailable**, never a fabricated `0`. - **The digest email.** Your daily/weekly [digest email](/guides/insights#the-digest-email) carries a **Business KPIs** section under the fleet-health tiles: your top few active KPIs by priority, each with its value, whether the target is **met / missed / unknown**, and a trend arrow. A KPI with no number yet shows **—**, never `0` or a red *missed*. - **The weekly narrative.** The weekly *"this week in review"* narrative now folds your active KPIs into its week-over-week movers, oriented by each KPI's direction — so a containment slip or a cost-per-case rise surfaces in prose alongside the fleet-health metrics. - **The Assistant and MCP.** Ask the [Assistant](/guides/assistant) *"how are our business KPIs doing?"* or call the `get_business_kpis` [MCP](/guides/mcp) tool: both return your KPIs with their value, target status and trend, and — for one KPI — the [failure clusters eroding it](#failure-impact--which-failures-are-eroding-a-kpi). A `null` value stays `null` and an unknown status stays `unknown` — the agent is never handed a fabricated number. - **Alerts.** Two shapes of [alert rule](/guides/alerts) watch a KPI: - a **threshold** rule on the measure — *"page me when containment drops below 85%"* (`containment_rate`, `resolution_time_p50`/`p90` and `cost_per_case` are first-class metrics); and - a **KPI trend** rule — *"page me when this KPI has regressed by 5% or more over the last 7 days"* — which watches the KPI's own recent [history](#history-and-trends) rather than a fixed line, and fires only on a genuine regression (an `unknown` trend never pages anyone). Test a threshold rule against live data before you trust it: containment is measured over *decided* cases, so a rule on an agent with 8% coverage is watching a very small sample, and the test shows you that now rather than at 3am. **A widget scoped across agents that define a KPI differently reports *unavailable*, not an average.** If one agent reads a helpdesk `escalation` outcome and another reads a classifier label, one blended containment rate would describe nothing. Neens refuses and names the fix: scope the widget to one agent, or give those agents the same definition. Two agents that both kept the platform default still agree — saving the default is not a change of meaning. ### Not included (yet) Worth stating plainly so you don't go looking: - [Pre-prod eval gates](/guides/eval-gates) and the `neens eval` [gate policy](/guides/preprod-evals) gate on a run's **judge metrics**, not on catalogue measures. There is no `containment_rate` gate rule. - **A KPI screen for editing.** Promoting, editing and archiving commitments is API-only for now; the values behind them are already on the [Business KPIs page](#the-business-kpis-page), on dashboards, and in the surfaces above. ## Late-arriving outcomes The time window selects **cases**, by the start of their first trace. Outcomes are joined regardless of when they arrived. That is exactly what makes late outcomes work: a ticket closed three days after the conversation still decides that conversation's case, and lands in the window the *case* belongs to. The consequence: **the same window, re-queried later, can show a higher `decided` count and a different rate.** Nothing changed retroactively — more evidence arrived about the same cases. That is why every KPI carries `coverage.asOf`, and why a screenshot of a business KPI is only interpretable with it. It is also why [recorded history](#recent-days-get-corrected-as-outcomes-arrive) re-checks its most recent days instead of freezing each one the night it ends: a day's figure has to be allowed to catch up with the evidence about that day, and `revisions` on the point says when it did. For cost per case, both sides are bounded by the window: a span that started outside the window is not counted toward its case's cost. All timestamps on both sides are normalized to UTC. There is no local-timezone arithmetic anywhere in this path. ## When Neens shows nothing A blank is a result. None of these render a zero. | Situation | What you see | | --- | --- | | No definition configured | The platform defaults, marked *not configured*, with a **Define what counts as escalation** call to action. | | No snapshot for the window yet | `—` on the tile, with *"Value appears once snapshots exist"*. The match coverage is in the **Data health** strip above. | | The value could not be read | `—` with *"Couldn't load current values"* — a failed read, explicitly distinguished from "no snapshot yet" so you don't wait on a number that isn't coming. | | No decided cases in the window | `—`, plus *"No measured outcomes in this window."* | | No cases at all in the window | `—`, plus *"No cases in this window."* Never `0`, never `$0.00`. | | Classifier mode, asking for resolution time | *Unavailable*, with the reason: a classifier records no resolution instant. | | Some models unpriced | `cost_per_case` renders with a **partial** badge; the value is a **floor**. | | Agents in scope disagree on the definition | *Unavailable*, naming the fix (scope to one agent, or align the definitions). | | Your only outcome kind is resolution | Containment reports **no coverage** — a resolution says nothing about whether the agent handled the case. | ## Troubleshooting | Symptom | Cause | Fix | | --- | --- | --- | | Containment is exactly `0%` | The escalation signal you declared isn't in your data, so every decided case reads as escalated. | On the **Definitions** tab, check the picker — a kind or label marked *not seen yet* decides nothing. | | Containment is exactly `100%` | Nothing in your data matches the escalation signal. | Same check. In classifier mode, confirm the escalation labels are the exact strings the classifier emits — they're matched literally, including case and spacing. | | Containment is `—` with a real feed | The kinds you send don't decide containment: only containment/escalation signals do, and only with a readable yes/no. A resolution-only feed decides nothing here. | Send a containment or escalation fact, or point the definition at the kind that carries it. | | The number looks right but coverage is tiny | Your outcome match rate is low, or your feed only covers some cases. | [Data sources → match coverage](/guides/business-outcomes#why-your-match-rate-is-not-100). | | Resolution time is *unavailable* | The KPI is in classifier mode, or its duration kind isn't a duration. | Switch resolution time to outcome mode and point it at a duration kind. | | Cost per case is lower than expected | Some models are unpriced, so the ratio is a floor. | Set prices — see [Cost & model pricing](/guides/cost-and-model-pricing). | | A tile shows `—` on a new agent | No snapshot yet for the window. | Nothing to fix — it fills in as snapshots accrue. For a number today, see [Cost optimization](/guides/cost-optimization). | | A tile says the value **couldn't be loaded** | The read failed — a permissions problem or a transient error — as opposed to there being no snapshot. | Reload; if it persists, check that your role still has read access to the agent. | | The **Definitions** tab is read-only for you | You're not an admin. | Ask an admin to declare the definition. | | A cross-agent widget says *unavailable* | The agents in scope define the KPI differently. | Scope the widget to one agent, or align the definitions. | | A KPI reads `targetStatus: "unknown"` | Either the measure produced no value this window, or the KPI has no `target` yet. Both are `unknown` on purpose — never `missed`. | Read `unavailable` on the response: it names which one. If it's coverage, the fix is upstream; if it's the target, `PATCH` a figure onto it. | | A KPI reads `measure_removed` | The custom measure it was promoted from has been deleted. | Re-create the measure, or archive the KPI. The commitment is not silently reporting a wrong number in the meantime. | | A KPI reads `measure_redefined` | A custom measure with the same key exists, but it is **not** the one this KPI was promoted from — the slug was freed by a delete and then reused for a measure that means something else. | Restore the original definition and the KPI resumes scoring by itself, or archive the KPI and promote the new measure deliberately. A target agreed for one measure is never scored against a different one. | | `trend` is `null` with `no_snapshots` | Nothing has been recorded for this KPI yet — it was promoted a moment ago, or it is still a draft. | Set it `active` and wait for the next recording pass, or `POST /kpis/{id}/snapshot` to record it now. | | `trend` is `null` with `insufficient_history` | Fewer than two comparable days a week apart carry a value. | Wait. A KPI promoted on Monday gets its first trend the following Monday; `POST /kpis/{id}/snapshot?days=…` backfills what your data can support. | | `trend` is `null` with `definition_changed` | The [definition](#configure-it) behind the number changed inside the window, so the two ends measure different things. | Nothing is broken — `breaks` on `GET /kpis/{id}/history` names the day. Re-record the window if the change was a correction rather than a redefinition. | | A day in `history` shows `value: null` | That day produced no number — no cases, no decided cases, or the measure could not resolve. | Read `unavailableReason` on the point. It is recorded deliberately: `targetStatus` is `unknown`, never `missed`. | | Yesterday is the newest day in `history` | Only [completed UTC days](#only-completed-days-are-recorded) are recorded — today is still filling up. | Nothing to fix. Use `GET /kpis/{id}` for today's live figure. | | A recorded day's value changed | [Late outcomes](#recent-days-get-corrected-as-outcomes-arrive) landed and the recent window was re-recorded. `revisions` went up; `computedAt` did not move. | Nothing to fix — that is the history catching up with the evidence. | | `GET /kpis/{id}/history` says `truncated: true` | You asked for more days than the workspace serves in one call. | Read `days` on the response for what you actually got. | | `POST /kpis` returns `409` | Either that measure already has a live KPI, or the agent is at its cap. | The message names which, and which KPI. Edit or archive the existing one. Archived KPIs don't block a re-commit. | | `GET /kpis/summary` says `truncated: true` | There are more active KPIs than the agent's cap allows — the cap was lowered under existing data, and the tail of the list was dropped. | Archive down to the cap. The flag is there so a shortened scorecard never passes for a complete one. | | A cost KPI reads `met` but the figure looks low | The window is only partly priced, so the value is a floor — `pricing` on the response is non-null. | Price the missing models under **Settings → Model pricing**, then re-read. | | `POST /kpis` returns `422` on `target` | A ratio target was typed as a percentage (`85` instead of `0.85`), or a target went negative on a count/duration/currency measure. | Send the figure in the measure's own `unit` — the `unit` field on `GET /kpis/{id}` and `GET /kpis/options` tells you which. | ## Related - [Business outcomes](/guides/business-outcomes) — the measured signal behind the Data sources tab, and how to raise your match rate. - [Connect your helpdesk](/guides/connectors/outcome-inlets) — pull outcomes from Zendesk, Intercom, Salesforce or Jira Service Management on a schedule. - [Derived measures](/guides/derived-measures) — zero-setup business numbers computed straight from your traces (turns per case, tool success rate, case duration, …), promotable to KPIs. - [Custom measures](/guides/custom-measures) — define your own KPI when these three aren't yours. - [Cost & model pricing](/guides/cost-and-model-pricing) — where the dollars in `cost_per_case` come from, and what *partial* means. - [Metrics catalogue](/guides/metrics) — the full measure × dimension model these four measures join. - [Issues & failure modes](/guides/issues-and-failure-modes) — the failure taxonomy whose labels can also back an [inferred KPI](#inferred-kpis-the-ready-made-classifiers). - [Judges](/guides/judges) — where you turn the ready-made escalation, resolution and sentiment classifiers on, alongside your scorers. - [Fix outcomes](/guides/post-merge-efficacy) — where the [fixes that moved this KPI](#fixes-that-moved-this-kpi) are measured, before vs after each merged fix. - [Dashboards](/guides/dashboards) and [Alert rules](/guides/alerts) — chart a KPI, or get paged on it. ================================================================================ # Derived measures Source: /docs/guides/derived-measures/ ================================================================================ # Derived measures Most business numbers need something from you first: an [outcome feed](/guides/business-outcomes) from your helpdesk, a label from a [classifier](/guides/issues-and-failure-modes), or a [definition](/guides/business-kpis#define-a-kpi) of what "escalated" means. **Derived measures need none of that.** They are computed straight from the traces you already ingest — no external data, no LLM, no configuration — so they have a value the moment your agent sends its first traces. They are the answer to "what can I measure on day one?". Turns per case, how long a case takes, what a turn costs, whether your tools succeed or thrash — all of it is already in your spans, and Neens reads it back to you as first-class **Business** measures you can chart, alert on, and [promote to a KPI](#promote-a-derived-measure-to-a-kpi). ## At a glance | | | | --- | --- | | **What** | Measures computed purely from your ingested traces — a new [provenance tier](#the-four-provenance-tiers): **derived**. | | **Setup** | None. They work as soon as you send traces. No feed, no definition, no LLM. | | **Where** | The **Business** group in the [dashboard](/guides/dashboards) widget gallery and the [metrics catalogue](/guides/metrics); promotable on the [Business KPIs](/guides/business-kpis) page. | | **Grain** | Per **case** (a conversation), per **tool call**, per **span**, or per **trace** — one row per measure below says which. | | **Provenance** | Always **Derived** — reproducible arithmetic over your spans, badged so nobody mistakes it for a system-of-record fact. | | **Honesty** | When there isn't enough signal, the value reads **—**, never a misleading `0`. See [how a derived number stays honest](#how-a-derived-number-stays-honest). | ## The four provenance tiers Every business measure in Neens carries a **provenance** badge that says where its number came from — and derived is the newest of the four. The badge is *derived from the source*, never chosen, so it can never claim a number is stronger than the evidence behind it. | Provenance | Where the number comes from | Needs from you | | --- | --- | --- | | **Measured** | Your **system of record** reported it — a [business outcome](/guides/business-outcomes) from your helpdesk, CRM or warehouse. The strongest claim available. | An outcome feed. | | **Emitted** | Your **agent asserted it about itself**, in the trace metadata. Useful, but nothing outside the agent confirmed it. | Your agent to write the claim onto its traces. | | **Inferred** | An **LLM classifier read the transcript and decided**. A directional signal when you have nothing better yet. | A [classifier](/guides/issues-and-failure-modes). | | **Derived** *(new)* | **Neens computed it** from your traces with a deterministic rule — no external system, no LLM, no configuration. | **Nothing. Just send traces.** | **Derived is the zero-setup tier.** The other three each depend on something arriving — an outcome, an agent claim, a classifier label — before they can show a number. Derived measures depend only on the traces you're already sending to observe your agent, so a brand-new agent has real business numbers on its first day. `cost_per_case` on the [Business KPIs](/guides/business-kpis#cost-per-case) page is zero-setup in the same spirit — computed straight from your traces — and this release brings that day-one, no-configuration treatment to the whole family of measures below. ## The derived measures All of them live in the **Business** category and carry the **Derived** badge. They group by the grain they're computed at — the "thing" each row counts. ### Case-level — the shape of a conversation One **case** is a whole conversation (`conversation_id` when the trace carries one, the trace's own id when it doesn't) — the same grain the [Business KPIs](/guides/business-kpis#what-a-case-is-and-why-the-kpi-is-per-case) use. These measure what a case *cost you to handle*. You can slice them by the case's first-trace **agent**, **status**, **source** and metadata, plus time. | Measure | Unit | What it means | Watch for | | --- | --- | --- | --- | | **Turns per case** (`turns_per_case`) | count | Average number of conversation turns it took to handle a case. | **Rising** — the agent is taking more back-and-forth to get to the same place. | | **Traces per case** (`traces_per_case`) | count | Average number of traces (agent runs) that make up one case. | A climb often means retries or re-runs stacking up inside a single conversation. | | **Multi-trace case rate** (`multi_trace_case_rate`) | ratio | Share of cases that needed **more than one** trace — a cheap "came back" / re-contact proxy. | **Rising** — more customers are having to re-engage to finish. | | **Case duration (p50)** (`case_duration_p50`) | ms | Median wall-clock time from a case's first activity to its last. | The everyday experience drifting slower. | | **Case duration (p90)** (`case_duration_p90`) | ms | The slow **tail** — the 10% of cases that take longest. This is the number your unhappiest customers feel. | A p90 pulling away from p50: a subset of cases is getting badly stuck. | | **Cost per turn** (`cost_per_turn`) | usd | LLM spend across the window divided by conversation turns. | **Rising** — each turn is getting more expensive (bigger prompts, a pricier model, more tool chatter). | ### Tool-level — is your agent's tooling healthy Computed per **tool call**. Slice them by **tool name** (plus time) to find the one tool dragging the average down. | Measure | Unit | What it means | Watch for | | --- | --- | --- | --- | | **Tool call volume** (`tool_call_volume`) | count | How many tool calls your agent makes. | A jump with no matching rise in traffic — the agent is working harder per case. | | **Tool success rate** (`tool_success_rate`) | ratio | Share of tool calls that completed **without an error**. | **Falling** — a dependency is flaky, or the agent is calling a tool wrong. | | **Tool retry rate** (`tool_retry_rate`) | ratio | Share of tool calls that immediately repeated the **same** tool — a thrash / retry-loop signal. | **Rising** — the agent is stuck re-trying instead of moving on. | ### Span & latency — the mechanics of a run Computed over **spans** (the individual steps in a trace). Span error rate slices by span **kind** and **status**; first-response latency is a per-trace figure. | Measure | Unit | What it means | Watch for | | --- | --- | --- | --- | | **Span error rate** (`span_error_rate`) | ratio | Share of spans that errored. Slice by kind/status to see *which* step. | **Rising** — a step in the run is failing more often, even if the case still finishes. | | **First response latency (p50)** (`first_response_latency_p50`) | ms | Median time from the start of a **trace** to the agent's **first model response** — how long a customer waits before anything happens. | A climb: the agent is slower to say its first word, which reads as unresponsive. | **Ratios are fractions, not percentages.** A `ratio` measure like tool success rate reads `0.98`, not `98`. Durations (`case_duration_*`, `first_response_latency_p50`) are in **milliseconds**. These are the units you'll set a target in when you promote one. ## Promote a derived measure to a KPI A derived measure is a number you can watch. A **KPI** is a number you've made a promise about — a target, a direction that says which way is good, an owner, and a daily history so you can see the trend. Because a derived measure already has a value with no setup, promoting one is the fastest way to a tracked commitment: there's no feed to wire up and no definition to declare first (unlike the measured and inferred paths, which need an [outcome feed](/guides/business-outcomes) or a [classifier](/guides/issues-and-failure-modes) before they can show anything). The mechanics — the fields, the one-live-KPI-per-measure rule, reading the value back — are exactly the same as for any other measure, and are documented in full under [Promote a measure to a KPI](/guides/business-kpis#promote-a-measure-to-a-kpi). The short version: ### Find the derived measure in the promotable catalogue The catalogue lists every measure you can promote, each marked whether the agent can already get a number out of it. A derived measure reads **ready** with no setup, alongside a proposed direction you can keep or flip. ```bash curl -sf "$NEENS_BASE_URL/api/kpis/options" -H "Authorization: Bearer $NEENS_API_KEY" ``` ```json { "projectId": "your-project", "measures": [ { "key": "tool_success_rate", "label": "Tool success rate", "unit": "ratio", "category": "business", "provenance": "derived", "defaultDirection": "higher_is_better", "promoted": false, "ready": true, "unavailable": null } ] } ``` ### Set a target and a direction Send the measure key, a label your team uses, a target in the measure's own unit, and the direction that says which way is good. Everything else takes a conservative default. ### It starts tracking, with daily history Once the KPI is `active`, Neens records its value once per completed day on its own, so it grows a trend without you starting anything. See [History and trends](/guides/business-kpis#history-and-trends). ### Worked example — a tool-health KPI You want your agent's tools to succeed **at least 98% of the time**, so you promote **Tool success rate** with a target of `0.98`. Higher is better, which is the proposed direction, so you keep it: ```bash curl -sf -X POST "$NEENS_BASE_URL/api/kpis" \ -H "Authorization: Bearer $SESSION_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "measureKey": "tool_success_rate", "label": "Tool reliability", "description": "Tools must succeed 98% of the time. If this slips, the agent starts failing cases it could handle.", "direction": "higher_is_better", "target": 0.98, "owner": "Agent Platform (@rota-agents)", "status": "active", "reviewCadence": "weekly" }' ``` The tile now reads its live value, judged `met` at or above `0.98` and `missed` below it, and starts building a daily line you can watch for erosion. ### Worked example — a re-contact KPI, direction flipped You want **at most 15%** of cases to need a second trace — a cheap proxy for customers coming back. **Multi-trace case rate** is proposed `higher_is_better` by default, but for this measure *lower* is what you want, so you override the direction and set `target: 0.15`: ```bash curl -sf -X POST "$NEENS_BASE_URL/api/kpis" \ -H "Authorization: Bearer $SESSION_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "measureKey": "multi_trace_case_rate", "label": "Re-contact rate", "direction": "lower_is_better", "target": 0.15, "status": "active" }' ``` **The proposed direction is a starting point, not a verdict.** Neens pre-fills a direction from the measure's name — a *cost* or *latency* or *error* measure is proposed `lower_is_better`, a *success* rate `higher_is_better` — but you always decide. For measures like turns per case, tool retry rate and multi-trace case rate, where *less is better*, set `lower_is_better` yourself. Whatever you save is what every later read judges against; nothing re-guesses at render time. ## How a derived number stays honest A derived measure follows the same honesty rule as every other number in Neens: **when there isn't enough signal to answer, it shows `—`, never a misleading `0`.** A dash means *we couldn't measure this*, which is a different fact from *this is zero*, and Neens keeps them apart. | Situation | What you see | | --- | --- | | A case whose last activity has no end time | It's **excluded** from the duration percentile and counted in coverage — never treated as a zero-length case. | | A window with no tool calls at all | Tool success and retry rates read **—**, not a perfect `1.0` or a bogus `0`. | | A window with no cases | Every per-case measure reads **—**, never `$0.00` cost per turn or `0` turns. | | Not enough data yet | The measure reads **—** with the reason, and fills in as traffic arrives. | **A dash is a result, and it never renders as zero.** "We have no tool calls this window" and "our tools failed every time" are opposite facts — collapsing the first into a `0` would manufacture a crisis out of a quiet window. When you promote a derived measure to a KPI, that same rule carries through: a window with no signal reads `targetStatus: unknown`, never `missed`. Nobody can miss a target nobody could measure. See [Met, missed, and the answer most dashboards get wrong](/guides/business-kpis#met-missed-and-the-answer-most-dashboards-get-wrong). ## Related - [Business KPIs](/guides/business-kpis) — promote any measure to a commitment, and the full KPI lifecycle, history and trends. - [Metrics catalogue](/guides/metrics) — the measure × dimension model these join, and everything else you can chart. - [Custom measures](/guides/custom-measures) — define a measure in your own vocabulary when these don't cover it. - [Cost & model pricing](/guides/cost-and-model-pricing) — where the dollars in cost per turn come from. - [Dashboards](/guides/dashboards) and [Alert rules](/guides/alerts) — chart a derived measure, or get paged on it. ================================================================================ # Custom measures Source: /docs/guides/custom-measures/ ================================================================================ # Custom measures [Business KPIs](/guides/business-kpis) ships three named numbers: containment rate, resolution time and cost per case. Three named KPIs are enough to prove the idea. They are not enough to describe *your* business, because **"containment" is not one thing**: | Your team | What "handled it" actually means | | --- | --- | | Support | Resolved without a human ever touching it | | Sales | Qualified without an SDR call | | Internal IT | The ticket was never created in the first place | | Collections | Payment arranged without a callback | A predicate hard-coded for the first row is silently wrong for the other three. So instead of adding a fourth, Neens lets you **declare a measure of your own** — your label, your source, your grain, your target — and then treats it exactly like one of ours: it appears in the measure gallery, charts on a dashboard, backs an alert rule, and carries its provenance and coverage everywhere it renders. **A custom measure is a *selection*, not a query language.** You pick a source Neens already stores, a grain, and an aggregation. There is no SQL box, no expression field, and no way for a definition to reach data outside your own tenant. That is deliberate: the point is a KPI your whole company can trust, not a reporting tool one person can break. ## At a glance | | | | --- | --- | | **Where** | The **Custom measures** tab of the **[Business KPIs](/guides/business-kpis)** page | | **Who** | Admins (`MANAGE_MEASURES`); Silver plan and above | | **Scope** | **Company-wide** — one definition, shared by every agent. See [Scope](#scope-the-definition-is-company-wide-what-reads-it-is-not) | | **Reads** | Anyone with `READ` — a viewer can see what a number means | | **Sources** | A business outcome, a trace metadata field, or a classifier/judge metric | | **Grains** | One **trace**, or one **case** (a conversation — many traces) | | **Aggregations** | `count`, `sum`, `avg`, `p50`, `p90`, `p99`, `rate` | | **Appears in** | Measure gallery · dashboard widgets · [alert rules](/guides/alerts) | | **API** | `/custom-measures` | --- ## The three sources, and why the badge matters Every measure reads from exactly one of three places, and **the source decides the provenance badge** — you do not get to choose it. A number from your helpdesk and a number an LLM guessed from a transcript are different kinds of claim, and someone reading a dashboard six months from now has no other way to tell them apart. | Source | Reads | Badge | What it means | | --- | --- | --- | --- | | **Business outcome** | A [business outcome](/guides/business-outcomes) kind | **Measured** | Your system of record recorded this. The strongest claim available. | | **Trace metadata** | A field in the trace's metadata | **Emitted** | Your agent asserted this *about itself*. Useful, but nothing confirmed it. | | **Classifier / judge** | A [score](/guides/scores) metric + its labels | **Inferred** | A model read the transcript and decided. An opinion, not a fact. | **Emitted is not measured.** If your agent writes `resolved: true` into trace metadata, that is the agent's opinion of its own work — the same agent whose failures you are trying to find. It is a perfectly reasonable thing to measure, and it is badged `Emitted` so nobody mistakes it for the customer's own confirmation. A custom measure carries **three** of the four provenance tiers — **Measured**, **Emitted** and **Inferred** — one per source above. The fourth tier, **Derived** (Neens computing a number from your traces with a deterministic rule, like `cost_per_case`), belongs to the platform KPIs and isn't a source you pick here. See [the four provenance tiers](/guides/business-kpis#the-four-provenance-tiers). --- ## Define your first measure ### Open the Custom measures tab Admin-only. On the **[Business KPIs](/guides/business-kpis)** page, open the **Custom measures** tab. If it's read-only, either you are not an admin or your plan does not include custom measures (Silver and above). Everyone can still *read* the measures your admins define. What you define here is **company-wide**: every agent sees it, and so does every later edit or deletion. [Scope](#scope-the-definition-is-company-wide-what-reads-it-is-not) explains what that means before you change one. ### Name it in your own words The **name** is what appears on every chart, alert and digest — write it the way your team says it out loud ("Deflection rate", "Happy customers", "Time to first useful answer"). The **identifier** becomes the measure key (`custom:deflection-rate`) and is **fixed once created**, because alert rules and dashboard widgets store that key. Renaming the measure later is free; renaming the identifier is not offered, because it would silently detach everything pointing at it. ### Pick where the number comes from The pickers only offer things this agent has **actually recorded** — the outcome kinds you have received, the metadata fields your traces carry, the labels your classifier has really emitted. That is not a convenience. A measure pointing at an outcome kind nobody sends saves without complaint and then renders a confident, permanent `0%`. Neens refuses to let you create one. ### Choose the grain - **A trace** — one agent run. - **A case** — one conversation, however many traces it took. Business numbers almost always belong at the **case** grain. An agent that takes six turns to fail must not look six times worse than one that fails in a single turn. (An outcome-sourced measure is case-grain automatically — an outcome is recorded *about a case*, so counting it per trace would count the same fact once per turn.) ### Set the aggregation `rate` is the common one for a KPI: "the fraction of measured cases where …". Pick the comparison and the threshold — `>= 4` for *CSAT 4 or better*. For a classifier source, a `rate`'s numerator is the **labels you tick**. Neens refuses to save a classifier rate with no labels selected, because that rate is `0%` forever — and a permanent zero is a number, which renders, charts, and satisfies an alert threshold. An error is safer. ### Say which way is good, and set a target `Higher is better` / `Lower is better` drives the trend arrow and the default alert comparator. The optional target renders as a goal line. For a `rate`, express the target as a fraction — `0.85`, not `85`. ### Press **Test it** This runs the real resolver over your real data for the last 7 days and shows you the number *and its coverage* before anything is saved. Nobody should have to publish a measure to a live exec dashboard to find out whether it returns anything. ### Save --- ## Worked example: deflection rate for a support org Your helpdesk already pushes a `containment` outcome per ticket (see [Business outcomes](/guides/business-outcomes)). You want the number your VP of Support reports. | Field | Value | | --- | --- | | Name | `Deflection rate` | | Identifier | `deflection-rate` | | Source | Business outcome → `containment` | | One row is | A case | | Aggregation | `rate`, counts when the value is `=` `1` | | Which way is good | Higher is better | | Target | `0.72` | Badge: **Measured**. It appears in the gallery under **Business** as `Deflection rate`. ```bash curl -X POST https://your-neens-host/api/custom-measures \ -H "Authorization: Bearer $NEENS_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "slug": "deflection-rate", "label": "Deflection rate", "description": "Share of decided cases our agent closed without a human.", "source": "outcome", "outcomeKind": "containment", "grain": "case", "agg": "rate", "unit": "ratio", "rateOp": "=", "rateValue": 1, "direction": "higher_is_better", "target": 0.72 }' ``` ```json { "measure": { "key": "custom:deflection-rate", "label": "Deflection rate", "provenance": "measured", "grain": "case", "dimensions": ["agent", "conversation", "org", "project", "queue", "source", "status", "time"], "target": 0.72 }, "warnings": [] } ``` ### More examples "What share of rated cases were happy ones." Measured, because it comes from your CSAT feed. ```json { "slug": "csat-good", "label": "Happy customers", "source": "outcome", "outcomeKind": "csat", "outcomeSource": "zendesk", "grain": "case", "agg": "rate", "unit": "ratio", "rateOp": ">=", "rateValue": 4, "direction": "higher_is_better", "target": 0.8 } ``` `outcomeSource` pins the signal to one system of record — useful when two systems disagree about the same case. Leave it out to accept any source. "How long the agent thinks a turn took." Emitted, because the agent wrote it about itself. ```json { "slug": "agent-handle-time", "label": "Agent handle time (p90)", "source": "metadata", "metadataPath": "handle_seconds", "grain": "trace", "agg": "p90", "unit": "seconds", "direction": "lower_is_better" } ``` Percentiles use **nearest rank**, so a p90 over two values returns the slower one — not the faster one, which is what a naive index would give you. For an agent with no system-of-record feed yet. Inferred, and badged as such everywhere. ```json { "slug": "escalation-rate", "label": "Escalations (inferred)", "source": "classifier", "classifierMetricKey": "issue_class", "classifierLabels": ["escalated", "transferred_to_human"], "grain": "case", "caseReducer": "first", "agg": "rate", "unit": "ratio", "direction": "lower_is_better", "target": 0.15 } ``` `caseReducer: "first"` means *the classifier's verdict on the case's first trace inside the window*. The alternatives (`sum`, `avg`, `min`, `max`) collapse across all of the case's traces. --- ## Coverage: the part everyone forgets Every custom measure reports **how many units it could actually read**, alongside the number: ```json "coverage": { "rows": 412, "measured": 168, "rate": 0.41, "asOf": "2026-07-27T09:14:22Z" } ``` That says: 412 cases fell in the window, 168 of them carried a readable signal, and the number above was computed over those 168. **A unit with no signal is excluded, never counted as a zero.** This is the single most important rule in the feature, and it runs the other way from most reporting tools: | | Neens | The tempting alternative | | --- | --- | --- | | 412 cases, 168 with a CSAT score, 84 of them ≥ 4 | **50%**, coverage 41% | 20% (84 ÷ 412) | The second number is not more conservative — it is *wrong*, and it is wrong in a direction that changes decisions. Neens reports the honest rate and tells you the denominator it used. **Read the coverage before you act on the number.** A 92% deflection rate over 8% of your cases is not a 92% deflection rate. The UI captions every partial number; the API returns the block above on every read. ### When there is nothing to measure A measure with no measured units resolves to **`null`**, and every surface renders `—`. It never renders `0`, because a zero looks like a real value: it charts, it trends, and it satisfies an alert threshold. --- ## Where your measure shows up Once saved, a custom measure is a first-class catalogue measure: - **Measure gallery** — under its category (default: **Business**), with its provenance badge. - **Dashboard widgets** — pick it like any other measure. The widget tile shows *your* label, not the raw key. - **[Alert rules](/guides/alerts)** — "page me when Deflection rate drops below 72%". A `rate` measure's threshold is a fraction (`0.72`), and the rule builder pre-fills your declared direction and target. - **[Promoted to a KPI](/guides/business-kpis#promote-a-measure-to-a-kpi)** — when the measure stops being something you watch and becomes something you *committed to*, promote it: an agent-scoped KPI adds a target, an owner, a status and a review cadence on top of the measure, without redefining or recomputing it. Your measure's declared direction is what the KPI pre-fills, so you are not asked the same question twice. Deleting a custom measure leaves any KPI promoted from it reading `measure_removed` rather than a wrong number — one more reason to check the [blast radius](#before-you-delete-a-shared-measure) first. **Editing** the measure's rule is a smaller version of the same event: the KPI's [recorded history](/guides/business-kpis#a-definition-change-breaks-the-series) marks the edit as a series break and refuses to draw a trend across it, because the days either side are two different quantities. ### Slicing A measure can be broken down by the dimensions its shape supports — `agent`, `time`, `project`, `org`, `status`, `source`, `conversation`, plus `meta:` for a trace metadata field. An outcome-sourced measure can *also* slice by `queue` and `outcome_meta:`, because only it has your outcome rows in scope. Slicing by **agent** is the one worth calling out: a KPI you cannot break down per agent cannot tell you *which* agent moved it. ### The daily digest If you have the [persona digest](/guides/insights#the-digest-email) turned on, your measures ride along in it — up to four of them, after the four platform tiles, with your labels. A morning email is a glance rather than a report, so the cap is deliberate; the dashboard is where all of them live. **One surface is not automatic yet.** Custom measures flow into the measure gallery, dashboard widgets, alert rules and the persona digest. The SDK's `gate-as-code` `metric` rule does **not** pick them up: that rule gates a pre-prod run's judge rollup and has no catalogue path at all today, so wiring it is a separate change rather than a flag. The weekly narrative also still reads a fixed measure list. --- ## Scope: the definition is company-wide, what reads it is not A custom measure has **no agent scope**. One definition belongs to the whole company, and every agent in it sees the same measure under the same key. That is on purpose. A measure is your company's shared vocabulary: "containment" has to mean the same thing in staging as it does in production, or the two numbers are not comparable. It is also what lets an **Org**- or **Company**-scoped [dashboard](/guides/dashboards) chart one measure across several agents at once — impossible if each agent had its own private definition. The things that *point at* a measure are the opposite — they are agent-scoped: | Object | Scope | What it stores | | --- | --- | --- | | The **custom measure** | The company | The definition (source, grain, aggregation, target) | | An **[alert rule](/guides/alerts)** | One agent | The measure **key** (`custom:deflection-rate`) | | A **[dashboard widget](/guides/dashboards)** | One agent | The measure **key** | So a change made from inside one agent lands everywhere: - **Deleting** removes the definition for the entire company. Every alert rule that named it becomes a rule that can never fire again, and every widget that named it stops rendering a number — the tile shows an error where the chart used to be. - **Editing** the definition redefines it for the entire company. A redefinition re-points every sibling agent's alert rules and widgets at the new definition — the same rule, now watching a different number. **Nobody is told.** An orphaned alert rule stays *enabled* and looks healthy: it simply resolves no value, so it never fires and never notifies you that it stopped. A widget at least shows its error, but only to whoever next opens that dashboard — which may be in an agent you never look at. That is why the confirmation tells you the whole blast radius *before* you commit — see [Before you delete a shared measure](#before-you-delete-a-shared-measure). --- ## Editing, and what stays fixed Every row below applies **company-wide**, not just to the agent you are standing in. | Change | Effect | | --- | --- | | Rename the label / description | Free. Cached values are **not** invalidated — the number did not change. | | Change the target or direction | Free. Presentation only. | | Change the source, kind, threshold, filters or reducer | The number changes, so every cached value is invalidated **immediately** — in every agent. | | Change the identifier | Not offered. Alert rules and widgets store the key. | That split is deliberate. Renaming a KPI should not blow every cached chart; redefining one must never keep serving yesterday's answer. --- ## Before you delete a shared measure Deleting (or destructively redefining) a measure asks you to confirm, and the confirmation reports the **full company-wide blast radius**: - everything in **your current agent**, **by name** — each alert rule, each dashboard widget; - everything in **other agents**, as **bare counts only** — how many alert rules, how many widgets, across how many other agents. Names, ids and dashboard titles from other agents are deliberately withheld. They belong to agents you may have no right to read, and "how much you are about to break" is answerable without disclosing *what* somebody else called it. **Neens warns; it does not refuse.** After you confirm, the delete goes through even when the blast radius is non-zero. A mis-defined measure — one pointed at the wrong outcome kind, or one that has been rendering a wrong number for a month — has to remain removable, so the platform's job here is to make sure nobody deletes it *believing* nothing else read it. ### Worked example: checking first The confirmation dialog runs the same pre-flight check you can run yourself, against `GET /custom-measures/references/{identifier}`. Run it as an admin session, with the agent header selecting the agent whose names you want spelled out: ```bash curl https://your-neens-host/api/custom-measures/references/deflection-rate \ -H "Authorization: Bearer nk_sess_..." \ -H "X-Neens-Project-Id: proj_support_prod" ``` ```json { "references": { "alertRules": [ { "id": "ar_1", "name": "Containment below 60%" } ], "widgets": [ { "id": "w_9", "title": null, "dashboardId": "dash_ops", "dashboardName": "Ops" } ], "outOfScope": { "alertRules": 3, "widgets": 2, "projects": 2 } } } ``` Read that as: **one** alert rule and **one** widget in *this* agent, named — plus **three** more alert rules and **two** more widgets spread across **two other** agents in the company, counted. Deleting `deflection-rate` breaks all **seven** of them. The `DELETE /custom-measures/{identifier}` response carries the same `references` block, so a script that deletes without checking first can still log exactly what it just broke. ### How to read `outOfScope` `outOfScope` has three states, and two of them look alike at a glance: | What you get | What it means | What to do | | --- | --- | --- | | `{"alertRules": 3, "widgets": 2, "projects": 2}` | Other agents **do** depend on this measure | Coordinate before deleting (below) | | `{"alertRules": 0, "widgets": 0, "projects": 0}` | Nothing outside your agent references it | Safe to delete once the named local references are handled | | *(the key is absent)* | **Not established** — either not disclosed to your credential, or the count could not be completed. Says nothing about whether references exist | Re-check with a credential that can perform the delete; if it stays absent, treat the impact as unknown | `outOfScope` is only disclosed to a caller who could actually perform the delete or the edit — an admin (`MANAGE_MEASURES`). An agent-scoped `nk_live_` API key can read the route, so it still sees the named references *in its own agent*, but it never receives the `outOfScope` key at all: ```bash curl https://your-neens-host/api/custom-measures/references/deflection-rate \ -H "Authorization: Bearer nk_live_9f2c1a7b4e05d8..." ``` ```json { "references": { "alertRules": [{ "id": "ar_1", "name": "Containment below 60%" }], "widgets": [{ "id": "w_9", "title": null, "dashboardId": "dash_ops", "dashboardName": "Ops" }] } } ``` That response is **not** evidence that nothing else references the measure. It is the same response you would get if forty widgets in six other agents depended on it. **Absent is not zero.** If you are scripting a cleanup, test for the key's *presence* before you trust its contents — `if "outOfScope" not in refs: abort()`. Treating a missing `outOfScope` as "nothing else uses it" is exactly how an automated tidy-up deletes the measure another team's paging rule was built on, and nothing anywhere raises an error. ### When the blast radius is non-zero You have two honest options, and neither is "delete it and see who complains": 1. **Coordinate first.** The counts tell you how many owners to talk to (`projects`) and how much they lose (`alertRules`, `widgets`). Ask each agent's admins to repoint their alert rules and widgets at a replacement measure, or to remove them — then re-run the check and delete once `outOfScope` reads all zeros. 2. **Keep the key, change nothing structural.** If the problem is the *name*, rename the label: that is free, company-wide, and breaks nothing. If the problem is the *definition*, remember that a redefinition re-points every one of those rules and widgets at the new number rather than blanking them — often a better outcome than deleting, but only if the other agents are expecting it. In your own agent, handle the named references directly: edit each alert rule to watch a different measure (or delete it), and repoint or remove each widget on its dashboard. --- ## Platform measures stay ours The measures Neens ships (`error_rate`, `spend_usd`, `containment_rate`, …) cannot be edited or overwritten by a tenant. Attempting it returns a **409**. This is not gatekeeping — it is a correctness boundary. `spend_usd` is a token roll-up through a dated price table; `error_rate` is a predicate over a trace's status column. None of that is expressible in the source grammar above, so an "editable copy" would compute something *different* under a familiar name. If you want a differently-defined version, define your own measure against your own source and give it whatever label you like — including the same one. Your measures are namespaced `custom:`, so a collision with a platform key is impossible by construction. --- ## Validation: why a save can fail Neens validates a definition against the data your agent actually has, at **save** time. | Result | When | What happens | | --- | --- | --- | | **Error** | You reference an outcome kind / metric / label that does not exist, and this agent *has* others | Save is refused, 422, the offending field is named and the real options are listed | | **Warning** | This agent has recorded *nothing* of that type yet | Save succeeds, with a note that it will render blank until data arrives | | **Skipped** | We could not enumerate that vocabulary at all | No finding — a lookup that could not run must never masquerade as a definition that is wrong | The middle row matters: on a fresh agent, "wrong" and "not yet" are indistinguishable, and refusing would make it impossible to define a KPI the day before a feed goes live. Other refusals, each because the alternative is a plausible-looking wrong number: - A **classifier rate with no labels** — it would be `0%` forever. - A **`ratio` unit on a non-rate aggregate** — a percent sign over a number that is not a fraction. - An **outcome measure at trace grain** — it would count one case's outcome once per turn. - **Labels on a numeric aggregate** — they would be silently ignored, leaving you believing you had filtered. --- ## API reference All routes are admin-gated for writes (`MANAGE_MEASURES`) and `READ` for reads. | Method | Path | Purpose | | --- | --- | --- | | `GET` | `/custom-measures` | Your measures + the platform catalogue + your remaining quota | | `GET` | `/custom-measures/options` | The grammar **and** what this agent has actually recorded | | `POST` | `/custom-measures` | Define one | | `PATCH` | `/custom-measures/{identifier}` | Edit one — **company-wide** | | `DELETE` | `/custom-measures/{identifier}` | Delete one — **company-wide**; returns what referenced it, including `outOfScope` | | `GET` | `/custom-measures/references/{identifier}` | [What reads it](#how-to-read-outofscope), *before* you change anything: local references by name, other agents as counts | | `POST` | `/custom-measures/preview` | Resolve an **unsaved** definition over a bounded window | ### Preview ```bash curl -X POST https://your-neens-host/api/custom-measures/preview \ -H "Authorization: Bearer $NEENS_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "definition": { "slug": "draft", "label": "Draft", "source": "outcome", "outcomeKind": "csat", "grain": "case", "agg": "rate", "unit": "ratio", "rateOp": ">=", "rateValue": 4 }, "days": 7, "dimensions": ["agent"] }' ``` ```json { "rows": [ { "agent": "billing-bot", "custom:draft": 0.83 }, { "agent": "shipping-bot", "custom:draft": 0.51 } ], "meta": { "custom": { "provenance": "measured", "coverage": { "rows": 412, "measured": 168, "rate": 0.41, "asOf": "2026-07-27T09:14:22Z" } } }, "windowDays": 7, "windowClamped": false, "problems": [] } ``` Preview windows are capped at 30 days because an unsaved definition has no fingerprint to cache against, so every call is a real query. A **saved** measure has no such cap — it rides the normal read cache like everything else. --- ## Limits and plan Each company can define up to **50** custom measures; creating one past that limit returns a **409** naming the cap (delete one before adding another). The **Test it** preview window is capped at **30 days** — a saved measure has no such cap, since it rides the normal read cache. Authoring requires the `custom_measures` feature (**Silver** and above). Reading and charting an existing custom measure is available on every plan — a downgrade never blanks a dashboard you already built, it only stops you adding more. --- ## See also - [Business KPIs](/guides/business-kpis) — the three named KPIs and the definitions behind them - [Promote a measure to a KPI](/guides/business-kpis#promote-a-measure-to-a-kpi) — turn this measure into a commitment with a target, an owner and a review cadence - [KPI history and trends](/guides/business-kpis#history-and-trends) — what a promoted measure's daily record looks like, and what editing this measure does to its trend - [Business outcomes](/guides/business-outcomes) — getting the `measured` data in - [Metrics catalogue](/guides/metrics) — the platform measures and dimensions - [Alert rules](/guides/alerts) — paging on a measure crossing a threshold - [Dashboards](/guides/dashboards) — putting a measure on a board ================================================================================ # Cost & model pricing Source: /docs/guides/cost-and-model-pricing/ ================================================================================ # Cost & model pricing Every dollar figure in Neens — the **Spend** measure on a dashboard, the **Cost** column on the Traces list, the cost delta on a pre-prod comparison — is computed from the token counts your traces already carry, multiplied by a **price for that specific model**. This page explains where those prices come from, why some models are deliberately shown as *unpriced* rather than guessed at, and how to enter your own negotiated or self-hosted rates. ## At a glance | | | |---|---| | **What it is** | A dated, per-model price table — Neens-shipped list prices plus your own overrides | | **Where it lives** | **Settings → Model pricing** | | **Key API routes** | `GET/POST /model-prices`, `PATCH/DELETE /model-prices/{id}`, `GET /model-prices/observed` | | **Who can change it** | Company **admins** (the `manage_model_prices` permission); everyone with read access can *see* the table | | **Unit** | US dollars **per million tokens** (`$/MTok`) — the unit every vendor publishes | | **What it feeds** | The `spend_usd` [measure](/guides/metrics), persona cost dashboards, the trace **Cost** column, the [Agent Map](/guides/agent-map) node costs, pre-prod `deltas.cost`, and the eval CLI's `cost` gate rule | **There is no generic fallback rate.** If Neens does not have a price for a model, it says so instead of inventing one. See [Unpriced is a real state](#unpriced-is-a-real-state-not-a-bug). ## How Neens computes cost Cost is **derived, never stored**. There is no cost column on a trace waiting to be read — Neens computes the number at query time, every time: 1. Your traces record **input and output token counts** per span (and roll them up per trace), plus the **model** each model call used. Those come straight from your instrumentation — see [Send traces](/guides/send-traces). 2. At read time, Neens sums tokens grouped by model, resolves each model against the **price table**, and multiplies. So a trace on `claude-sonnet-5` with 12,000 input and 3,000 output tokens costs: ``` (12,000 / 1,000,000) × $3.00 = $0.036 input (3,000 / 1,000,000) × $15.00 = $0.045 output ------- $0.081 ``` Two consequences worth internalizing: - **Correcting a price corrects history immediately.** Nothing was baked in at ingest, so entering a rate today makes every existing trace show the right number (subject to [effective dates](#effective-dates-and-historical-accuracy) — a price change does not rewrite spend from *before* the change). - **One price table, everywhere.** The per-trace estimate and the aggregate roll-up resolve through the same table, so a trace's **Cost** and the dashboard **Spend** it contributes to can never disagree. ### The surfaces cost feeds | Surface | What it shows | |---|---| | `spend_usd` [catalogue measure](/guides/metrics) | Total spend, sliceable by `model`, org, project, and time — the basis of every cost widget | | Persona cost dashboards | The **Finance cost explorer** and the spend tiles on the Executive digest, all built from `spend_usd` | | **Cost** column + trace inspector | Per-trace estimated USD on the [Traces list](/guides/traces-and-sessions) and in the trace drawer | | [Agent Map](/guides/agent-map) | Per-node cost, and which node dominates spend across a cohort | | [Pre-prod comparison](/guides/preprod-evals) | Average cost per item for the candidate and baseline, and the `deltas.cost` between them | | Eval CLI gate | The `cost` rule's `max_avg_usd` / `max_delta` / `max_delta_pct` budgets | ## Where prices come from A price is a **row in a table**, not a constant in the product: it names a model, carries input, cached-input, and output rates in `$/MTok`, and is stamped with an **effective date** and a **named source**. Every price you see in Neens is in one of three states, and the UI labels which: | State | Badge | What it means | |---|---|---| | **List price** | `list · eff 2026-07-01` | A price Neens ships, from a dated published source. Read-only — you override it, you don't edit it. | | **Your rate** | `your rate` | A price *you* entered: a negotiated discount, a committed-use rate, a hosted-provider price, or `$0` for self-hosted. Wins over the list price. Scoped to the **agent** it was created under — set it in each agent whose spend it should apply to. | | **Unpriced** | `unpriced` | Neens has no price for this model. Its tokens are counted; its dollars are not. | ### How a model resolves to a price For each model Neens sees, it looks for a price in this order — and within each step, **your rate beats the shipped list price**. Note that the steps are tried in order: an exact list price is reached before any family-prefix rate, including one of yours, so state your override against the exact model id you run rather than a family prefix: ### Exact match The model id as recorded on the span, lowercased — `claude-sonnet-5`, `gpt-4o`, `gpt-oss:20b`. ### Family prefix If there's no exact row, Neens falls back to a **family** row — e.g. `claude-opus-` matches `claude-opus-4-6-20260315`. This is what keeps a newly released dated snapshot priced sanely instead of dropping to unpriced the day a vendor ships one. ### Unpriced No exact row and no family row ⇒ the model is **unpriced**. Not $0, not a guess — unpriced. A model with no name at all (spans that never recorded `gen_ai.request.model`, `gen_ai.response.model`, or `llm.model_name`) is always unpriced, and appears in the unpriced list under an empty name. If you see that, the fix is instrumentation, not pricing — see [Send traces](/guides/send-traces#which-attributes-neens-reads). ## Unpriced is a real state, not a bug This is deliberate, and it is the single most important thing to understand about cost in Neens: - An unpriced model contributes **$0 to the total** — it is excluded, not estimated. - Its tokens are still **counted and reported separately**, so you can see exactly how much traffic is missing a price. - Every dollar figure that includes unpriced traffic is labelled **partial**, with the number of unpriced models: *⚠ partial · 2 models unpriced*. Its ⓘ names the models and links to **Settings → Model pricing** — see [Fix a spend figure that says *partial*](#fix-a-spend-figure-that-says-partial). - A single unpriced value renders as **—**, never `$0.00`, so "we don't know" never masquerades as "it was free". **Neens will never silently apply a generic rate to a model it does not know.** A confidently wrong dollar figure is worse than a visibly missing one: you can't tell it's wrong, and you'll plan against it. If a number is incomplete, Neens says so — and tells you exactly which model to price. Aggregate responses carry the partial state as machine-readable metadata alongside the value, so dashboards, alerts, and the digest all surface the same caveat: ```json "pricing": { "partial": true, "unpricedModels": ["gpt-4o", ""], "unpricedInputTokens": 91000, "unpricedOutputTokens": 12000, "version": "42:2026-07-26T00:00:00Z" } ``` ## Fix a spend figure that says *partial* A spend figure reads **partial** when some traces in the window ran on a model with no rate. This section is the whole fix, end to end: what the label means, how to find out *which* models are unpriced, and how to price them. It takes about a minute per model, and you do it once per model — not once per dashboard. **A partial total is a floor, not an estimate.** The unpriced tokens contribute **$0** — they are excluded, never priced at a guess. So real spend is **higher** than the number on screen, by whatever the unpriced traffic actually cost you. The error only ever runs in that one direction. ### Where you'll see the badge The badge sits next to the figure it qualifies, so the caveat travels with the number: | Surface | Where the badge appears | |---|---| | **Overview** | Under the **Spend** tile, and in the **Spend** column of the breakdown table below it | | [Dashboard widgets](/guides/dashboards) | Next to the value on any currency widget — `spend_usd`, and `cost_per_case`, whose ratio is a floor for the same reason | | [Insights](/guides/insights) | On the spend tile of the insights header | | [Agent Map](/guides/agent-map) | In the graph footer next to the cohort cost | | [Model sweeps](/guides/model-sweeps) | On the arm leaderboard and the cost estimate panel | | [Pre-prod comparison](/guides/preprod-evals) | On each run's average cost | In dense chrome — a table cell, a graph footer — it renders compactly as *partial (2)*; on a headline tile it renders in full as *⚠ partial · 2 models unpriced*. When the backend flags the window as partial without enumerating the models (the Agent Map payload does), it says *⚠ partial · some models unpriced* rather than inventing a count. ### Read the popover — it names the models Every badge carries an **ⓘ**. **Hover it to read it, click it to pin it open** so you can move the pointer onto the contents, and **`Escape` or a click outside** dismisses it. It's a real button, so you can also reach it with `Tab` and open it from the keyboard. Pinning matters because the popover is not just an explanation — it contains **the link you need**: > **Partial spend** > Some traces in this window ran on a model with no rate. Their tokens are excluded from the total > rather than priced at a guess, so real spend is higher than shown. > **Unpriced models:** `gpt-4o`, `llama-3.3-70b-instruct` > **Set prices in Settings → Model pricing** *(a link)* That model list is the answer to "which ones?", and the link goes straight to `/settings?tab=pricing`. A model that recorded no name at all is listed as **(unknown model)** — that one is an [instrumentation fix](/guides/send-traces#which-attributes-neens-reads), not a pricing one. ### Then price them ### Open Settings → Model pricing Follow the link in the popover, or navigate to **Settings → Model pricing** yourself. A warning banner at the top of the tab restates the scale of the gap — *2 models have no rate — spend totals are partial until you set one* — and it is the same count as `unpricedCount` from the API. ### Find the unpriced rows — they're already at the top The table lists **the models this agent actually ran first**, and among those, **unpriced ones first**, ordered by token volume. So the model costing you the most accuracy is the top row. Each unpriced row shows the `unpriced` badge, the tokens seen in the window (*1,204,000 in / 88,000 out*), an em dash for every rate, and a **Set price** button. Models you *haven't* run in the window are marked **Not seen in this window** and sort below — they're not what made your figure partial. ### Enter the rate Click **Set price**. The model id is pre-filled and locked to the row you clicked. Fill in **Input ($ / 1M tokens)** and **Output ($ / 1M tokens)** — both required — optionally **Cached input**, pick a **Provider**, set **Effective from** (defaults to today, UTC), and use **Notes** to record where the number came from. Then **Save price**. Two worked examples, both taken off an invoice rather than guessed: | Model | Input | Cached | Output | Notes | |---|---|---|---|---| | `gpt-4o` | `2.50` | `1.25` | `10.00` | *list rate, invoice 2026-07* | | `llama-3.3-70b-instruct` | `0` | *(blank)* | `0` | *self-hosted on our own GPUs* | **`0` is a rate, not a blank.** Enter it for any model where you pay for GPU compute rather than tokens — a local Ollama model, a vLLM deployment. Leaving the field empty is an error (*Input and output rates are required. Enter 0 for a self-hosted model.*); typing `0` stores a real, resolving price of zero. **Rates are per 1M tokens**, matching how every vendor publishes them. ### Re-check the figure The badge flips to `your rate`, the banner's count drops, and — because cost is computed at query time, never stored — **every existing figure recomputes on the next read**, including historical ones back to the rate's **Effective from** date. Reload the dashboard: if that was the last unpriced model in the window, the *partial* badge is gone and the total is complete. Repeat for each name the popover listed. When `unpricedCount` reaches `0`, nothing in the agent is partial any more. **Not an admin?** Changing prices needs the company **admin** role, and the **Model pricing** tab isn't in your Settings at all without it. The badge and its popover are visible to everyone — that's deliberate, so whoever is reading the number knows it's incomplete — but the fix is an admin action. Send them this page and the list of model names from the popover. ### The same fix, from the API If you'd rather not click, `GET /model-prices/observed` gives you the identical list — every model your spans carried over the window, with `priced: false` on the ones making figures partial — and `POST /model-prices` sets a rate. Both are covered in [Set rates from the API](#set-rates-from-the-api). Watching `unpricedCount` in CI is the durable version of this fix: it fails the day someone routes traffic to a model you haven't priced, instead of the day someone notices a dashboard looks low. ## Which models ship priced The shipped catalogue is deliberately small: **the Anthropic family, and the two `gpt-oss` local models at $0.** Everything else is unpriced until you say otherwise — Neens only ships a price it can date, name a source for, and stand behind. ### Anthropic — priced The Anthropic family ships with dated list prices (source `anthropic-list-2026-07-01`, effective **2026-07-01**), in `$/MTok`: | Model | Input | Cached input | Output | |---|---|---|---| | `claude-fable-5` | $10.00 | $1.00 | $50.00 | | `claude-mythos-5` | $10.00 | $1.00 | $50.00 | | `claude-opus-5` | $5.00 | $0.50 | $25.00 | | `claude-opus-4-8` | $5.00 | $0.50 | $25.00 | | `claude-opus-4-7` | $5.00 | $0.50 | $25.00 | | `claude-opus-4-6` | $5.00 | $0.50 | $25.00 | | `claude-opus-4-5` | $5.00 | $0.50 | $25.00 | | `claude-sonnet-5` | $3.00 | $0.30 | $15.00 | | `claude-sonnet-4-6` | $3.00 | $0.30 | $15.00 | | `claude-sonnet-4-5` | $3.00 | $0.30 | $15.00 | | `claude-haiku-4-5` | $1.00 | $0.10 | $5.00 | Plus three **family fallback** rows so an unrecognized dated snapshot still prices sanely: `claude-opus-` → $5 / $0.50 / $25, `claude-sonnet-` → $3 / $0.30 / $15, `claude-haiku-` → $1 / $0.10 / $5. If you buy Anthropic capacity through a committed-use agreement or a reseller, **your rate is not the list rate** — [override it](#set-your-own-rate) so your spend numbers are yours, not the sticker price. ### `gpt-oss` — priced at $0 Exactly two open-weight ids ship with a price: **`gpt-oss:20b`** and **`gpt-oss:120b`**, at **$0.00 per million tokens**, source `self-hosted-zero-cost`. That is not "unknown" — it is a real, deliberate price. These are local-first models: they run on hardware you already pay for, so **you're billed for GPU compute and hours, not for tokens.** Attaching a per-token price to them produces a number with no relationship to any invoice you will ever receive. **This is a correction, not a cop-out.** Earlier releases priced any unrecognized model at a generic mid-tier rate. That got self-hosted models badly wrong *in the expensive direction* — a local `gpt-oss:20b` workload could show thousands of dollars of "spend" that nobody was ever billed for. If your historical self-hosted spend looks like it collapsed, that's the bug being fixed. ### Everything else — unpriced on purpose **Every other model resolves to `unpriced` out of the box.** That covers two groups, for two closely related reasons. **Third-party hosted models — GPT, Gemini, and friends.** `gpt-4o`, `gpt-4o-mini`, `gemini-2.5-pro` and their siblings ship with no price. We will not state a third-party list price we cannot date and verify, and we especially will not ship one that quietly drifts out of date while your finance dashboard keeps reporting it as fact. Third-party pricing also varies by tier, region, batch mode, and contract — the number *you* are billed is genuinely not knowable from here. **Open-weight models beyond `gpt-oss` — Llama, Qwen, Mistral, Mixtral, DeepSeek, Phi, Gemma.** `llama-3.3-70b-instruct`, `mistral-large-latest`, `deepseek-chat`, `gemma-3-27b-it`, `phi-4`, `qwen2.5:14b` and the rest are **unpriced**, deliberately — Neens does *not* blanket them at $0. The reason is specific: **the same model name can be free or expensive, and the model string alone doesn't say which.** `mistral-large-latest` might be running on your own GPUs (genuinely $0) or through Mistral's La Plateforme, Bedrock, Together, Fireworks, or Groq — where every token is invoiced. A blanket $0 would silently price real, billed spend at nothing. **An under-report is more dangerous than an over-report.** If Neens overstates spend, someone notices and complains. If it understates spend to zero, the number looks fine and nobody investigates — you find out from the invoice. That's exactly the fabricated-constant bug this release exists to kill, merely inverted, so Neens asks instead of guessing. **The remedy is one action, either way, and both answers are honest:** - **Self-hosting Llama on your own GPUs?** Set it to `0` once in **Settings → Model pricing**. It's a real price, it sticks, and you never think about it again. - **Paying Together, Bedrock, Groq, or DeepSeek per token?** Enter the rate off your invoice, once. Now your dashboards show what you're actually spending. Until you do, those models' tokens are counted but contribute $0, and every spend figure that includes them is labelled partial — so the gap is visible rather than silent. Follow [Set your own rate](#set-your-own-rate) below; it takes about a minute per model. ## Set your own rate **Settings → Model pricing** is the admin surface. The table lists **Model · Provider · Input · Cached · Output · Source**, with the models you actually run listed first — including unpriced ones, each with a **Set price** button. **Admin only.** Changing prices requires the company **admin** role (the `manage_model_prices` permission, the same tier as [data retention](/administration/data-retention)). Members and viewers don't see the tab, and the write endpoints return `403`. Every change is recorded in the [audit log](/administration/audit-log). ### Price a model Neens ships unpriced Find the model (e.g. `gpt-4o`) — it shows the `unpriced` badge — and click **Set price**. Enter **Input**, **Output**, and optionally **Cached input**, all in **dollars per million tokens**, straight off your provider invoice or contract. Pick an **Effective from** date (defaults to today) and save. The badge flips to `your rate`, and every spend figure that included that model stops being partial. For a model you haven't run yet — pricing ahead of a migration, say — use **Add model price** and type the model id yourself. ### Override a shipped list price A row showing `list · eff 2026-07-01` is a platform price. It has an **Override** button, not an inline edit — clicking it creates *your* row on top of the list row. Enter your negotiated rate (e.g. `claude-sonnet-5` at $2.40 / $12.00 under a committed-use agreement), add a **note** like "committed-use agreement — renewal 2027-01", and save. Your rate now wins wherever that model resolves. The list row is still there underneath, unchanged. ### Enter `$0` for a self-hosted model `0` is a **valid, storable price**, not "unset". Use it for any model where you pay for compute rather than tokens — a local Ollama model, a vLLM deployment, a model on your own GPUs. This is the step for **self-hosted open-weight models**: apart from `gpt-oss:20b` and `gpt-oss:120b`, Neens ships no $0 rows, because it can't tell a locally-served `llama-3.3-70b-instruct` from one billed by a hosted provider. Set it to `0` once and the model is priced correctly from then on — same one-time action as entering a paid rate, and just as honest. ### Revert to the list price On a row showing `your rate`, click **Revert to list price** and confirm. Your override is deleted and the model falls back to whatever it resolves to without it — the shipped list price, or `unpriced` if there isn't one. **Platform list prices are immutable to you.** You override them; you never edit them. Attempting to `PATCH` or `DELETE` a platform row returns `409`. That keeps the shipped catalogue upgradeable — when Neens ships a newer dated price list, your overrides are preserved and never clobbered. ## Set rates from the API Every Settings action has a REST equivalent, so rates can live in your IaC repo or be applied from a CI job. Routes are available bare and under the canonical `/api/` prefix; use `/api/` for anything programmatic. Reads need any read-capable credential (a `nk_live_` agent key works); writes need an **admin-role** credential — an admin's session, or a `nk_live_` agent key created with the `admin` role. This is *not* the operator `nk_admin_` control-plane key, which carries no agent scope. The `$NEENS_API_KEY` in the examples below is an admin-role agent key. ### Which models am I running, and which are unpriced? `GET /model-prices/observed` is the "state of the world" call: the distinct models actually seen in your agent's spans over a window, joined against the resolved price table. ```bash curl -G https://your-neens-host/api/model-prices/observed \ -H "Authorization: Bearer nk_live_your_key_here" \ --data-urlencode "range=30d" ``` ```json { "models": [ { "model": "gpt-oss:20b", "priced": true, "matchedBy": "exact", "provenance": "platform", "effectiveFrom": "2026-07-01", "source": "self-hosted-zero-cost", "inputPerMillion": 0.0, "cachedInputPerMillion": null, "outputPerMillion": 0.0, "inputTokens": 1204000, "outputTokens": 88000, "spendUsd": 0.0, "priceId": "mp-platform-gpt-oss-20b-2026-07-01" } ], "unpricedCount": 2, "version": "42:2026-07-26T00:00:00Z" } ``` `unpricedCount` is the number to watch: **anything above zero means your spend figures are partial.** A nightly CI check that fails when `unpricedCount > 0` is a cheap way to keep a new model from silently landing outside your cost reporting. ### Set a rate ```bash curl -X POST https://your-neens-host/api/model-prices \ -H "Authorization: Bearer $NEENS_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "modelId": "gpt-4o", "provider": "openai", "displayName": "GPT-4o", "inputPerMillion": 1.88, "cachedInputPerMillion": 0.94, "outputPerMillion": 7.5, "effectiveFrom": "2026-07-01", "notes": "committed-use discount" }' ``` Returns `201` with the created row. Re-running the exact same call is a **no-op upsert**, not an error — so re-applying your pricing manifest on every deploy is safe. Field reference | Field | Required | Notes | |---|---|---| | `modelId` | ✅ | The exact vendor model id, or a family prefix (e.g. `claude-opus-`). Lowercased server-side. | | `inputPerMillion` | ✅ | USD per million input tokens. Must be `>= 0`; **`0` is valid and meaningful.** | | `outputPerMillion` | ✅ | USD per million output tokens. Must be `>= 0`. | | `cachedInputPerMillion` | | USD per million cached-input tokens. Omit to bill cached input at the input rate. | | `provider` | | `anthropic`, `openai`, `google`, `bedrock`, `meta`, `mistral`, `self_hosted`, `other`. | | `displayName` | | What the table shows. | | `effectiveFrom` | | ISO date. Defaults to today (UTC). | | `notes` | | Free text — record the contract, invoice, or ticket the rate came from. | `422` on a negative rate, a blank `modelId`, or a malformed date. The rest of the surface | Call | Does | |---|---| | `GET /model-prices` | The resolved table. `?includeHistory=true` returns superseded rows too; `?provider=` and `?q=` filter. The response carries a `version` fingerprint that changes whenever any price changes. | | `PATCH /model-prices/{id}` | Edit a rate, `effectiveFrom`, `effectiveTo`, `displayName`, `provider`, or `notes` on one of *your* rows. **`409` on a platform row.** | | `DELETE /model-prices/{id}` | Remove your override — the model reverts to the list price, or to unpriced. `204`. **`409` on a platform row.** | | `GET /model-prices/observed` | The models seen in your spans + their resolved price and token volume. | Cached spend figures are keyed on the price table's `version`, so an edit is reflected on the next read — you never have to wait out a cache TTL to see a corrected number. ## Effective dates and historical accuracy Every price row carries a window: **`effectiveFrom` (inclusive)** and **`effectiveTo` (exclusive, empty = still current)**. A trace is costed at the price that was **in force when it ran** — not at today's price. That means a price change never retroactively rewrites last quarter's spend. Your Q3 number stays your Q3 number after you renegotiate in Q4. ### Worked example You run `claude-sonnet-5` on the shipped list price of $3.00 / $15.00, effective `2026-07-01`. On 1 September you sign a committed-use agreement at 20% off, so you post your rate: ```bash curl -X POST https://your-neens-host/api/model-prices \ -H "Authorization: Bearer $NEENS_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "modelId": "claude-sonnet-5", "provider": "anthropic", "inputPerMillion": 2.4, "outputPerMillion": 12.0, "effectiveFrom": "2026-09-01", "notes": "committed-use agreement, 20% off list" }' ``` From then on, the same 12,000-in / 3,000-out trace costs: | Trace ran on | Price applied | Cost | |---|---|---| | 2026-08-14 | List, $3.00 / $15.00 (your rate wasn't in force yet) | $0.081 | | 2026-09-14 | Your rate, $2.40 / $12.00 | $0.0648 | A **30-day dashboard spanning 1 September mixes both** — each trace priced by its own date. That is the point: the total is what you were actually billed, not a re-projection of today's rate over history. When the agreement ends, don't delete the row — **close it**. `PATCH` it with `"effectiveTo": "2027-01-01"` and the window becomes `[2026-09-01, 2027-01-01)`: a trace on 31 December 2026 still uses your rate, a trace on 1 January 2027 falls back to whatever comes next. Your history stays intact. ## Cached input Prompt caching is billed at a different rate from fresh input, so a price row carries a separate **cached input** rate. Cached tokens are treated as a *subset* of the input tokens: the cached portion is priced at the cached rate and the remainder at the full input rate. If a row leaves **Cached input** blank, cached tokens are simply **billed at the input rate** — a safe over-estimate rather than a silent discount. The shipped Anthropic rows set cached input at one tenth of the input rate (the cache-read rate) — $0.50/MTok on `claude-opus-5`, $0.30 on `claude-sonnet-5`. When you enter your own rate, fill in **Cached input** if your contract prices it separately; leave it blank if it doesn't. ## Troubleshooting | Symptom | Cause | Fix | |---|---|---| | A spend figure shows *⚠ partial · N models unpriced* | Some traffic ran on a model with no price | [Fix a spend figure that says *partial*](#fix-a-spend-figure-that-says-partial) — the popover names the models; `GET /model-prices/observed` lists them all | | The badge's popover closes before you can click the link in it | You moved the pointer off the ⓘ | **Click** the ⓘ to pin the popover open, then click the link; `Escape` or a click outside closes it | | A trace's **Cost** shows `—` | That trace's model is unpriced | Set a price for it — the value is missing, not zero | | `gpt-oss` spend "dropped to zero" | Correct — `gpt-oss:20b` / `gpt-oss:120b` are priced at `$0` because you pay for compute, not tokens | Nothing to do; if you're billed per token for them by a hosted provider, override the rate | | Llama / Mistral / DeepSeek / Qwen show as unpriced | Deliberate — Neens can't tell a self-hosted run from a hosted, invoiced one from the model name | Enter `0` if you self-host, or your provider's rate if you don't — once, in **Settings → Model pricing** | | An unpriced model with **no name** in the list | Spans didn't record a model attribute | Emit `gen_ai.request.model` / `gen_ai.response.model` / `llm.model_name` — see [Send traces](/guides/send-traces) | | `409` editing a price | You're editing a shipped list price | Use **Override** (a `POST` of your own row) instead | | `403` on a write | You're not a company admin | Ask an admin, or see [Members & roles](/administration/members-and-roles) | | A pre-prod run reports the cost gate as *skipped* | Neither side had a priced session, so an honest cost delta can't be computed | Price the candidate's model; the gate would otherwise be comparing a fabricated number | ## Related - [Metrics catalogue](/guides/metrics) — the `spend_usd`, `tokens_in`, and `tokens_out` measures and the `model` slice. - [Model comparison](/guides/model-comparison) — the other half of the argument: whether the cheaper model still passes your evals, sliced by model and by agent × model. - [Business KPIs](/guides/business-kpis) — `cost_per_case`, the same spend divided by the cases it handled (and a floor, not a total, while any model is unpriced). - [Dashboards](/guides/dashboards) — build cost widgets, including the Finance cost explorer. - [Traces & sessions](/guides/traces-and-sessions#token-usage--cost) — the per-trace **Cost** column. - [Pre-prod evaluations](/guides/preprod-evals#gate-as-code-a-richer-policy-file) — the `cost` gate rule. - [Usage metering](/administration/usage-metering) — feature-level usage counters, a separate signal from LLM spend. ================================================================================ # Cost & Quality Source: /docs/guides/cost-and-quality/ ================================================================================ # Cost & Quality **Cost & Quality** puts spend and quality on the same screen and answers one question for each of your two big LLM bills: *could a cheaper model do this job without losing quality I can actually measure?* It leads with the single recommended move and the one button that matters — **Test before switching** — and keeps the raw sweeps and scorer comparisons a click away as evidence. The page has three tabs: | Tab | Answers | | --- | --- | | **Overview** | What am I spending, what could I save this month, and what is the one move to make next? | | **Agent Model** | Can a cheaper model *run the agent* and still pass my golden suite? | | **Judge Model** | Can a cheaper model *judge* the agent and still agree with my gold labels? | ## At a glance | | | | --- | --- | | **Where it lives** | **Cost & Quality** in the left nav, under *Evaluate* → `/cost-quality` | | **Tabs** | **Overview** · **Agent Model** · **Judge Model** (`?tab=overview\|agent\|judge`) | | **Key API routes** | `GET /cost-optimization/summary`, `GET /cost-optimization/judges`, `GET /cost-optimization/frontier`, `GET /cost-optimization/moves` | | **Needs** | Ingested traces (agent spend), at least one LLM judge that has run (judge spend), [gold labels](/guides/annotations-and-review) to compare *judge* models, a finished [model sweep](/guides/model-sweeps) to compare *agent* models | | **Scope** | The agent (project) you're viewing. Anyone with read access can see it | | **Reads only** | Nothing here spends money and nothing switches a model for you | | **Dollars** | Every dollar amount on these tabs is shown as a **whole number, rounded up** — no cents | **Old links still work.** `/model-bench` opens this page, `/model-sweeps` opens the **Agent Model** tab, and `/cost-optimization` opens **Overview**. Individual sweeps (`/model-sweeps/{id}`, `/model-sweeps/{id}/compare`) and scorer comparisons (`/scorer-comparison/{id}`) are unchanged. ## The version slice A Neens project **is** one agent, so "which agent?" is never a question this page asks. The useful sub-slice is **which version of that agent** produced the trace, read from the `neens.version_label` attribute your traces carry: ```python span.set_attribute("neens.version_label", "v2.4.0") ``` - Every label you have sent appears in the version selector, newest first. The label with the most recent trace is **live**, and it is the default selection on the tabs that have a version control — you land on the build you are actually running, not on an "all versions" blend. - Traces with no label form their own **unversioned** row. They are never folded into the live version: "we don't know which build this was" and "this was the current build" are different facts. - Nothing breaks if you never send the attribute — you get one unversioned row and everything else works. Adding it later starts the split from that point forward; it does not rewrite history. **Where the version control appears differs by tab**, because the useful slice is different: | Tab | Time range | Version | | --- | --- | --- | | **Overview** | Picker in the header | Header select, defaults to **live** | | **Agent Model** | Picker in the header | In-body dropdown, defaults to **live** | | **Judge Model** | *None* | Header select, defaults to **live** | **The Judge Model tab has no time-range picker, on purpose.** A judge is scored against a fixed set of gold labels, not against a rolling stream of traffic — so "the last 7 days" is meaningless for judge *agreement*. The one traffic-based number on that tab (flag rate, below) is computed over the judge's recent scored sessions and does not need you to choose a window. ## Overview Overview merges spend and recommendations into one visual screen: a hero move, three stat tiles, a spend trend, a spend breakdown, and the ranked list of moves. ### The hero move The top of the tab is the **single most valuable move we can prove**, presented as the page's main call to action. If a cheaper model clears the bar, this is a **Switch** card whose button reads **Test before switching** (see [Test before switching](#test-before-switching)). The copy is explicit that pressing it does *not* change your agent — it opens a gated test first. If there is nothing to recommend, the hero says so honestly rather than inventing an action. ### The three tiles | Tile | What it is | | --- | --- | | **Agent spend** | What the agent's own LLM calls cost over the window, with a daily sparkline | | **Judge spend** | What your judges actually spent scoring, shown big — with a smaller **"At full coverage ~$X"** projection and a **coverage %** chip beside it | | **Savings available** | The **projected monthly** saving from the moves we can prove, labelled as monthly/projected | Two of these need reading carefully. **Judge spend** leads with the real number — what your judges actually spent — because that is what lands on the bill. Underneath it, **"At full coverage ~$X"** answers a different question: what *would* it cost if this judge scored every trace instead of sampling a share of them? The coverage chip tells you how much of your traffic the judge is currently scoring, so you can read the two together: a low actual cost next to a high full-coverage projection just means the judge is sampling, not that it is cheap to run everywhere. The **at-full-coverage** projection is shown as **—** whenever a judge with spend in the window has no price in your [price table](/guides/cost-and-model-pricing). A partial total presented as a whole one is worse than no total. **Savings available** is the number the earlier version of this page got wrong, so it is worth being precise about what it now means. **Savings available is a monthly projection, independent of the time range you have selected.** Changing the window from 7 days to 30 days changes how much you *spent*, but it does not change how much a proven switch would *save you per month* — that is a property of the price gap between the two models and your typical monthly volume, not of the window you happen to be looking at. The figure is derived from a per-unit saving projected onto a steady monthly volume, so it stays stable as you move the picker. Only **switch** moves that we can prove contribute to it; holds, untested versions, and moves with no quality signal add nothing. ### Spend trend and breakdown **Spend trend** stacks judge spend on agent spend, one bar per complete UTC day. Today's bucket is marked partial rather than shown as a finished day that happens to look cheap. **Spend by version | model** breaks the same total down as bars — the **total dollars** spent per agent version, or per model. It is deliberately kept at the dollars-you-paid level; there is no per-trace or per-1,000 framing here, because for a "where did the money go" view the absolute total is the honest unit. ### Recommended moves Below the charts is the ranked list of every move we found — at most one per version and one per judge — sorted by projected monthly saving: | Status | Means | | --- | --- | | **Switch** | A cheaper model clears the bar with no measurable quality loss. This card has a **Test before switching** action | | **Hold** | You are already on the best proven option. No action, and no card pretending there is one | | **Untested** | There is no finished, comparable sweep for this version yet. The action is to run one | | **No quality signal** | There is nothing to measure quality against — usually no gold labels. The action is to add some | ## Agent Model **On this tab, quality means a stable pass rate on your golden suite.** Neens replays the same frozen dataset against each candidate model *k* times and judges every run with your judges. "Stable" is the key word: a model that passes on two of three replays is **not** counted as passing. This pass^k reading is what stops a lucky single run from recommending a model that is actually flaky — you are switching on behaviour that repeats, not on one good roll. Pick a version from the in-body dropdown (it defaults to your live one) and the tab shows the newest finished, comparable [sweep](/guides/model-sweeps) for it. - **Candidate bars** — one row per model, named exactly as it appears in your traces with the provider underneath. The **quality axis is the stable pass rate on your golden suite**, and a **pass-rate bar** is drawn across the chart at the threshold the candidate has to clear. Models that fall below the bar are drawn visually distinct so a failing candidate can never be mistaken for a passing one. The model you run today is chipped **Current**; the frontier's pick is chipped **Recommended**. - **Verdict card** — the plain-language read of the comparison: how much a switch saves per month, the change in pass rate (with an honest *not distinguishable at n=…* when the sample is too small to separate two models), and whether the recommended model clears the bar. - **Dataset version** — a **link** straight to the [dataset](/guides/datasets) version the sweep replayed, so you can open the exact items behind the number instead of taking the pass rate on faith. - **Evidence** — the underlying sweep, a **Share report** link a colleague can open without an account, and the option to launch a new sweep. A worked example, from the shipped demo — a 30-item golden set replayed 3× with a 90% bar: | Model | Stable pass rate | Monthly cost | Read | | --- | --- | --- | --- | | `claude-sonnet-4-6` | 3/3 green — clears | $45 | What you run today (**Current**) | | `claude-haiku-4-5` | 3/3 green — clears | $15 | **Recommended** — a third of the cost, no measurable loss | | `gpt-4o-mini` | 3/3 green — clears | **—** | Clears, but unpriced, so it can't be recommended | | `claude-sonnet-4-5` | 2/3 green — fails | $45 | Two of three is not a pass | | `gpt-oss:120b` | 0/3 green — fails | $0 (self-hosted) | Cheapest possible, nowhere near the bar | Two rules make the recommendation trustworthy rather than merely confident, and the demo ships an example of each: 1. **An unpriced model can never win.** A model with no price shows **—** for cost, never `$0`, and is never recommended — you cannot claim a saving you cannot compute. A model priced at a real **$0** (self-hosted, where you pay for the GPU not for tokens) *can* win, but only if it also clears the bar. 2. **A measurement carries its uncertainty.** "Not measurably worse" means the candidate's confidence interval overlaps the current model's — not that its point estimate happened to land higher. Too small a sample to tell them apart reads as *not distinguishable*, not as a winner. A sweep whose arms did not run on comparable inputs is **void**: Neens refuses to draw a comparison it cannot stand behind rather than showing a misleading chart. See [Sweep decisions](/guides/sweep-decisions). ## Judge Model **On this tab there are two quality numbers, and which one you can act on depends on whether you have gold labels.** This is the single most important thing to understand about the Judge Model tab, so read it before the table. | Number | What it measures | Needs gold? | What it drives | | --- | --- | --- | --- | | **Fail rate** *(vs gold)* | `1 − agreement`: how often the judge's verdict *disagrees* with the human verdict on the same session. Lower is better | **Yes** | The recommendation — a cheaper judge is only proposed when its fail-rate-vs-gold is not distinguishably worse | | **Flag rate** *(on traffic)* | The share of the sessions this judge actually scored where it returned a *failing* verdict. Available for every judge | No | Nothing on its own — it describes the judge's *behaviour*, not its *accuracy* | The distinction matters because the two can look similar and mean opposite things. A judge with a high **flag rate** is failing a lot of your traffic — which is fine if your traffic really is failing, and a problem if the judge is trigger-happy. You cannot tell which from the flag rate alone. Only the **fail rate vs gold** — comparing the judge to a human on the same sessions — tells you whether the judge is *right*. That is why: **A cheaper judge cannot be recommended without gold labels.** With no gold labels, there is no way to know whether the cheaper model *agrees with a human* — only how often it flags traffic, which says nothing about accuracy. So a judge with no gold shows **—** in the fail-rate column and its status reads **Add gold labels to compare models**. The recommendation is withheld, not guessed. Add labels in [Review](/guides/annotations-and-review) and the recommendation appears once verdicts exist. ### The lens toggle A toggle at the top-right of the table switches which number both quality columns show: - **Fail rate** (default) — `1 − agreement` with gold. This is the lens that drives recommendations. Judges without gold read **—**. - **Flag rate** — the fail-verdict rate on traffic, available for *every* judge. In this lens the **recommended-model column reads —**, because a model you have not deployed has no live traffic to flag; the recommendation itself stays gold-based whichever lens you are viewing. ### All judges at a glance The table lists every judge with its current model beside the cheapest model that still holds the bar, so the comparison is one row: | Judge | Current model | Fail rate | Cost | Cheapest that holds | Fail rate | Cost | Status | | --- | --- | --- | --- | --- | --- | --- | --- | | Groundedness Check | `claude-sonnet-4-5` | 7% | $12 | `claude-haiku-4-5` | 7% | $4 | **Switch** — same agreement, a third of the cost | | Reasoning Consistency | `claude-sonnet-4-5` | 9% | $11 | `claude-haiku-4-5` | 27% | $4 | **Hold** — the cheap model is genuinely worse here | | Harmful Content Screen | `claude-haiku-4-5` | — | $4 | — | — | — | **Add gold labels to compare models** | Read the first two rows together: the *same* cheap model is the right answer for one judge and the wrong answer for another. That is exactly why this is measured per judge rather than decided once for the whole workspace. The recommendation is always the cheapest scorer model whose fail-rate-vs-gold is not distinguishably worse and still clears the bar. ### The docked comparison chart The full per-model comparison for a single judge is **docked at the bottom of the tab and only opens when you select a judge row**. Selecting a row draws that judge's candidate bars — one per scorer model — with the **agreement threshold bar (default 0.80)** drawn across them; models below the bar are visually distinct, and cost is shown as a whole-dollar label. A close control hides the dock again. A judge with no gold labels shows an **add-gold-labels** empty state in the dock instead of bars, because there is no agreement axis to plot. To compare scorer models you have not run yet, the **Run a scorer comparison** button at the top of the tab opens the launch dialog directly — the same targets scored by several models side by side, which is how you get gold-grounded evidence for a judge before you switch it. ## Test before switching Every recommended switch on this page leads to the same button — **Test before switching** — and never to a button that flips the model for you. Neens does not edit your agent or redeploy your judge; it hands you a gated test and gets out of the way. For an **agent-model** switch, that test is a [pre-prod evaluation](/guides/preprod-evals) run against the candidate model: ### Press **Test before switching** The pre-prod eval dialog opens **pre-filled** from the evidence on screen: the same golden dataset version the sweep replayed, the same judges, the same pass-rate bar, and the candidate model as the version under test. ### Check the prefill Everything is editable. Widen the dataset, add a judge, raise the bar — the prefill is a starting point drawn from the comparison, not a locked form. ### Save and let it run Saving creates the pre-prod eval, which **reruns your golden suite through your judges against the candidate model and checks the result against your pass-rate bar**. If the candidate regresses — its stable pass rate drops below the bar — the gate goes red and the switch is **blocked**. ### Ship behind a green gate Only when the gate is green do you switch the model in your own deployment. You can wire the same gate into CI with [eval gates](/guides/eval-gates) so a regression can never merge. **Why the gate protects you.** The sweep on the Agent Model tab is *evidence* — a comparison of past runs. The pre-prod gate is *proof for this switch* — a fresh, controlled rerun of your own golden suite against the exact candidate you are about to ship, with the same bar it will be held to. It is the difference between "this model looked good last week" and "this model clears my bar right now", and it is the last thing standing between a promising cheaper model and a silent quality regression in production. For a **judge-model** switch, the equivalent proof is a [scorer comparison](/guides/cost-optimization#compare-scorer-models-side-by-side) against your gold labels — the same "measure before you move" principle, because a judge you cannot check against a human is a judge you cannot trust to be cheaper *and* right. ## How-tos ### Cut judge cost without losing agreement ### Open **Judge Model** and keep the **Fail rate** lens The default lens compares each judge to your gold labels. This is the only lens that can tell you a cheaper judge is still *correct*, not just quieter. ### Find a **Switch** row A judge is a switch candidate when a cheaper scorer model holds the bar at a fail rate not distinguishably worse than the current model's. If a judge you care about reads **Add gold labels to compare models**, label a handful of its sessions in [Review](/guides/annotations-and-review) first — without gold there is no recommendation to make. ### Select the row and read the docked chart Confirm the cheaper model sits above the 0.80 agreement bar with an interval that overlaps your current model's. If it clears the bar and saves real dollars, this is a safe switch. ### Prove it with a scorer comparison Press **Run a scorer comparison** to score the same targets with both models side by side against gold before you redeploy the judge. ### Prove a cheaper agent model on your golden set before switching ### Open **Agent Model** and pick the version you run The dropdown defaults to your live version. The tab shows the newest comparable sweep for it. ### Read the pass-rate bar, not just the ranking The **Recommended** model must sit above the pass-rate bar on the *stable* pass rate (pass^k) — a model that only passes some of its replays sits below the bar and is not a candidate, however cheap. ### Open the dataset version Follow the **Dataset version** link to see the exact golden items behind the number. If the suite is too small or too narrow for the decision, widen it before you trust the pass rate. ### Press **Test before switching** Run the pre-prod gate against the candidate. Ship only when it is green, and wire it into CI with an [eval gate](/guides/eval-gates) so the cheaper model can't quietly regress later. ## Troubleshooting | Symptom | Cause | Fix | | --- | --- | --- | | Every cost shows **—** | No prices for the models in your traces | Add them in [Settings → model pricing](/guides/cost-and-model-pricing). Neens never guesses a rate | | **Savings available** is `$0` with moves on screen | Those moves are holds, untested, or lack a quality signal — only proven **switch** moves contribute | Read each card's status; the tile deliberately sums proven switches only | | **Savings available** looks unchanged when I change the time range | It is a **monthly projection**, independent of the window by design | This is correct — the window changes *spend*, not the per-month saving of a switch | | A judge's **fail rate** reads **—** and its status says *Add gold labels* | No gold labels for that judge's scored sessions | Label sessions in [Review](/guides/annotations-and-review); the fail rate and recommendation appear once verdicts exist | | A judge shows a **flag rate** but no recommendation | Flag rate is behaviour on traffic, not accuracy vs gold — it can't justify a switch | Add gold labels so the fail-rate lens can compare the cheaper model to a human | | The **Agent Model** tab says *untested* | No finished, comparable sweep for that version | Run one from the tab, or see [Model sweeps](/guides/model-sweeps) | | A version you expect is missing | No trace in the window carried that `neens.version_label` | Widen the time range, or check the attribute is set on a span your agent emits | | Judge spend looks impossibly low | The judge is sampling | Read the **at-full-coverage** projection and the coverage chip, not the actual — that is what they are for | | Today's bar looks small | Today is a partial day | It is marked partial; compare complete days | ## Related - [Cost & model pricing](/guides/cost-and-model-pricing) — where every rate on this page comes from, and what unpriced means - [Cost–quality frontier](/guides/cost-quality-frontier) — the statistics behind "clears the bar" and "not measurably worse" - [Model sweeps](/guides/model-sweeps) / [Sweep decisions](/guides/sweep-decisions) — the agent-model evidence - [Judges](/guides/judges) — deployments, sampling modes and scorer models - [Annotations & review](/guides/annotations-and-review) — the gold labels that give the judge tab its recommendation - [Pre-prod evaluations](/guides/preprod-evals) / [Eval gates](/guides/eval-gates) — how a switch is tested before it ships ================================================================================ # Cost Optimization Source: /docs/guides/cost-optimization/ ================================================================================ # Cost Optimization **Cost Optimization** answers a running operational question: *what is this agent spending on LLM calls, and how much of that is evaluation?* It splits spend into two halves — what your **agent** spends producing answers, and what your **LLM judges** spend scoring them — over a time window you choose, alongside how much of your traffic actually got scored. It holds your agent and its model fixed: this is "what does the current setup cost", not a model experiment (for that, see the **Sweeps** tab, [Model sweeps](/guides/model-sweeps), and [Model comparison](/guides/model-comparison)). ## At a glance | | | | --- | --- | | **What it is** | Agent LLM cost vs. LLM-judge cost for an agent, plus % of traces scored, over a window | | **Where it lives** | The **Spend** tab on the **Cost & Quality** page | | **Key API routes** | `GET /cost-optimization/summary`, `GET /cost-optimization/judges` | | **Needs** | Ingested traces (for agent cost) and at least one LLM judge that has run (for judge cost). Prices come from your [model price table](/guides/cost-and-model-pricing) | | **Scope** | The agent you're viewing; anyone with read access can see it | | **Default window** | The last **7 days**; change it with the time-range picker at the top right | Every dollar figure here is computed the same way as everywhere else in Neens — token counts multiplied by the price for that specific model. A model with **no price** is shown as unpriced (**—**), never as `$0`. See [Cost & model pricing](/guides/cost-and-model-pricing). **Cost Optimization moved twice.** It used to be its own page in the left nav, then the **Cost** tab on Model Bench. It is now the **Spend** tab on [**Cost & Quality**](/guides/cost-and-quality), alongside **Overview**, **Agent model** and **Judge model**. An old `/cost-optimization` link still works: it opens that tab. Read [Cost & Quality](/guides/cost-and-quality) first — it covers the whole page, including the **Absolute $ | Per 1,000 traces** unit toggle and the three ways judge cost is reported. This page is the deeper reference for the spend breakdowns and for adding cheaper scorer LLMs. **Looking for the "switch & save" recommendation?** It now lives on the **Judge model** tab, which leads with a [**cost–quality frontier**](/guides/cost-quality-frontier) — the chart that plots each scorer model by cost against its agreement with your experts and names the cheapest safe switch. That view needs [human gold labels](/guides/annotations-and-review) to have a quality axis; the breakdowns on *this* page work with cost alone. ## What the page shows ### The KPI row Four tiles summarize the window: | Tile | What it is | | --- | --- | | **Agent cost** | What your agent's own model calls cost — summed from the token counts on your ingested traces. | | **LLM-judge cost** | What your LLM judges spent scoring those traces — the tokens the *scorer* model used, separate from the agent. | | **Total cost** | Agent + judge, for the window. If one side is unpriced it's excluded and the total is marked partial (see below). | | **Traces scored** | The share of traces in the window that carry at least one score, shown as a percentage with the underlying `{scored} of {total} traces`. | ### The breakdowns - **Cost by judge** — one row per judge and the scorer model it ran on, with the judge's cost, the share of traces it scored, its number of scores, and the distribution of its verdicts. Deterministic (classifier) and external-API judges appear with **no scorer cost** — they don't call an LLM to score, so there's nothing to charge. That blank is honest, not missing data. - **Agent cost by model** and **LLM-judge cost by model** — the same spend split by the individual models involved, with input/output token counts. This is where you see, for example, that most of your judge spend is on one scorer model and you could move some judges to a cheaper one. ## The judge model is not the agent model A score carries **two** independent models, and Cost Optimization keeps them apart: - the **agent model** — the one that produced the answer being graded (this drives **Agent cost**), and - the **scorer model** — the one the judge used to *do* the grading (this drives **LLM-judge cost**). They're tracked separately on purpose. Switching a judge to a cheaper scorer model changes your **LLM-judge cost** without touching your agent or its answers at all — which is exactly the lever this page exists to surface. ## Reading unpriced and partial results Neens never invents a price. When a model isn't in your [price table](/guides/cost-and-model-pricing), its spend shows as **—** and the affected total is flagged **partial** rather than silently under-counted. - A single **—** in a by-model or by-judge row means *that* model has no price yet. - A **partial** total means at least one contributing model is unpriced, so the number you see is a floor — real spend is that much *plus* whatever the unpriced models cost. To turn a **—** into a number, add a rate for that model in **Settings → Model pricing**. The next time the page loads, that model's spend is priced and folds into the totals. Self-hosted / open-weight models are a legitimately priced `$0` (you run them yourself) — that's a real zero, distinct from an unpriced **—**. The full model of where prices come from, including your own negotiated overrides, is in [Cost & model pricing](/guides/cost-and-model-pricing). **Low "Traces scored"?** That's a coverage signal, not an error. It just means many traces in the window have no score yet — either no judge is deployed to score them, or scoring hasn't caught up. See [Continuous evaluation](/guides/continuous-evaluation) to score traces automatically as they arrive. ## Add more scorer LLMs out of the box The cheapest way to lower **LLM-judge cost** is often to run judges on a cheaper scorer model. Neens speaks the OpenAI-compatible API, so you can point judges at aggregators and proxies that expose hundreds of models behind one endpoint — **OpenRouter** and a **LiteLLM** proxy are both first-class API formats. Add one as an [LLM connection](/administration/llm-connections), then select it on a judge. ### Open Settings → LLM providers Click **Add connection** and choose the **openrouter** API format. The base URL is filled in for you (`https://openrouter.ai/api/v1`) — you don't need to type it. ### Paste your key Paste your OpenRouter API key into the credential field. It's write-only: encrypted at rest and never shown again. ### Load models and pick one Click **Load models**. Neens asks OpenRouter for its catalogue and turns the model field into a searchable list — pick, say, `openai/gpt-4o-mini` or `google/gemini-flash-1.5`. If the list can't be fetched you can still type the model id by hand; listing is a convenience, not a requirement. ### Test and save Click **Test** to confirm the connection reaches OpenRouter, choose its visibility (agent, org, or company), and save. A [LiteLLM](https://docs.litellm.ai/) proxy gives you one OpenAI-compatible endpoint in front of many providers, with your own keys and routing. ### Open Settings → LLM providers Click **Add connection** and choose the **litellm** API format. ### Enter your proxy base URL Point it at your running proxy, e.g. `https://litellm.mycompany.com` (whatever host you deployed it on). Unlike OpenRouter there's no default — it's your proxy, so you supply the address. ### Add the proxy key, then Load models Paste the proxy's master/virtual key if it requires one, then click **Load models** to pull the models your proxy is configured to serve and pick one. As always you can type the model id directly if you prefer. ### Test and save **Test** confirms reachability, then choose visibility and save. If your LiteLLM proxy runs on a **private, internal, or `localhost`** address, an administrator has to allowlist that host before Neens will call it — the same egress guard that protects every outbound connection. A public proxy URL needs no special handling. See [LLM connections](/administration/llm-connections). ### Point a judge at the new scorer, then compare Once the connection exists, open a [judge](/guides/judges), assign it the new connection, and let it score. Come back to Cost Optimization and the **Cost by judge** breakdown now shows that judge on the new scorer model — with its own cost line, so you can read the before/after directly. **Worked example.** Say your **Helpfulness** judge runs on a premium scorer and shows up in **Cost by judge** at a few dollars for the week, scoring 70% of your traces. You add an OpenRouter connection (steps above), pick `openai/gpt-4o-mini`, and reassign Helpfulness to it. Next week its row shows the cheaper scorer model and a much smaller **Cost** — while **Traces scored** for that judge stays around 70%, because you changed *what* does the grading, not *how much* grading happens. If `gpt-4o-mini` isn't in your price table yet, its cost shows as **—** (partial) until you add a rate in **Settings → Model pricing** — after which the savings are quantified, not just implied. ## Compare scorer models side by side Reassigning a judge to a cheaper scorer (above) tells you the new **cost**, but not whether the cheaper model *agrees* with the one you trust. A **scorer comparison** answers both at once: it holds your agent and your judge's prompt fixed and varies only the **scorer LLM**, scoring the same window of traces with two or more candidate models and laying their results out **side by side** — distributions, cost, and coverage in one view. There's no reference "correct" judge and no automatic winner: you read the columns and decide. This is the natural follow-on to the breakdowns above. **Cost Optimization** tells you what your current scorer costs; a **scorer comparison** tells you what a *different* scorer would score and cost on the very same traces — the evidence you need before you switch. ### What it is Pick one **judge** (its rubric and prompt are frozen — every candidate grades with the identical instructions), a **time window** of traces, and **two or more scorer LLM connections** to put head-to-head. Neens scores each trace with each candidate and shows you, per candidate: - its **score distribution** — the same verdict breakdown you see on the Scores page (pass/fail for a numeric judge, or the category split for a labelled one); - its **cost** for the window — priced exactly like everywhere else, and shown as **—** when the model has no price yet (never `$0`); - its **coverage** — how many of the window's traces it scored, as a percentage; - shared with all candidates: the **shared-targets** count — the number of traces *every* candidate scored, which is the honest apples-to-apples base for reading the columns against each other. ### How to create one ### Open the Comparison tab From **Cost & Quality → Spend**, follow **Compare scorer models →**, or open the **Comparison** tab directly on the [Judges](/guides/judges) page. Press **New comparison** in the top right to open the launch dialog, then fill it in with the steps below. ### Pick the judge Choose the judge whose scoring you want to shop around. Its prompt and rubric are locked for the run, so the only thing changing between columns is the model doing the grading. ### Choose the window Set the time range of traces to score — the same time-range picker used across Neens. A recent, representative window (say the last 7 days) is usually enough to see the models diverge. ### Select two or more scorer LLMs Pick the candidate scorer connections to compare — for example your current premium scorer against an [OpenRouter](#add-more-scorer-llms-out-of-the-box) `gpt-4o-mini` or a cheaper open-weight model. Add an [LLM connection](/administration/llm-connections) first if the one you want isn't listed. ### Launch and read the columns Start the run. It fills in as it goes; when it finishes, read the side-by-side distributions, costs, and coverage against the shared-targets base to decide whether a cheaper model grades closely enough to switch. **Already-scored traces aren't re-run.** If a candidate model has already scored some of the traces in the window under this judge, Neens reuses those existing scores instead of paying to grade them again, and only backfills the traces it hasn't seen. A comparison that includes the scorer you're already running therefore costs little or nothing extra for that column. ### Reading the result — a worked example Say your **Response Risk** judge (it labels each answer `none` / `low` / `medium` / `high` risk) currently runs on a premium scorer, and you want to know whether `gpt-4o-mini` could take over. You compare the two over the last week's traces. The result: | | Premium scorer | gpt-4o-mini | | --- | --- | --- | | **Distribution** | none 14 · low 10 · medium 10 · high 6 | none 12 · low 13 · medium 8 · high 7 | | **Cost (week)** | **$0.04** | **—** (unpriced) | | **Coverage** | 100% | 100% | | **Shared targets** | 40 | 40 | Both models scored all 40 traces in the window, so **shared targets** is 40 — the whole set is common ground and the two columns are directly comparable. Read it like this: - **Do the distributions agree?** Over the 40 shared traces the two models land on nearly the same risk breakdown — a handful of traces shift by one level, none swing from `none` to `high`. For a triage signal that's close enough to trust the cheaper model. - **What do you save?** The premium column shows a real dollar figure; the `gpt-4o-mini` column shows **—** because that model isn't in your price table yet. Add a rate in **Settings → Model pricing** and re-open the comparison to turn that **—** into a number and quantify the saving. - **Is the comparison fair?** Both models scored all 40 traces, so **shared targets** is 40 and you are comparing like with like. If one model had only covered 30, you'd read the columns knowing the distributions rest on different populations. If the cheaper model's distribution tracks the incumbent's and its cost is lower, switch that judge to it (see [Point a judge at the new scorer](#add-more-scorer-llms-out-of-the-box) above) and the saving shows up on your **LLM-judge cost** the next time the page loads. If the distributions diverge in a way that matters, you've caught it *before* changing what grades your production traffic. ## How it works - **Agent cost** is summed from the input/output token counts already on your ingested traces, grouped by the model each call used, then priced. Nothing extra is recorded — it's derived at read time from data you already send. - **LLM-judge cost** comes from each score a judge writes: Neens records the scorer model and the tokens it spent, so judge spend can be summed per judge and per scorer model and priced the same way. Classifier and external-API judges record no scorer tokens — they don't call an LLM — so they carry no judge cost. - **Traces scored** counts distinct traces in the window that have at least one score, over the total number of traces in the window. - Changing the window recomputes everything for that range; there's nothing to refresh or rebuild. ## Troubleshooting | Symptom | Likely cause | Fix | | --- | --- | --- | | **LLM-judge cost is empty** | No LLM judge has scored traces in this window | Deploy an `llm_prompt` [judge](/guides/judges) and let it run; deterministic/external-API judges never produce judge cost | | A judge or model shows **—** | That model isn't in your price table | Add a rate in **Settings → Model pricing** ([Cost & model pricing](/guides/cost-and-model-pricing)) | | **Total cost** says *partial* | At least one contributing model is unpriced | Price the unpriced models; the total then includes them | | **Traces scored** is low | Few traces have scores yet | Turn on [continuous evaluation](/guides/continuous-evaluation) or run judges over the window | | **Load models** returns nothing | The provider didn't return a catalogue (or the key/base URL is wrong) | Type the model id by hand, and re-check the key and base URL with **Test** | ## Related - [Cost & model pricing](/guides/cost-and-model-pricing) — where prices come from and how to set your own - [LLM connections](/administration/llm-connections) — every API format and how credentials are handled - [Judges](/guides/judges) and [Continuous evaluation](/guides/continuous-evaluation) — what produces judge cost; the **Comparison** tab on that page is the scorer comparison documented above - [Cost & Quality](/guides/cost-and-quality) — the page this tab lives on: recommended moves, the unit toggle, and both proof tabs - [Cost–quality frontier](/guides/cost-quality-frontier) — the statistics behind "clears the bar", and where the "switch & save" recommendation comes from - [Model sweeps](/guides/model-sweeps) (the **Agent model** tab, next door on **Cost & Quality**) / [Model comparison](/guides/model-comparison) — when you *do* want to change the agent's model ================================================================================ # Cost–quality frontier Source: /docs/guides/cost-quality-frontier/ ================================================================================ # Cost–quality frontier The proof tabs on [**Cost & Quality**](/guides/cost-and-quality) rest on one picture: a **cost–quality frontier**. Every candidate model is a dot — the cheaper it is, the further left; the better it scores, the higher up. The line through the best dots is the *efficient frontier*: the most quality you can buy at each price. It answers the question underneath both proof tabs — *are we overpaying for quality, and what is the single move that saves money without moving the quality needle?* The same maths drives two tabs, pointed at two different models: - The **Judge model** tab ranks your **LLM judges' scorer models** — quality is how well each scorer agrees with your human ground truth. See [Cost & Quality](/guides/cost-and-quality) and [Cost Optimization](/guides/cost-optimization). - The **Agent model** tab ranks candidate **agent models** — quality is a stable pass rate on a frozen golden suite. See [Model sweeps](/guides/model-sweeps) and [Sweep decisions](/guides/sweep-decisions). Both read the *same* frontier, so **"clears the bar"**, **"recommended"**, and **"no measurable quality loss"** mean exactly one thing across the two tabs. ## At a glance | | | | --- | --- | | **Where** | The **Judge model** and **Agent model** tabs on the **Cost & Quality** page | | **Key API routes** | `GET /cost-optimization/frontier` (judge scorer models), `GET /model-sweeps/{id}/frontier` (agent models) | | **Both need a quality axis** | The Cost frontier needs **human gold labels** (see below); the Sweeps frontier needs a finished [sweep](/guides/model-sweeps) with judges | | **Reads only** | Nothing is stored and nothing is spent — every number is computed on read. Neens never switches a model for you | | **Scope** | The agent you're viewing; anyone with read access can see it | **A frontier is a recommendation, not an action.** Neens names the cheapest model that clears your bar without a measurable quality loss; it never switches a scorer, promotes an agent model, or changes a deployment. You make the move — and the honest way to make it is behind a [pre-prod eval](/guides/preprod-evals) so it can't regress in production. ## How to read the frontier The axes are always the same shape — **cost on the X-axis (right = more expensive), quality on the Y-axis (up = better)** — and every dot falls into one role. | On the chart | What it means | | --- | --- | | **Your model now** | The one you're running today — the incumbent every recommendation is measured against. | | **On frontier** | Nothing cheaper is at least as good. You're getting the best quality available at that price — there's no free saving here. | | **Recommended** | The single best move: the cheapest model that is strictly cheaper than your current one, clears the acceptance bar, and is **not measurably worse** (see the noise band below). | | **Overpaying** | A model *off* the frontier: something cheaper is at least as good, so the extra spend buys you nothing. | | **Quality risk / below the bar** | Below the acceptance bar — cheap, but not good enough to consider. | | **Quality unknown** | Its cost is known but there's no quality signal for it yet, so it can't be placed on the frontier. It's shown, never guessed at. | ### The acceptance bar The **acceptance bar** is the minimum quality a model must clear to be a candidate at all — a horizontal line on the chart. A model below it is a **quality risk** no matter how cheap. - **Cost tab (judges):** the bar is **agreement with your experts**, default **80%**. Override it per view with the bar control (or `?bar=` on the API, a `0`–`1` rate). - **Sweeps tab (agent models):** the bar is the sweep's **pass-rate gate** — the one you committed to when you launched it, defaulting to the **90%** deployment success threshold. See [Set the quality bar](/guides/sweep-decisions#set-the-quality-bar). ### The noise band — why "cheaper" needs a confidence interval Quality is measured on a sample, so it is never a single exact number — it's a point estimate with a **confidence interval** around it (a 95% **Wilson** interval, the same statistic the [sweep comparison](/guides/sweep-decisions#sample-size-and-confidence-interval) uses). That interval is the **noise band**. This is what makes an honest "no quality loss" claim possible: - If the cheaper model's interval **overlaps** your current model's, the two are **not statistically distinguishable** at this sample size — switching is a **no measurable quality loss** move. That is the honest version of *"cheaper, without compromising quality."* - If the intervals **don't overlap**, Neens shows the **real, signed quality delta** instead of hiding it — the cheaper model either measurably **improves** or measurably **regresses**. - A model that is *distinguishably worse* is **never recommended**, however much it saves. **Unpriced is never `$0`, and never wins "cheapest."** If a model has no rate in your [price table](/guides/cost-and-model-pricing), Neens doesn't know its cost — it's shown as unpriced (**—**), excluded from the frontier line, and can never be named the cheapest move. A self-hosted model you run yourself is a legitimately priced `$0`, which is a different thing entirely. Add a rate under **Settings → Model pricing** to bring an unpriced model onto the chart. ## Cost tab: the judge frontier On the **Cost** tab, each dot is a **scorer model** one of your [judges](/guides/judges) can run on, and quality is that scorer's **agreement with your experts**. The chart answers: *can this judge do its grading on a cheaper model without disagreeing with my humans any more than it already does?* There is **one independent frontier per judge** — you can only swap a judge's scorer model within that judge, so a recommendation never proposes moving one judge's grading onto a different judge's model. ### You need gold labels first **Without human gold labels there is no quality axis, so there is no frontier.** Judge quality here *is* judge↔expert agreement, and that only exists once experts have labelled sessions. A judge with no gold-backed labels shows **quality unknown** for every scorer model and proposes no switch — an honest empty state, never a fabricated agreement number. To give the Cost frontier a quality axis, produce expert truth: ### Label sessions in the Review queue Open the **Review** queue and record **pass**/**fail** verdicts on the sessions it prioritizes. See [Annotations & review](/guides/annotations-and-review). ### Make the decisive ones gold Mark authoritative labels **gold**, or have a **principal** (expert) reviewer label them — gold and principal labels are the expert truth the frontier measures against. ### Let alignment compute Judge↔expert **alignment** recomputes nightly (and on demand). Once a judge has gold-backed alignment on **two or more** scorer models, its frontier can propose a switch between them. ### Find and apply a "switch & save" ### Open the Judge model tab **Cost & Quality → Judge model.** Candidate scorer models are ranked against the bar; the **Recommended moves** list on the **Overview** tab ranks the available moves across every judge by monthly saving, each showing the quality delta against that judge's noise band. ### Read the recommended move The recommendation names the cheapest scorer model that is strictly cheaper than the judge's current one, clears the agreement bar, and sits inside the current model's noise band. It comes with the **projected monthly saving** (the window's spend scaled to 30 days) and the **quality delta** — with a **no measurable quality loss** claim when the intervals overlap, or the real signed delta when they don't. ### Corroborate before you trust it Read the point's failure-class metrics — **precision** and **recall** — next to the headline agreement (see the [honest limitation](#an-honest-limitation-you-should-know) below). A high agreement on a stream with few real failures deserves a second look. ### Make the switch, then verify it Neens doesn't flip the scorer for you. Reassign the judge to the cheaper scorer model yourself (**Cost Optimization → [Point a judge at the new scorer](/guides/cost-optimization#add-more-scorer-llms-out-of-the-box)**), and gate the change behind a [pre-prod evaluation](/guides/preprod-evals) so it can't quietly regress your scoring in production. The saving shows up on your **LLM-judge cost** the next time the page loads. ### An honest limitation you should know **The two scorer models are compared over the sessions each happened to score, not a shared holdout.** Neens keeps one score per session, so a judge's scorer models are each measured over *their own* gold-labelled sessions — the comparison is confounded by *which* sessions each one scored. This is surfaced, not hidden: each point carries its own **gold count** (a thin point is visible), and a recommendation is only ever "cheaper *and* within the noise band." Treat a recommended switch as a **strong hint, corroborated by precision/recall**, not a proof — and confirm it with a pre-prod eval before you rely on it. The **current** scorer model shown is a heuristic: the busiest one in the window. ## Sweeps tab: the agent-model frontier On the **Sweeps** tab, each dot is a **[model sweep](/guides/model-sweeps) arm** — a candidate model for your *agent* — and quality is the arm's **stable-pass rate** (the composite judge score over the frozen eval set). The chart answers: *which cheaper model still clears the bar for this agent?* Because a sweep already measures quality with a Wilson interval, cost per case, and a pass-rate gate, the frontier **reuses those numbers directly** — it recomputes nothing. Read it exactly like the judge frontier: - The **recommended** arm is the cheapest one that beats your **baseline** (the declared incumbent — the arm you listed first, overridable), clears the bar, and is inside the baseline's noise band. - Because a sweep is an experiment, not production traffic, the recommendation reports a **per-case saving** and, honestly, **no monthly figure** — inventing production volume for an experiment would be a lie. - An arm with **no data** (its endpoint was unreachable) is shown as such, never as 0%. For the full leaderboard, per-agent verdicts, regression drill-down, and the shareable decision, use the comparison view — see [Sweep decisions](/guides/sweep-decisions). **A void sweep draws no frontier.** If a sweep's arms didn't run under [identical conditions](/guides/model-sweeps#identical-conditions-and-what-void-means), it is **void**: the arms' own numbers are still shown as facts, but the frontier is empty, no arm is classified, and no switch is proposed. A comparison of arms that measured different things is the most convincing wrong answer this feature could produce, so Neens declines it rather than draw a chart that would be screenshotted into a decision doc. Fix what drifted and run a fresh sweep. ## How it works - **Cost frontier** (`GET /cost-optimization/frontier`) joins the per-scorer cost from [Cost Optimization](/guides/cost-optimization) with judge↔expert agreement from [alignment](/guides/annotations-and-review#judge--human-alignment), one frontier per judge, over a window (`?window=7d` by default, or `?start=&end=`). Agreement carries a Wilson confidence interval — the noise band. - **Sweeps frontier** (`GET /model-sweeps/{id}/frontier`) is derived entirely from the sweep's own [comparison](/guides/sweep-decisions) — same reconcile, same bar, same baseline — and recomputes no quality. - **The recommended move** is, in both cases, the cheapest priced model that is strictly cheaper than your current one, clears the acceptance bar, and whose quality interval overlaps the current one's (or is better). Overlapping intervals ⇒ *no measurable loss*; a measurable gap ⇒ the real delta; a distinguishably worse model ⇒ no recommendation. - **Everything is computed on read.** Changing the window or the bar recomputes the frontier; there's nothing to refresh or rebuild, and nothing is spent. ## Troubleshooting | Symptom | Cause | Fix | | --- | --- | --- | | The Cost frontier is empty / every dot says **quality unknown** | No gold/expert labels for these judges yet | Label sessions and mark them gold in the [Review queue](/guides/annotations-and-review), then let alignment compute | | A judge shows a single dot and no switch | It has only run on one scorer model, or only one has gold-backed quality | Score the same judge on a second scorer, over sessions that have expert labels | | A model shows **—** in cost and never wins | Its model has no price in your catalogue | Add a rate under **Settings → [Model pricing](/guides/cost-and-model-pricing)**; it is *unknown*, not free | | The recommendation says "no measurable quality loss" but the delta isn't zero | The quality intervals overlap, so the sample can't distinguish the two | That *is* the honest claim. For more certainty, gather more gold labels (Cost) or golden items (Sweeps) | | The Sweeps frontier is blank with a void banner | The sweep is [void](/guides/model-sweeps#identical-conditions-and-what-void-means) | Fix what drifted mid-sweep and run a fresh one; a void sweep is intentionally not comparable | | You applied a recommended switch and quality dropped | A recommendation is a strong hint, not a proof — especially on the Cost tab's disjoint-sample join | Always gate a switch behind a [pre-prod eval](/guides/preprod-evals); revert if it regresses | ## Related - [Cost Optimization](/guides/cost-optimization) — the Cost tab: what your agent and judges spend, and how to add a cheaper scorer - [Model sweeps](/guides/model-sweeps) / [Sweep decisions](/guides/sweep-decisions) — the Sweeps tab: running the comparison and reading the verdict - [Annotations & review](/guides/annotations-and-review) — how to produce the gold labels the Cost frontier needs - [Cost & model pricing](/guides/cost-and-model-pricing) — where a price comes from, and how unpriced models are handled - [Pre-prod evaluations](/guides/preprod-evals) — how to ship a model switch behind a gate that blocks a real regression ================================================================================ # Model comparison Source: /docs/guides/model-comparison/ ================================================================================ # Model comparison Swapping an agent onto a cheaper model is easy to justify on cost and hard to justify on quality. Neens closes that gap: every score records **which model produced the answer it graded** and **how that was known**, so `eval_pass_rate` can be broken down per model — and per agent × model — from the same judges you already run in production. The other half of the argument is on [Cost & model pricing](/guides/cost-and-model-pricing): that page tells you what each model costs, this one tells you whether it still hits the bar. ## At a glance | | | |---|---| | **What it is** | Model + agent attribution recorded on every score, so the quality measures slice by model | | **Where it lives** | The **Quality** tab of the widget gallery on any [dashboard](/guides/dashboards) — presets **Pass rate by model**, **Pass rate by agent × model**, **Scores by model attribution** | | **Measures it applies to** | `eval_pass_rate`, `avg_score`, `score_count` (see the [metrics catalogue](/guides/metrics)) | | **New dimensions** | **Model**, **Agent**, and **Model attribution** at score grain | | **Key API routes** | `GET /measures/catalogue`, `POST /dashboards/{id}/widgets`, `GET /dashboards/{id}/widgets/{wid}/data` | | **Needs** | Traces that record the model on their LLM spans, and at least one [judge](/guides/judges) producing scores. No LLM call, no configuration. | **Attribution is recorded, not inferred at read time.** The model is stamped onto the score row when the score is written. That is what makes a comparison reproducible: a pre-prod run's model is frozen when the run is created, so *"Haiku passed at 94% in June"* keeps meaning the same thing after somebody edits the connection in July. ## Answer: "does the cheaper model still hit our bar?" ### Open a dashboard and add the preset On the **Dashboards** page, open (or create) a board scoped to the agent you're comparing, click **Add widget**, and pick the **Quality** tab. Drop in **Pass rate by model** — one click adds it fully configured (`eval_pass_rate` grouped by **Model**). ### Set the window The board's time picker drives the tile (**Today · Last 24h · 7 days · 30 days · All time · Custom**). Pick a window that covers *both* models — a comparison over a window where the new model only ran for the last two days is a comparison of sample sizes, not of models. ### Read the bars Each bar is one model id, and its value is the share of scores in that window that met **their own threshold**. Two bars you may not have expected are drawn with a **hatched** fill, and a legend under the chart says why: - **Mixed** — traces whose LLM spans named more than one model. - **Unknown** — scores with no attributable model. Neither is folded into a real model's bar, and neither is dropped — dropping them would raise the apparent pass rate of the models that remain. ### Check the sample behind each bar Add a second tile from the **Custom** tab — measure **Total scores**, dimension **Model** — or drop in the **Scores by model attribution** preset. A 100% pass rate over eleven scores is not evidence. ### Decide with the cost half Put a **Spend by model** tile (from the **Cost** tab) on the same board. The claim you can now make is a two-tile claim: *this model costs X and passes at Y*. See [Cost & model pricing](/guides/cost-and-model-pricing). **Narrow to one metric before you quote a number.** `eval_pass_rate` grouped by model averages across every metric that scored in the window — a helpfulness judge and a safety judge are not the same bar. Slice or filter by **Metric** (`metric_key`) so the two models are compared on the same question. The same applies to **Score source**: pre-prod runs write scores too, so a production comparison should filter to the sources you mean. ## Where a score's model comes from Every score carries a **Model attribution** value naming the evidence used. It is a closed vocabulary, listed here in precedence order — the first one that resolves wins: | Attribution | Shown as | The model is | When | |---|---|---|---| | `preprod_run` | **Frozen on the run** | the model **frozen on the pre-prod run** when it was created | The score came from a [pre-prod evaluation](/guides/preprod-evals) whose agent connection declared a model. The strongest evidence there is: it records what the candidate was configured to run, not what a trace happened to look like. | | `span` | **From the scored span** | that span's own model | The score graded a single span rather than a whole trace. | | `session_uniform` | **Uniform across the trace** | that one model | Every model-bearing span in the graded trace named the **same** model. | | `mixed` | **Mixed** | the **Mixed** bucket | The graded trace's spans named **two or more distinct** models. | | `unknown` | **Unknown** | nothing | No model could be derived: the trace recorded none, or the score predates model attribution. | The left column is the value stored on the score and used in API filters; the middle column is how it's worded on a chart. Only spans that actually name a model vote. A tool call or a retrieval step carries no model, so it never supplies one and never makes a trace look "mixed". **`preprod_run` beats the trace.** If a pre-prod run froze a model, its scores attribute to that model even when the captured trace's spans say something else — the run row is the record of what you deliberately tested. A pre-prod run with no declared model falls through to the captured trace's own spans; it is never given an invented one. ## Reading Mixed and Unknown — and what to do about them These are the two buckets people want to make disappear. Both are answerable. ### Mixed **Mixed** means the graded trace ran on more than one model — a supervisor on one model with workers on another is the common shape. Neens does not pick a winner between them (see [why](#why-neens-refuses-to-pick-a-dominant-model)), so the trace gets its own bucket and the score row keeps the list of distinct models it saw. What to do: - **If it's expected** (a genuine multi-model pipeline), compare at a level where the answer is single-model: score **spans**, or split the pipeline into separate agents and use the [agent × model cross-tab](#worked-example-agent--model) below. - **If it's not expected**, it usually means a fallback or retry silently switched models. The [Agent Map](/guides/agent-map) keys its LLM nodes by model, so a trace that used two shows two nodes — that's the fastest way to see which step defected. ### Unknown **Unknown** means no model was derivable. There are two causes and they have different fixes: | Cause | Fix | |---|---| | **The traces don't record a model.** Your instrumentation never set `gen_ai.response.model` / `gen_ai.request.model` / `llm.model_name` on the LLM spans. | Fix the instrumentation — see [Which attributes Neens reads](/guides/send-traces#which-attributes-neens-reads). This also fixes cost, which reads the same field. | | **The scores predate model attribution.** Scores written before this feature existed were stamped `unknown` and deliberately **not** guessed at. | These stay **Unknown** — they are never re-derived retroactively. Exclude them from comparisons; scores written from now on are attributed automatically as they land. | Until then, exclude them: a pass rate computed over rows whose model you don't know is not a statement about any model. ### Filter to the attributions you trust **Model attribution** exists so you can do exactly that. Add the **Scores by model attribution** preset first — it answers "how much of this comparison rests on evidence I trust?" — then filter. **Filtering `model` to `"Unknown"` or `"Mixed"` works.** Neens normalizes the `model` dimension the same way for the chart *and* for the filter, so a filter selects exactly the rows the bar is drawn from — clicking through a bucket round-trips. Use the labels as shown (`Unknown`, `Mixed`), not the stored values: the raw `__mixed__` sentinel never survives normalization and matches nothing. Filter `model_source` when you want to narrow by *how* the attribution was established (`preprod_run` / `span` / `session_uniform` / `mixed` / `unknown`) rather than by which model ran. The widget builder's **Custom** tab groups by one dimension and does not author filters, so a filtered tile is created through the API — the same call the gallery makes, with a `filters` object: ```bash curl -X POST https://your-neens-host/api/dashboards/dsh_your_dashboard/widgets \ -H "Authorization: Bearer nk_sess_your_session_token" \ -H "Content-Type: application/json" \ -d '{ "type": "bar", "measure": "eval_pass_rate", "dimensions": ["model"], "filters": { "model_source": ["preprod_run", "span", "session_uniform"], "metric_key": "primary_score", "score_source": "llm_judge" }, "range": "30d" }' ``` That widget reads: *pass rate on the Primary Score metric, per model, over the last 30 days, counting only scores whose model attribution is a single known model.* Filter values are always bound as parameters, and an invalid measure × dimension combination is rejected with `422` — a widget can never query outside the [catalogue](/guides/metrics). Fetch its rows with `GET /dashboards/{id}/widgets/{wid}/data`: ```json { "rows": [ { "model": "claude-haiku-4-5", "eval_pass_rate": 0.94 }, { "model": "claude-sonnet-4-5", "eval_pass_rate": 0.96 } ] } ``` Filter values for **Model attribution** `model_source` accepts a single value or a list. The values are the stored vocabulary strings — `preprod_run`, `span`, `session_uniform`, `mixed`, `unknown` — not the wording the chart shows (**Frozen on the run**, **From the scored span**, **Uniform across the trace**, **Mixed**, **Unknown**). The **Mixed** bucket can also be selected on the `model` dimension itself, using the label `Mixed` — *not* the raw `__mixed__` sentinel, which is normalized away before the filter is applied and therefore matches nothing. ## Worked example: agent × model The per-agent half of the comparison is a **cross-tab** — agents down the side, models across the top — so you can see that a cheaper model is fine for one agent and not for another. Any widget with two dimensions renders this way, whichever visualization it asks for: a bar chart keyed on the agent alone would show the same agent several times with conflicting values. Add the **Pass rate by agent × model** preset from the **Quality** tab, or build it from the API: ```bash curl -X POST https://your-neens-host/api/dashboards/dsh_your_dashboard/widgets \ -H "Authorization: Bearer nk_sess_your_session_token" \ -H "Content-Type: application/json" \ -d '{ "type": "table", "measure": "eval_pass_rate", "dimensions": ["agent", "model"], "filters": { "metric_key": "primary_score" }, "range": "30d" }' ``` The rows come back one per pair: ```json { "rows": [ { "agent": "support-triage", "model": "claude-haiku-4-5", "eval_pass_rate": 0.95 }, { "agent": "support-triage", "model": "claude-sonnet-4-5", "eval_pass_rate": 0.96 }, { "agent": "refund-resolver", "model": "claude-haiku-4-5", "eval_pass_rate": 0.71 }, { "agent": "refund-resolver", "model": "claude-sonnet-4-5", "eval_pass_rate": 0.93 }, { "agent": "refund-resolver", "model": "Mixed", "eval_pass_rate": 0.80 } ] } ``` …and render as one cell per pair: | | claude-haiku-4-5 | claude-sonnet-4-5 | Mixed | |---|---|---|---| | **support-triage** | 95% | 96% | — | | **refund-resolver** | 71% | 93% | 80% | Read that as a decision, not a score: `support-triage` loses a point of pass rate on the cheaper model and can move; `refund-resolver` loses twenty-two and cannot. That is the rollout plan the aggregate "94% vs 96%" would have hidden — and it is the reason agent and model are recorded on the same row rather than left on two different tables. An empty cell renders as an **em dash**: that pair has no scores in the window — it is **not** a zero, and it is not a failure. An agent value of **Unknown** means the score's target has no resolvable trace (so no agent name), the same explicit bucket the model side uses. ## Pre-prod runs record the model they ran on A [pre-prod evaluation](/guides/preprod-evals) is the cleanest model comparison available, because you control both sides: replay the same golden dataset against the same code on two connections and the only difference is the model. When a run is created, Neens copies the model and provider off the run's agent connection and **freezes them on the run**. The run header carries a **Model** badge showing it — or an explicit *Model not recorded* when there is nothing honest to show. The run detail (`GET /preprod-evals/{id}`) carries the same values: ```json { "id": "ppr_…", "versionLabel": "pr-482", "runnerMode": "push", "model": "claude-haiku-4-5", "modelProvider": "agent_http", "status": "completed" } ``` Two honest details: - **`model` is `null` for a run with no declared model** — most commonly a run you drive yourself, where your own harness serves the replay and Neens genuinely does not know what served it. Those scores fall back to the captured trace's own spans, and are **Unknown** if the trace recorded no model either. - **`modelProvider` is the connection's provider**, i.e. the transport Neens used. When Neens calls your agent at an HTTP endpoint that is `agent_http`, not an LLM vendor — the vendor is not knowable from an HTTP endpoint, and recording what was actually used beats recording a guess. ## Why Neens refuses to pick a dominant model A trace that used two models has no single correct model, and the tempting fix — attribute it to whichever model produced the most tokens — is the bug this feature exists to prevent. Under token-weighted attribution, a supervisor on an expensive model coordinating workers on a cheap one becomes "a cheap-model trace", and the cheap model inherits a pass rate it did not earn. That number then survives every review, because nothing about it looks wrong. So the rules are fixed: - **Two or more models is its own bucket.** *Mixed* is a real, chartable value, not a rounding step. - **Unknown is visible, never blank.** A score with no derivable model says **Unknown** and is excluded from a model comparison rather than quietly merged into one — which would inflate whichever model it landed next to. - **Only model-bearing spans vote.** A tool span is not evidence of a model. - **Nothing is re-derived at read time.** The attribution and the evidence behind it are recorded together, so two queries over the same window can never disagree about which model produced what. This is the same posture Neens takes with [unpriced models](/guides/cost-and-model-pricing#unpriced-is-a-real-state-not-a-bug): a visibly missing number beats a confidently wrong one, because you can act on the first and you will plan against the second. **Do not quote a pass rate computed over `unknown` rows.** It is not a claim about a model — it is a claim about a set of scores whose model you don't know. Filter attribution first, then quote. ## Troubleshooting | Symptom | Cause | Fix | |---|---|---| | Every bar is **Unknown** | The scores predate model attribution, or the traces never recorded a model. | Check the **Scores by model attribution** chart. Scores predating attribution stay `unknown` and are never guessed at; if *new* scores are still `unknown`, fix the [model attribute](/guides/send-traces#which-attributes-neens-reads) on your LLM spans. | | A large **Mixed** bucket | The graded traces genuinely used more than one model — often a fallback or a supervisor/worker split. | Expected for multi-model pipelines. If not expected, open a trace's [Agent Map](/guides/agent-map): LLM nodes are keyed by model, so the second model is visible as a second node. | | **Model** isn't offered in the widget builder | The measure isn't score-grain or span-sourced. `model` applies to `eval_pass_rate` / `avg_score` / `score_count` and to `spend_usd` / `tokens_in` / `tokens_out`. | Pick one of those measures — the builder only ever offers valid combinations. | | Pass rates look higher than the Scores page | The window includes `preprod` scores, or several metrics with different thresholds. | Filter `score_source` and `metric_key` — see the [warning above](#answer-does-the-cheaper-model-still-hit-our-bar). | | A model comparison flipped after somebody edited a connection | It shouldn't — a pre-prod run's model is frozen at create time. If a *production* comparison moved, the traces themselves changed models. | Slice by **Time** to find when the model changed, and cross-check with [What changed](/guides/what-changed). | | Filtering `model` to `__mixed__` returns nothing | `__mixed__` is the stored sentinel, and Neens normalizes it to the label **Mixed** before the filter is applied. | Filter `model` to `Mixed` (or `model_source` to `mixed`). The labels **Unknown** and **Mixed** are what the filter accepts, and they match exactly the rows their bars are drawn from. | ## Related - [Cost & model pricing](/guides/cost-and-model-pricing) — what each model costs, and why an unpriced model is reported rather than guessed at. The other half of "cheaper *and* still passing". - [Metrics catalogue](/guides/metrics) — every measure and dimension, including the score-grain slices used here. - [Dashboards](/guides/dashboards) — building and sharing the boards these widgets live on. - [Scores](/guides/scores) — where scores come from and what a threshold means. - [Pre-prod evaluations](/guides/preprod-evals) — the controlled comparison: same dataset, same code, one model changed. - [Sweep decisions](/guides/sweep-decisions) — the same question asked of several candidate models at once, answered with a verdict per agent instead of a chart to interpret. - [Agent Map](/guides/agent-map) — LLM nodes keyed by model, for seeing a mixed trace step by step. ================================================================================ # Insights Source: /docs/guides/insights/ ================================================================================ # Insights Insights are the things Neens notices for you: error-rate and latency anomalies, eval-score regressions, spiking failure modes, novel failure patterns. Detectors run on a schedule, deduplicate what they find, track each signal's lifecycle, and deliver it where you'll see it — the in-app **Insights** page, Slack, an outbound webhook, or a daily digest email. ## At a glance | | | |---|---| | **Where** | The **Insights** page — a **Fleet briefing** at the top plus three feeds (Observe / Diagnose / Fix) | | **Key API routes** | `GET /insights/observe`, `GET /insights/diagnose`, `GET /insights/fix`, `GET /insights/summary`, `POST /insights/summary/briefing`, `GET /insights/summary/briefing/{id}`, `POST /insights/refresh`, `POST /insights/{id}/feedback` | | **When detectors run** | Nightly at 02:30 (server clock), and on demand via the page's **Refresh** action | | **Scope** | Cross-agent, always limited to the orgs/agents you can access | | **Needs** | Nothing for detection or for the briefing's numbers; the *AI-written* briefing and the digest need an [LLM connection](/administration/llm-connections) and an email provider respectively | ## What the detectors watch Each detector compares a fixed **recent 24-hour window** against the 24 hours before it, per agent. The thresholds are deliberately simple and documented: | Detector | Watches | Fires when | |---|---|---| | **Anomaly** | Error rate and p99 latency, per agent + agent name | Recent error rate is at least **2×** the baseline *and* at least **5%** — or recent p99 latency is at least 2× baseline *and* at least **2 s**. Windows with fewer than 5 traces are ignored. | | **Regression** | Eval pass rate, per agent + metric | Pass rate drops by **5 points or more** vs the prior window, with at least 5 scored targets on each side. A drop of 15+ points is high severity. | | **Issue spike** | Classified failure-mode ("issue") volume, per agent + issue | Recent classified sessions reach at least **2×** the baseline, with at least 3 sessions (10+ is high severity). Muted/resolved issues never fire. | Two more insight types are surfaced on demand rather than by the windowed pass: - **Novel failure** — a trace that doesn't match any known failure cluster (a candidate new failure mode), surfaced by the online cluster assigner. - **Pre-prod regression** — a [pre-prod evaluation run](/guides/preprod-evals) where the candidate version failed prompts the baseline passed. ## The three feeds The Insights page mirrors the platform's pillars: - **Observe** — a fleet-health line (traces, error %, p99, spend) plus the open **anomalies**. - **Diagnose** — the top problems, ranked and capped at 20: failure clusters, open issues, and score regressions, each linking to the right drill-down. See [Failure clustering](/guides/clustering). - **Fix** — ranked remediation recommendations drawn from failure clusters (by confidence × sessions affected, capped at 20), linking to the evidence and to [Remediations](/guides/remediations). A **scope picker** narrows all three feeds to all agents, one org, or one agent; the header **time picker** (default: last 24h) sets the window for the fleet stats and issue rollups. Everything is intersected with your access — asking for an agent you can't see just drops it. ### Refreshing Detectors run nightly, and the page's **Refresh** action (`POST /insights/refresh`) runs them immediately over your accessible scope — useful right after ingesting new data. **Refresh** also re-fetches all four sections and asks for a fresh briefing narrative, because new detector output changes the story the briefing is describing. ## The fleet briefing At the top of the Insights page, under the **Fleet briefing** masthead, is a short newspaper-style lede over the current window: a headline, two or three sentences, and four KPI tiles — **Traces**, **Error rate**, **p99 latency**, **Spend**. There are two ways that lede can be written, and the page always shows you which one you're reading: - a **computed** briefing, assembled from the real numbers with no model involved. It is always available, it is never blank, and it is what you see first, every time; - an **AI-written** briefing over the same numbers, produced by your agent's [LLM connection](/administration/llm-connections) in the background and swapped in when it's ready. ### What loads when Nothing on the page waits for anything else. Each of the four sections is on its own clock and paints the moment its own data lands, behind a placeholder shaped like its real content. ### Immediately `GET /insights/summary` returns without calling a model, so the computed briefing and the KPI tiles appear as fast as your numbers can be aggregated. The tiles fill from whichever of the briefing or the **Observe** feed answers first — they carry the same four figures. ### In parallel **Observe**, **Diagnose** and **Fix** stream in independently. A slow or failing feed shows a retryable error in **its own section** and leaves the rest of the page working. ### In the background If an AI briefing is worth writing, the page asks for one and polls for it. While it is being written you get a small **"Neens AI is writing a fuller briefing…"** note next to the by-line and the article dims slightly — never a spinner in place of text, because there is already a complete answer on screen. ### When it lands The AI headline and body replace the computed ones in place and the by-line flips to **Written by Neens AI**. Nothing else on the page moves. Changing the time range or the scope picker cancels everything in flight. A response for the window you just left is discarded rather than painted, so you never see last week's briefing above this week's numbers. ### Telling an AI briefing from a computed one The by-line under the article says so explicitly — different icon *and* different words: | By-line | Icon | Meaning | |---|---|---| | **Written by Neens AI · ``** | sparkle | A model wrote this text, and that is the model that wrote it | | **Written by Neens AI** | sparkle | A model wrote it; the model id wasn't recorded | | **Auto-generated summary** | calculator | Computed from the numbers. No model was involved | Two more qualifiers can appear beside it: **Updated \** (when the text was generated) and **cached** (the text was served from storage rather than composed during this request — including a computed briefing that was stored earlier). ### Why a briefing you'd disagree with is never shown The briefing is stored with a fingerprint of the signals it describes — the fleet KPIs plus the identity of the top anomalies, issues and remediations. On every page load Neens recomputes that fingerprint and compares. If it doesn't match, or the stored text is older than the freshness window (**10 minutes** by default), the AI text is **not** shown. You get the computed briefing instead, and a fresh narrative is requested in the background. This is deliberate. A narrative that says "error rate is holding steady" directly above a tile reading 22% is worse than a plain sentence of arithmetic: it teaches you to distrust the page. So a briefing that no longer matches its numbers is downgraded, never displayed. This means an AI briefing can disappear and come back as a computed one — after **Refresh**, after new traces land, or after a detector pass changes the top issue. That is the guard working, not a fault. ### Driving the briefing over the API Three routes. `GET /insights/summary` is the one you read; the other two exist to *ask for* the AI narrative and to watch it being written. ### Read the briefing ```bash curl -G https://your-neens/api/insights/summary \ -H "Authorization: Bearer nk_live_…" \ --data-urlencode "range=24h" ``` This route **never calls a model and never queues work**, so it is safe to call on every page load. ```json { "headline": "12,480 traces · 4.1% errors · 3 open issues", "body": "The fleet ran 12,480 traces at a 4.1% error rate, p99 latency of 3,180 ms and $41.62 in spend. Top anomaly: error rate moved from 0.021 to 0.058 on Support Agent. The leading issue is \"Unsupported order-status claims\" (214 affected).", "llm": false, "model": null, "empty": false, "stats": { "sessions": 12480, "errorRate": 0.041, "p99Ms": 3180, "spendUsd": 41.62, "anomalyCount": 1, "issueCount": 3, "remediationCount": 2 }, "cached": false, "generatedAt": "2026-07-30T08:14:02.418Z", "briefing": { "id": null, "status": "missing", "refreshing": false, "source": "deterministic", "signalDigest": "9c41ab77e2d0f3b18a4e5d62", "error": null } } ``` `headline`/`body` are always populated — unless `empty` is `true`, meaning there is nothing in the window to report, in which case they are `null`, the UI drops the article (the KPI tiles still render), and `briefing.status` is `unavailable`. `llm` is literal: `true` only when a model wrote the text you were just handed. The `briefing` object tells you what to do next. ### Ask for an AI narrative Only when `briefing.status` is `missing` or `stale`. Send the **same window and scope** you read the summary with — the briefing is identified by them, and a different window claims a different briefing. ```bash curl -X POST https://your-neens/api/insights/summary/briefing \ -H "Authorization: Bearer nk_live_…" \ -H "Content-Type: application/json" \ -d '{"range": "24h", "from": null, "to": null, "projectIds": null}' ``` The call is **idempotent and single-flighted**: however many clients ask for the same window and the same signals at the same moment, exactly one generation is started, so a hundred open dashboards cost one LLM call, not a hundred. ```json { "briefing": { "id": "brf_4b1e77c0a9d2f5386e0c14ab", "status": "queued", "refreshing": true, "source": "deterministic", "signalDigest": "9c41ab77e2d0f3b18a4e5d62", "error": null, "headline": null, "body": null, "model": null, "generatedAt": null, "stats": null } } ``` Fields: `range` (default `24h`), `from`/`to` (only meaningful with `range: "custom"`), `projectIds` (omit or `null` for your whole accessible scope). ### Poll until it's written ```bash curl https://your-neens/api/insights/summary/briefing/brf_4b1e77c0a9d2f5386e0c14ab \ -H "Authorization: Bearer nk_live_…" ``` This is a single-row read — no aggregation behind it — so polling it every few seconds is cheap. The in-app page polls every **3 seconds** and gives up after **40 tries** (about two minutes), keeping the computed briefing if it runs out. ```json { "briefing": { "id": "brf_4b1e77c0a9d2f5386e0c14ab", "status": "ready", "refreshing": false, "source": "llm", "signalDigest": "9c41ab77e2d0f3b18a4e5d62", "error": null, "headline": "Order-status errors double on Support Agent overnight", "body": "Error rate on Support Agent jumped from 2.1% to 5.8% across 12,480 traces, driven by 214 sessions making unsupported order-status claims. p99 latency held at 3.2 s and spend was $41.62. Two remediations are already waiting for review.", "model": "claude-sonnet-4-5", "generatedAt": "2026-07-30T08:14:09.902Z", "stats": { "sessions": 12480, "errorRate": 0.041, "p99Ms": 3180, "spendUsd": 41.62, "anomalyCount": 1, "issueCount": 3, "remediationCount": 2 } } } ``` `stats` is the snapshot of the numbers the narrative was actually written from, so you can render the text and the figures together and know they agree. A briefing id is scoped like everything else. If your key or membership cannot see every agent the briefing covers, `GET /insights/summary/briefing/{id}` returns **404** — including for a briefing written over the whole workspace when you only have some agents. ### Briefing statuses `briefing.status` is the whole client contract. There are seven values: | `status` | Meaning | What a client should do | |---|---|---| | `ready` | A stored briefing matches the current numbers and is inside the freshness window, and it is what you were served. Check `source` — it is `llm` for an AI narrative and `deterministic` for a stored computed one | Nothing. Render it | | `queued` | A generation has been claimed and is waiting for a worker | Poll `GET /insights/summary/briefing/{id}`. **Do not** POST again | | `running` | A worker is writing it now | Poll. **Do not** POST again | | `stale` | An AI briefing exists but describes signals that have moved on, or has aged past the freshness window | POST once to request a fresh one, then poll | | `missing` | Nothing has ever been written for this window and scope | POST once to request one, then poll | | `failed` | Generation was attempted and failed. `error` carries a short reason | Show the computed briefing. Retry later or after **Refresh**; do not hammer it | | `unavailable` | No usable LLM connection for this scope, or there is nothing to report | **Do not poll and do not POST.** The computed briefing is the final answer | `unavailable` is terminal by design, not a transient condition to retry through. A client that polls on `unavailable` will poll forever against an agent that will never have a narrative. Treat `ready`, `failed` and `unavailable` alike as *stop*. Two extra fields matter. `refreshing` is `true` when a newer narrative is being written *behind* the text you were just served — that's the cue for the small "writing a fuller briefing…" note; keep showing what you have and poll. And `source` (`llm` or `deterministic`) is the only honest way to know who wrote the text you are holding: **do not infer it from `status`**, because a computed briefing can be stored and served exactly like an AI one. A stored *computed* briefing that still matches the numbers is served as-is rather than being re-written — a fresh, correct briefing is not withheld on the grounds that a model might phrase it better. So right after you add an LLM connection to an agent that already has one stored, the by-line can stay **Auto-generated summary** until the freshness window lapses (10 minutes by default) or the signals move — **Refresh** re-runs the detectors, which usually does move them. Over the API you can force the question immediately: a stored *computed* briefing is deliberately not treated as a hit by `POST /insights/summary/briefing`, so that call starts a real generation. ### When generation fails A `failed` briefing always carries a short, fixed reason code in `error` — never a raw error message. The computed briefing stays on the page throughout. | `error` | What happened | What to check | |---|---|---| | `llm_error` | The provider rejected the call or errored | The connection under **Settings → Connections**: credential still valid, model id still offered by the provider, provider not rate-limiting you | | `timeout` | The provider didn't answer within **30 s** | Provider status; a very slow self-hosted endpoint. Egress from the Neens host if you self-host | | `unusable_output` | The model answered, but not with a usable headline and body | The model on that connection — very small or heavily quantized models often can't hold the requested JSON shape. Try a stronger model | | `max_attempts` | The same briefing failed the retry budget (**3** attempts by default) for the same story | The underlying cause of the earlier failures. The budget resets by itself as soon as the signals change, so this never means "never again" | | `empty_scope` | By the time the worker ran, the window had nothing in it to narrate | Usually benign — a scope whose traces aged out of the window, or an agent that stopped sending. Confirm ingest is still arriving | | `internal_error` | An unexpected server-side error | The server log for that time window; if self-hosting, the worker's log | Note that "this agent has no LLM connection" is **not** in that list. It is not a failure — see below. ### No LLM connection is a supported end state Neens holds **no platform API key**. Every LLM feature — the briefing included — runs on the connection *you* configure under **Settings → Connections**, on your own account and your own spend. An agent with no connection gets the computed briefing, permanently. `briefing.status` reports `unavailable`, no generation is ever queued, nothing is retried, and nothing is logged as broken. The page is complete: real headline, real body, real numbers, honest by-line. Adding a connection later is all it takes for the AI narrative to start appearing — there is nothing else to switch on. ## Lifecycle: dedup, recurrence, resolution Every insight carries a stable identity (agent + signal, never the run), so re-detection updates one row instead of stacking duplicates: - First detection ⇒ **new**. Detected again after being resolved ⇒ **recurred**. An acknowledged insight stays acknowledged while its signal persists. - When a detector-owned signal **stops firing**, the insight is automatically marked **resolved** on the next pass — feeds only show open insights. - Feedback survives re-detection because it's keyed to the same identity. ### Feedback and muting Each insight accepts a verdict — `useful`, `actioned`, `not_useful`, `wrong`, or `dismissed`. A negative verdict (**not useful / wrong / dismissed**) **mutes** the insight: it disappears from the feeds and stops notifying, until the signal fully resolves and then recurs fresh — at which point it surfaces again as a genuinely new occurrence. Positive feedback is kept for ranking and audit and never suppresses anything. ## Persona-aware routing Detectors stamp every insight with an intended **audience** (engineer, PM, exec, support) and **routing tags** (`reliability`, `latency`, `cost`, `quality`, `compliance`, `safety`, `pii`, `issue`). Regressions on safety-flavored metrics (toxicity, PII leakage, bias, hallucination, …) automatically pick up `compliance`/`safety` tags. Persona lenses use these to decide which insights *lead* for a user: an **Executive** lens leads with reliability/quality/cost signals, **Finance** with cost, **Compliance** with compliance/safety/PII signals and pre-prod regressions, **Product manager** with quality regressions. Catch-all lenses (developer, admin, full workspace) see everything. Routing shapes the [digest email](#the-digest-email) below — the in-app feeds always show the full picture, and a lens never hides data you have access to. ## Notification sinks When an insight **newly surfaces** (first appearance, or recurrence after resolve/mute), it's dispatched to the configured sinks: | Sink | Default | What it delivers | |---|---|---| | **In-app** | Always on | The Insights page itself — the store is the feed | | **Slack** | Off | High-severity insights only, posted to a Slack incoming webhook as a short summary | | **Webhook** | Off | The full structured insight JSON (agent, evidence, impact, recommendation, tags) POSTed to any URL — the hook for GRC/ticketing/automation workflows | Delivery is best-effort: a down Slack or webhook endpoint is logged and never breaks detection. Want to alert on a specific number you choose — *"page me when eval pass-rate < 90%"* — rather than the detectors' automatic signals? Define an [**alert rule**](/guides/alerts). Alert rules fire through these same sinks. ## The digest email The digest is an opt-in, per-user email — "your fleet, every morning": - **Opt in** under **Settings › Account** (or `PUT /auth/digest`), choosing **daily** or **weekly**. It's off by default, and capped at one email per day per user. - **Contents:** the fleet-health tiles — **Traces, Error rate, Eval pass rate, Spend** — computed through the same [metrics catalogue](/guides/metrics) the dashboards use, plus up to five open "needs attention" insights routed to *your* persona lens, each deep-linking into the app. When your agent has committed to [Business KPIs](/guides/business-kpis), the digest adds a **Business KPIs** section — your top few by priority, each with its value, target status (met / missed / unknown) and trend. A KPI with no number yet shows **—**, never a zero. - **Scope-safe:** every number and insight is limited to the agents you could see in the app. - **Send time:** the daily job fires at 13:00 UTC. Weekly recipients get theirs on the same tick once a week. - **Never noisy:** a digest with nothing to say (no activity, no insights) is skipped, not sent empty. - **Try it:** **Send me a test digest** in Settings (or `POST /auth/digest/test`) composes and sends yours immediately, ignoring the daily cap. ## Troubleshooting | Symptom | Cause | Fix | |---|---|---| | Feeds are empty right after ingesting | Detectors run nightly | Use **Refresh** to run them now | | A known problem never appears | It's below the documented floors (e.g. fewer than 5 traces, or a <5-point drop) | The thresholds guard against noise; check the raw data via [Failure clustering](/guides/clustering) | | A dismissed insight came back | Its signal resolved and then recurred fresh | That's by design — a genuinely new occurrence resurfaces; dismiss it again if it's still noise | | No digest arrives | Not opted in, or nothing to report | Opt in under **Settings › Account**; a digest with no activity or insights is skipped rather than sent | | Slack is silent | Nothing high-severity surfaced | The Slack sink only posts high-severity, *newly surfaced* insights | | The briefing always says **Auto-generated summary** | No [LLM connection](/administration/llm-connections) for that agent | Configure a connection under **Settings → Connections**. An agent with no connection keeps the computed briefing — a supported end state, not a fault | | The AI briefing keeps reverting to the computed one | The signals changed, so the stored narrative no longer matches the numbers next to it | Expected. A fresh narrative is requested automatically; the computed text holds the page until it lands | | "Writing a fuller briefing…" never finishes | On a Celery deployment, the eval worker fleet isn't running | Check the `nq.scorers.*` consumers. If judge and eval runs are also stuck, fix the worker | | A short reason appears next to the by-line | Generation failed | Look the code up in [When generation fails](#when-generation-fails) — each one names what to check | ## Related - [Dashboards](/guides/dashboards) — build the charts these signals point you at. - [Metrics catalogue](/guides/metrics) — the measures behind the fleet stats and digest tiles. - [Failure clustering](/guides/clustering) and [Remediations](/guides/remediations) — where insight links land. ================================================================================ # Alert rules Source: /docs/guides/alerts/ ================================================================================ # Alert rules The Neens [detectors](/guides/insights) find anomalies and regressions automatically, but sometimes you know exactly what you want to watch: *"page me when the toxicity pass-rate drops below 90%"* or *"tell me when the error rate goes over 5%."* **Alert rules** let you say that directly — a metric, a threshold, and a window — and route the result through the same notification sinks as insights. ## At a glance | | | |---|---| | **What it is** | A user-defined check: watch a [metric](/guides/metrics) over a window and fire when it crosses a threshold | | **Key API routes** | `GET/POST /alerts`, `GET/PATCH/DELETE /alerts/{id}`, `GET /alerts/catalogue`, `POST /alerts/{id}/test` | | **When it runs** | Every few minutes, per agent, on the server's schedule | | **Where it fires** | The in-app **Insights** feed, plus any configured Slack / webhook sink — the same plumbing insights use | | **Who can set them** | Any **member** (viewers are read-only) | ## Anatomy of a rule Every rule is four choices plus a couple of options: | Field | Meaning | |---|---| | **Metric** | Any [catalogue measure](/guides/metrics) — eval pass rate, error rate, p99 latency, spend, tokens, average score, the [business KPIs](/guides/business-kpis) (containment rate, resolution time, cost per case), and any [custom measure](/guides/custom-measures) your company has defined — those are company-wide definitions, so deleting one breaks every rule that names it | | **Comparator** | `<`, `≤`, `>`, or `≥` | | **Threshold** | The number to compare against (a fraction like `0.9` for rate metrics — the builder shows it as a percentage) | | **Window** | `today`, `24h`, `7d`, or `30d` — the period the metric is measured over | | **Metric key** *(optional)* | For **eval pass rate**, narrow to one judge's metric (e.g. `toxicity`) so you alert on that scorer alone | | **Severity** | `high` / `medium` / `low` — high-severity alerts are the ones the Slack sink posts | | **Sinks & tags** *(optional)* | Restrict which sinks this rule uses, and attach routing tags for the webhook sink | | **Cooldown** | Minimum minutes between repeat notifications while the metric stays breached (default 60) | The rule **fires** when `metric threshold`. An empty or idle window (no data) is never a breach — an alert fires on real signal, not on the absence of it. ## Creating a rule Create one with `POST /alerts`: ```bash curl -X POST https://your-neens/api/alerts \ -H "Authorization: Bearer $NEENS_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "name": "Toxicity pass-rate floor", "measureKey": "eval_pass_rate", "metricKey": "toxicity", "comparator": "lt", "threshold": 0.9, "window": "24h", "severity": "high" }' ``` `GET /alerts/catalogue` returns the alertable measures, comparators, windows, and severities the rule-builder offers, so a UI never has to hardcode them. **"Page me when containment drops below 85%"** is the same shape: `"measureKey": "containment_rate"`, `"comparator": "lt"`, `"threshold": 0.85`. Containment is measured over *decided* cases, so **Test** the rule first — it shows you the current value, and whether the agent's coverage is large enough for the threshold to mean anything. See [Business KPIs](/guides/business-kpis). ### Test before you trust it `POST /alerts/{id}/test` evaluates the rule **right now** and reports the current value and whether it would breach — **without** recording anything, notifying anyone, or touching the rule's state. Use it to sanity-check a threshold against live data. ```json { "value": 0.83, "display": "83%", "breached": true, "comparator": "lt", "threshold": 0.9, "window": "24h" } ``` ## Trend alerts — page me when a KPI slips A threshold rule compares a number to a fixed line. Sometimes what you care about is the *direction of travel*: not "is containment below 85%?" but "**has containment fallen** from where it was last week?". A **KPI trend** rule watches a [Business KPI](/guides/business-kpis)'s own recent history and fires when it has **regressed** by at least a percentage you set, over a window you set. A trend rule references a KPI (not a raw measure), because a trend needs the KPI's recorded [daily history](/guides/business-kpis#history-and-trends) and its declared **direction** — so a *drop* in containment and a *rise* in cost per case are both "regressed", each oriented correctly. ```bash curl -X POST https://your-neens/api/alerts \ -H "Authorization: Bearer $NEENS_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "name": "Containment is slipping", "ruleType": "kpi_trend", "config": { "kpiId": "a68e0cbb059f4b0ba38f409ee2145b19", "lookbackDays": 7, "minDeltaPct": 5 }, "severity": "high" }' ``` | Field | Meaning | |---|---| | `ruleType` | `kpi_trend` (the default is `threshold`). A trend rule needs no `measureKey`/`comparator`/`threshold` — it derives them from the KPI | | `config.kpiId` | The KPI to watch — from `GET /kpis/summary` or the Business KPIs page. Must be a KPI in your agent | | `config.lookbackDays` | How far back the comparison point sits — `7` compares this week to last | | `config.minDeltaPct` | The regression that pages you, as a percentage. `5` means "a 5% or larger adverse move" — small enough to catch a real slip, large enough not to fire on noise | **A trend rule only fires on a genuine regression — never on "we don't know".** If the KPI has too little [recorded history](/guides/business-kpis#history-and-trends) to draw a trend, or its [definition changed](/guides/business-kpis#a-definition-change-breaks-the-series) inside the window, the trend reads `unknown` — and an `unknown` trend is **not** a breach. An alert that paged you because Neens couldn't measure something would be worse than silence. ## What happens on a breach When the periodic evaluator finds a rule breaching, it: 1. **Records an alert** in the [Insights](/guides/insights) store (type `alert`, in the Observe feed) — so it appears in-app alongside detector insights, with the same lifecycle and feedback. 2. **Dispatches it to the sinks** — the in-app feed always, plus Slack (for high-severity rules) and any webhook whose tags match. This is the exact [notification-sink plumbing](/guides/insights#notification-sinks) insights use; no new configuration is needed. 3. **Respects the cooldown** — while the metric stays breached, the rule won't re-notify until its cooldown elapses, so a persistent problem doesn't spam you. When the metric **recovers** (stops breaching), the rule's open alert is automatically resolved and drops out of the feed — mirroring how detector insights self-resolve. Each rule keeps a short **history** (`GET /alerts/{id}/events`) of its breaches and recoveries, and its last evaluated value and state on the rule itself. Alert rules reuse the exact same notification sinks as insights, so **Slack and webhook delivery is configured once**, centrally — see [Insights › Notification sinks](/guides/insights#notification-sinks). There is no separate alert egress path and no new credentials. ## Troubleshooting | Symptom | Cause | Fix | |---|---|---| | A rule never fires | The window is idle (no data ⇒ no breach), or the threshold direction is inverted | Use **Test** to see the current value and confirm the comparator | | A rule on a `custom:…` measure never fires, and **Test** returns nothing | Somebody deleted the [custom measure](/guides/custom-measures) — definitions are company-wide, so an admin in another agent can remove one your rule depends on. The rule stays enabled and resolves no value | Point the rule at an existing measure, or have the measure redefined. See [Before you delete a shared measure](/guides/custom-measures#before-you-delete-a-shared-measure) | | No Slack message on a breach | The Slack sink only posts **high**-severity insights, or isn't configured | Set the rule's severity to `high` and configure the [Slack sink](/guides/insights#notification-sinks) | | A `metricKey` is rejected | Only **eval pass rate** supports narrowing to a specific scorer metric | Drop `metricKey`, or switch the rule's metric to eval pass rate | ================================================================================ # Assistant Source: /docs/guides/assistant/ ================================================================================ # Assistant **Neens Assistant** is an in-app chat that answers questions about your workspace's data in plain language. Ask what's happening with your agents — it looks up the [traces and sessions](/guides/traces-and-sessions), scores, issues, and fixes you already have and answers with real numbers, streaming the reply as it writes. It also answers **how-to and "can Neens do X?"** questions by searching this documentation, so its guidance stays grounded in features Neens actually has rather than guesswork. It can also **do** a few things, not just describe them: curate a dataset from failing traces, record a ground-truth label, create and deploy a judge, start an eval run, or move a suggested fix along. Every one of those is shown to you as a proposal and runs only when you press **Approve** — see [Changes it can make](#changes-it-can-make). ## At a glance | | | | --- | --- | | **Where** | The floating **Sparkles** launcher, bottom-right of every page | | **Key API routes** | `POST /chat/stream` (streamed answers), `GET`/`PUT /chat/settings` (per-agent connection) | | **Needs** | An LLM connection with tool-calling support ([LLM connections](/administration/llm-connections)) | | **Scope** | Answers and changes stay within your own permissions and the currently selected agent | | **Changes** | 8 write actions, each proposed for approval; nothing deletes | ## Open it and ask Click the **Sparkles** launcher in the bottom-right corner to open the panel. Use the header controls to **Maximize** it into a larger centered window (**Esc** restores) or **Minimize** it back to the launcher. Type a question and press **Enter** (**Shift+Enter** for a new line). While it answers you can press **Stop** to end the turn and keep what it has written so far; if a turn fails, a **Retry** button re-sends your last question. An empty panel greets you with suggested questions you can click — tailored to whether an agent is selected. As it works, the Assistant shows small activity chips ("Counting matching traces…", "Reviewing judges and scorers…") so you can see what it looked at on the way to an answer. ## Questions it can answer Each capability below is a tool the Assistant can call. Every tool reads live data from your agent — it never answers data questions from the model's own memory, and if a lookup comes back empty or fails, it says so plainly. Questions about how Neens *works* are answered the same way: the Assistant searches this documentation and grounds its reply on what it finds, rather than inventing feature names, pages, or step-by-step flows. If the docs don't cover something, it tells you it isn't certain instead of guessing. | Ask it… | What it looks at | | --- | --- | | "Which scorers are enabled, and how are they trending?" | The score catalogue: metric, source, average, pass rate, and period-over-period delta | | "What failure modes exist? Any new ones?" | The latest failure-clustering run: top failure modes, session counts, NEW flags, root-cause hypotheses | | "Tell me more about this failure cluster" | One cluster's label, description, root cause, suggested fix, and example sessions | | "What issues are open right now?" | The Issues taxonomy: name, lifecycle state, severity, session counts, last seen | | "How many traces failed the relevancy score this week?" | An exact trace count for any filter combination — status, agent, score, issue, enrichment value, time window | | "Show me examples of traces that errored" | Up to 10 matching traces with agent, status, start time, and failure mode | | "What agents / models / tools show up in my traces?" | The live filter values present in your agent's traces | | "Which enrichments run here, and what values do they produce?" | The agent's enrichments and their observed output fields | | "How is the agent doing?" | KPI snapshot: trace/session/span counts, error rate, p50/p99 latency, plus a 14-day daily volume/error series | | "How are our business KPIs doing?" | Your [Business KPIs](/guides/business-kpis) — each commitment's current value, whether the target is met, and the recent trend; ask about one KPI to also see the failure clusters eroding it. A missing number stays *unknown*, never a fabricated zero | | "What judges exist? When did X last run?" | Each judge's type, scope, latest version, and deployment status/last run | | "What should we fix first?" | Suggested remediations ranked by priority, with status and confidence | | "How is everything doing across agents?" | Org-level KPIs and a per-agent health table (works with no agent selected) | | "Anything unusual lately?" | The Observe / Diagnose / Fix insight feeds (works with no agent selected) | | "Which agents and orgs do I have?" | The agents and orgs your account can access | | "How do I set up a daily digest? Can Neens do X?" | This product documentation — feature and how-to answers are grounded on the docs, with a link, and it says so plainly when the docs don't cover something (works with no agent selected) | ## Changes it can make The Assistant has eight write actions. It never runs one on its own initiative: it **proposes** the change, the panel shows you exactly what would happen, and nothing is written until you press **Approve**. | Ask it… | What it does | Permission it needs | | --- | --- | --- | | "Put these failing traces in a golden dataset" | **Create dataset** — creates a dataset, optionally populated from the traces you were just looking at, optionally as an immutable golden version 1 | Member | | "Add these to the Golden Q&A set" | **Add to dataset** — adds traces/sessions to an existing dataset by id or name; reports how many were added vs skipped as duplicates | Member | | "Mark this session as a fail — it looped on refunds" | **Add annotation** — records a pass/fail ground-truth label with an optional critique, optionally golden, optionally tied to a failure mode | Member | | "Create a judge that checks refund correctness" | **Create judge** — creates an LLM-prompt judge and its version 1 (experimental) | Member | | "Turn that judge on" | **Deploy judge** — deploys a judge version with a `manual` or `on_new_trace` trigger. You don't have to name a version: it deploys the latest unless you ask for a specific one | Member | | "Score the recent traces with it" | **Run eval** — starts an evaluation run for a deployed judge. Naming the judge is enough when it has one deployment | Member | | "Suggest a fix for this cluster" | **Generate remediation** — generates a grounded suggested fix for a failure cluster or failure mode | Member | | "Accept that fix and mark it in progress" | **Update remediation** — transitions a remediation's status, work state, or labels | Member | **Nothing deletes.** There is no tool that deletes a dataset, judge, label, remediation, agent, or trace — on this surface or on the [MCP server](/guides/mcp). Deleting is an admin action you take yourself in the UI. ### Approve is the only confirmation The approve/decline card **is** how the Assistant asks. It will not ask you to type a confirmation first — no "reply *Yes, deploy it* and I'll proceed", no plan restated for your sign-off before the card appears. Once you've asked for a change, the next thing you see is the card, and one click on **Approve** runs it. It asks a question only when a genuinely **required** detail is missing — what to call a new dataset, say. Optional details it leaves out are filled in for you: ask it to turn a judge on without naming a version and it deploys the latest one; ask it to score with a judge that has a single deployment and it finds that deployment itself. So a two-action request costs two clicks, not two clicks plus four typed replies. ### A worked example #### Ask for the change > **You:** Add the failing traces from the "Refund loop" failure mode to a golden dataset. The Assistant looks things up first — you'll see its usual activity chips ("Reviewing latest clustering results…", "Sampling matching traces…") — and then stops. #### Read the proposal A card appears in the conversation: > ✎ **The assistant wants to make a change** > Create dataset "Refund loop failures" and add 6 traces to it as an immutable golden version 1 > **[ Approve ] [ Decline ]** Runs as you, with your permissions The sentence is built from the actual arguments the Assistant chose, so it always describes the call that would run — not a generic "perform a write". Nothing has happened yet. #### Approve — or decline Press **Approve** and the card turns into a running, then completed, activity chip, and the Assistant continues its answer with the result: > ✓ Created **Refund loop failures** with 6 items (golden v1). You can open it under > **Datasets**. Press **Decline** and the card reads *"Declined — nothing ran."* The Assistant is told you declined and answers around it, usually by offering an alternative. Typing a new question instead of answering also leaves the change unmade. ### Worked example: create, deploy and run a judge Standing up a [judge](/guides/judges) is three changes, so it is three cards — and three clicks, with nothing to type in between. #### Ask for all three at once > **You:** Create a judge that checks refunds quote the real amount, turn it on, and score the > last week of traces with it. #### Approve the judge The Assistant proposes the first change immediately — it does not ask you to confirm the plan first: > ✎ **The assistant wants to make a change** > Create judge "Refund amount accuracy" scoped to each trace, criteria: "The reply must state the > refund amount returned by the refund tool, and must not…" > **[ Approve ] [ Decline ]** Runs as you, with your permissions The criteria it drafted are on the card, because the rubric *is* the judge — you're approving the substance, not just the name. Press **Approve** once. The judge and its version 1 are created. #### Approve the deployment The next card follows straight on. Notice it names no version id: > ✎ **The assistant wants to make a change** > Deploy judge "judge_a41f" at its latest version with the manual trigger > **[ Approve ] [ Decline ]** You never had to look a version id up in the UI, and the Assistant never had to guess one — it reads the judge's ids straight out of its own lookup, and omitting the version means "the latest one", resolved when you approve. Press **Approve**. #### Approve the run > ✎ **The assistant wants to make a change** > Start an eval run for the enabled deployment of judge "judge_a41f" > **[ Approve ] [ Decline ]** Same story for the deployment: the judge has exactly one enabled deployment, so naming the judge is enough. Press **Approve** and the run starts: > ✓ Created **Refund amount accuracy**, deployed version 1 with a manual trigger, and started an > eval run over 128 traces. It's running now — I'll have scores under **Scores** when it > finishes. Three changes, three **Approve** clicks, nothing typed. If you'd rather stop after the judge is created, **Decline** the next card — each change is approved on its own, and declining one doesn't undo the ones you already approved. If a judge has **more than one** enabled deployment, "score with it" is ambiguous, and the Assistant says so and asks which one rather than picking for you. If it has none, it offers to deploy it first. ### What a viewer sees The Assistant offers the same tools to everyone, because the permission check happens where it always does — inside the Neens API, on your own credentials. If your role can't make the change, approving it returns a plain permission error rather than a failure: > ✕ The current user's role does not permit this action on this project. Which is the same answer the button in the UI would give you. Ask an admin if you need a broader role; see [Roles and permissions](/administration/members). ## Scope and safety guarantees **No silent changes.** A change is always proposed, described in one line, and made only after you approve it — and it is made **as you**, on your own credentials, so it can never do something your role doesn't already allow. Every change is **attributed to you**: the row it creates or updates records you as the author, and where the resource keeps an audit-trail entry (datasets and judges) that entry names you and records that the change came from the Assistant. Deletes are not available at all. The Assistant sees exactly what you can see and does exactly what you could do, and nothing more: - **Your credentials, re-checked per call.** Each tool call re-enters the regular Neens APIs in-process, forwarding only your own authorization and agent/tenant headers. Tenancy resolution, agent scoping, and permission checks run again on **every** tool call — read or write — exactly as if you had made the request from your browser. There is no separate data path the Assistant could use to bypass them, and approving a change is not what authorizes it: a call you aren't permitted to make returns a permission error to the model, which reports it. - **One agent per conversation.** A chip in the header shows the active scope — the agent name, or **All agents** when none is selected. With an agent selected, agent tools are available; without one, only the org-level tools are. Switching agents clears the conversation, so answers never mix scopes. - **Data is data, not instructions.** Tool results are treated as workspace data; directives embedded in trace content are ignored by instruction. ## How it works `POST /chat/stream` runs an agent loop over your configured LLM: the model streams its answer, optionally requesting tool calls; Neens executes each tool, feeds the compacted result back, and repeats until the model answers without tools — up to **8** rounds per question, after which it closes out honestly with what it found. Responses stream over Server-Sent Events, so text appears as it's generated and tool activity shows in real time. A **write** breaks that loop. When the model asks for one, Neens does not run it: it emits the proposal and ends the turn. Your answer travels on the *next* request — the panel re-sends the conversation together with the exact call you approved, Neens re-checks that call against the tool's own schema, runs it, and the loop resumes with the result. That means there is no pending change stored anywhere: a proposal you never answer simply never happened. Tool results are aggressively compacted before reaching the model (lists capped, long strings truncated, verbose blobs dropped; at most 6,000 characters per result and 30,000 per question) — the Assistant is built for summaries and pointers, not for dumping a full trace into chat. When a tool fails in a way nobody wrote a message for — a database that isn't reachable, an upstream that refuses the connection — the Assistant says *"the tool failed unexpectedly"* and shows a short reference id instead of the underlying error. That is deliberate: the raw message tends to carry server paths and internal hostnames, and it would end up in the model's context and the conversation transcript. The full detail, with the traceback and the same reference id, goes to the server log (the `neens.agent_tools` logger) — quote the id when you report the problem. It is the same contract the [MCP server](/guides/mcp#an-unexpected-failure-is-redacted-to-a-reference-id) uses, because both surfaces run the same tool substrate. Only `user` and `assistant` turns are accepted from the client; the system prompt and tool results are always server-authored. A request may carry at most **40** messages; the server keeps the most recent 30 (up to 24,000 characters) as context. ## Configuration The Assistant uses your agent's **LLM connection** to generate answers — like every LLM-powered feature in Neens, it makes no calls of its own without one. By default it uses the agent's default connection; an agent admin can pin a specific one under **Settings → Assistant** (the **Connection** field, defaulting to **Agent default connection**). Programmatically this is `GET`/`PUT /chat/settings`; updating it requires the manage-LLM-connections permission, and only connections visible to the agent can be selected. **The model must support tool calling.** The Assistant answers data questions by calling tools. If the configured connection's model rejects tools, the Assistant reports it and asks you to pick a different connection under **Settings → Assistant**. **No connection yet?** If an agent has no usable LLM connection, the Assistant doesn't error — it tells you and points you to **Settings → LLM connections**. See [LLM connections](/administration/llm-connections) to set one up. ## Troubleshooting | Symptom | Cause | Fix | | --- | --- | --- | | "No LLM connection is configured for this project" | The agent has no LLM connection, or the Assistant override points at a removed one | Add a connection in **Settings → LLM connections**, or reset the override in **Settings → Assistant** | | "…doesn't support tool calling" | The connection's model rejects the tools parameter | Pick a tool-capable model/connection in **Settings → Assistant** | | "…does not permit this action on this project" in an answer | A tool call hit your real permission boundary | Expected — the Assistant can't see past your access, or make a change your role doesn't allow; ask an admin if you need broader scope | | "the tool failed unexpectedly (ref: …)" | A tool hit an unanticipated server-side error; the detail is withheld from chat on purpose | Give an operator the `ref` — the full exception and traceback are in the server log under that id (`neens.agent_tools`) | | The Assistant says it will change something but nothing happens | The proposal card is still waiting on you | Press **Approve** on the card; a change is never made without it | | It offers to delete something | It shouldn't — there is no delete tool | Nothing was deleted; do it yourself under the relevant page | | Answers say data is missing that you expect | It hasn't been ingested, scored, or clustered yet in this agent | Check the source page directly ([Traces](/guides/traces-and-sessions), [Judges](/guides/judges)) — the Assistant only reports what exists | ================================================================================ # MCP server Source: /docs/guides/mcp/ ================================================================================ # MCP server Neens exposes a **Model Context Protocol (MCP)** server so any MCP-compatible agent — Claude Desktop, an IDE assistant, or your own automation — can both **read** and **write** the failure→fix loop with tools instead of hand-written HTTP calls. This is the foundation of the "agents that fix agents" workflow: an agent can inspect what's failing, curate a dataset, record a ground-truth label, stand up a judge, and move a remediation forward — all through the same permission and tenancy model your team uses. ## At a glance | | | | --- | --- | | **Endpoint** | `POST /mcp` — MCP **Streamable HTTP** (JSON-RPC 2.0), stateless | | **Server name** | `neens` — the name the server reports in `initialize`, and the name to register it under in your client (tools then appear as `mcp__neens__`) | | **Auth** | `Authorization: Bearer nk_live_…` — your [agent API key](/administration/api-keys) | | **Scope** | Everything is scoped to the key's **agent**; deletes are never exposed | | **Tools** | **52** — 34 read + 18 write, [listed below](#what-an-agent-can-do) | The MCP server reuses the existing Neens REST endpoints **in-process**, so every tool re-runs the same authorization and tenancy checks as a normal API request. A tool can only do what the credential you connect with could already do. ## Connect **Using Claude Code?** Prefer the per-**user** browser sign-in over a shared key: run `claude mcp add --transport http neens /mcp` with no header, sign in through your browser, and every action is attributable to **you** and governed by your role. See [Connect Claude Code (MCP)](/guides/connect-claude-code). The agent-key method below stays the right choice for headless automation with no human to sign in. The server speaks MCP over Streamable HTTP at `POST /mcp` on the same origin as the app. Every client needs the same two things: the endpoint URL and your agent key sent as a bearer token (`Authorization: Bearer nk_live_…`). Pick your client below. Your agent API key is shown in **Settings → API keys** (and is printed by the seed scripts for local demos). It resolves to exactly one company + agent, so the agent can only ever see and change **that agent's** data. Add the server with the Claude Code CLI, using the HTTP transport and an `Authorization` header: ```bash claude mcp add --transport http neens \ https://YOUR_NEENS_HOST/mcp \ --header "Authorization: Bearer nk_live_your_project_key" ``` This registers a `neens` MCP server for your current agent. Start a new session and the tools become available, namespaced by the server name you registered — `mcp__neens__list_traces`, `mcp__neens__get_fix_bundle`, and so on. Add a `.cursor/mcp.json` in your project root (or the equivalent under Cursor's global settings) with a `neens` entry pointing at the endpoint and carrying the bearer header: ```json { "mcpServers": { "neens": { "url": "https://YOUR_NEENS_HOST/mcp", "headers": { "Authorization": "Bearer nk_live_your_project_key" } } } } ``` Reload Cursor; the `neens` server appears under **Settings → MCP**. Any Streamable-HTTP MCP client (including Claude Desktop's custom-connector config) needs just the endpoint and the header: ``` Endpoint: https://YOUR_NEENS_HOST/mcp (POST, JSON-RPC 2.0) Authorization: Bearer nk_live_your_project_key ``` ```json { "mcpServers": { "neens": { "url": "https://YOUR_NEENS_HOST/mcp", "headers": { "Authorization": "Bearer nk_live_your_project_key" } } } } ``` **Verify the connection.** Once connected, ask the client to list the Neens tools (an MCP `tools/list` call). You should see **52 tools** — 34 read and 18 write — described in [What an agent can do](#what-an-agent-can-do). Seeing *none* means the connection isn't working — re-check the URL and that the `Authorization` header is being sent. The key is **agent-scoped**: it can read and perform the documented writes within its one agent, and nothing else — deletes are never exposed. The note below spells out exactly what an agent key may write. That scope covers **by-id** calls, not just listings. `get_trace` (and every other tool that takes an id) refuses an object belonging to a different agent — even a sibling agent of the same company, and even when the id is exact. The tool reports an error rather than returning data: ```jsonc // tools/call get_trace {"trace_id": ""} { "error": "This credential is not permitted to perform this action (project keys can read + write within their project, but cannot delete; some actions require a member/admin role).", "status": 403 } ``` Tools whose lookup filters the agent directly report `404` instead, so an out-of-scope id looks the same as one that does not exist. See [API keys](/administration/api-keys#reads-are-scoped-too-including-by-id) for the HTTP-level equivalent. The key's **role** governs writes, exactly as in the UI. An agent key can read and create / update within its agent, but **cannot delete** — deleting judges, datasets, or remediations still requires an admin over the regular API. Creating datasets requires the **Datasets** feature (Silver tier and above). ## What an agent can do The 52 tools are grouped below by the job they do, in roughly the order a loop uses them. Every group is labelled **Read** or **Write** per tool: a read tool carries MCP's `readOnlyHint` annotation so a host can auto-approve it, a write tool doesn't. **No tool deletes anything.** | Group | Tools | | --- | --- | | [Your connection](#your-connection) | 1 | | [Traces and sessions](#traces-and-sessions) | 2 | | [Failure modes and clusters](#failure-modes-and-clusters) | 4 | | [Scores, judges and eval runs](#scores-judges-and-eval-runs) | 8 | | [Datasets, versions and exports](#datasets-versions-and-exports) | 9 | | [Annotations](#annotations) | 1 | | [Pre-prod evaluations](#pre-prod-evaluations) | 5 | | [Comparing results](#comparing-results) | 4 | | [The remediation backlog](#the-remediation-backlog) | 4 | | [The fix loop](#the-fix-loop) | 6 | | [Orchestrated fix runs](#orchestrated-fix-runs) | 2 | | [Prompt optimization](#prompt-optimization) | 2 | | [Model sweeps](#model-sweeps) | 3 | ### Your connection | Tool | Kind | Does | | --- | --- | --- | | `get_current_project` | Read | Reports which **agent** this connection is scoped to — `projectName`, `projectId`, `orgName`, `companyName` — plus `credentialKind`, `role`, and `scopePinned` (whether the agent is fixed for the connection or picked per request). Returns `projectId: null` with a `note` when no single agent is pinned | An agent API key and an SSO/OAuth MCP token each **pin one agent**, so `get_current_project` is the direct answer to "which agent am I connected to?". Call it at the start of a session — for example right after signing in through SSO — to confirm you landed in the agent you expected before reading or writing anything: ```jsonc // tools/call get_current_project {} { "projectName": "Help agent", "projectId": "project_7c2a…", "orgName": "Support", "companyName": "Acme", "credentialKind": "user", // "project" for a project API key; "user" for an SSO/OAuth token "role": "member", "scopePinned": true // the agent is fixed for this connection } ``` The server also names the scoped agent in its `initialize` reply's `instructions`, so a capable client knows the agent from the handshake without an extra call. The tool never reveals another agent: the answer comes from your own credential, and a `null` result simply means no single agent is pinned (a user session that selects its active agent per request). ### Traces and sessions | Tool | Kind | Does | | --- | --- | --- | | `list_traces` | Read | Searches the agent's [traces](/guides/traces-and-sessions). Filters: `cluster_id`, `status`, `agent_name`, `score_metric` + `score_status` + `score_label`, `issue_mode`, `failure_set`, `model`, `tool_name`, `tags`, `source`, `q`, `started_after` / `started_before`, `conversation_id`, `min_duration_ms` / `max_duration_ms`, `min_tokens` / `max_tokens`. Returns the total match count and up to `limit` (1–50, default 20) summary rows | | `get_trace` | Read | One trace by `trace_id`: agent, status, timing, cost, error message, and a summary of its spans and tool calls | **Use `cluster_id`, not the cluster's name.** `model`, `tool_name` and `tags` take arrays and match any of the values given; everything else is a single value. `total` is how many traces matched, `traces` holds at most `limit` of them. ### Failure modes and clusters | Tool | Kind | Does | | --- | --- | --- | | `list_failure_modes` | Read | The latest clustering run: readiness, plus the top [failure modes](/guides/issues-and-failure-modes) with session counts, NEW flags and root-cause hypotheses | | `get_failure_mode` | Read | One cluster by `cluster_id`: label, description, root cause, remediation summary and member session ids | | `list_clusters` | Read | The active clusters as a flat list — id, label, description, session count, L1 bucket. The lightweight way to get the `cluster_id` that `list_traces` takes | | `get_cluster_exemplars` | Read | The members nearest a cluster's centroid, with their distance — the shortest read that shows what a failure mode actually looks like | ### Business KPIs | Tool | Kind | Does | | --- | --- | --- | | `get_business_kpis` | Read | The agent's [Business KPIs](/guides/business-kpis) — each commitment's current value, `unit`, whether the target is `met`/`missed`/`unknown`, and its recent `trend`. Pass a `kpi_id` to drill into one KPI, which also returns the [failure clusters eroding it](/guides/business-kpis#failure-impact--which-failures-are-eroding-a-kpi). A `null` value and an `unknown` status are passed through verbatim — never a fabricated `0` | ### Scores, judges and eval runs | Tool | Kind | Does | | --- | --- | --- | | `list_scores` | Read | The [scorers](/guides/scores) active in the agent and how each trends — metric key, source, target type, average, pass rate, period-over-period delta. Optional `range`: `24h`, `7d` (default), `30d`, `all` | | `list_score_rows` | Read | **Individual** score rows with the judge's score, label and reason. Filter by `metric_key`, `status` (`pass`/`fail`), `target_id` / `target_type`, `source`, a specific eval `run_id`, `min_score` / `max_score`, `lifecycle`, `sort` and `range`. This is how you find *which* traces a judge failed. `run_id` reads that run's own snapshot, which applies no metric/source/time filter — so it cannot be combined with `metric_key`, `source` or `range`, and asking for both is refused rather than answered with rows that only look filtered | | `list_judges` | Read | The agent's judges with **every id the write tools need**: `currentVersionId`, `latestVersionId` and a `versions[]` list of `{id, versionNum}` (newest first, up to 5), plus `deployments[]` where each entry carries the `id` that `run_eval` and `create_preprod_run` need alongside its `versionId`, trigger, status and last run | | `get_judge_runs` | Read | One judge's eval runs, newest first: status, targets scored, average score and pass rate per run, plus an active/historic summary | | `get_eval_run` | Read | One eval run by id: status, per-status task counts, success rate, and a page of its scored **tasks** (target id, score, label, reason, error). `taskTotal` is the run's whole size; `tasks` is at most `limit` rows from `offset` | | `create_judge` | Write | An LLM-prompt [judge](/guides/judges) plus its version 1 (experimental), from a `prompt_template` that interpolates `{target}`, optional `criteria` and `required_data` | | `deploy_judge` | Write | Deploys a judge version so it can run. Only `judge_id` is required — omit `version_id` for the judge's **latest** version, or pass a version id *or* a version number like `"2"`. `trigger_policy: "manual"` scores on demand, `"on_new_trace"` scores every new trace. See [Deploying without a version id](#deploying-without-a-version-id) | | `run_eval` | Write | Starts an evaluation run and returns the run id, status, `resolvedDeploymentId` and `total`. Pass a `deployment_id`, **or** just `judge_id` to use that judge's single enabled deployment. Optional per-run overrides: `sample_size` caps the run at the N most-recent eligible traces (leaving the deployment's sampling filter untouched); `success_threshold` overrides the pass cutoff. See [Running an eval from a judge id](#running-an-eval-from-a-judge-id) | #### One read is enough to deploy and run `list_judges` is the only call you need before either write tool. Each judge it returns carries the version ids **and** the deployment ids, so nothing has to be looked up in the UI and nothing has to be guessed: ```json { "id": "judge_a41f", "name": "Grounding check", "currentVersionId": "jver_9c02", "latestVersionId": "jver_9c02", "latestVersionNum": 2, "versions": [ {"id": "jver_9c02", "versionNum": 2}, {"id": "jver_31ab", "versionNum": 1} ], "deployments": [ {"id": "dep_5e7d", "versionId": "jver_9c02", "trigger": "manual", "status": "enabled"} ] } ``` `versions` is newest-first and holds up to the 5 most recent; `latestVersionId` is always the newest one. A judge with no `deployments` entry whose `status` is `enabled` has nothing running — that is the one to call `deploy_judge` on. (It may still have `disabled` entries: those are deployments that were turned off, not judges that were never deployed.) #### Deploying without a version id `judge_id` is the only required argument. Omit `version_id` and the judge's latest version is resolved for you: ```json {"judge_id": "judge_a41f", "trigger_policy": "manual"} ``` A version **number** works too, so `{"judge_id": "judge_a41f", "version_id": "1"}` deploys version 1 — you don't have to translate it into `jver_31ab` yourself. Version *ids* are matched before numbers, so an id made only of digits still wins. The reply names what was actually deployed, which matters when you let the server pick: ```json { "id": "dep_5e7d", "judgeId": "judge_a41f", "versionId": "jver_9c02", "resolvedVersionId": "jver_9c02", "trigger": "manual", "status": "enabled", "successThreshold": null, "alreadyDeployed": false } ``` Deploying a version that is **already** enabled with the same trigger and the same pass cutoff returns that existing deployment with `alreadyDeployed: true` rather than creating a second one, so a retry is safe. Changing `trigger_policy` or `success_threshold` does create a new deployment — different behaviour is not a duplicate, and silently reusing the old row would drop the argument you changed. Omitting `success_threshold` expresses no preference, so it still matches a deployment that has one. If a version genuinely can't be resolved, the error lists the judge's real version ids (`Its versions are: version 2 = jver_9c02; version 1 = jver_31ab`) so the next call succeeds. Never retry `deploy_judge` with an invented id. #### Running an eval from a judge id `run_eval` takes `deployment_id`, but if you only have the judge, pass that instead and its single enabled deployment is used: ```json {"judge_id": "judge_a41f"} ``` The reply carries `resolvedDeploymentId` alongside the run `id`, `status` and `total`, so you always know which deployment actually ran. Two cases are refused rather than guessed at: | Situation | What `run_eval` answers | | --- | --- | | The judge has **no** enabled deployment | An error saying so, telling you to call `deploy_judge` for that judge first | | The judge has **more than one** enabled deployment | An error naming every candidate — deployment id, its version id and its trigger — for you to pass as `deployment_id` | By default a run covers what the deployment's sampling filter selects. Pass `sample_size` to cap **this** run at the N most-recent eligible traces without editing the deployment — e.g. `sample_size=1000` to score 1000 traces. The response `total` is the number actually queued, which is smaller when fewer traces are eligible. `success_threshold` independently overrides what counts as a pass for that one run. ### Datasets, versions and exports | Tool | Kind | Does | | --- | --- | --- | | `list_datasets` | Read | The agent's [datasets](/guides/datasets): id, name, item count, source kind, tags, and the cluster a from-cluster dataset was built from | | `get_dataset_items` | Read | A dataset's **live** items (input, expected output, captured output, source trace), with optional `q` search and `kind` filter. Reports `total` / `returned` / `complete` so a page is never mistaken for the set | | `list_dataset_versions` | Read | A dataset's immutable versions, newest first, and which one is **golden** (returned separately as `goldenVersion` / `goldenVersionId`) | | `get_dataset_version_items` | Read | One version's frozen snapshot items — same shape as `get_dataset_items`, but it never changes under a running evaluation. `version` is the integer version **number** | | `export_dataset` | Read | A dataset export as a **manifest** — see [Exports return a manifest](#exports-return-a-manifest) | | `export_dataset_version` | Read | One immutable version's export, as the same manifest plus which version it is and whether it is golden | | `create_dataset` | Write | A dataset. With `session_ids` / `trace_ids` it is created **and** populated from that selection, optionally as an immutable golden v1 (`golden: true`); with neither, an empty manual dataset | | `add_to_dataset` | Write | Adds `session_ids` / `trace_ids` to an existing `dataset` (by id or name); reports how many were added versus skipped as duplicates | | `create_golden_version` | Write | Snapshots a dataset's current items into a new immutable version and marks it golden (pass `golden: false` for a plain snapshot). One golden version per dataset — goldening a new one un-goldens the previous | `create_golden_version` is the step between curating a dataset and evaluating against it: a [pre-prod evaluation](/guides/preprod-evals) and a [model sweep](/guides/model-sweeps) both replay a *frozen* version, so results stay comparable across runs. If `create_preprod_run` refuses with *"dataset has no golden version"*, this is the tool you missed. ### Annotations | Tool | Kind | Does | | --- | --- | --- | | `add_annotation` | Write | A pass/fail [ground-truth label](/guides/annotations-and-review) on a trace or session, with an optional `critique`, `is_gold`, and a `failure_mode_id` / `cluster_id` tie-in. Re-labelling the same target supersedes the prior label | ### Pre-prod evaluations Use these to run a [pre-prod evaluation](/guides/preprod-evals) yourself. When you are proving one specific remediation's fix, `run_verification` in [the fix loop](#the-fix-loop) does the same job in one call. | Tool | Kind | Does | | --- | --- | --- | | `list_preprod_runs` | Read | The agent's runs, newest first: status, version label, dataset and **frozen dataset version id**, gate, progress, regression count. The dataset version id is what tells you which runs are comparable. Filter by `status` or `dataset_id` | | `get_preprod_run_items` | Read | The run's frozen prompt list — one row per golden item with its `itemId`, input and expected output. `itemId` is what `get_preprod_trajectory` diffs on | | `create_preprod_run` | Write | Creates a run over a dataset's golden version at a candidate `version_label`. Created in `awaiting_traces` — it runs nothing until you start it. `runner_mode: "push"` means Neens calls your endpoint itself and **requires** `agent_connection_id`; `"runner"` means Neens waits for your own harness's traces. Gate it with `min_pass_rate` / `max_regressions` | | `start_preprod_run` | Write | Starts a created run, routing itself on the run's own `runner_mode` — you do not have to pick. Refuses a run that is already terminal | | `cancel_preprod_run` | Write | Cancels a run that is still awaiting traces or running. A terminal run is refused rather than rewritten | ### Comparing results | Tool | Kind | Does | | --- | --- | --- | | `get_preprod_comparison` | Read | The candidate-vs-**baseline** verdict for one run: the candidate's pass rate, the regression set (baseline passed, candidate failed), the new passes, and cost/latency/step deltas. The baseline is the run's own — a prior run or a prod window — and `baseline` overrides which | | `get_preprod_metrics` | Read | Per-metric average and pass rate for one run. This enforces *"faithfulness must stay ≥ 0.8"* where the run's single aggregate pass rate cannot | | `compare_preprod_runs` | Read | Lines up **two or more** runs over the same frozen golden version: per-run pass rate, average score and cost/latency/step traits, plus a matrix keyed by golden prompt with one cell per run. Name one as `baseline_run_id` and each row also reports which runs regressed against it | | `get_preprod_trajectory` | Read | Diffs **how** the agent behaved on one golden prompt between two runs — each run's captured trajectory as an ordered tool-call/LLM step list, aligned, with added/removed/changed steps. Use it to explain *why* an item regressed | Three reading rules these tools depend on you honouring: - **`compare_preprod_runs` requires runs that snapshot the same dataset version.** Mismatched runs are refused rather than aligned on prompts that are not the same prompts. - **Read `regressionCount`, not the length of `regressions`.** The row lists are capped evidence; the count is the finding. - **`null` is not zero.** A `passRate` of `null` means nothing was scored yet, a `cost` delta of `null` means neither side had a priced session, and a `sessionId` of `null` in a trajectory means that run captured nothing for the prompt — which is not the same as "no change". ### The remediation backlog | Tool | Kind | Does | | --- | --- | --- | | `list_remediations` | Read | The [remediation](/guides/remediations) backlog ranked by priority, with a stats rollup. Filter by `status`, `workState`, `clusterId`, `failureModeId` or `type` | | `get_remediation` | Read | One remediation with its simulations and [what-changed](/guides/what-changed) correlation — the evidence to review before accepting it | | `generate_remediation` | Write | Generates a grounded fix for a `cluster_id` or `failure_mode_id`. Degrades deterministically when the agent has no LLM connection | | `update_remediation` | Write | Transitions a remediation's `status` (proof lane) and/or `work_state` (triage lane) and/or `labels` | ### The fix loop The six tools that let a coding agent pull a fix, prove it, and report the result back — see [Fix loop over MCP](#fix-loop-over-mcp) for the round trip. | Tool | Kind | Does | | --- | --- | --- | | `list_open_remediations` | Read | Open fixes ranked by priority — excludes verified/closed and archived unless `include_resolved: true`. Each row carries its lifecycle status, any PR/commit already reported, and the last verification run id | | `get_fix_bundle` | Read | The [fix bundle](/guides/fix-bundles) for a `remediation_id`: the paste-ready markdown brief, the pre-generated `neens eval run` CI command, the gate policy, the proof evals and the acceptance criteria | | `run_verification` | Write | Runs a real [pre-prod PUSH evaluation](/guides/preprod-evals) against your preview deploy (`endpoint_url` or `agent_connection_id`) at a required `version_label`, scores it with the proof judges, and gates on regressions. Polls up to `wait_seconds` (default 90, max 300) for a verdict | | `get_verification_run` | Read | Polls a verification `run_id` for the terminal verdict: status, gate pass/fail, reasons, regressions, pass rate and score aggregate | | `report_fix_status` | Write | Sets a remediation's `status` / `work_state` and records the `pr_url` and `commit_sha` where the fix landed | | `record_fix_merge` | Write | Tells Neens a fix PR was **merged** so it can open a [post-merge efficacy](/guides/post-merge-efficacy) close-out watch and measure the failure mode's real production volume before versus after | ### Orchestrated fix runs | Tool | Kind | Does | | --- | --- | --- | | `start_fix_run` | Write | Launches a Neens-orchestrated, [eval-verified fix run](/guides/eval-verified-fix) for an accepted `remediation_id`: Neens proposes the patch, applies it on a branch, verifies it with pass^k pre-prod runs, and opens a PR **only** if green | | `get_fix_run` | Read | Polls a fix `run_id`: status (`queued`, `applying`, `verifying`, `pr_opened`, `failed`, `drafted`, `cancelled`), attempt count, pass^k eval report, verifying judges, PR url, failure report | ### Prompt optimization | Tool | Kind | Does | | --- | --- | --- | | `start_prompt_optimization` | Write | Launches an offline, GEPA-style [prompt-optimization](/guides/prompt-optimization) run over a failure mode's historical traces — no live traffic. A winner is emitted as a normal `prompt_change` remediation, never a deploy | | `get_prompt_optimization` | Read | Polls an optimization `run_id`: status, baseline versus best held-out score, rollouts and iterations spent, the candidate lineage, the verdict, and the emitted remediation id when the winner cleared the bar | ### Model sweeps | Tool | Kind | Does | | --- | --- | --- | | `start_model_sweep` | Write | Launches a [model sweep](/guides/model-sweeps): one frozen golden set, N `arms` (each `{label, agent_connection_id}`), k runs each, pass^k per arm. Refused if the estimate exceeds `budget_usd` or the deployment's ceilings | | `get_model_sweep` | Read | Polls a `sweep_id`: status, the pinned conditions, the cost estimate, and a per-arm summary (model, status, greens/k, run counts) | | `get_model_sweep_comparison` | Read | The [sweep's verdict](/guides/sweep-decisions): which model to ship, the per-arm leaderboard with sample size, 95% confidence interval and cost per case, and a verdict **per agent** | ### Exports return a manifest, not the file `export_dataset` and `export_dataset_version` deliberately **never** put the export body in the tool result. A real golden export is far past the 20,000-character result cap, so inlining one would go through the [shrinking](#results-are-compacted-never-truncated-mid-json) described below and hand you a file that is missing rows but still parses as valid JSON. An export you cannot tell is incomplete is worse than no export tool, so you get everything needed to act on it — and a URL for the bytes: ```json { "datasetId": "ds-7f2a91", "datasetName": "checkout-timeout-v1", "format": "json", "rowCount": 4821, "sampleFields": ["id", "kind", "traceId", "sessionId", "input", "expectedOutput", "output"], "csvColumns": ["kind", "trace_id", "session_id", "input", "expected_output", "output"], "itemsChecksum": "sha256:9d31c0…", "bytes": 3841902, "sampleRowCount": 3, "sample": [ /* the first 3 rows, each field capped at 300 characters */ ], "complete": false, "retrieval": { "method": "GET", "path": "/api/datasets/ds-7f2a91/export", "csvPath": "/api/datasets/ds-7f2a91/export?format=csv", "note": "…" } } ``` | Field | What it is good for | | --- | --- | | `rowCount` | The **exact** number of rows in the export. Counted from the full body, never from `sample` | | `complete` | `true` only when `sample` **is** every row *and* no field in it was truncated. On `false`, treat `sample` as a preview and fetch the export | | `itemsChecksum` | `sha256` over the canonical **row list**, in order. Two exports carry the same rows if and only if these match — the cheapest way to tell whether a dataset changed since you last pulled it. It is **not** the digest of the downloaded file: `retrieval.path` wraps the rows in a `{dataset, items}` envelope, and `csvPath` is a different format again, so hashing either will not reproduce it | | `bytes` | Size of the canonical row list, so you can decide whether to fetch the export at all | | `sampleFields` | The fields each row of `sample` carries. The JSON export additionally carries `raw` and `createdAt` (plus `versionId` on a version export) | | `csvColumns` | The CSV header served at `retrieval.csvPath` — six snake_case columns, and note there is **no `id`** | | `retrieval.path` / `retrieval.csvPath` | Where to GET the whole export | **To fetch the full export**, call the `retrieval.path` over ordinary HTTP with the same bearer token you connected the MCP server with — no extra credential, and the same agent scope: ```bash curl -H "Authorization: Bearer nk_live_your_project_key" \ "https://YOUR_NEENS_HOST/api/datasets/ds-7f2a91/export" -o dataset.json # …or as CSV curl -H "Authorization: Bearer nk_live_your_project_key" \ "https://YOUR_NEENS_HOST/api/datasets/ds-7f2a91/export?format=csv" -o dataset.csv ``` If you only need a few rows and not a file, `get_dataset_items` (or `get_dataset_version_items`) pages them properly and reports `total` alongside `returned`. ## Walkthroughs Three end-to-end flows you can run today. Each shows the prompt you'd type at your coding agent and the wire traffic underneath, so the same recipe works whether you're driving a chat client or writing the JSON-RPC by hand. ### Triage a failure mode and curate a golden dataset The everyday loop: find what's failing, look at real examples, record verdicts, and freeze the result into a dataset your judges can evaluate against forever. > **Prompt.** *"Using the neens tools, find the biggest new failure mode in this agent, pull > 20 example traces from its failure set, mark the ones that are genuinely broken as `fail`, and > collect them into a golden dataset called `checkout-timeout-v1`."* #### Find the failure mode `list_failure_modes` takes no arguments. It returns the clustering run's `readiness` plus a ranked `modes` array — each with `id`, `label`, `sessionCount`, `isNew` and `rootCauseHypothesis` — so the agent can pick a target without a second call. #### Read its root cause `get_failure_mode` with the `cluster_id` from the previous step returns the full root cause, the suggested remediation, and up to ten `memberSessionIds` to start from. ```json {"cluster_id": "cl_7f2a91"} ``` #### Pull example traces `list_traces` with `failure_set: true` scopes the search to the clustering failure set. Here is the full `tools/call` request and reply, so you can see exactly what crosses the wire: ```json { "jsonrpc": "2.0", "id": 3, "method": "tools/call", "params": { "name": "list_traces", "arguments": { "failure_set": true, "issue_mode": "Checkout tool timeout", "status": "error", "limit": 20 } } } ``` The reply is a standard MCP result whose single text block is the JSON tool result: ```json { "jsonrpc": "2.0", "id": 3, "result": { "content": [{ "type": "text", "text": "{\"total\":83,\"traces\":[{\"id\":\"sess_4d19\",\"agentName\":\"checkout-agent\",\"status\":\"error\",\"startedAt\":\"2026-07-28T09:14:22Z\",\"turnCount\":6,\"failureMode\":\"Checkout tool timeout\"}]}" }], "isError": false } } ``` `total` is the number of traces that matched, not the number returned — `traces` holds at most `limit` rows. #### Record verdicts `add_annotation`, once per trace the agent judged. `target_type` is `session` by default; `is_gold` marks the label authoritative, and tying it to the cluster keeps the evidence linked: ```json { "target_type": "session", "target_id": "sess_4d19", "verdict": "fail", "critique": "Tool call to /checkout exceeded the 30s budget; agent retried instead of failing over.", "is_gold": true, "cluster_id": "cl_7f2a91" } ``` #### Freeze the dataset `create_dataset` in one shot — creating it and populating it from the selection, with `golden: true` to snapshot an immutable version 1: ```json { "name": "checkout-timeout-v1", "description": "Traces from the checkout tool-timeout failure mode, July 2026.", "session_ids": ["sess_4d19", "sess_51c0", "sess_6b77"], "golden": true } ``` Grow it later with `add_to_dataset` (same `session_ids` / `trace_ids`, plus `dataset` — the id or the name), which reports `added` and `skipped` so re-running it is safe. #### Score against it `create_judge` → `deploy_judge` → `run_eval` turns the dataset into a repeatable score, and each step hands the next one the id it needs. `create_judge` returns the judge id; `deploy_judge` needs nothing but that (`{"judge_id": "judge_a41f"}` deploys the latest version) and returns the deployment id; `run_eval` accepts either that deployment id or the judge id on its own. Use `list_judges` instead of `create_judge` to score with a judge that already exists — one call returns its version ids and its deployment ids together. See [Deploying without a version id](#deploying-without-a-version-id). #### Read what the judge decided `run_eval` returns a run id. `get_eval_run` opens it — per-status task counts, success rate, and a page of scored tasks with each target's score, label and reason: ```json {"run_id": "run_9c21", "limit": 20} ``` For the failures across *every* judge rather than one run, `list_score_rows` filters the raw score rows — `{"status": "fail", "metric_key": "primary_score", "sort": "lowest"}` puts the worst first. `get_judge_runs` lists a judge's history when you need an older run's id. ### From a failure cluster to a pre-prod comparison The full evaluation loop, using nothing but MCP tools: pick a failure cluster, freeze its traces into a golden set, evaluate a candidate against it, and compare the result to a baseline. > **Prompt.** *"Using the neens tools, take the biggest failure cluster in this agent, freeze > its traces into a golden dataset, run a pre-prod evaluation of branch `fix/checkout-timeout` > against it, and tell me whether anything regressed versus the current baseline."* #### Pick the cluster `list_clusters` takes no required arguments and returns each cluster's `id`, `label`, `description` and `sessionCount`. `get_cluster_exemplars` then shows the members nearest its centroid so you can confirm you picked the right one before pulling hundreds of traces: ```json {"cluster_id": "cl_7f2a91", "limit": 5} ``` #### Pull every trace in it `list_traces` with `cluster_id` returns the cluster's exact membership. This is the filter to reach for — matching a cluster's *label* through `issue_mode` silently returns the wrong set as soon as a label is regenerated: ```json {"cluster_id": "cl_7f2a91", "limit": 50} ``` Narrow further with any of the filters in [Traces and sessions](#traces-and-sessions) — `min_duration_ms` for the slow tail, `model` for one model's share of the cluster, `tool_name` for the tool that failed. #### Freeze them into a golden set `create_dataset` collects the trace ids, then `create_golden_version` snapshots them into an immutable version. Evaluations replay the *frozen* version, which is what keeps two runs comparable: ```json // create_dataset {"name": "checkout-timeout-v1", "session_ids": ["sess_4d19", "sess_51c0", "sess_6b77"]} // create_golden_version {"dataset": "checkout-timeout-v1", "name": "v1", "notes": "Traces from cluster cl_7f2a91"} ``` Check the result with `list_dataset_versions` — it reports `goldenVersion` and `goldenVersionId` explicitly, which is exactly what the next step resolves. #### Create the run `create_preprod_run` takes the dataset (id **or** name) and the candidate `version_label` — the branch, commit or deploy under test. With no `dataset_version_id` it resolves the golden version for you. Add the gate you want enforced: ```json { "name": "checkout-timeout candidate", "dataset_id": "checkout-timeout-v1", "version_label": "fix/checkout-timeout@a1b2c3d", "runner_mode": "push", "agent_connection_id": "conn_preview", "min_pass_rate": 0.9, "max_regressions": 0 } ``` It comes back in `awaiting_traces` and runs nothing yet. `get_preprod_run_items` shows the frozen prompt list it will replay. #### Start it `start_preprod_run` with `{"run_id": "ppr_9d31"}`. It reads the run's own `runner_mode` and does the right thing: a `push` run is executed by Neens against your endpoint, a `runner` run moves to `running` and waits for your harness's traces. Poll `list_preprod_runs` for progress, or `cancel_preprod_run` to stop one. #### Read the verdict `get_preprod_comparison` with the run id gives the candidate's pass rate, the regression set, the new passes, and cost/latency/step deltas versus the baseline: ```json { "runId": "ppr_9d31", "candidate": {"scored": 30, "passed": 27, "passRate": 0.9}, "regressionCount": 2, "newPassCount": 5, "matrixTotal": 30, "regressions": [ /* capped evidence rows */ ], "deltas": {"latencyMs": {"before": 1400, "after": 1180, "delta": -220}} } ``` `get_preprod_metrics` adds the per-metric breakdown — the one that enforces a threshold on a single metric rather than on the aggregate. #### Compare candidates against each other Once you have a second run over the **same** golden version, `compare_preprod_runs` lines them up side by side and marks per-row regressions against whichever you declare the baseline: ```json {"run_ids": ["ppr_9d31", "ppr_9d44"], "baseline_run_id": "ppr_9d31"} ``` For any row that regressed, `get_preprod_trajectory` diffs how the agent actually behaved on that one prompt between the two runs — the step list, aligned, with what was added, removed or changed: ```json {"run_id": "ppr_9d44", "baseline_run_id": "ppr_9d31", "item_id": "dvi_0875bc"} ``` That is the answer to *why* it regressed, not just *that* it did. ### Close the loop on a fix The differentiator: your coding agent writes the fix in **your** repo, and Neens proves it with a real evaluation against your preview deploy. > **Prompt.** *"Pick the highest-priority open remediation from neens, get its fix bundle, > implement it on a branch, deploy it to preview, then ask neens to verify the fix at that branch > and report the PR back."* #### Pick a fix `list_open_remediations` — no required arguments — returns the open backlog ranked by priority, each row carrying `id`, `title`, `type`, `status`, `workState`, `priority`, and any `prUrl`, `commitSha` or `verificationRunId` already recorded. #### Get the brief `get_fix_bundle` with that `remediation_id` returns everything the coding agent implements from: a paste-ready `markdown` brief (root cause, typed before/after fix spec, redacted failing exemplars), the pre-generated `evalCommand`, the `gatePolicy`, the `proof` evals, and the `acceptanceCriteria`. #### Implement it — in your repo Your coding agent makes the change with its own model and your credentials, and pushes a branch to a preview deploy. Neens is not involved and never sees your source. #### Prove it `run_verification` replays the remediation's golden dataset against the preview endpoint and gates the result. `version_label` is required — it's the branch, commit or preview id under test: ```json { "jsonrpc": "2.0", "id": 9, "method": "tools/call", "params": { "name": "run_verification", "arguments": { "remediation_id": "rem_2c81", "version_label": "fix/checkout-timeout@a1b2c3d", "endpoint_url": "https://pr-482.preview.example.com", "request_shape": "openai_chat", "min_pass_rate": 0.9, "max_regressions": 0, "wait_seconds": 120 } } } ``` If the run finishes inside `wait_seconds` you get the verdict directly: ```json {"runId":"pre_9d31","status":"completed","passed":true,"reasons":[],"regressions":0,"passRate":0.94,"aggregate":{"scored":50,"passed":47}} ``` If it doesn't, you get `{"runId": "pre_9d31", "status": "running", "pending": true, "message": "Run still executing; poll get_verification_run with this runId."}` — poll `get_verification_run` with `{"run_id": "pre_9d31"}` until `status` is terminal. #### Report it back `report_fix_status` links the remediation to the code that resolved it: ```json { "remediation_id": "rem_2c81", "status": "verified", "work_state": "done", "pr_url": "https://github.com/acme/checkout-agent/pull/482", "commit_sha": "a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0" } ``` #### After a human merges `record_fix_merge` with `{"remediation_id": "rem_2c81"}` (the `pr_url` and `commit_sha` default from the remediation, `merged_at` defaults to now) opens a [post-merge efficacy](/guides/post-merge-efficacy) close-out watch: after the post-deploy window Neens compares the failure mode's real production volume before and after, and auto-verifies — or flags a regression of — the fix. **Prefer Neens to write the patch too?** Swap the middle three steps for `start_fix_run` → `get_fix_run`: Neens proposes the patch from the same bundle, verifies it with pass^k pre-prod runs, and opens a PR **only** if it goes green. A human still merges. See [eval-verified fix](/guides/eval-verified-fix). ## Fix loop over MCP The six **[fix-loop tools](#the-fix-loop)** let a coding agent do more than investigate: it can pull a fix off the Neens backlog, implement it in **your** repository, ask Neens to **prove** the fix works, and report the result back — all without Neens ever touching your code. Neens does the diagnosis and the verification; your coding agent, in your repo, with your credentials, does the edit and opens the pull request. A human still reviews and merges. This is what sets the Neens MCP apart: other observability MCP servers stop at read-only triage. The Neens `run_verification` turns "here's a suggested fix" into "here's a fix that **provably** passes the same evals the failure originally broke." ### The round trip | # | Step | Tool | | --- | --- | --- | | 1 | Pull a fix off the backlog — anything not yet verified or closed, and not archived | `list_open_remediations` | | 2 | Get the full brief: root cause, typed before/after fix, anonymized failing examples, a ready-to-run proof-eval command | `get_fix_bundle` | | 3 | **Apply it in your own repository** — your coding agent, your model, your credentials. Neens is not involved and never sees your source | *(none)* | | 4 | Point Neens at your preview deploy at a version label; it replays the remediation's golden dataset as a [pre-prod evaluation](/guides/preprod-evals) and gates the result | `run_verification` | | 5 | Poll for the terminal pass/fail and eval report if step 4 didn't finish in its wait budget | `get_verification_run` | | 6 | Attach the pull-request URL and commit SHA, so the remediation is permanently linked to the code that resolved it | `report_fix_status` | | 7 | After a human reviews and merges, open the [post-merge efficacy](/guides/post-merge-efficacy) close-out watch on real production volume | `record_fix_merge` | [Close the loop on a fix](#close-the-loop-on-a-fix) walks the same seven steps with real arguments and the wire traffic. `run_verification` calls **your** preview endpoint over HTTP. Neens never receives or stores repository credentials; the pull-request URL and commit SHA you attach are the only code references it keeps. ## How it works Each `tools/call` builds a short-lived in-process client that re-enters an existing Neens REST endpoint (for example, `list_traces` calls `GET /sessions`, `add_annotation` calls `POST /review`), forwarding your `Authorization` header. The Neens tenant resolution and each route's authorization gate run per call, so there is no separate access path and no way for an agent to exceed the connecting key's permissions. The server is **stateless** and advertises only the `tools` capability. Reads are available to any agent-scoped credential; writes are admitted for an agent key on the specific non-destructive write routes, staying leak-safe because the key is pinned to a single agent. ### Results are compacted, never truncated mid-JSON A tool result feeds a model's context window, not a UI, so every result is compacted on the way out. This is visible to you, and it is worth knowing before you build on a response shape. | Cap | Value | | --- | --- | | Maximum characters in one tool result | **20,000** | | Strings longer than this are truncated (with a trailing `…`) | **600** characters | | Lists are capped at this many entries | **30** | | Fields dropped entirely | `embedding`, `sparkline`, `confidenceBasis` | | Error message lifted out of a failed response | first **400** characters | The result is **always valid JSON**. If the compacted payload still exceeds 20,000 characters, Neens repeatedly halves every list and re-serializes — up to 20 passes — rather than cutting the string off somewhere arbitrary. Any result that went through that shrinking carries a top-level marker so a client can tell: ```json {"total": 4821, "traces": [ /* … */ ], "truncated": "…(truncated)"} ``` So a `"truncated"` key means *there was more* — narrow the query (a tighter `limit`, a time bound, a single id) rather than assuming you saw everything. In the pathological case where there are no lists left to shrink, a **successful** payload becomes `{"truncated": "…(truncated)", "data": ""}` — still parseable, deliberately obvious. Any top-level `error`, `ref` or `status` it carries stays a real key beside those two rather than going into `data`. A **failed** call is handled differently, and never loses its reason: see [A truncated failure still tells you why](#a-truncated-failure-still-tells-you-why). ### "The call failed" vs. "the run I asked about failed" These are different facts and Neens reports them in different places. Read them in this order: 1. **`isError` on the JSON-RPC result** answers *did the tool call work?* 2. **Fields inside the payload** — `status`, `error` — answer *what did I learn about the thing I asked for?* A failed **call** is never a protocol error and never crashes the session: the reply carries `"isError": true` and the text block holds `{"error": ""}`, so the agent can read it and recover. A payload `status` is present only when the failure came back from the underlying REST route — a non-2xx adds `{"status": }`. The failures raised before any route is reached (an unknown tool name, a missing required argument, a tool that refuses its own arguments, an executor that threw) carry `error` alone, so key off `isError` and treat `status` as optional. A `403` always returns the same fixed sentence rather than echoing the route's own message — the credential's permissions are not something to probe by reading error text. **A payload field named `error` is data, not a verdict.** `get_fix_run`, `get_prompt_optimization`, `get_model_sweep` and their `start_…` siblings all project the run's own error string at the top level. It is `null` on a healthy run and a real message on a run that broke — and in *both* cases the tool call itself succeeded, so `isError` is `false`. A healthy run — the call worked, and so did the run: ```json { "jsonrpc": "2.0", "id": 7, "result": { "isError": false, "content": [{ "type": "text", "text": "{\"id\":\"fxr_9c21\",\"status\":\"pr_opened\",\"attempt\":1,\"passK\":2,\"prUrl\":\"https://github.com/acme/agent/pull/412\",\"failureReport\":null,\"error\":null}" }] } } ``` A run that failed — the call still worked, and `error` is how you find out why: ```json { "jsonrpc": "2.0", "id": 8, "result": { "isError": false, "content": [{ "type": "text", "text": "{\"id\":\"fxr_9c22\",\"status\":\"failed\",\"attempt\":3,\"passK\":2,\"prUrl\":null,\"failureReport\":\"pass^2 not met: 1/2 green\",\"error\":\"patch did not apply to origin/main\"}" }] } } ``` A call that failed — no payload to read, just the reason: ```json { "jsonrpc": "2.0", "id": 9, "result": { "isError": true, "content": [{ "type": "text", "text": "{\"error\":\"fix run not found\",\"status\":404}" }] } } ``` **How to handle a `tools/call` reply, in order:** 1. If `isError` is `true`, the call did not happen — parse `error` (and `status`, if present) and decide whether to retry. Retrying is only useful for a `5xx`; a `403`, a `404` or a missing argument will fail identically forever. 2. Otherwise you have a real payload. If the tool reports on a *run* (`get_fix_run`, `get_prompt_optimization`, `get_model_sweep`), branch on the payload's `status` field, then read `error` for the reason when that status is a failure. `get_verification_run` is the exception: it reports a gate rather than a run error, so its detail is `passed` plus `reasons[]` and it projects no `error` field at all. 3. Never treat the *presence* of an `error` key as a failure — on a healthy run it is present and `null`. Up to and including Neens **0.10.0**, a payload carrying `"error": null` was misreported as `"isError": true`, so a successful `get_fix_run` / `get_prompt_optimization` / `get_model_sweep` looked like a failed call. If your client works around that by ignoring `isError` for those tools, remove the workaround — `isError` is now the authoritative answer for every tool. `get_trace` is the one read that spells the field differently: a trace's own error message rides as `errorMessage` (with per-tool-call `error` strings inside `toolCalls[]`). That name predates the rule above and is kept for compatibility. The in-app [Assistant](/guides/assistant) runs the same substrate over its own, tighter policy — 6,000 characters per result, 300-character strings, 20-entry lists, and it additionally drops the evidence, artifact, proposal, proof and raw-output fields that the MCP surface keeps. That is deliberate: the Assistant summarizes for a human reading chat, while an MCP client asked for those payloads on purpose and is the one that has to act on them. ### An unexpected failure is redacted to a reference id A tool can fail in a way nobody wrote a message for — the database is unreachable, an upstream connection is refused, a query names a column that isn't there. Those failures still come back as an ordinary failed call (`"isError": true`, a text block holding `{"error": …}`), but the message is deliberately the same fixed sentence every time, plus a `ref`: ```json { "jsonrpc": "2.0", "id": 11, "result": { "isError": true, "content": [{ "type": "text", "text": "{\"error\":\"the tool failed unexpectedly (ref: 4b1c8ad02e7f)\",\"ref\":\"4b1c8ad02e7f\"}" }] } } ``` The underlying exception is **not** relayed. Its message routinely carries the server's database file path, an internal hostname and port, or the name of a tenant's ClickHouse database — and a tool result travels straight into an agent's context window and whatever that client stores, which is a one-way trip. So the detail stays server-side and the `ref` is the handle to it. For an agent, `ref` is what to quote. There is nothing to parse and nothing to probe: treat the call as failed, do **not** retry it blindly (an unexpected failure repeats until an operator fixes it), and if you report the problem to a human, include the `ref` verbatim. The same rule applies one layer up. If the JSON-RPC dispatch itself fails, the protocol error is `{"code": -32603, "message": "Internal error (ref: )"}` — same handle, same reason. **Operators: finding the detail.** Every redacted failure writes one `ERROR` line to the `neens.agent_tools` logger on the server that handled the call, carrying the full exception message, its traceback, the tool name and the same `ref`. Grep your application logs for the id the agent quoted: ```bash # container / compose docker compose logs web | grep 4b1c8ad02e7f # kubernetes kubectl logs deploy/neens-web | grep 4b1c8ad02e7f ``` The line reads `agent tool failure ref= where=tool:: : `, followed by the traceback. There is no flag that turns the detail back on for the client — the redaction is the contract, and the log is where the answer lives. Failures Neens *does* have words for are unchanged and still say what to do: an unknown tool name, a missing required argument, a tool refusing its own arguments, a `403`, a `404` and every other non-2xx all come back with their own message (and a `status`, when a REST route produced it). Only the unanticipated ones are redacted. ### A truncated failure still tells you why Compaction and failure reporting are two rules that used to be able to contradict each other. They no longer can, and the guarantee is worth writing your client against: > **Whenever `isError` is `true`, the text block parses to a JSON object with a top-level `error` > key holding a non-empty string.** Truncation can shorten that string — never move it, never drop > it. The reason is the *last* thing a result gives up. When a failure is too large for the cap, Neens sacrifices in this order: the bulk payload is halved and then demoted into a `data` string, the `truncated` marker goes, `status` goes, and what remains is the reason — shortened with the usual trailing `…` if it has to be, but never emptied. A redacted failure's `ref` (above) is never sliced, because half a correlation id identifies nothing: it survives whole past `status`, and is given up whole only after the reason has already been shortened. So a large failure looks like this: ```json { "jsonrpc": "2.0", "id": 11, "result": { "isError": true, "content": [{ "type": "text", "text": "{\"error\":\"upstream refused the patch: pass^2 not met on attempt 3 of 3\",\"status\":502,\"truncated\":\"…(truncated)\",\"data\":\"{\\\"raw\\\":{\\\"attempts\\\":[…\"}" }] } } ``` `error` and `status` are real keys you can read directly. `truncated` says context was dropped, and `data` is whatever was demoted — a **sliced JSON dump carried as a plain string**, so treat it as opaque diagnostic text and never as something to parse. **For a client author:** 1. Branch on `isError` first. If it is `true`, `JSON.parse` the text block and read `error` — it is always there and always non-empty, so there is no "unknown failure" branch to write. 2. Read `status` if present to decide whether to retry (`5xx` yes; `403`/`404`/a bad argument will fail identically forever). Its absence means the failure happened before any route was reached, not that the status was hidden. 3. Treat `truncated` as informational: context was dropped and `data`, if present, is partial. Do not parse `data`, and do not read the failure out of it. Note that `truncated` present means the *reason* is whole — the marker is given up before the message is, so the two never appear abridged together. 4. A reason ending in `…` is a shortened message, not a different one — it is always a prefix of the full text. It also means every droppable key had already been dropped, so do not read the absence of `status` as a signal in that case. The in-app [Assistant](/guides/assistant) makes the same promise under a 6,000-character cap, so a failure the Assistant reports is readable for exactly the same reason. ================================================================================ # Connect Claude Code Source: /docs/guides/connect-claude-code/ ================================================================================ # Connect Claude Code (MCP) Neens exposes a [Model Context Protocol (MCP) server](/guides/mcp) so Claude Code can read your failure→fix loop and write back to it with tools instead of hand-written HTTP calls. The **secure, recommended** way to connect is a per-**user** browser sign-in: you run one command, Claude Code opens a browser to sign in to Neens, you pick an agent, and Claude Code is then authorized **as you** — not as the bearer of a shared key. Every tool call it makes is attributable to your account and governed by your role. ## At a glance | | | | --- | --- | | **Command** | `claude mcp add --transport http neens /mcp` | | **How you sign in** | A browser window to Neens (which may itself be [SSO](/administration/single-sign-on)) | | **What you're authorized as** | **You** — your account, your role, on the agent you pick | | **Manage authorizations** | **Settings → MCP access** — view and revoke every connection | | **No shared secret** | Nothing to paste, nothing to store; the authorization is bound to you | This replaces pasting a shared `nk_live_` agent key into the client. A key names an agent but no person; the OAuth sign-in binds the connection to **you**, so what Claude Code can do is exactly what you can do in that agent, and you can revoke it at any time. The [agent-key method](/guides/mcp#connect) still works for headless automation that has no human to sign in. ## Connect ### Add the Neens MCP server Run this in your terminal, substituting your Neens URL: ```bash claude mcp add --transport http neens https://YOUR_NEENS_HOST/mcp ``` Note there is **no** `--header` and no key — the whole point is that you authorize interactively. ### Sign in through your browser The first time Claude Code uses the server it opens a browser window to sign in to Neens. Sign in the way you normally do — with your email and password, or **Continue with SSO** if your company uses [single sign-on](/administration/single-sign-on). If your company federates login, you sign in through your own identity provider and never enter a separate Neens password — see [Sign in with your company's single sign-on](#sign-in-with-your-companys-single-sign-on). ### Pick an agent and approve Neens shows a short consent screen: **Authorize Claude Code to act as you**, with a picker for which of your agents to scope this connection to. Choose the agent and click **Allow**. The browser hands control back to Claude Code, which is now connected. ### Verify Start a session and ask Claude Code to list the Neens tools. You should see the Neens MCP tools appear, namespaced as `mcp__neens__…`. Seeing none means the authorization didn't complete — re-run the sign-in. The authorization is scoped to the **one agent** you picked, exactly like an agent key — Claude Code can read and write within that agent and nothing else, and it can never delete. To work in a different agent, add a second connection (give it a different server name, e.g. `neens-staging`) and pick that agent during its sign-in. ## Sign in with your company's single sign-on If your company routes its email domains to an identity provider — Okta, Azure AD / Microsoft Entra, Google Workspace, or any [SSO connection](/administration/single-sign-on) — connecting Claude Code uses that same login. There is **no separate Neens password** to create or remember. ### Run the same command ```bash claude mcp add --transport http neens https://YOUR_NEENS_HOST/mcp ``` Nothing changes on the command line — federation is decided by your email domain, not by a flag you pass here. ### Sign in through your own identity provider Claude Code opens the browser and Neens recognizes your company by your work email. Instead of a Neens password prompt you go **straight to your identity provider** and sign in there exactly as you do for every other company app — including whatever MFA your IdP enforces. If your company also requires a Neens second factor on top of the IdP, you enter that code before you continue. ### Approve and pick an agent Your IdP returns you to Neens, which shows the same consent screen — **Authorize Claude Code to act as you**, with the agent picker — and you click **Allow**. Claude Code is now connected as you. **Neens brokers the connection; it never holds your IdP credentials.** Your identity provider authenticates you, and *Neens* then issues Claude Code its own short-lived access token bound to you, your role, and the one agent you picked. Claude Code never receives a token from your IdP, and your IdP never sees Claude Code — you get an ordinary Neens authorization that your role governs and that you can revoke, established through a login you already trust. ## Staying connected Your authorization is **time-limited**, but Claude Code refreshes it for you — you do **not** have to sign in again every hour. Under the hood the access token is short-lived and Claude Code silently renews it in the background with a longer-lived refresh token, so a working connection keeps working across a day of use with no interruption. Eventually the refresh window itself ends (or you [revoke](#manage-your-mcp-authorizations) the connection, or an admin removes your access). When that happens Claude Code reports something like **your session expired, sign in again** on its next tool call. That is expected and harmless — re-run the sign-in and Claude Code reconnects: ```bash claude mcp add --transport http neens https://YOUR_NEENS_HOST/mcp ``` Nothing you do keeps a token alive past its limits — that is deliberate. A connection that could never expire would be a standing key; a bounded one that quietly refreshes while you work, and asks you to sign in again when the window closes, is what keeps every action attributable to a live login. ## What Claude Code can do over MCP Once connected, Claude Code drives the same [MCP tools](/guides/mcp#what-an-agent-can-do) any MCP client gets — read your traces, failure modes and clusters, curate datasets, stand up judges, run pre-prod evaluations, move remediations forward, and close the fix loop by verifying a fix against your preview deploy. The difference is only in **who** it acts as: every one of those actions is attributable to you and admitted by your role, the same as if you'd done it in the app. Reads are open to your credential; the writes it may perform are the same non-destructive ones the [MCP server guide](/guides/mcp#what-an-agent-can-do) lists, and **no tool deletes anything**. ## Manage your MCP authorizations Every connection you approve is listed under **Settings → MCP access** as an **active MCP connection**, showing the client, the agent it's scoped to, and when it was created, last used, and expires. - **Revoke** any connection to disconnect that Claude Code session immediately — the next tool call it attempts is rejected and it will have to sign in again to reconnect. - The list is **yours**: it shows your own authorizations only, and revoking one never affects another person's connections. - Authorizations are time-limited and drop off the list once they expire; revoking is the way to end one early. Revoke a connection you no longer recognize the same way you'd revoke any credential. Because the authorization is bound to your account, a revoke is complete — there is no shared key still floating around that would keep working. ## Related - [MCP server](/guides/mcp) — the full tool catalogue, the fix loop over MCP, and the agent-key connection method for headless use. - [Single sign-on (SSO)](/administration/single-sign-on) — the browser sign-in this flow uses when your company federates login to an IdP. - [Assistant](/guides/assistant) — the same tools, in-app, over a chat interface. - [API keys](/administration/api-keys) — agent keys for scripts and CI that have no human to sign in. ================================================================================ # API reference Source: /docs/api-reference/ ================================================================================ # API reference The complete Neens HTTP API — **469 endpoints**, generated from the live OpenAPI schema (version `0.16.6`) so it can't drift from the running service. ## Authentication Programmatic and ingest calls authenticate with a **project API key** (`nk_live_…`) sent as a bearer token: ```bash curl https://YOUR-NEENS-HOST/sessions \ -H "Authorization: Bearer nk_live_your_key_here" ``` Create keys under **Settings → API keys** (see [API keys](/administration/api-keys)). The in-app UI uses `nk_sess_…` session cookies instead. A key resolves to exactly one company + project, so it can't read or write another tenant's data. ## Explore it Every page below is generated from the live FastAPI schema, published at [`/docs/openapi.json`](/openapi.json). Regenerate the reference with `make docs`. - **Read it here** — browse by pillar in the sidebar; each endpoint lists its parameters, request body, and responses. - **Download the spec** — [`/docs/openapi.json`](/openapi.json) for Postman, code generation, or feeding an agent. - **Try it live** — on a local or development run, a Neens instance serves interactive **Swagger UI** at `/api-docs` and **ReDoc** at `/api-redoc`. On a hosted instance these interactive explorers may not be reachable — use the downloadable spec above instead. ## For agents & LLMs Neens publishes [llms.txt](https://llmstxt.org) files so coding agents and LLM tools can discover and read the docs efficiently: - [`/docs/llms.txt`](/llms.txt) — a curated map of every documentation page. - [`/docs/llms-full.txt`](/llms-full.txt) — the entire documentation as one plain-text file, ready to paste into a model's context. Point your agent at these alongside the [`openapi.json`](/openapi.json) above. ## Versioning This reference tracks platform release **`0.16.6`** (the schema's `info.version`). Breaking changes ship with a release bump. The OTLP ingest path `/v1/traces` follows the stable OpenTelemetry contract and is safe to hard-code. ## Sections - [Ingest & observe](/api-reference/ingest-observe) — Send traces and read back sessions, conversations, agents, and the agent topology. - [Diagnose](/api-reference/diagnose) — Failure clustering, the failure-mode taxonomy and Issues, topics, and insights. - [Evaluate](/api-reference/evaluate) — Judges, scores, enrichments, datasets, human annotations & review, and pre-prod eval runs. - [Fix & ship](/api-reference/fix) — Typed remediations, the evals-from-failures flywheel, and deploy-event correlation. - [Analyze](/api-reference/analyze) — Custom dashboards, the metrics catalogue, and saved analyses. - [Assistant & prompts](/api-reference/assistant) — The in-app chat assistant — questions plus approval-gated writes — and the versioned prompt registry. - [Administration](/api-reference/administration) — Tenancy, members & auth, personas, settings & LLM connections, retention, and the audit log. - [More endpoints](/api-reference/more) — Additional endpoints not grouped above. ================================================================================ # Ingest & observe — API Source: /docs/api-reference/ingest-observe/ ================================================================================ # Ingest & observe — API reference Send traces and read back sessions, conversations, agents, and the agent topology. Generated from the live OpenAPI schema (version `0.16.6`). Download the full spec at [`/docs/openapi.json`](/openapi.json). ## ingest ### `POST /ingest/batch` Ingest Batch High-volume batch ingest: many raw sessions in one request. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `sync` | query | boolean | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /ingest/openinference` Ingest Openinference Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `sync` | query | boolean | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /ingest/otlp` Ingest Otlp Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `sync` | query | boolean | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /ingest/raw` Ingest Raw Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `sync` | query | boolean | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /v1/traces` Otlp Traces OTLP/HTTP trace receiver. Accepts protobuf or JSON; enqueues and returns an OTLP response. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `sync` | query | boolean | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ## traces ### `GET /agent-graph` Cohort Agent Graph Cross-cohort aggregate agent/tool topology. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `status` | query | string[] | no | | | `agent_name` | query | string \| null | no | | | `source` | query | string \| null | no | | | `min_turns` | query | integer \| null | no | | | `max_turns` | query | integer \| null | no | | | `min_duration_ms` | query | integer \| null | no | | | `max_duration_ms` | query | integer \| null | no | | | `min_spans` | query | integer \| null | no | | | `max_spans` | query | integer \| null | no | | | `min_tokens` | query | integer \| null | no | | | `max_tokens` | query | integer \| null | no | | | `min_input_tokens` | query | integer \| null | no | | | `max_input_tokens` | query | integer \| null | no | | | `min_output_tokens` | query | integer \| null | no | | | `max_output_tokens` | query | integer \| null | no | | | `model` | query | string[] | no | | | `tool_name` | query | string[] | no | | | `span_kind` | query | string[] | no | | | `span_status` | query | string[] | no | | | `tags` | query | string[] | no | | | `cluster_id` | query | string \| null | no | | | `failure_set` | query | boolean | no | | | `bucket` | query | string \| null | no | | | `conversation_id` | query | string \| null | no | | | `session_id` | query | string \| null | no | | | `score_metric` | query | string \| null | no | | | `score_status` | query | string \| null | no | | | `score_label` | query | string \| null | no | | | `issue_mode` | query | string \| null | no | | | `version` | query | string \| null | no | | | `hour` | query | string \| null | no | | | `started_after` | query | string \| null | no | | | `started_before` | query | string \| null | no | | | `enrichment` | query | string[] | no | | | `q` | query | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /sessions` List Sessions Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `status` | query | string[] | no | | | `agent_name` | query | string \| null | no | | | `source` | query | string \| null | no | | | `page` | query | integer | no | | | `page_size` | query | integer | no | | | `min_turns` | query | integer \| null | no | | | `max_turns` | query | integer \| null | no | | | `min_duration_ms` | query | integer \| null | no | | | `max_duration_ms` | query | integer \| null | no | | | `min_spans` | query | integer \| null | no | | | `max_spans` | query | integer \| null | no | | | `min_tokens` | query | integer \| null | no | | | `max_tokens` | query | integer \| null | no | | | `min_input_tokens` | query | integer \| null | no | | | `max_input_tokens` | query | integer \| null | no | | | `min_output_tokens` | query | integer \| null | no | | | `max_output_tokens` | query | integer \| null | no | | | `model` | query | string[] | no | | | `tool_name` | query | string[] | no | | | `span_kind` | query | string[] | no | | | `span_status` | query | string[] | no | | | `tags` | query | string[] | no | | | `cluster_id` | query | string \| null | no | | | `failure_set` | query | boolean | no | | | `bucket` | query | string \| null | no | | | `conversation_id` | query | string \| null | no | | | `session_id` | query | string \| null | no | | | `score_metric` | query | string \| null | no | | | `score_status` | query | string \| null | no | | | `score_label` | query | string \| null | no | | | `issue_mode` | query | string \| null | no | | | `version` | query | string \| null | no | | | `hour` | query | string \| null | no | | | `started_after` | query | string \| null | no | | | `started_before` | query | string \| null | no | | | `enrichment` | query | string[] | no | | | `q` | query | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /sessions/filter-options` Filter Options Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `GET /sessions/trend` Sessions Trend Volume trend for the Traces page: per-day (or per-hour) trace counts, token usage (input and output), p50 latency and estimated spend over the selected time window, plus a period-over-period summary. Counts individual traces. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `range` | query | string \| null | no | | | `from` | query | string \| null | no | | | `to` | query | string \| null | no | | | `started_after` | query | string \| null | no | | | `started_before` | query | string \| null | no | | | `grain` | query | string \| null | no | | | `status` | query | string \| null | no | | | `model` | query | string \| null | no | | | `agent_name` | query | string \| null | no | | | `source` | query | string \| null | no | | | `version` | query | string \| null | no | | | `q` | query | string \| null | no | | | `group_by` | query | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /sessions/{session_id}` Get Session Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `session_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /sessions/{session_id}/agent-graph` Agent Graph Aggregate agent/tool topology for one run. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `session_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /sessions/{session_id}/feedback` Submit Feedback Record end-user feedback on a conversation — a CSAT rating or a thumbs up/down — without a helpdesk integration. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `session_id` | path | string | yes | | **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `rating` | number \| null | no | | | `thumb` | string \| null | no | | | `comment` | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `201` | Successful Response | | `422` | Validation Error | ### `GET /sessions/{session_id}/similar` Similar Sessions Sessions most similar to ``session_id`` ("more like this"), up to ``limit``. Returns ``[]`` when similarity search is unavailable. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `session_id` | path | string | yes | | | `limit` | query | integer | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ## conversations ### `GET /conversations` List Conversations Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `status` | query | string[] | no | | | `agent_name` | query | string \| null | no | | | `source` | query | string \| null | no | | | `conversation_id` | query | string \| null | no | | | `session_id` | query | string \| null | no | | | `page` | query | integer | no | | | `page_size` | query | integer | no | | | `min_turns` | query | integer \| null | no | | | `max_turns` | query | integer \| null | no | | | `min_duration_ms` | query | integer \| null | no | | | `max_duration_ms` | query | integer \| null | no | | | `min_spans` | query | integer \| null | no | | | `max_spans` | query | integer \| null | no | | | `min_tokens` | query | integer \| null | no | | | `max_tokens` | query | integer \| null | no | | | `min_input_tokens` | query | integer \| null | no | | | `max_input_tokens` | query | integer \| null | no | | | `min_output_tokens` | query | integer \| null | no | | | `max_output_tokens` | query | integer \| null | no | | | `model` | query | string[] | no | | | `tool_name` | query | string[] | no | | | `span_kind` | query | string[] | no | | | `span_status` | query | string[] | no | | | `tags` | query | string[] | no | | | `cluster_id` | query | string \| null | no | | | `score_metric` | query | string \| null | no | | | `score_status` | query | string \| null | no | | | `score_label` | query | string \| null | no | | | `issue_mode` | query | string \| null | no | | | `version` | query | string \| null | no | | | `hour` | query | string \| null | no | | | `started_after` | query | string \| null | no | | | `started_before` | query | string \| null | no | | | `q` | query | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /conversations/trend` Conversations Trend Volume trend for the Sessions page: per-day (or per-hour) session counts, token usage (input and output), p50 latency and estimated spend over the selected time window, plus a period-over-period summary. Counts distinct sessions. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `range` | query | string \| null | no | | | `from` | query | string \| null | no | | | `to` | query | string \| null | no | | | `started_after` | query | string \| null | no | | | `started_before` | query | string \| null | no | | | `grain` | query | string \| null | no | | | `status` | query | string \| null | no | | | `model` | query | string \| null | no | | | `agent_name` | query | string \| null | no | | | `source` | query | string \| null | no | | | `version` | query | string \| null | no | | | `q` | query | string \| null | no | | | `group_by` | query | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /conversations/{conversation_id}` Get Conversation Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `conversation_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /conversations/{conversation_id}/agent-graph` Conversation Agent Graph Aggregate agent/tool topology for a whole conversation. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `conversation_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /conversations/{conversation_id}/annotations` Add Conversation Annotation Create one annotation (targetType 'session', targetId = member trace id) per member trace of the conversation, mirroring api/annotations.py's contract + reviewer attribution. Returns the created count. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `conversation_id` | path | string | yes | | **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `value` | string | yes | | | `comment` | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /conversations/{conversation_id}/labels` Add Conversation Label Apply a tag to EVERY member trace of the conversation (idempotent, mirroring the single-trace ``POST /sessions/{id}/labels`` path). The written ``session_labels`` rows make the tag show up in the Sessions/Traces "Tags" filter. Returns the applied count. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `conversation_id` | path | string | yes | | **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `label` | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ## agents ### `GET /agents` List Agents Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ## labels ### `GET /labels` List Labels Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `POST /sessions/{session_id}/labels` Add Label Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `session_id` | path | string | yes | | **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `label` | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `DELETE /sessions/{session_id}/labels/{label}` Remove Label Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `session_id` | path | string | yes | | | `label` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ## activity ### `GET /activity/runs` List Runs Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `kind` | query | string \| null | no | | | `state` | query | string \| null | no | | | `status` | query | string \| null | no | | | `started_after` | query | string \| null | no | | | `started_before` | query | string \| null | no | | | `sort_by` | query | string \| null | no | | | `sort_dir` | query | string \| null | no | | | `limit` | query | integer | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /activity/trend` Activity Trend Event-volume trend for the Activity page: per-day (or per-hour) count of activity runs over the selected window, plus a period-over-period total. Scoped to the current project like the feed. Queued-but-unstarted runs are excluded — they have no place on a time axis yet. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `range` | query | string \| null | no | | | `from` | query | string \| null | no | | | `to` | query | string \| null | no | | | `grain` | query | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /eval-runs/{run_id}/metrics` Run Metrics Real pipeline metrics for a score run, derived from activity_runs + eval_tasks (no synthetic data). ``run_id`` is the activity-run id (``act_eval_{eval_run_id}``). Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `run_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ## stats ### `GET /stats` Get Stats Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `range` | query | string | no | | | `from` | query | string \| null | no | | | `to` | query | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /stats/timeseries` Get Timeseries Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `range` | query | string | no | | | `from` | query | string \| null | no | | | `to` | query | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ## overview ### `GET /overview/kpis` Overview Kpis Hero KPI row: spine counts + scope-wide trace/session/error/spend totals + deltas. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `range` | query | string | no | | | `from` | query | string \| null | no | | | `to` | query | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /overview/segments` Overview Segments Segmentation table: scope sliced by org | project | agent. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `by` | query | string | no | | | `range` | query | string | no | | | `from` | query | string \| null | no | | | `to` | query | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /overview/timeseries` Overview Timeseries Tenant trends: per-day traces / sessions / errors / eval pass-rate / spend over the range. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `range` | query | string | no | | | `from` | query | string \| null | no | | | `to` | query | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ================================================================================ # Diagnose — API Source: /docs/api-reference/diagnose/ ================================================================================ # Diagnose — API reference Failure clustering, the failure-mode taxonomy and Issues, topics, and insights. Generated from the live OpenAPI schema (version `0.16.6`). Download the full spec at [`/docs/openapi.json`](/openapi.json). ## clusters ### `GET /clusters` List Clusters Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `POST /clusters/assign` Assign Sessions Classify sessions against the project's existing failure clusters without re-clustering. Each result is ``assigned`` / ``novel`` / ``unassigned`` / ``no_model`` / ``unavailable`` / ``not_found``; a ``novel`` session raises a new-failure-mode insight. Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `session_ids` | string[] | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /clusters/configs` List Configs List the project's clustering configs. ``lifecycle`` filters by stage (``finalized`` default | ``experimental`` | ``all``) — but the ACTIVE config is ALWAYS included (never hide what runs). Sort: active first, then updated_at desc. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `lifecycle` | query | string | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /clusters/configs` Create Config Create a new config (stamped ``experimental``, ``is_active=0``, knobs defaulting from ``CLUSTER_SETTINGS_DEFAULTS`` and clamped like the settings patch). 422 when name is blank. Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `name` | string | yes | | | `description` | string \| null | no | | | `enabled` | boolean \| null | no | | | `samplingPct` | number \| null | no | | | `windowDays` | integer \| null | no | | | `maxSessions` | integer \| null | no | | | `minFailureSessions` | integer \| null | no | | | `minClusterSize` | integer \| null | no | | | `scoreThreshold` | number \| null | no | | | `selectionMode` | string \| null | no | | | `dedupLabels` | boolean \| null | no | | | `scoreMetricKey` | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /clusters/configs/estimate` Estimate Config Cheap eligible-failure-set count for an UNSAVED knob set — powers the fine-tune preview (a count only, no ML dry-run). Degrades to ``{eligible: 0,...}`` on any error. Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `enabled` | boolean \| null | no | | | `samplingPct` | number \| null | no | | | `windowDays` | integer \| null | no | | | `maxSessions` | integer \| null | no | | | `minFailureSessions` | integer \| null | no | | | `minClusterSize` | integer \| null | no | | | `scoreThreshold` | number \| null | no | | | `selectionMode` | string \| null | no | | | `dedupLabels` | boolean \| null | no | | | `scoreMetricKey` | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /clusters/configs/{config_id}` Get Config A single config. 404 when it belongs to another project. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `config_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `PATCH /clusters/configs/{config_id}` Patch Config Edit a config's name/description/knobs (same clamps + nullable-meaningful scoreMetricKey as the settings patch). 404 out-of-scope; 422 on a blanked name. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `config_id` | path | string | yes | | **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `name` | string \| null | no | | | `description` | string \| null | no | | | `enabled` | boolean \| null | no | | | `samplingPct` | number \| null | no | | | `windowDays` | integer \| null | no | | | `maxSessions` | integer \| null | no | | | `minFailureSessions` | integer \| null | no | | | `minClusterSize` | integer \| null | no | | | `scoreThreshold` | number \| null | no | | | `selectionMode` | string \| null | no | | | `dedupLabels` | boolean \| null | no | | | `scoreMetricKey` | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `DELETE /clusters/configs/{config_id}` Delete Config Delete a config. Refused (409) when it is the ACTIVE one (activate another first) or the project's ONLY config. 404 out-of-scope. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `config_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /clusters/configs/{config_id}/activate` Activate Config Make this config the project's active one — atomically deactivating every other config in the project (enforces the exactly-one-active invariant). 404 out-of-scope. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `config_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `PATCH /clusters/configs/{config_id}/lifecycle` Patch Config Lifecycle Move a config through its curation lifecycle: experimental → finalized → archived. A LABEL ONLY — never changes what runs, only the default list visibility. 404 out-of-scope; 422 on an unknown stage. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `config_id` | path | string | yes | | **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `stage` | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /clusters/failure-modes` Failure Modes Hero The Failure Modes hero page (plan B1) in one call: readiness (zero-state), the ranked active failure-mode clusters enriched with root-cause hypothesis / typed-remediation chip / daily-volume trend / NEW flag, plus honest freshness ("Last analyzed") and sample-based estimate reporting. Project-scoped. Purely additive — no storage change (the Cluster model already carries root_cause/remediation/first_seen_at/status). Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `range` | query | string \| null | no | today\|24h\|7d\|30d\|all\|custom | | `from` | query | string \| null | no | | | `to` | query | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /clusters/readiness` Readiness Backs the Failure Modes zero-state (plan B1): the windowed failure-set count vs the first-run threshold, plus whether the first run has landed. Project-scoped. Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `POST /clusters/run` Run Clustering Manually trigger a clustering run. Project-scoped. Details **Request body** (`application/json`) Schema: `object`. **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /clusters/scatter` Scatter Points Cluster-member scatter. Each point carries its cluster's L1 ``bucket`` key and the response carries a finite ``buckets`` legend ``[{key, label, count}]`` (count = points, sorted desc) so the UI can colour by ~10 buckets instead of an unbounded label set. Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `GET /clusters/settings` Get Settings Endpoint Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `PATCH /clusters/settings` Patch Settings Endpoint Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `enabled` | boolean \| null | no | | | `samplingPct` | number \| null | no | | | `windowDays` | integer \| null | no | | | `maxSessions` | integer \| null | no | | | `minFailureSessions` | integer \| null | no | | | `minClusterSize` | integer \| null | no | | | `scoreThreshold` | number \| null | no | | | `selectionMode` | string \| null | no | | | `dedupLabels` | boolean \| null | no | | | `scoreMetricKey` | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /clusters/{cluster_id}` Get Cluster Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `cluster_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `PATCH /clusters/{cluster_id}` Rename Cluster Endpoint Rename a cluster and PIN the name: a pinned label survives future clustering relabels. Requires write access, object-level scope 404, 422 on a blank label. Declared after ``/settings`` + ``/configs`` so PATCH ``/{cluster_id}`` never shadows them. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `cluster_id` | path | string | yes | | **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `label` | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /clusters/{cluster_id}/exemplars` Cluster Exemplars Representative sessions of a cluster, most typical first, capped at ``limit``. A cluster in another project returns 404. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `cluster_id` | path | string | yes | | | `limit` | query | integer | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /clusters/{cluster_id}/merge` Merge Clusters Endpoint Manually merge one or more source clusters INTO ``cluster_id``. Mirrors the taxonomy failure-mode merge: MANAGE_TAXONOMY-gated, object-level scope (target + each source must be in the caller's project — else 404), a cross-project guard (a source in a different project 404s), and 422 on an empty source set or a self-merge. Members are reassigned + deduped by ``storage.merge_clusters``; sources are soft-resolved. Returns the merged target cluster. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `cluster_id` | path | string | yes | | **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `source_ids` | string[] \| null | no | | | `source_id` | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ## taxonomy ### `GET /taxonomy/failure-modes` List Failure Modes Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `status` | query | string \| null | no | | | `severity` | query | string \| null | no | | | `provenance` | query | string \| null | no | | | `q` | query | string \| null | no | Case-insensitive substring match on name | | `sort_by` | query | string \| null | no | | | `sort_dir` | query | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /taxonomy/failure-modes` Create Failure Mode Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `cluster_id` | string \| null | no | | | `name` | string | yes | | | `definition` | string \| null | no | | | `severity` | string \| null | no | | | `owner` | string \| null | no | | | `status` | string \| null | no | | | `provenance` | string | no | | | `compliance_tags` | string[] \| null | no | | | `exemplar_session_ids` | string[] \| null | no | | | `judge_id` | string \| null | no | | | `dataset_id` | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /taxonomy/failure-modes/import` Import Failure Modes Endpoint Bulk-import a starter taxonomy (Phase C1). Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `modes` | FailureModeImportItem[] | no | | | `include_starter_library` | boolean | no | | | `status` | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /taxonomy/failure-modes/import-file` Import Failure Modes File Bulk-import a taxonomy from an uploaded **file** (CSV or JSON) — the enterprise "upload our failure taxonomy" path. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `format` | query | string \| null | no | csv\|json — inferred from Content-Type/content when omitted | | `status` | query | string | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /taxonomy/failure-modes/import-template` Failure Modes Import Template Download the CSV import template (header + two illustrative rows). Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `GET /taxonomy/failure-modes/{mode_id}` Get Failure Mode Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `mode_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `PATCH /taxonomy/failure-modes/{mode_id}` Patch Failure Mode Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `mode_id` | path | string | yes | | **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `name` | string \| null | no | | | `definition` | string \| null | no | | | `severity` | string \| null | no | | | `owner` | string \| null | no | | | `status` | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `DELETE /taxonomy/failure-modes/{mode_id}` Delete Failure Mode Hard-delete a failure mode and its cluster-evidence links (project-scoped, audited). Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `mode_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /taxonomy/failure-modes/{mode_id}/links` Add Link Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `mode_id` | path | string | yes | | **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `cluster_id` | string | yes | | | `weight` | number \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `DELETE /taxonomy/failure-modes/{mode_id}/links/{cluster_id}` Delete Link Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `mode_id` | path | string | yes | | | `cluster_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /taxonomy/failure-modes/{mode_id}/merge` Merge Failure Mode Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `mode_id` | path | string | yes | | **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `source_id` | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /taxonomy/failure-modes/{mode_id}/split` Split Failure Mode Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `mode_id` | path | string | yes | | **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `name` | string | yes | | | `cluster_ids` | string[] | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /taxonomy/issues` List Issues Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `range` | query | string | no | today\|24h\|7d\|30d\|all\|custom | | `from` | query | string \| null | no | | | `to` | query | string \| null | no | | | `state` | query | string \| null | no | Filter by lifecycleState | | `severity` | query | string \| null | no | | | `q` | query | string \| null | no | Case-insensitive substring match on name | | `include_muted` | query | boolean | no | | | `include_resolved` | query | boolean | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /taxonomy/issues/{mode_id}` Get Issue Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `mode_id` | path | string | yes | | | `range` | query | string | no | | | `from` | query | string \| null | no | | | `to` | query | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `PATCH /taxonomy/issues/{mode_id}/lifecycle` Patch Issue Lifecycle Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `mode_id` | path | string | yes | | **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `state` | string | yes | | | `note` | string \| null | no | | | `owner` | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ## failure-modes ### `GET /failure-modes` Get Failure Modes Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ## topics ### `POST /topics` Create Topic Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `GET /topics/atlas` Get Atlas Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `space` | query | string \| null | no | | | `q` | query | string \| null | no | Case-insensitive substring match on name | | `severity` | query | string \| null | no | | | `source` | query | string \| null | no | | | `sort_by` | query | string \| null | no | | | `sort_dir` | query | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /topics/spaces` Create Space Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `GET /topics/{topic_id}` Get Topic Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `topic_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `PATCH /topics/{topic_id}` Patch Topic Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `topic_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /topics/{topic_id}/datasets` Create Dataset Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `topic_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /topics/{topic_id}/examples` Get Examples Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `topic_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /topics/{topic_id}/merge` Merge Topic Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `topic_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /topics/{topic_id}/split` Split Topic Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `topic_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ## insights ### `GET /insights/diagnose` Diagnose The Diagnose feed: ``{issues:[{projectId, type:'cluster'|'regression', label, affected, confidence, link, asOf}]}`` — failure clusters and score regressions across the accessible projects, filterable by ``range`` / ``from`` / ``to`` and repeatable ``project_id``. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `range` | query | string | no | | | `from` | query | string \| null | no | | | `to` | query | string \| null | no | | | `project_id` | query | string[] \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /insights/fix` Fix The Fix feed: ``{remediations:[{projectId, clusterLabel, summary, likelyCause, recommendation, confidence, affected}]}`` — suggested remediations for the latest failure clusters, highest-impact first. Repeatable ``project_id`` narrows the scope. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `range` | query | string | no | | | `from` | query | string \| null | no | | | `to` | query | string \| null | no | | | `project_id` | query | string[] \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /insights/observe` Observe ``{fleet:{sessions, errorRate, p99Ms, spendUsd}, anomalies:[…]}``. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `range` | query | string | no | | | `from` | query | string \| null | no | | | `to` | query | string \| null | no | | | `project_id` | query | string[] \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /insights/refresh` Refresh Run the anomaly + regression detectors now, within the caller's pinned tenant schema, over their accessible project set (``None`` ⇒ all projects in the schema). Persists to the insight store + dispatches newly-surfaced insights to the configured sinks. Returns ``{detected, surfaced}`` counts. This is the synchronous counterpart to the nightly scheduler job — useful for the demo and for a 'refresh now' UI affordance. Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `GET /insights/summary` Summary ``{headline, body, llm, model, stats, empty, cached, generatedAt, briefing}`` — a short briefing over the current fleet KPIs + top anomalies / issues / remediations for the scope + window. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `range` | query | string | no | | | `from` | query | string \| null | no | | | `to` | query | string \| null | no | | | `project_id` | query | string[] \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /insights/summary/briefing` Request Briefing Request generation of the LLM narrative for a scope + window. **Idempotent.** Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `range` | string \| null | no | | | `from` | string \| null | no | | | `to` | string \| null | no | | | `projectIds` | string[] \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /insights/summary/briefing/{briefing_id}` Get Briefing The cheap poll: ONE indexed row read, no measure or cluster queries — this is polled every few seconds while a narrative generates. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `briefing_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /insights/{insight_id}/feedback` Get Feedback List the feedback recorded for an insight (most recent first). A scoped member/viewer may only read feedback for an insight in a project they can access; an unknown id (incl. one outside the pinned tenant schema) returns an empty list — no existence leak. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `insight_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /insights/{insight_id}/feedback` Submit Feedback Record user feedback on a surfaced insight. ``verdict`` ∈ ``useful|actioned|not_useful|wrong|dismissed`` — a negative verdict (``not_useful``/ ``wrong``/``dismissed``) mutes the insight on the next detection pass until its signal resolves and recurs. 404 if the insight id is unknown. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `insight_id` | path | string | yes | | **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `verdict` | string | yes | | | `comment` | string \| null | no | | | `reviewer` | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ================================================================================ # Evaluate — API Source: /docs/api-reference/evaluate/ ================================================================================ # Evaluate — API reference Judges, scores, enrichments, datasets, human annotations & review, and pre-prod eval runs. Generated from the live OpenAPI schema (version `0.16.6`). Download the full spec at [`/docs/openapi.json`](/openapi.json). ## judges ### `POST /eval-runs` Create Eval Run Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `deployment_id` | string | yes | | | `pool_connection_ids` | string[] \| null | no | | | `success_threshold` | number \| null | no | | | `sample_size` | integer \| null | no | Cap THIS run at this many of the deployment's eligible traces (most-recent first), leaving the deployment's stored sampling filter unchanged — e.g. 1000 to score the 1000 most-recent eligible traces without editing the deployment. The response `total` is the number actually queued (smaller than sample_size when fewer traces are eligible). Omit to let the deployment's filter decide the count. | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /eval-runs/export` Export Eval Run Scores Export a judge run's (or several runs') scores WITH each target's input + output. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `runs` | query | string | yes | Comma-separated eval_run ids | | `format` | query | string | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /eval-runs/{run_id}` Get Eval Run Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `run_id` | path | string | yes | | | `limit` | query | integer | no | | | `offset` | query | integer | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /eval-runs/{run_id}/cancel` Cancel Eval Run Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `run_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /eval-targets/estimate` Estimate Targets Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `filter` | object | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /judges` List Judges Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `view` | query | string | no | | | `q` | query | string \| null | no | | | `type` | query | string[] \| null | no | | | `target` | query | string[] \| null | no | | | `level` | query | string[] \| null | no | | | `version` | query | string[] \| null | no | | | `configured` | query | string \| null | no | | | `trigger` | query | string[] \| null | no | | | `status` | query | string[] \| null | no | | | `running` | query | boolean \| null | no | | | `sort_by` | query | string \| null | no | | | `sort_dir` | query | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /judges` Create Judge Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `name` | string | yes | | | `type` | string | yes | | | `scope` | string | no | | | `description` | string | no | | | `level` | string | no | | | `definition` | object | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /judges/preview` Preview Judge Prompt Render the EXACT prompt a judge would send to the LLM, hydrated on a real sample session. Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `definition` | object | no | | | `target_type` | string \| null | no | | | `sample_session_id` | string \| null | no | | | `type` | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /judges/run-status` List Run Status Poll-friendly, status-ONLY projection of every enabled deployment. Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `POST /judges/test-run` Test Run Judge Prompt Execute the in-progress judge prompt against a real sample session ON THE FLY and return the model's verdict — WITHOUT persisting anything. Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `definition` | object | no | | | `target_type` | string \| null | no | | | `sample_session_id` | string \| null | no | | | `type` | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `PATCH /judges/{judge_id}` Patch Judge Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `judge_id` | path | string | yes | | **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `name` | string \| null | no | | | `description` | string \| null | no | | | `type` | string \| null | no | | | `scope` | string \| null | no | | | `level` | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `DELETE /judges/{judge_id}` Delete Judge Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `judge_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /judges/{judge_id}/compare` Compare Judge Runs Compare two eval runs of the SAME judge — typically two different versions. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `judge_id` | path | string | yes | | | `runA` | query | string | yes | | | `runB` | query | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /judges/{judge_id}/deployments` Create Deployment Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `judge_id` | path | string | yes | | **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `version_id` | string | yes | | | `scope` | string \| null | no | | | `trigger_policy` | string | no | | | `filter` | object | no | | | `pool_connection_ids` | string[] | no | | | `success_threshold` | number \| null | no | | | `run_target_id` | string \| null | no | | | `dedupe` | boolean | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `PATCH /judges/{judge_id}/deployments/{deployment_id}` Patch Deployment Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `judge_id` | path | string | yes | | | `deployment_id` | path | string | yes | | **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `status` | string \| null | no | | | `trigger_policy` | string \| null | no | | | `scope` | string \| null | no | | | `filter` | object \| null | no | | | `pool_connection_ids` | string[] \| null | no | | | `success_threshold` | number \| null | no | | | `run_target_id` | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `DELETE /judges/{judge_id}/deployments/{deployment_id}` Delete Deployment Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `judge_id` | path | string | yes | | | `deployment_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `PATCH /judges/{judge_id}/lifecycle` Patch Judge Lifecycle Promote/demote a judge through its lifecycle: experimental → finalized → archived. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `judge_id` | path | string | yes | | **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `stage` | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /judges/{judge_id}/runs` List Judge Runs All eval runs for a judge (current + historical), in the Activity-run shape so the UI can reuse the same row layout. Joins eval_runs → judge_deployments and left-joins activity_runs (`act_eval_{run_id}`) for name / duration / errors. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `judge_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /judges/{judge_id}/targets` List Judge Targets Per-binding aggregate for the scorer's "results by target" table: one row per enabled deployment with its Run Target summary and the latest scored/avg/passRate over that target's population. Project-scoped; NULL-not-zero on an empty population. A legacy inline binding (no Run Target) still appears, with ``runTarget: null``. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `judge_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /judges/{judge_id}/versions` Create Version Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `judge_id` | path | string | yes | | **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `definition` | object | no | | | `note` | string \| null | no | | | `name` | string \| null | no | | | `description` | string \| null | no | | | `type` | string \| null | no | | | `scope` | string \| null | no | | | `level` | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ## scores ### `GET /scores` List Scores Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `metric_key` | query | string \| null | no | | | `range` | query | string | no | | | `from` | query | string \| null | no | | | `to` | query | string \| null | no | | | `targetId` | query | string \| null | no | | | `targetType` | query | string \| null | no | | | `status` | query | string \| null | no | | | `min_score` | query | number \| null | no | | | `max_score` | query | number \| null | no | | | `source` | query | string \| null | no | | | `lifecycle` | query | string | no | | | `runId` | query | string \| null | no | | | `page` | query | integer | no | | | `limit` | query | integer | no | | | `sort` | query | string | no | | | `sort_by` | query | string \| null | no | | | `sort_dir` | query | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /scores/display-settings` Get Display Settings Resolved score-display config for the current project (scale + green/yellow/red bands + per-metric overrides). Readable by anyone who can read the project — the UI needs it to render every score. Falls back to platform defaults when nothing is persisted. Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `PATCH /scores/display-settings` Patch Display Settings Update the project's score-display config. Admin-only (``MANAGE_SCORE_DISPLAY``) — it changes what every viewer sees. Presentation only: it never alters stored scores or grading. Values are clamped to safe ranges and bands kept non-inverted (``warn <= pass``). Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `scaleMax` | number \| null | no | | | `passThreshold` | number \| null | no | | | `warnThreshold` | number \| null | no | | | `metricOverrides` | object \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /scores/distribution` Distribution The label breakdown for one score type — the drill-down's answer to `/scores/timeseries`. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `metric_key` | query | string | yes | | | `source` | query | string \| null | no | | | `targetType` | query | string \| null | no | | | `range` | query | string | no | | | `from` | query | string \| null | no | | | `to` | query | string \| null | no | | | `lifecycle` | query | string | no | | | `buckets` | query | integer | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /scores/metrics` List Metrics Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `range` | query | string | no | | | `from` | query | string \| null | no | | | `to` | query | string \| null | no | | | `lifecycle` | query | string | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `PUT /scores/prefs` Upsert Pref Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `metricKey` | string | yes | | | `source` | string | yes | | | `targetType` | string | yes | | | `favorite` | boolean \| null | no | | | `hidden` | boolean \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /scores/timeseries` Timeseries Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `metric_key` | query | string | yes | | | `range` | query | string | no | | | `from` | query | string \| null | no | | | `to` | query | string \| null | no | | | `lifecycle` | query | string | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ## enrichments ### `GET /enrichment-catalog` Enrichment Catalog Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `targetType` | query | string | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /enrichment-fields` Enrichment Fields The enrichment output fields that actually have values in this project, with the distinct values observed (capped). Drives the Traces/Sessions enrichment filter and the "Create dataset from enrichment" picker. Only fields with at least one output are returned, so the UI never offers a filter that can't match anything. Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `POST /user-enrichment-runs` Create Run Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `ids` | string[] \| null | no | | | `filter` | object \| null | no | | | `dataset_id` | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /user-enrichment-runs/{run_id}` Get Run Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `run_id` | path | string | yes | | | `sample` | query | integer | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /user-enrichments` List Enrichments Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `name` | query | string \| null | no | | | `mode` | query | string \| null | no | | | `status` | query | string \| null | no | | | `target_type` | query | string \| null | no | | | `sort_by` | query | string \| null | no | | | `sort_dir` | query | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /user-enrichments` Create Enrichment Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `name` | string | yes | | | `mode` | string | yes | | | `targetType` | string | yes | | | `outputFields` | OutputField[] | no | | | `prerequisites` | object | no | | | `definition` | object | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `PATCH /user-enrichments/{enrichment_id}` Update Enrichment Edit an enrichment. Metadata (name / output fields / prerequisites / status) mutates in place; a changed ``definition`` (prompt or LLM connection) creates a new immutable version and promotes it (version history stays append-only, matching ``POST.../versions``). Disable = ``status='draft'`` (also drops the on-ingest trigger back to manual); re-enable = ``status='enabled'``. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `enrichment_id` | path | string | yes | | **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `name` | string \| null | no | | | `outputFields` | OutputField[] \| null | no | | | `prerequisites` | object \| null | no | | | `status` | string \| null | no | | | `definition` | object \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `DELETE /user-enrichments/{enrichment_id}` Delete Enrichment Hard-delete an enrichment and everything it owns — versions, runs, computed outputs, and its Activity-feed reflections — project-scoped so a scoped caller can't delete another project's enrichment. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `enrichment_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /user-enrichments/{enrichment_id}/dataset` Create Dataset From Enrichment The enrichment→dataset→judge bridge: materialize a dataset whose membership is every session whose enrichment output ``field == value``. The dataset carries an ``enrichment_filter`` source config, so (when ``streaming``) hourly sync keeps the cohort fresh as new traces are enriched — then any judge eval-run can target it via ``dataset_id``. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `enrichment_id` | path | string | yes | | **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `name` | string | yes | | | `description` | string \| null | no | | | `field` | string | yes | | | `value` | string | yes | | | `streaming` | boolean | no | | | `fields` | string[] \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /user-enrichments/{enrichment_id}/deployments` Create Deployment Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `enrichment_id` | path | string | yes | | **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `schedule` | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /user-enrichments/{enrichment_id}/versions` Create Version Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `enrichment_id` | path | string | yes | | **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `definition` | object | no | | | `note` | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ## datasets ### `GET /datasets` List Datasets Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `POST /datasets` Create Dataset Details **Request body** (`application/json`) — required Schema: `object`. **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /datasets/from-selection` Create From Selection One-call "create dataset from selected rows": create a ``manual`` dataset, bulk-add the selected sessions/traces, and — when ``golden`` — snapshot an immutable v1 marked golden. Returns the dataset dict with a ``versions`` summary (newest first). Details **Request body** (`application/json`) — required Schema: `object`. **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /datasets/preview` Preview Dataset Mirror ``/eval-targets/estimate`` for dataset sources: count eligible + sampled sessions for a source config, without persisting anything. Project-scoped. Details **Request body** (`application/json`) — required Schema: `object`. **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `PATCH /datasets/{id_or_name}` Update Dataset Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `id_or_name` | path | string | yes | | **Request body** (`application/json`) — required Schema: `object`. **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `DELETE /datasets/{id_or_name}` Delete Dataset Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `id_or_name` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /datasets/{id_or_name}/export` Export Dataset Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `id_or_name` | path | string | yes | | | `format` | query | string | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /datasets/{id_or_name}/items` List Items List a dataset's items with optional text search (input/output/expected), kind + captured-field-presence filters, and whitelisted server-side sort. All params are additive: with none supplied the historical ``ORDER BY created_at ASC`` is preserved. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `id_or_name` | path | string | yes | | | `q` | query | string \| null | no | | | `kind` | query | string \| null | no | | | `has` | query | string[] | no | | | `sort_by` | query | string \| null | no | | | `sort_dir` | query | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /datasets/{id_or_name}/items` Add Item Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `id_or_name` | path | string | yes | | **Request body** (`application/json`) — required Schema: `object`. **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `DELETE /datasets/{id_or_name}/items` Delete Item Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `id_or_name` | path | string | yes | | | `trace_id` | query | string \| null | no | | | `session_id` | query | string \| null | no | | | `item_id` | query | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /datasets/{id_or_name}/items/bulk` Bulk Add Items Bulk-add sessions/traces to an existing dataset, de-duped against what's already in it. Sessions are field-captured (respecting the dataset's ``fields`` unless the body overrides ``fields``); traces are added as reference rows. Returns ``{added, skipped}``. Project-scoped. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `id_or_name` | path | string | yes | | **Request body** (`application/json`) — required Schema: `object`. **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `PATCH /datasets/{id_or_name}/items/{item_id}` Update Item Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `id_or_name` | path | string | yes | | | `item_id` | path | string | yes | | **Request body** (`application/json`) — required Schema: `object`. **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /datasets/{id_or_name}/sync` Sync Dataset Re-run a filter/cluster dataset's stored source config, adding newly-matching sessions not already present. Respects sampling and (for streaming) the ``last_synced_at`` high-water mark. Returns ``{added, total}``. 404 unless the dataset is a filter/cluster source. Project-scoped. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `id_or_name` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /datasets/{id_or_name}/versions` List Versions Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `id_or_name` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /datasets/{id_or_name}/versions` Create Version Snapshot the dataset's current items into a new immutable version (``version = max+1``). Body ``{name?, notes?, golden?}``. When ``golden`` is set, sibling versions are un-goldened (one golden per dataset). Returns the version row. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `id_or_name` | path | string | yes | | **Request body** (`application/json`) — required Schema: `object`. **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `PATCH /datasets/{id_or_name}/versions/{version}` Update Version Mutate version METADATA only — ``{name?, notes?, golden?}``. Snapshot items are immutable. Setting ``golden:true`` un-goldens sibling versions (one golden per dataset). Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `id_or_name` | path | string | yes | | | `version` | path | integer | yes | | **Request body** (`application/json`) — required Schema: `object`. **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /datasets/{id_or_name}/versions/{version}/export` Export Version Export an immutable version's snapshot (json|csv), same shape as the dataset export (plus a ``version`` summary in the JSON body). Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `id_or_name` | path | string | yes | | | `version` | path | integer | yes | | | `format` | query | string | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /datasets/{id_or_name}/versions/{version}/items` List Version Items List an immutable version's snapshot items — paged/searchable/sortable exactly like the live ``/items`` endpoint. Read-only. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `id_or_name` | path | string | yes | | | `version` | path | integer | yes | | | `q` | query | string \| null | no | | | `kind` | query | string \| null | no | | | `has` | query | string[] | no | | | `sort_by` | query | string \| null | no | | | `sort_dir` | query | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ## annotations ### `GET /annotations` List Annotations Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `targetType` | query | string \| null | no | | | `value` | query | string \| null | no | | | `reviewer` | query | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /annotations` Create Annotation Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `targetType` | string | yes | | | `targetId` | string | yes | | | `value` | string | yes | | | `comment` | string \| null | no | | | `reviewer` | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /annotations/reviewers` List Reviewers Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `GET /annotations/trend` Annotations Trend Annotation-throughput trend for the Review Queue page: per-day (or per-hour) count of annotations created over the selected window, plus a period-over-period total. Scoped to the current project — the review loop's output volume over time. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `range` | query | string \| null | no | | | `from` | query | string \| null | no | | | `to` | query | string \| null | no | | | `grain` | query | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ## review ### `POST /review` Create Review Write a ground-truth label. Reviewer + role resolved from the request identity. Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `target_type` | string | no | | | `target_id` | string | yes | | | `verdict` | string | yes | | | `critique` | string \| null | no | | | `failure_mode_id` | string \| null | no | | | `cluster_id` | string \| null | no | | | `is_gold` | boolean | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /review-queue` Review Queue Active-learning-ranked sessions a human should review next. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `limit` | query | integer \| null | no | | | `novelty_budget` | query | integer \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /review/adjudicate` Adjudicate Principal override: supersede every conflicting active label on a target and write the authoritative one. Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `target_id` | string | yes | | | `verdict` | string | yes | | | `critique` | string \| null | no | | | `is_gold` | boolean | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /review/alignment` List Alignment Persisted judge↔expert alignment rows (the trend), newest first. Project-scoped. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `judge_id` | query | string \| null | no | | | `range` | query | string | no | | | `from` | query | string \| null | no | | | `to` | query | string \| null | no | | | `latest_only` | query | boolean | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /review/alignment/disagreements` Alignment Disagreements Sessions where this judge disagreed with expert ground truth, split FN/FP. Project-scoped. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `judge_id` | query | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /review/alignment/refresh` Refresh Alignment Recompute + persist judge↔expert alignment for every judge that has scores. Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `GET /review/gold` List Gold List the active gold-standard labels (``is_gold = 1``), newest first. Project-scoped. Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `GET /review/label-options` Label Options Grouped options a reviewer can tag a verdict with: confirmed taxonomy failure modes and discovered clusters (each carrying its finite L1 ``bucket``/``bucketLabel`` + session count). Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `GET /review/labels` List Labels Unified human-label history — the source the Annotations tab renders. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `targetType` | query | string \| null | no | | | `verdict` | query | string \| null | no | | | `reviewer` | query | string \| null | no | | | `failureModeId` | query | string \| null | no | | | `clusterId` | query | string \| null | no | | | `isGold` | query | boolean \| null | no | | | `includeSuperseded` | query | boolean | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /review/leaderboard` Leaderboard Per-reviewer QUALITY = agreement with the expert/consensus on **gold** items, not volume. Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `GET /review/queue-config` Get Queue Config Current review-queue config for the project (defaults synthesized when unset). Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `PUT /review/queue-config` Put Queue Config Upsert the project's review-queue config (gated ``require_action('review')``). Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `statuses` | string[] \| null | no | | | `scoreFilters` | object[] \| null | no | | | `includeFailureModeIds` | string[] \| null | no | | | `includeClusterIds` | string[] \| null | no | | | `includeTaxonomy` | boolean \| null | no | | | `onlyUnlabeled` | boolean \| null | no | | | `noveltyBudget` | integer \| null | no | | | `queueLimit` | integer \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /review/reviewers` List Label Reviewers Distinct reviewers that have written a label in this project. Project-scoped. Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ## preprod-evals ### `GET /preprod-compare` Compare Runs Aligned N-run comparison over a shared golden version. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `runs` | query | string \| null | no | comma-separated run ids (A,B,C…) | | `run` | query | string[] \| null | no | repeatable run id param | | `baseline` | query | string \| null | no | a run id (must be in the set) to mark per-row regressions against | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /preprod-compare/trajectory` Trajectory Diff Diff the candidate's captured trajectory against the baseline run's captured trajectory for the SAME golden prompt. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `run` | query | string | yes | candidate run id | | `item` | query | string | yes | dataset_version_item_id of the golden prompt | | `baseline` | query | string | yes | baseline run id to diff against | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /preprod-evals` List Preprod Evals List pre-prod eval runs (project-scoped) with progress counts. Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `POST /preprod-evals` Create Preprod Eval Create a pre-prod eval run + snapshot the golden version's items into pending item rows. Details **Request body** (`application/json`) — required Schema: `object`. **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /preprod-evals/trend` Preprod Trend Run-cadence and pass-rate trend for the Pre-prod Evals page. Primary series: per-day (or per-hour) count of runs created over the window. Secondary: the pass-rate across scored items in the same window. A window with no scored items reports a null pass-rate, never a fabricated 0%. Scoped to the current project. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `range` | query | string \| null | no | | | `from` | query | string \| null | no | | | `to` | query | string \| null | no | | | `grain` | query | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /preprod-evals/verify` Verify Fix Verification webhook — PROVE a fix against a candidate endpoint/branch. Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `endpoint_url` | string \| null | no | | | `agent_connection_id` | string \| null | no | | | `auth_token` | string \| null | no | | | `request_shape` | string \| null | no | | | `dataset_id` | string \| null | no | | | `remediation_id` | string \| null | no | | | `version_label` | string | yes | | | `min_pass_rate` | number \| null | no | | | `max_regressions` | integer \| null | no | | | `name` | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /preprod-evals/{run_id}` Get Preprod Eval Run detail: the run header + per-item summary + an aggregate rollup. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `run_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /preprod-evals/{run_id}/cancel` Cancel Preprod Eval Cancel a run (terminal). A completed run cannot be cancelled. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `run_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /preprod-evals/{run_id}/comparison` Comparison Candidate-vs-baseline comparison. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `run_id` | path | string | yes | | | `baseline` | query | string \| null | no | override baseline kind | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /preprod-evals/{run_id}/gate` Gate Evaluate the run's ``gate_json`` against the candidate results → pass/fail + reasons. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `run_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /preprod-evals/{run_id}/items` Get Preprod Items The frozen prompt list for the harness/runner: ``[{item_id, input, expected_output}]``. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `run_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /preprod-evals/{run_id}/metrics` Metrics Per-metric score rollup for a run's captured+scored sessions — the CI-gate surface. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `run_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /preprod-evals/{run_id}/run` Run Preprod Eval Route Trigger a PUSH run: Neens HTTP-calls the run's agent endpoint for every frozen golden prompt, captures + links each response, then scores it — no user harness. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `run_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /preprod-evals/{run_id}/start` Start Preprod Eval Transition ``awaiting_traces`` → ``running`` (idempotent). Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `run_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ================================================================================ # Fix & ship — API Source: /docs/api-reference/fix/ ================================================================================ # Fix & ship — API reference Typed remediations, the evals-from-failures flywheel, and deploy-event correlation. Generated from the live OpenAPI schema (version `0.16.6`). Download the full spec at [`/docs/openapi.json`](/openapi.json). ## remediations ### `GET /remediations` Get Remediations Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `GET /remediations/agent-context` Get Agent Context Return the project's optional agent context (nulls when unset). Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `PUT /remediations/agent-context` Put Agent Context Upsert the project's agent context (improves future grounded generations). Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `systemPrompt` | string \| null | no | | | `repoUrl` | string \| null | no | | | `notes` | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /remediations/efficacy-report` Efficacy Report Per-project post-merge efficacy + MTTR report from LIVE close-out data: failure volume before/after, fixes shipped/verified/regressed/pending, median MTTR, and a monthly trend. Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `POST /remediations/generate` Generate Remediation Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `clusterId` | string \| null | no | | | `failureModeId` | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /remediations/items` List Remediations Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `clusterId` | query | string \| null | no | | | `failureModeId` | query | string \| null | no | | | `failure_mode_id` | query | string \| null | no | | | `status` | query | string \| null | no | | | `workState` | query | string \| null | no | | | `type` | query | string \| null | no | | | `label` | query | string \| null | no | | | `q` | query | string \| null | no | | | `sort` | query | string \| null | no | | | `includeArchived` | query | boolean | no | | | `include_ungrounded` | query | boolean | no | | | `include_advisories` | query | boolean | no | | | `actionability` | query | string \| null | no | | | `failureLocus` | query | string \| null | no | | | `failure_locus` | query | string \| null | no | | | `view` | query | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /remediations/items/{rid}` Get Remediation Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `rid` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `PATCH /remediations/items/{rid}` Patch Remediation Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `rid` | path | string | yes | | | `force` | query | boolean | no | | | `reason` | query | string \| null | no | | **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `status` | string \| null | no | | | `workState` | string \| null | no | | | `labels` | string[] \| null | no | | | `prUrl` | string \| null | no | | | `commitSha` | string \| null | no | | | `acknowledgedBy` | string \| null | no | | | `note` | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `DELETE /remediations/items/{rid}` Delete Remediation Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `rid` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /remediations/items/{rid}/closeout` Get Closeout The single most-recent close-out for this remediation, or 404. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `rid` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /remediations/items/{rid}/efficacy` Efficacy Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `rid` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /remediations/items/{rid}/fix-bundle` Get Fix Bundle Assemble a remediation's root cause + typed fix spec + proof evals + anonymized failing exemplars into ONE coding-agent-ready pack (markdown + structured JSON + a pre-generated ``neens eval run`` CI command). Pure assembly of data Neens already computes — no LLM, no egress, no repo access. Read-only; mirrors the sibling GET-detail routes. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `rid` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /remediations/items/{rid}/record-merge` Record Merge Route Record that a Neens-opened fix PR for this remediation was MERGED (the manual + MCP path): stamp a deploy event + a pending close-out watch. The webhook (``POST /webhooks/github``) is the automated sibling. Returns the close-out dict. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `rid` | path | string | yes | | **Request body** (`application/json`) Schema: `object`. **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /remediations/items/{rid}/regenerate` Regenerate Remediation Item Re-run the CURRENT grounded engine for this remediation's source (cluster or failure mode), producing a FRESH remediation and archiving this one as a superseded predecessor. Force-capable on ANY status — the old proposal (and its PR/verification state) is preserved, never mutated in place and never deleted. Runs synchronously (one generation). 404 if ``rid`` is not in the caller's project scope; 409 if it is already superseded. Returns the NEW remediation dict with ``regeneratedFromId`` set and an embedded compact ``predecessor``. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `rid` | path | string | yes | | **Request body** (`application/json`) Schema: `object`. **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /remediations/items/{rid}/simulate` Simulate Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `rid` | path | string | yes | | **Request body** (`application/json`) Schema: `object`. **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /remediations/items/{rid}/simulation-plan` Simulation Plan Describe what a simulate run WOULD do WITHOUT running any replay. Shares the exact target-selection + judge/model resolution used by ``POST.../simulate``. The optional override query params mirror ``SimulateRequest`` (minus the explicit cohort — the UI holds the selected ids client-side) so the Configure panel gets a live preview of a tuned run. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `rid` | path | string | yes | | | `judgeId` | query | string \| null | no | | | `threshold` | query | number \| null | no | | | `maxTargets` | query | integer \| null | no | | | `sampling` | query | string \| null | no | | | `connectionId` | query | string \| null | no | | | `strategy` | query | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /remediations/regenerate-bulk` Regenerate Remediations Bulk Regenerate a SET of remediations with the current engine. Resolves the target id list under project scope from exactly one selector (``remediation_ids`` | ``failure_mode_id`` | ``all_open``), caps it at a server-side maximum, then: Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `remediationIds` | string[] \| null | no | | | `failureModeId` | string \| null | no | | | `allOpen` | boolean \| null | no | | | `reason` | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /remediations/stats` Remediation Stats Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `GET /remediations/trend` Remediations Trend Throughput trend for the Remediations page: per-day (or per-hour) count of fix tasks created over the selected window, plus a period-over-period total. Scoped to the current project. Creation volume is counted regardless of later archive state, so a historical trend doesn't drift as tasks are archived. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `range` | query | string \| null | no | | | `from` | query | string \| null | no | | | `to` | query | string \| null | no | | | `grain` | query | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ## flywheel ### `POST /flywheel/failure-modes/{mode_id}/generate-eval` Generate Eval Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `mode_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /flywheel/gates` List Gates Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `status` | query | string \| null | no | | | `failureModeId` | query | string \| null | no | | | `name` | query | string \| null | no | | | `sort_by` | query | string \| null | no | | | `sort_dir` | query | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /flywheel/gates/{gate_id}` Get Gate Rich single-gate detail: the stored gate row PLUS the provenance it binds together (failure mode / judge / dataset / deployment) and a bounded history of its recent eval runs, so the UI can explain *what this gate checks*, *when it last ran*, and *how it has been trending* on one click. Every cross-table read is best-effort and project-scoped. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `gate_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `PATCH /flywheel/gates/{gate_id}` Patch Gate Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `gate_id` | path | string | yes | | **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `status` | string \| null | no | | | `baselinePassRate` | number \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /flywheel/gates/{gate_id}/run` Run Gate Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `gate_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ## deploy-events ### `GET /deploy-events` List Deploy Events Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `agentName` | query | string \| null | no | | | `since` | query | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /deploy-events` Create Deploy Event Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `kind` | string | yes | | | `title` | string | yes | | | `agentName` | string \| null | no | | | `oldValue` | string \| null | no | | | `newValue` | string \| null | no | | | `deployedAt` | string \| null | no | | | `deployedBy` | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /deploy-events/what-changed` What Changed Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `at` | query | string | yes | ISO-8601 reference instant | | `agentName` | query | string \| null | no | | | `windowHours` | query | integer | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ================================================================================ # Analyze — API Source: /docs/api-reference/analyze/ ================================================================================ # Analyze — API reference Custom dashboards, the metrics catalogue, and saved analyses. Generated from the live OpenAPI schema (version `0.16.6`). Download the full spec at [`/docs/openapi.json`](/openapi.json). ## dashboards ### `GET /dashboards` List Dashboards Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `POST /dashboards` Create Dashboard Details **Request body** (`application/json`) — required Schema: `object`. **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /dashboards/by-slug/{slug}` Get Dashboard By Slug Resolve a dashboard by its stable ``slug`` within the caller's tenant. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `slug` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /dashboards/favorites` List Favorites The caller's favorited dashboards. Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `POST /dashboards/favorites/{dash_id}` Add Favorite Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `dash_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `DELETE /dashboards/favorites/{dash_id}` Remove Favorite Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `dash_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /dashboards/{dash_id}` Get Dashboard Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `dash_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `PATCH /dashboards/{dash_id}` Update Dashboard Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `dash_id` | path | string | yes | | **Request body** (`application/json`) — required Schema: `object`. **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `DELETE /dashboards/{dash_id}` Delete Dashboard Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `dash_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /dashboards/{dash_id}/clone` Clone Dashboard Deep-clone a dashboard the caller can see into a fresh **private** copy they own. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `dash_id` | path | string | yes | | **Request body** (`application/json`) Schema: `object`. **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /dashboards/{dash_id}/widgets` Add Widget Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `dash_id` | path | string | yes | | **Request body** (`application/json`) — required Schema: `object`. **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /dashboards/{dash_id}/widgets/reorder` Reorder Widgets Persist a new widget display order. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `dash_id` | path | string | yes | | **Request body** (`application/json`) — required Schema: `object`. **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `PATCH /dashboards/{dash_id}/widgets/{wid}` Update Widget Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `dash_id` | path | string | yes | | | `wid` | path | string | yes | | **Request body** (`application/json`) — required Schema: `object`. **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `DELETE /dashboards/{dash_id}/widgets/{wid}` Delete Widget Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `dash_id` | path | string | yes | | | `wid` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /dashboards/{dash_id}/widgets/{wid}/clone` Clone Widget Duplicate a widget within the same dashboard (config + color preserved). Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `dash_id` | path | string | yes | | | `wid` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /dashboards/{dash_id}/widgets/{wid}/data` Widget Data Resolve a widget's data LIVE via ``run_measure`` — the render-time data fetch. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `dash_id` | path | string | yes | | | `wid` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /measures/catalogue` Measure Catalogue The measure catalogue: each measure + the dimensions valid for it + the widget types. Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `GET /measures/fields` Measures Fields Discover dynamic, sliceable ``session.metadata.*`` keys for the current scope. Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ## analyses ### `POST /analyses/{name}/run` Run Analysis Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `name` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ================================================================================ # Assistant & prompts — API Source: /docs/api-reference/assistant/ ================================================================================ # Assistant & prompts — API reference The in-app chat assistant — questions plus approval-gated writes — and the versioned prompt registry. Generated from the live OpenAPI schema (version `0.16.6`). Download the full spec at [`/docs/openapi.json`](/openapi.json). ## chat ### `GET /chat/settings` Get Chat Settings Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `PUT /chat/settings` Put Chat Settings Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `connectionId` | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /chat/stream` Chat Stream Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `messages` | ChatMessage[] | yes | | | `approved_tool_call` | ApprovedToolCall \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ## prompts ### `GET /prompts` List Prompts Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `q` | query | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /prompts` Create Prompt Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `name` | string | yes | | | `description` | string \| null | no | | | `promptType` | string | no | | | `content` | object | yes | | | `config` | object \| null | no | | | `commitMessage` | string \| null | no | | | `sourceType` | string \| null | no | | | `sourceRef` | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /prompts/by-name/{name}` Resolve By Name SDK/judge-facing resolution: the content of a prompt by NAME at a label/version/newest. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `name` | path | string | yes | | | `label` | query | string \| null | no | | | `version` | query | integer \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /prompts/from-remediation` From Remediation Lift a ``prompt_change`` remediation's fixed system prompt into a registry version. Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `remediationId` | string | yes | | | `promptId` | string \| null | no | | | `name` | string \| null | no | | | `commitMessage` | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /prompts/playground/fix-simulation` Playground Fix Simulation Re-score an edited system prompt against a real failing trace's judges by replaying it. Returns a queued id to poll. Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `sessionId` | string | yes | | | `editedSystemPrompt` | string | yes | | | `judgeIds` | string[] \| null | no | | | `connectionId` | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /prompts/playground/fix-simulation/{fixloop_id}` Get Playground Fix Simulation Poll a fix-loop's status + (when completed) its before/after result. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `fixloop_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /prompts/playground/run` Playground Run Execute a prompt against the project's default LLM connection and return the output. Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `promptId` | string \| null | no | | | `label` | string \| null | no | | | `version` | integer \| null | no | | | `promptType` | string \| null | no | | | `content` | object \| null | no | | | `config` | object \| null | no | | | `variables` | object \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /prompts/{prompt_id}` Get Prompt Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `prompt_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `PATCH /prompts/{prompt_id}` Patch Prompt Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `prompt_id` | path | string | yes | | **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `name` | string \| null | no | | | `description` | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `DELETE /prompts/{prompt_id}` Delete Prompt Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `prompt_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `PUT /prompts/{prompt_id}/labels/{label}` Set Label Point a movable deploy tag (``production``/``staging``/…) at a specific version. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `prompt_id` | path | string | yes | | | `label` | path | string | yes | | **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `version` | integer | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `DELETE /prompts/{prompt_id}/labels/{label}` Delete Label Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `prompt_id` | path | string | yes | | | `label` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /prompts/{prompt_id}/versions` List Versions Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `prompt_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /prompts/{prompt_id}/versions` Create Version Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `prompt_id` | path | string | yes | | **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `content` | object | yes | | | `config` | object \| null | no | | | `commitMessage` | string \| null | no | | | `sourceType` | string \| null | no | | | `sourceRef` | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ================================================================================ # Administration — API Source: /docs/api-reference/administration/ ================================================================================ # Administration — API reference Tenancy, members & auth, personas, settings & LLM connections, retention, and the audit log. Generated from the live OpenAPI schema (version `0.16.6`). Download the full spec at [`/docs/openapi.json`](/openapi.json). ## auth ### `POST /auth/accept-invite` Accept Invite Activate an invited user: set their password from a one-time token, log them in. Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `token` | string | yes | | | `password` | string | yes | | | `displayName` | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /auth/change-password` Change Password Change the signed-in user's password after re-verifying the current one. Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `currentPassword` | string | yes | | | `newPassword` | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `PUT /auth/digest` Set Digest Set the signed-in user's persona digest email preference. Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `optIn` | boolean | yes | | | `frequency` | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /auth/digest/test` Send Test Digest Compose the signed-in user's persona digest NOW and send it, ignoring the daily cadence. Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `POST /auth/forgot-password` Forgot Password Begin a forgotten-password reset for a registered user. Always returns a uniform 200. Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `email` | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `PUT /auth/locale` Set Locale Persist the signed-in user's preferred UI language. Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `locale` | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /auth/login` Login Verify email + password, issue a session token, return it with the user profile. Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `email` | string | yes | | | `password` | string | yes | | | `challengeToken` | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /auth/logout` Logout Revoke the presented session token (best-effort; idempotent). Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `GET /auth/me` Me Return the currently authenticated user, or 401 if there is no valid session. Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `GET /auth/mfa` Mfa Status The caller's own MFA state. Reachable on an ``mfa_pending`` session — it is what the enrolment screen reads to know what to show. Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `POST /auth/mfa/challenge` Mfa Challenge Exchange a ``nk_mfa_…`` token + a second factor for a real session. **Unauthenticated.** Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `mfaToken` | string | yes | | | `code` | string \| null | no | | | `recoveryCode` | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /auth/mfa/disable` Mfa Disable Turn the second factor off, after re-proving BOTH the password and the factor itself. Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `password` | string | yes | | | `code` | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /auth/mfa/enroll` Mfa Enroll Issue a fresh TOTP secret + provisioning URI. **Does not enable anything.** Details **Request body** (`application/json`) — required Schema: `EnrollBody`. **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /auth/mfa/recovery-codes` Mfa Regenerate Recovery Codes Replace every outstanding recovery code. Returns the new set, shown once. Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `password` | string | yes | | | `code` | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /auth/mfa/verify` Mfa Verify Confirm a code against the PENDING secret, then enable MFA and mint the recovery codes. Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `password` | string | no | | | `code` | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /auth/password-policy` Password Policy The set-password rules the server will actually enforce: ``{minLength, breachCheck}``. Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `PUT /auth/persona` Set Persona Record the signed-in user's persona lens. Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `personaKey` | string \| null | no | | | `dismissed` | boolean | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /auth/reset-password` Reset Password Complete a forgotten-password reset from a one-time token, then sign the user in. Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `token` | string | yes | | | `newPassword` | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /auth/reset-password/validate` Validate Reset Token Cheap pre-flight check the reset page runs before showing its form. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `token` | query | string | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /auth/sessions` List Sessions The caller's OWN live sessions — "where am I signed in". Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `POST /auth/sessions/revoke-all` Revoke All Sessions Sign out everywhere. Returns ``{"revoked": n}`` — how many sessions were actually ended. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `includeCurrent` | query | boolean | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `DELETE /auth/sessions/{session_id}` Revoke Session Revoke ONE of the caller's own sessions — the "I don't recognise that device" action. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `session_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /auth/sso/connections` List Connections List this company's SSO connections (client secrets never returned). Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `POST /auth/sso/connections` Create Connection Create an SSO connection for this company. Protocol-specific required fields are validated before the store persists (and Fernet-encrypts any client secret). Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `GET /auth/sso/connections/{connection_id}` Get Connection Read one connection (company-scoped; client secret never returned). Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `connection_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `PATCH /auth/sso/connections/{connection_id}` Update Connection Patch a connection (company-scoped). A raw ``oidcClientSecret`` is encrypted by the store; a supplied required field may not be blanked. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `connection_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `DELETE /auth/sso/connections/{connection_id}` Delete Connection Delete a connection and its domain mappings (company-scoped). Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `connection_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `PUT /auth/sso/connections/{connection_id}/domains` Set Connection Domains Set the email domains routed to this connection (home-realm discovery). A domain already claimed by another company hits the UNIQUE index — surfaced as a clean 409. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `connection_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /auth/sso/oidc/callback` Sso Oidc Callback OIDC authorization-code callback: consume state, exchange the code with PKCE, fully validate the id_token, JIT-provision, and deliver the session (or an MFA challenge) to the SPA. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `code` | query | string | no | | | `state` | query | string | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /auth/sso/providers` Sso Providers Which SSO connection (if any) serves this email's domain — the login page's "Sign in with SSO" affordance. No secrets in the response. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `email` | query | string | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /auth/sso/saml/acs` Sso Saml Acs SAML Assertion Consumer Service (SP-initiated POST binding): validate the signed assertion and deliver the session (or an MFA challenge) to the SPA. Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `GET /auth/sso/saml/{connection_id}/metadata` Sso Saml Metadata SP metadata XML for the IdP administrator (application/xml). Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `connection_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /auth/sso/{connection_id}/start` Sso Start Begin an SSO login: build the authz redirect and 302 to the IdP. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `connection_id` | path | string | yes | | | `redirect` | query | string | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /me/workspace` Workspace Single bootstrap contract the SPA renders its shell from. Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ## tenancy ### `GET /companies` List Companies Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `GET /memberships` List Memberships Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `user_id` | query | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /orgs` List Orgs Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `company_id` | query | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /orgs` Create Org Create an org in the caller's company (admin-gated; enforces the tier's org cap). Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `name` | string | yes | | | `slug` | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `PATCH /orgs/{org_id}` Update Org Rename an org (or update its slug) within the caller's company. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `org_id` | path | string | yes | | **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `name` | string \| null | no | | | `slug` | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /overview` Overview Tenancy tree (companies → orgs → projects) with per-project session counts. Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `GET /projects` List Projects Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `org_id` | query | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /projects` Create Project Create a new project under an org. Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `name` | string | yes | | | `org_id` | string \| null | no | | | `slug` | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /projects/current` Current Project Report the project THIS connection is scoped to. Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `GET /projects/{project_id}` Get Project Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `project_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `PATCH /projects/{project_id}` Update Project Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `project_id` | path | string | yes | | **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `name` | string \| null | no | | | `slug` | string \| null | no | | | `archived` | boolean \| null | no | | | `org_id` | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `DELETE /projects/{project_id}` Delete Project Delete a project and every data-plane row scoped to it. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `project_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `PUT /projects/{project_id}/prefs` Set Project Pref Toggle the current user's per-user preference for a project (favorite on/off). Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `project_id` | path | string | yes | | **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `favorite` | boolean | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /users` List Users Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `company_id` | query | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ## members ### `GET /members` List Members List the caller company's members (directory users + their status/role). Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `POST /members/bulk-orgs` Bulk Set Orgs Assign org access to several members at once. Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `userIds` | string[] | yes | | | `orgIds` | string[] \| null | no | | | `allOrgs` | boolean | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /members/bulk-persona` Bulk Assign Persona Set (or clear) the persona DEFAULT for a set of the caller company's members at once. Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `userIds` | string[] | yes | | | `personaKey` | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /members/invite` Invite Member Invite a new member: create an ``invited`` directory user + a one-time activation link. Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `email` | string | yes | | | `displayName` | string \| null | no | | | `role` | string | no | | | `orgIds` | string[] \| null | no | | | `allOrgs` | boolean | no | | | `personaKey` | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `PATCH /members/{user_id}` Update Member Change a member's role or status (e.g. disable). Scoped to the caller's company. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `user_id` | path | string | yes | | **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `role` | string \| null | no | | | `status` | string \| null | no | | | `personaDefault` | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `DELETE /members/{user_id}` Remove Member Remove a member (delete the directory user + revoke their sessions). Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `user_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /members/{user_id}/activity` Member Activity Lifecycle timeline for one member: the tenant audit events targeting them. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `user_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `PUT /members/{user_id}/orgs` Set Member Orgs Assign a member to one or more orgs. A full REPLACE of the member's org-scoped membership grants, scoped to the caller's company. Admin-gated. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `user_id` | path | string | yes | | **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `orgIds` | string[] \| null | no | | | `allOrgs` | boolean | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /members/{user_id}/resend-invite` Resend Invite Rotate an invited member's activation token and return a fresh one-time link. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `user_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ## personas ### `GET /personas` Get Personas List the personas (lenses) available to the caller's tenant — powers the switcher/modal. Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `POST /personas` Create Persona Route Create a tenant-authored (``custom``) persona lens. Admin + ``custom_personas`` (Silver+). Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `name` | string | yes | | | `description` | string \| null | no | | | `focusAreas` | string[] | no | | | `home` | string | no | | | `defaultRange` | string | no | | | `nav` | PersonaNavInput | no | | | `dashboardIds` | string[] | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `PATCH /personas/{key}` Update Persona Route Edit a persona. Editing a **platform** lens flips it to ``custom`` provenance (freezing it — it stops receiving code upgrades); an already-custom lens is edited in place. The always-on ``full_workspace`` escape hatch can't be customized. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `key` | path | string | yes | | **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `name` | string \| null | no | | | `description` | string \| null | no | | | `focusAreas` | string[] \| null | no | | | `home` | string \| null | no | | | `defaultRange` | string \| null | no | | | `nav` | PersonaNavInput \| null | no | | | `dashboardIds` | string[] \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `DELETE /personas/{key}` Delete Persona Route Delete a **custom** persona. Deleting a customized *platform* key resets it to the shipped default (re-seeded on the next ensure); deleting a purely-custom key also clears it from any member who had it as a default (they fall back to their role-derived lens). Platform lenses that were never customized, and ``full_workspace``, can't be deleted. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `key` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ## settings ### `GET /llm-connections` List Llm Connections Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `POST /llm-connections` Create Llm Connection Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `name` | string \| null | no | | | `provider` | string \| null | no | | | `model` | string \| null | no | | | `baseUrl` | string \| null | no | | | `credential` | string \| null | no | | | `credentialMethod` | string \| null | no | | | `secretRef` | string \| null | no | | | `requestsPerMinute` | integer \| null | no | | | `scopeType` | string \| null | no | | | `scopeIds` | string[] \| null | no | | | `isDefault` | boolean \| null | no | | | `options` | object \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /llm-connections/list-models` List Llm Connection Models List the model ids a connection's provider advertises, for the model picker. Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `id` | string \| null | no | | | `provider` | string \| null | no | | | `baseUrl` | string \| null | no | | | `credential` | string \| null | no | | | `credentialMethod` | string \| null | no | | | `secretRef` | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /llm-connections/probe` Probe Llm Connection Live connection probe: makes a real completion through the provider and returns reachability. Used by the UI 'Test' action. Never persists anything. A completion that only ran out of token budget (finish_reason=length) still counts as reachable. Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `id` | string \| null | no | | | `provider` | string \| null | no | | | `model` | string \| null | no | | | `baseUrl` | string \| null | no | | | `credential` | string \| null | no | | | `credentialMethod` | string \| null | no | | | `secretRef` | string \| null | no | | | `requestShape` | string \| null | no | | | `invokePath` | string \| null | no | | | `auth` | string \| null | no | | | `outputJsonpath` | string \| null | no | | | `method` | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `PATCH /llm-connections/{conn_id}` Update Llm Connection Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `conn_id` | path | string | yes | | **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `name` | string \| null | no | | | `provider` | string \| null | no | | | `model` | string \| null | no | | | `baseUrl` | string \| null | no | | | `credential` | string \| null | no | | | `credentialMethod` | string \| null | no | | | `secretRef` | string \| null | no | | | `requestsPerMinute` | integer \| null | no | | | `scopeType` | string \| null | no | | | `scopeIds` | string[] \| null | no | | | `isDefault` | boolean \| null | no | | | `options` | object \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `DELETE /llm-connections/{conn_id}` Delete Llm Connection Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `conn_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /settings` Get Settings Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `GET /settings/api-keys` List Api Keys Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `POST /settings/api-keys` Create Api Key Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `name` | string | yes | | | `role` | string | no | | | `mfaCode` | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /settings/api-keys/revoke` Revoke Api Key Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `id` | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ## preferences ### `GET /preferences/table-columns` Get Table Columns Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `view` | query | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `PUT /preferences/table-columns` Put Table Columns Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `view` | string | yes | | | `columns` | string[] | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ## onboarding ### `POST /onboarding/complete` Complete Onboarding Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `POST /onboarding/dismiss` Dismiss Onboarding Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `POST /onboarding/provision` Provision Onboarding Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `GET /onboarding/status` Get Onboarding Status Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `GET /onboarding/tenant` Get Tenant Onboarding Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `POST /onboarding/tenant/ack` Ack Tenant Step Mark a checklist step acknowledged by the admin (idempotent). Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `step` | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /onboarding/tenant/complete` Complete Tenant Onboarding Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `POST /onboarding/tenant/dismiss` Dismiss Tenant Onboarding Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ## retention ### `GET /retention/erasure` List Erasures List erasure requests for the tenant, newest first (compliance evidence register). Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `limit` | query | integer | no | | | `offset` | query | integer | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /retention/erasure` Create Erasure Submit a GDPR Art. 17 erasure. Validates the selector is non-empty (an empty selector can never fan out to "erase everything"), records a ``pending`` request, and starts the erasure asynchronously — poll the request for its terminal status and manifest. Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `session_ids` | string[] \| null | no | | | `conversation_ids` | string[] \| null | no | | | `subject_key` | string \| null | no | | | `subject_value` | string \| null | no | | | `project_id` | string \| null | no | | | `reason` | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /retention/erasure/{request_id}` Get Erasure Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `request_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /retention/policies` List Policies List the tenant's retention policies + a preview of the effective retention (days) resolved per project (most-specific-first: project override → company policy → platform env default). Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `PUT /retention/policies` Upsert Policy Create or update a retention policy for a scope. A ``company`` policy is forced to the caller's own company (its ``scope_id`` can't be spoofed); a ``project`` policy requires an explicit ``scope_id`` (the project). ``retention_days`` is clamped to ``>= 0`` (0 = keep forever). Upserts on the unique ``(scope_type, scope_id)`` pair. Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `scope_type` | string | yes | | | `scope_id` | string \| null | no | | | `retention_days` | integer | no | | | `enabled` | boolean | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `DELETE /retention/policies/{policy_id}` Delete Policy Delete a retention policy (the scope reverts to the next-coarser effective window). Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `policy_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ## audit ### `GET /audit` List Audit Admin-only tenant audit log. Filterable by action/actor/target/org/project/status, the canonical time ticker, and a free-text ``q`` (substring over summary/target name/ action). Returns ``{data, meta:{total, limit, offset, facets}}``. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `action` | query | string \| null | no | | | `actorId` | query | string \| null | no | | | `actorType` | query | string \| null | no | | | `targetType` | query | string \| null | no | | | `orgId` | query | string \| null | no | | | `projectId` | query | string \| null | no | | | `status` | query | string \| null | no | | | `q` | query | string \| null | no | | | `range` | query | string | no | | | `from` | query | string \| null | no | | | `to` | query | string \| null | no | | | `sortBy` | query | string | no | | | `sortDir` | query | string | no | | | `limit` | query | integer | no | | | `offset` | query | integer | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /audit/trend` Audit Trend Event-volume trend for the admin audit log: per-day (or per-hour) count of audit events over the selected window. Scoped to the events you can see — your company's, plus shared-schema events — never another company's. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `range` | query | string | no | | | `from` | query | string \| null | no | | | `to` | query | string \| null | no | | | `grain` | query | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ================================================================================ # More endpoints — API Source: /docs/api-reference/more/ ================================================================================ # More endpoints — API reference Additional endpoints not grouped above. Generated from the live OpenAPI schema (version `0.16.6`). Download the full spec at [`/docs/openapi.json`](/openapi.json). ## alerts ### `GET /alerts` List Alerts List the project's alert rules (newest first). Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `POST /alerts` Create Alert Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `name` | string | yes | | | `measureKey` | string \| null | no | | | `comparator` | string \| null | no | | | `threshold` | number \| null | no | | | `window` | string | no | | | `metricKey` | string \| null | no | | | `severity` | string | no | | | `sinks` | string[] \| null | no | | | `tags` | string[] \| null | no | | | `cooldownMinutes` | integer | no | | | `enabled` | boolean | no | | | `ruleType` | string | no | | | `config` | object \| null | no | | | `kpiId` | string \| null | no | | | `lookbackDays` | integer \| null | no | | | `minDeltaPct` | number \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /alerts/catalogue` Alert Catalogue The alertable measures + comparators + windows the rule-builder picker renders. Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `GET /alerts/{rule_id}` Get Alert Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `rule_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `PATCH /alerts/{rule_id}` Update Alert Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `rule_id` | path | string | yes | | **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `name` | string \| null | no | | | `measureKey` | string \| null | no | | | `comparator` | string \| null | no | | | `threshold` | number \| null | no | | | `window` | string \| null | no | | | `metricKey` | string \| null | no | | | `severity` | string \| null | no | | | `sinks` | string[] \| null | no | | | `tags` | string[] \| null | no | | | `cooldownMinutes` | integer \| null | no | | | `enabled` | boolean \| null | no | | | `ruleType` | string \| null | no | | | `config` | object \| null | no | | | `kpiId` | string \| null | no | | | `lookbackDays` | integer \| null | no | | | `minDeltaPct` | number \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `DELETE /alerts/{rule_id}` Delete Alert Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `rule_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /alerts/{rule_id}/events` Get Alert Events Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `rule_id` | path | string | yes | | | `limit` | query | integer | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /alerts/{rule_id}/test` Test Alert Evaluate the rule NOW as a dry-run — resolve its metric over the window and report the value + whether it would breach — WITHOUT recording an insight, firing a sink, or touching bookkeeping. The 'Test' button in the rule editor. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `rule_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ## autonomy ### `GET /autonomy/decisions` List Decisions Route The decision trail — every autonomous admission, refusal, suspension, resume and cancel. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `limit` | query | integer | no | | | `offset` | query | integer | no | | | `decision` | query | string \| null | no | | | `remediationId` | query | string \| null | no | | | `reason` | query | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /autonomy/resume` Resume Route Resume a suspended project. 409 when it is not suspended — resuming what is already running would write a misleading ``resumed`` row into the decision trail. Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `GET /autonomy/settings` Get Settings Route The project's autonomy level, suspension state and auto-accept policy. Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `PATCH /autonomy/settings` Patch Settings Route Set the autonomy level and/or the auto-accept policy. Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `level` | string \| null | no | | | `policy` | object \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /autonomy/status` Status Route The SPA's single call: level + suspension + policy + the live health and budget signals the sweep will act on, plus the platform ceilings and what has already happened today. Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `POST /autonomy/suspend` Suspend Route The human kill switch: suspend autonomy for this project and cancel anything in flight. Details **Request body** (`application/json`) Schema: `object`. **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ## badges ### `POST /badge/mint` Mint Badge Mint a signed badge token + ready-to-paste markdown for the current project (+ optional judge). Details **Request body** (`application/json`) Schema: `object`. **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /badge/{token}.svg` Badge Svg PUBLIC eval-status badge. Verify the token, pin the tenant schema, render the SVG. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `token` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ## cost-optimization ### `GET /cost-optimization/frontier` Get Frontier Which cheaper model could judge your traces without losing your trust. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `window` | query | string \| null | no | | | `start` | query | string \| null | no | | | `end` | query | string \| null | no | | | `version` | query | string \| null | no | one agent version label, or `__unversioned__` for traces with no label | | `bar` | query | number \| null | no | agreement bar in 0–1; overrides the default | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /cost-optimization/judges` Get Judges What each LLM judge cost you over the window, and what it would cost at other coverages. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `window` | query | string \| null | no | | | `start` | query | string \| null | no | | | `end` | query | string \| null | no | | | `version` | query | string \| null | no | one agent version label, or `__unversioned__` for traces with no label | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /cost-optimization/moves` Get Moves The ranked list of ways to cut LLM cost without losing quality — at most one per subject. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `window` | query | string \| null | no | | | `start` | query | string \| null | no | | | `end` | query | string \| null | no | | | `version` | query | string \| null | no | one agent version label, or `__unversioned__` for traces with no label | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /cost-optimization/summary` Get Summary What your agent and your LLM judges cost over a window, and what one trace costs. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `window` | query | string \| null | no | | | `start` | query | string \| null | no | | | `end` | query | string \| null | no | | | `version` | query | string \| null | no | one agent version label, or `__unversioned__` for traces with no label | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ## custom-measures ### `GET /custom-measures` List Custom Measures This tenant's measures + the platform catalogue + the authoring limit. Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `POST /custom-measures` Create Custom Measure Define a new measure. Admin (``MANAGE_MEASURES``) + ``custom_measures`` tier (Silver+). Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `slug` | string | yes | | | `label` | string | yes | | | `source` | string | yes | | | `agg` | string | yes | | | `grain` | string \| null | no | | | `unit` | string \| null | no | | | `description` | string \| null | no | | | `outcomeKind` | string \| null | no | | | `outcomeSource` | string \| null | no | | | `metadataPath` | string \| null | no | | | `classifierMetricKey` | string \| null | no | | | `classifierLabels` | string[] \| null | no | | | `rateOp` | string \| null | no | | | `rateValue` | number \| null | no | | | `caseReducer` | string \| null | no | | | `filters` | object \| null | no | | | `direction` | string \| null | no | | | `target` | number \| null | no | | | `category` | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `201` | Successful Response | | `422` | Validation Error | ### `GET /custom-measures/options` Custom Measure Options The vocabulary an author can pick from — the shipped grammar **plus what this project has really observed**. Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `POST /custom-measures/preview` Preview Custom Measure Resolve an UNSAVED definition over a bounded window — the "does this actually work?" button. Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `definition` | object | yes | | | `days` | integer \| null | no | | | `dimensions` | string[] \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /custom-measures/references/{slug}` Custom Measure References What currently reads this measure — shown BEFORE a destructive edit, not after. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `slug` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `PATCH /custom-measures/{slug}` Update Custom Measure Edit a measure you own. Editing a built-in platform measure returns 409. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `slug` | path | string | yes | | **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `slug` | string \| null | no | | | `label` | string \| null | no | | | `description` | string \| null | no | | | `unit` | string \| null | no | | | `grain` | string \| null | no | | | `agg` | string \| null | no | | | `source` | string \| null | no | | | `outcomeKind` | string \| null | no | | | `outcomeSource` | string \| null | no | | | `metadataPath` | string \| null | no | | | `classifierMetricKey` | string \| null | no | | | `classifierLabels` | string[] \| null | no | | | `rateOp` | string \| null | no | | | `rateValue` | number \| null | no | | | `caseReducer` | string \| null | no | | | `filters` | object \| null | no | | | `direction` | string \| null | no | | | `target` | number \| null | no | | | `category` | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `DELETE /custom-measures/{slug}` Delete Custom Measure Delete a measure. Reports what still REFERENCES it rather than silently orphaning them. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `slug` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ## docs ### `GET /docs-search` Docs Search Search the shipped documentation. Returns ranked page snippets (may be empty). Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `q` | query | string | no | Search query over the product documentation. | | `limit` | query | integer | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ## fix-engine ### `GET /fix-engine/runs` List Fix Runs List fix runs (project-scoped), optionally filtered by remediation / status. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `remediationId` | query | string \| null | no | | | `status` | query | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /fix-engine/runs` Create Fix Run Launch a Neens-orchestrated fix run for an accepted remediation. Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `remediationId` | string | yes | | | `driver` | string \| null | no | | | `vcsInstallationId` | string \| null | no | | | `repoUrl` | string \| null | no | | | `baseBranch` | string \| null | no | | | `versionLabel` | string \| null | no | | | `endpointUrl` | string \| null | no | | | `agentConnectionId` | string \| null | no | | | `maxAttempts` | integer \| null | no | | | `passK` | integer \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /fix-engine/runs/{run_id}` Get Fix Run Fix-run detail (incl. eval/regression report + verifying judges + PR link). Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `run_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /fix-engine/runs/{run_id}/cancel` Cancel Fix Run Cancel a fix run (terminal). A run already in a terminal state cannot be cancelled. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `run_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /fix-engine/vcs-installations` List Vcs Installations List the project's VCS installations. NEVER returns the decrypted credential. Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `POST /fix-engine/vcs-installations` Create Vcs Installation Register a GitHub App / local-git installation the fix engine opens PRs against. The credential (GitHub App PEM / token) is Fernet-encrypted at rest and NEVER returned. Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `provider` | string | yes | | | `name` | string | yes | | | `repoUrl` | string \| null | no | | | `baseBranch` | string \| null | no | | | `config` | object \| null | no | | | `credential` | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `201` | Successful Response | | `422` | Validation Error | ### `DELETE /fix-engine/vcs-installations/{inst_id}` Delete Vcs Installation Delete a VCS installation. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `inst_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `204` | Successful Response | | `422` | Validation Error | ### `POST /fix-engine/vcs-installations/{inst_id}/test` Test Vcs Installation Bounded reachability/auth probe: for ``github_app`` mint an installation token; for ``local_git`` run ``git ls-remote``. Honest ``{ok, detail}``; never raises to a 500. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `inst_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ## kpi-definitions ### `GET /kpi-definitions` Get Kpi Definitions This project's EFFECTIVE KPI definitions (declared values merged over platform defaults). Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `PUT /kpi-definitions` Put Kpi Definitions Declare what this project's KPIs mean. Admin-only (``MANAGE_OUTCOMES``). Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `definitions` | object \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /kpi-definitions/options` Kpi Options The vocabulary an admin can actually pick from — the platform catalogue **plus what this project has really observed**. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `metricKey` | query | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ## kpis ### `GET /kpis` List Kpis This project's KPIs. ``status`` = ``live`` (default: draft + active) | ``active`` | ``draft`` | ``archived`` | ``all``. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `status` | query | string | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /kpis` Create Kpi Promote a measure to a commitment. Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `measureKey` | string \| null | no | | | `measure_key` | string \| null | no | | | `label` | string \| null | no | | | `description` | string \| null | no | | | `direction` | string \| null | no | | | `target` | number \| null | no | | | `owner` | string \| null | no | | | `status` | string \| null | no | | | `priority` | integer \| null | no | | | `reviewCadence` | string \| null | no | | | `review_cadence` | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `201` | Successful Response | | `422` | Validation Error | ### `GET /kpis/options` Kpi Options The measures this project may promote, each annotated with what the project actually has. Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `POST /kpis/snapshot` Snapshot All Kpis Materialize daily history for every ACTIVE KPI in the caller's project scope. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `days` | query | integer \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /kpis/summary` Kpi Summary Every ACTIVE KPI with its current value, in display order. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `range` | query | string | no | | | `from` | query | string \| null | no | | | `to` | query | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /kpis/{kpi_id}` Get Kpi One KPI plus its current value over ``?range=``. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `kpi_id` | path | string | yes | | | `range` | query | string | no | | | `from` | query | string \| null | no | | | `to` | query | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `PATCH /kpis/{kpi_id}` Patch Kpi Partially update a KPI. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `kpi_id` | path | string | yes | | **Request body** (`application/json`) — required Schema: `object`. **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /kpis/{kpi_id}/archive` Archive Kpi Retire a commitment. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `kpi_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /kpis/{kpi_id}/calibration` Kpi Calibration How well this KPI's LLM classifier agrees with human ground truth. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `kpi_id` | path | string | yes | | | `days` | query | integer | no | | | `window` | query | integer \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /kpis/{kpi_id}/eroding-clusters` Kpi Eroding Clusters The failure clusters eroding this KPI, worst first. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `kpi_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /kpis/{kpi_id}/history` Kpi History This KPI's materialized daily history, plus where the series BREAKS. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `kpi_id` | path | string | yes | | | `days` | query | integer | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /kpis/{kpi_id}/snapshot` Snapshot Kpi Materialize daily history for ONE KPI. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `kpi_id` | path | string | yes | | | `days` | query | integer \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ## mcp ### `GET /mcp` Mcp Get Server→client SSE stream is not offered (stateless server). Beating the SPA GET catch-all with an explicit 405 so a client's optional GET probe gets a clean answer, not the app shell. Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `POST /mcp` Mcp Endpoint MCP Streamable HTTP endpoint. Accepts a single JSON-RPC message or a batch array. Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ## mcp-oauth ### `GET /.well-known/oauth-authorization-server` Oauth Authorization Server RFC 8414 authorization-server metadata (fully wired). Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `GET /.well-known/oauth-protected-resource` Oauth Protected Resource RFC 9728 protected-resource metadata (fully wired). Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `GET /.well-known/oauth-protected-resource/mcp` Oauth Protected Resource Mcp RFC 9728 metadata under the resource path some clients probe (fully wired). Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `GET /auth/mcp/tokens` List My Mcp Tokens List the caller's OWN active ``nk_mcp_`` tokens (bearer ``nk_sess_``/``nk_mcp_`` auth). Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `DELETE /auth/mcp/tokens/{token_id}` Revoke My Mcp Token Revoke one of the caller's OWN ``nk_mcp_`` tokens by id (bearer-authenticated). Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `token_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /oauth/authorize` Oauth Authorize OAuth 2.1 authorization endpoint (the browser entry point from Claude Code). Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `response_type` | query | string | no | | | `client_id` | query | string | no | | | `redirect_uri` | query | string | no | | | `code_challenge` | query | string | no | | | `code_challenge_method` | query | string | no | | | `state` | query | string | no | | | `scope` | query | string | no | | | `resource` | query | string | no | | | `login_hint` | query | string | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /oauth/authorize/decision` Oauth Authorize Decision The SPA consent callback (bearer ``nk_sess_``-authenticated). Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `POST /oauth/introspect` Oauth Introspect RFC 7662 token introspection. Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `POST /oauth/register` Oauth Register Full Dynamic Client Registration (RFC 7591) + RFC 7592 registration credential. Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `GET /oauth/register/{client_id}` Oauth Register Get RFC 7592 client read — return the current registration (registration-token authed). Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `client_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `PUT /oauth/register/{client_id}` Oauth Register Put RFC 7592 client update — apply whitelisted metadata (registration-token authed). Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `client_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `DELETE /oauth/register/{client_id}` Oauth Register Delete RFC 7592 client delete — soft-disable the client (registration-token authed). 204. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `client_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /oauth/revoke` Oauth Revoke RFC 7009 token revocation — user-scoped. Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `POST /oauth/token` Oauth Token OAuth 2.1 token endpoint (``grant_type=authorization_code``). Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ## model-prices ### `GET /model-prices` List Model Prices The tenant's price table — platform list prices plus this project's own overrides. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `includeHistory` | query | boolean | no | | | `provider` | query | string \| null | no | | | `q` | query | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /model-prices` Create Model Price Create (or idempotently replace) a TENANT price override. Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `modelId` | string | yes | | | `provider` | string \| null | no | | | `displayName` | string \| null | no | | | `familyPrefix` | string \| null | no | | | `inputPerMillion` | number \| null | no | | | `cachedInputPerMillion` | number \| null | no | | | `outputPerMillion` | number \| null | no | | | `effectiveFrom` | string \| null | no | | | `effectiveTo` | string \| null | no | | | `notes` | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `201` | Successful Response | | `422` | Validation Error | ### `GET /model-prices/observed` Observed Models The models this project ACTUALLY ran over the window, joined against the resolved table. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `range` | query | string | no | | | `from` | query | string \| null | no | | | `to` | query | string \| null | no | | | `projectId` | query | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `PATCH /model-prices/{price_id}` Update Model Price Edit a TENANT price row. A ``platform`` row is immutable to the tenant → **409**. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `price_id` | path | string | yes | | **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `provider` | string \| null | no | | | `displayName` | string \| null | no | | | `inputPerMillion` | number \| null | no | | | `cachedInputPerMillion` | number \| null | no | | | `outputPerMillion` | number \| null | no | | | `effectiveFrom` | string \| null | no | | | `effectiveTo` | string \| null | no | | | `notes` | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `DELETE /model-prices/{price_id}` Delete Model Price Delete a TENANT override — the model reverts to the platform list price, or to *unpriced* if the catalogue does not name it. A ``platform`` row is immutable to the tenant → **409**. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `price_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `204` | Successful Response | | `422` | Validation Error | ## model-sweeps ### `GET /model-sweeps` List Model Sweeps List sweeps (project-scoped), newest first. Headers only — the per-arm detail is one read away and a list page must not fan out N reconciles. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `status` | query | string \| null | no | | | `limit` | query | integer | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /model-sweeps` Create Model Sweep Route Create a sweep + its arms and enqueue the launch onto the DEDICATED pre-prod fleet. Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `datasetId` | string \| null | no | | | `datasetVersionId` | string \| null | no | | | `scenarioSuiteId` | string \| null | no | | | `arms` | ModelSweepArmBody[] | no | | | `passK` | integer \| null | no | | | `judgeDeploymentIds` | string[] \| null | no | | | `budgetUsd` | number \| null | no | | | `name` | string | yes | | | `versionLabel` | string | yes | | | `gate` | object \| null | no | | | `baseline` | object \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /model-sweeps/preview` Preview Model Sweep Pre-flight cost estimate — **writes nothing, spends nothing**. Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `datasetId` | string \| null | no | | | `datasetVersionId` | string \| null | no | | | `scenarioSuiteId` | string \| null | no | | | `arms` | ModelSweepArmBody[] | no | | | `passK` | integer \| null | no | | | `judgeDeploymentIds` | string[] \| null | no | | | `budgetUsd` | number \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /model-sweeps/{sweep_id}` Get Model Sweep Sweep detail: header + arms + per-arm child runs, **reconciled live**. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `sweep_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /model-sweeps/{sweep_id}/arms/{arm_id}/regressions` Sweep Arm Regressions Where this arm breaks: the golden prompts the BASELINE stably passed and this arm did not. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `sweep_id` | path | string | yes | | | `arm_id` | path | string | yes | | | `baselineArm` | query | string \| null | no | | | `limit` | query | integer | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /model-sweeps/{sweep_id}/cancel` Cancel Model Sweep Cancel a sweep + every non-terminal child pre-prod run it launched. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `sweep_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /model-sweeps/{sweep_id}/comparison` Sweep Comparison The N-way comparison: per-arm quality + cost + regressions, the per-agent split, and ONE verdict — reconciled live so a running sweep is current. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `sweep_id` | path | string | yes | | | `bar` | query | number \| null | no | pass-rate bar in 0–1; overrides the sweep's gate | | `baselineArm` | query | string \| null | no | arm id to compare against (default: position 0) | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /model-sweeps/{sweep_id}/frontier` Sweep Frontier The agent-model cost–quality frontier for a sweep — per-arm cost (per case) vs stable-pass quality, with each arm classified (recommended / on_frontier / overpaying / quality_risk / unknown) and ONE recommended move. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `sweep_id` | path | string | yes | | | `bar` | query | number \| null | no | pass-rate bar in 0–1; overrides the sweep's gate | | `baselineArm` | query | string \| null | no | arm id to treat as the incumbent (default: pos 0) | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ## other ### `GET /app-config` App Config Front-end bootstrap: docs URL + release version. Public, non-tenant, cache-friendly. Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `GET /health` Health Liveness: the process is up. Always 200, cheap, no I/O. Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `GET /ready` Ready Readiness: probe the backends this role depends on. 200 if all ok, else 503. Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ## outcome-inlets ### `GET /outcome-inlets` List Connectors Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `POST /outcome-inlets` Create Connector Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `provider` | string | yes | | | `name` | string | yes | | | `baseUrl` | string | yes | | | `credential` | string \| null | no | | | `options` | object \| null | no | | | `enabled` | boolean \| null | no | | | `pollIntervalMinutes` | integer \| null | no | | | `backfillDays` | integer \| null | no | | **Responses** | Status | Description | | --- | --- | | `201` | Successful Response | | `422` | Validation Error | ### `PATCH /outcome-inlets/{inlet_id}` Update Connector Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `inlet_id` | path | string | yes | | **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `name` | string \| null | no | | | `baseUrl` | string \| null | no | | | `credential` | string \| null | no | | | `options` | object \| null | no | | | `enabled` | boolean \| null | no | | | `pollIntervalMinutes` | integer \| null | no | | | `backfillDays` | integer \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `DELETE /outcome-inlets/{inlet_id}` Delete Connector Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `inlet_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `204` | Successful Response | | `422` | Validation Error | ### `POST /outcome-inlets/{inlet_id}/sync` Sync Connector Run this connector's sync NOW (bounded), synchronously off the event loop. Never 500s on a connector error — returns ``{ok: false,...}``. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `inlet_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /outcome-inlets/{inlet_id}/test` Test Connector Bounded reachability/auth probe. Honest error string on failure; never raises to a 500. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `inlet_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ## outcomes ### `GET /outcomes` List Outcomes Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `kind` | query | string \| null | no | | | `source` | query | string \| null | no | | | `matchState` | query | string \| null | no | | | `correlationKey` | query | string \| null | no | | | `conversationId` | query | string \| null | no | | | `sessionId` | query | string \| null | no | | | `since` | query | string \| null | no | | | `until` | query | string \| null | no | | | `includeSuperseded` | query | boolean | no | | | `limit` | query | integer | no | | | `offset` | query | integer | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /outcomes` Push Outcomes Record one or many business outcomes. Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `correlationKey` | string \| null | no | | | `correlationType` | string \| null | no | | | `kind` | string \| null | no | | | `value` | object | no | | | `unit` | string \| null | no | | | `valueType` | string \| null | no | | | `occurredAt` | object | no | | | `metadata` | object | no | | | `idempotencyKey` | string \| null | no | | | `externalId` | string \| null | no | | | `source` | string \| null | no | | | `outcomes` | OutcomeIn[] \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /outcomes/coverage` Outcome Coverage Match coverage — the honest denominator. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `since` | query | string \| null | no | | | `until` | query | string \| null | no | | | `kind` | query | string \| null | no | | | `source` | query | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /outcomes/kinds` List Kinds The platform outcome catalogue. An unknown kind is still accepted on push (forward compatibility with tenant-defined KPIs) — this is the curated set the UI offers. Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `POST /outcomes/reconcile` Reconcile Now Re-attempt correlation for this project's parked outcomes, right now. Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `GET /outcomes/settings` Get Settings Route Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `PUT /outcomes/settings` Put Settings Route Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `externalKeyPaths` | string[] \| null | no | | | `matchWindowHours` | integer \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /outcomes/{outcome_id}` Get Outcome Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `outcome_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /outcomes/{outcome_id}/revisions` Get Outcome Revisions Every revision of the outcome's series, oldest → newest. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `outcome_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ## prompt-optimization ### `GET /prompt-optimization/preview` Preview Optimization Pre-flight: the resolved task set, the recovered baseline prompt, the graders, and an ESTIMATED rollout count — everything a user needs to judge the cost BEFORE spending a cent of their own LLM budget. Read-only; makes no LLM call and writes nothing. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `failureModeId` | query | string \| null | no | | | `clusterId` | query | string \| null | no | | | `maxTasks` | query | integer \| null | no | | | `valFraction` | query | number \| null | no | | | `minibatchSize` | query | integer \| null | no | | | `maxIterations` | query | integer \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /prompt-optimization/runs` List Optimization Runs List optimization runs (project-scoped), newest first. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `status` | query | string \| null | no | | | `failureModeId` | query | string \| null | no | | | `limit` | query | integer | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /prompt-optimization/runs` Create Optimization Run Launch a GEPA-style offline prompt-optimization run. Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `failureModeId` | string \| null | no | | | `clusterId` | string \| null | no | | | `name` | string \| null | no | | | `sessionIds` | string[] \| null | no | | | `judgeIds` | string[] \| null | no | | | `connectionId` | string \| null | no | | | `objective` | string \| null | no | | | `maxRollouts` | integer \| null | no | | | `maxIterations` | integer \| null | no | | | `minibatchSize` | integer \| null | no | | | `maxTasks` | integer \| null | no | | | `valFraction` | number \| null | no | | | `minImprovement` | number \| null | no | | | `autoEmit` | boolean \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /prompt-optimization/runs/{run_id}` Get Optimization Run Run detail: the run + its candidate lineage + the Pareto frontier (recomputed over the recorded per-task train scores, so the UI's badges are derived from the same pure function the loop used) + the outcome verdict + the markdown proof report. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `run_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /prompt-optimization/runs/{run_id}/cancel` Cancel Optimization Run Cancel a queued/running run (terminal). The worker re-reads the status at the top of every iteration, so an in-flight run stops at the next boundary instead of spending the rest of the budget; every terminal write it makes is guarded ``AND status NOT IN ('cancelled')``. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `run_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /prompt-optimization/runs/{run_id}/candidates/{candidate_id}` Get Optimization Candidate One candidate + its recorded rollouts (bounded), so a reviewer can read the actual replayed output and the judge's reasoning behind every per-task score. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `run_id` | path | string | yes | | | `candidate_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /prompt-optimization/runs/{run_id}/emit-remediation` Emit Remediation Manually emit the winning candidate as a ``prompt_change`` remediation (the ``autoEmit=false`` path). 409 when the run already emitted one; 422 when it did not clear the held-out bar — a run that never beat baseline has nothing to propose, and that is a legitimate result, not an error. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `run_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ## redaction ### `POST /redaction/preview` Preview Redaction Run the project's policy (± a mode override) over a sample. NEVER persists the text. Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `text` | string | yes | | | `mode` | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /redaction/settings` Get Redaction Settings The per-project policy row + the resolved effective modes (for the governance UI). Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `PUT /redaction/settings` Put Redaction Settings Set the per-project policy. Admin-gated (``MANAGE_RETENTION`` — the same privacy-admin persona that owns retention/erasure; deliberately no new authz action). Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `mode` | string | yes | | | `enforced` | boolean | no | | | `egressMode` | string | no | | | `allowlist` | string[] | no | | | `customPatterns` | CustomPatternBody[] | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ## run-targets ### `GET /run-targets` List Run Targets Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `POST /run-targets` Create Run Target Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `name` | string | yes | | | `kind` | string | yes | | | `grain` | string | yes | | | `filter` | object \| null | no | | | `sampling` | object \| null | no | | | `datasetId` | string \| null | no | | | `datasetVersionId` | string \| null | no | | | `followLatest` | boolean | no | | | `defaultTrigger` | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /run-targets/{run_target_id}` Get Run Target Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `run_target_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `PATCH /run-targets/{run_target_id}` Patch Run Target Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `run_target_id` | path | string | yes | | **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `name` | string \| null | no | | | `filter` | object \| null | no | | | `sampling` | object \| null | no | | | `datasetId` | string \| null | no | | | `datasetVersionId` | string \| null | no | | | `followLatest` | boolean \| null | no | | | `defaultTrigger` | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `DELETE /run-targets/{run_target_id}` Delete Run Target Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `run_target_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ## scenario-suites ### `GET /scenario-suites` List Suites Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `POST /scenario-suites` Create Suite Create a scenario suite from a set of failure modes + kick off generation. Details **Request body** (`application/json`) — required Schema: `object`. **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /scenario-suites/{suite_id}` Get Suite Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `suite_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `DELETE /scenario-suites/{suite_id}` Delete Suite Hard-delete a suite AND its owned synthetic dataset (items + versions). A launched pre-prod run already snapshotted its items, so it is unaffected. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `suite_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /scenario-suites/{suite_id}/launch` Launch Suite One-click: create a pre-prod eval run over this suite's synthetic golden dataset. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `suite_id` | path | string | yes | | **Request body** (`application/json`) Schema: `object`. **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /scenario-suites/{suite_id}/regenerate` Regenerate Suite Re-run generation for a suite (e.g. after adding exemplars or configuring an LLM). Optional body may override ``per_mode``/``strategies``/``include_replay``/``connection_id``. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `suite_id` | path | string | yes | | **Request body** (`application/json`) Schema: `object`. **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /scenario-suites/{suite_id}/scenarios` List Scenarios Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `suite_id` | path | string | yes | | | `limit` | query | integer | no | | | `offset` | query | integer | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ## scorer-comparisons ### `GET /scorer-comparisons` List Scorer Comparisons List comparisons (project-scoped), newest first — headers only. Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `POST /scorer-comparisons` Create Scorer Comparison Route Create a scorer comparison and start scoring. Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `name` | string | yes | | | `judgeId` | string | yes | | | `scorerConnectionIds` | string[] | no | | | `window` | string \| null | no | | | `windowStart` | string \| null | no | | | `windowEnd` | string \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /scorer-comparisons/{comparison_id}` Get Scorer Comparison Comparison detail: meta + status/progress + the per-scorer side-by-side aggregate. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `comparison_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /scorer-comparisons/{comparison_id}/cancel` Cancel Scorer Comparison Cancel a comparison that is still in progress. Scoring stops shortly after; any results already computed are kept. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `comparison_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /scorer-comparisons/{comparison_id}/run` Run Scorer Comparison Route (Re)run a comparison to fill in any missing scores. Already-computed results are kept; only traces that have not yet been scored by a given scorer are scored. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `comparison_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ## share-links ### `GET /share-links` List Share Links Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `POST /share-links` Create Share Link Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `resourceType` | string | yes | | | `resourceId` | string | yes | | | `ttlHours` | integer \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /share-links/settings` Get Share Settings Per-project toggle state + the global flag / TTL bounds (for the UI). Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `PUT /share-links/settings` Put Share Settings Flip the per-project share toggle. Admin-gated: enabling PUBLIC (even redacted) sharing is a sensitive project-config / data-governance decision, so it reuses the admin ``DELETE_RESOURCE`` capability. (A dedicated ``MANAGE_SHARE_LINKS`` action is the clean follow-up — deferred to avoid editing the shared authz matrix concurrently.) Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `enabled` | boolean | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `DELETE /share-links/{link_id}` Revoke Share Link Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `link_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /share/{token}` Public Share View Resolve a share token → pin the owning company's schema → load + REDACT the incident. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `token` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ## spend-budgets ### `GET /spend-budgets` List Budgets Route Every known feature's budget for this project (unconfigured features read as unlimited). Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `GET /spend-budgets/{feature}` Get Budget Route One feature's budget. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `feature` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `PUT /spend-budgets/{feature}` Put Budget Route Set a feature's spend limit + period. ``limitUsd: null`` clears the limit. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `feature` | path | string | yes | | **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `limitUsd` | number \| null | no | | | `period` | string \| null | no | | | `enabled` | boolean \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /spend-budgets/{feature}/usage` Budget Usage Route The current period's spend against the limit, plus the ledger rows behind it. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `feature` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ## trace-inlets ### `GET /trace-inlets` List Connectors Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ### `POST /trace-inlets` Create Connector Details **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `provider` | string | yes | | | `name` | string | yes | | | `baseUrl` | string | yes | | | `credential` | string \| null | no | | | `options` | object \| null | no | | | `enabled` | boolean \| null | no | | | `pollIntervalMinutes` | integer \| null | no | | | `backfillDays` | integer \| null | no | | **Responses** | Status | Description | | --- | --- | | `201` | Successful Response | | `422` | Validation Error | ### `PATCH /trace-inlets/{inlet_id}` Update Connector Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `inlet_id` | path | string | yes | | **Request body** (`application/json`) — required | Field | Type | Required | Description | | --- | --- | --- | --- | | `name` | string \| null | no | | | `baseUrl` | string \| null | no | | | `credential` | string \| null | no | | | `options` | object \| null | no | | | `enabled` | boolean \| null | no | | | `pollIntervalMinutes` | integer \| null | no | | | `backfillDays` | integer \| null | no | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `DELETE /trace-inlets/{inlet_id}` Delete Connector Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `inlet_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `204` | Successful Response | | `422` | Validation Error | ### `POST /trace-inlets/{inlet_id}/sync` Sync Connector Run this connector's sync NOW (bounded), synchronously off the event loop. Never 500s on a connector error — returns ``{ok: false,...}``. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `inlet_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `POST /trace-inlets/{inlet_id}/test` Test Connector Bounded reachability/auth probe. Honest error string on failure; never raises to a 500. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `inlet_id` | path | string | yes | | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ## usage ### `GET /usage` Get Usage Return the current tenant's usage aggregates, grouped by ``group_by``. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `since` | query | string \| null | no | Inclusive start day (YYYY-MM-DD); default 30d ago | | `until` | query | string \| null | no | Inclusive end day (YYYY-MM-DD); default today | | `project_id` | query | string \| null | no | Restrict to one project id | | `group_by` | query | string | no | feature \| day \| project \| connection | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /usage/kpis` Get Usage Kpis Headline KPI tiles for the tenant Usage page. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `since` | query | string \| null | no | Inclusive start day (YYYY-MM-DD); default 30d ago | | `until` | query | string \| null | no | Inclusive end day (YYYY-MM-DD); default today | | `project_id` | query | string \| null | no | Restrict to one project id | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ### `GET /usage/summary` Get Usage Summary Totals per feature for the window + a daily timeseries for the headline features. Details **Parameters** | Parameter | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `since` | query | string \| null | no | Inclusive start day (YYYY-MM-DD); default 30d ago | | `until` | query | string \| null | no | Inclusive end day (YYYY-MM-DD); default today | | `project_id` | query | string \| null | no | Restrict to one project id | **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | | `422` | Validation Error | ## webhooks ### `POST /webhooks/github` Github Webhook Handle a GitHub webhook delivery. ``ping`` ⇒ ``{ok:true}``; a MERGED ``pull_request`` closed event ⇒ record a close-out watch on the owning remediation. Auth is the HMAC signature only. Details **Responses** | Status | Description | | --- | --- | | `200` | Successful Response | ================================================================================ # Workspace, orgs & agents Source: /docs/administration/workspace/ ================================================================================ # Workspace, orgs & agents Your Neens workspace is a three-level hierarchy — the **company** (your whole workspace) contains **organizations**, and each organization contains **agents**. Everything you send, score, and diagnose lives in an agent; orgs group related agents; the company is the isolation boundary that keeps your data physically separate from every other tenant. ## At a glance | | | | --- | --- | | Where | **Agents** page in the left nav; workspace naming in **Getting started** | | Key API routes | `GET/POST /orgs`, `PATCH /orgs/{id}`, `GET/POST/PATCH/DELETE /projects/{id}`, `GET /onboarding/tenant` | | Who can create | Admins and members create orgs and agents; viewers are read-only | | Who can delete | Admins only — deletion cascades to all agent data | | Visibility | Admins see the whole company; members and viewers see only the orgs/agents they belong to | ``` Company (your workspace) └── Organization └── Agent ← traces, scores, judges, datasets, annotations ``` ## Create an org or agent Open **Agents** in the left nav: - **New org** — creates an organization inside your company. Admins and members can create orgs (`POST /orgs`); viewers cannot. The creator automatically becomes an admin of the new org, so it appears in their workspace immediately. - **New agent** — creates an agent inside an org (`POST /projects`). A brand-new agent starts empty — no traces, scores, or [API keys](/administration/api-keys) yet — but the Neens default judges are available to it right away. Your plan caps how many orgs and agents the company can hold. Creating one past the cap is rejected with a clear error — contact your operator to raise the limit. Click any agent card to make it the **active agent**; most pages (Traces, Scores, Datasets, …) show the active agent's data. Star a card to **favorite** it — favorites are personal to you and float to the top of the Agents page. ## Rename, move, and delete Each agent card has a **⋮** (more) menu: | Action | What it does | Who | | --- | --- | --- | | **Rename** | Change the agent's display name (`PATCH /projects/{id}`). | Admins & members | | **Move to org** | Move the agent to another org **in your own company** — the target org must exist in your workspace, so an agent can never move across tenants. | Admins & members | | **Delete** | Permanently remove the agent and **every trace, score, dataset, annotation, and API key** scoped to it. | Admins only | Orgs can be renamed the same way (`PATCH /orgs/{id}`) — renaming your org and first agent is also the first step of the [Getting started checklist](#the-getting-started-checklist), which is how you "rename the workspace" after provisioning. **Deleting an agent can't be undone.** The delete cascades through every data-plane table scoped to the agent. The workspace's original **Default** agent is protected and cannot be deleted (the API returns `409`). ## Who sees what Visibility inside a company is **membership-based**: - **Admins** see every org and agent in the company — no membership filter applies. - **Members and viewers** see only the orgs and agents they hold a membership grant for: - an **org** grant makes the org visible **and every agent inside it** accessible; - an **agent** grant makes that agent accessible and shows its org as a container (only the granted agents inside it appear). - A member invited with **no organizations** lands on an empty workspace (the "zero state") and is prompted to create their first org — they become its admin, so it appears for them immediately. Org assignment happens when a teammate is invited — see [Members & roles](/administration/members-and-roles). Enforcement is hard: list endpoints filter to the caller's grants, and requesting an agent you have no grant for is denied. **Narrower always means less, never more.** Every read — a session list, a dashboard tile, a spend or token roll-up, the filter dropdowns, an API or [MCP](/guides/mcp) call made with a agent [API key](/administration/api-keys) — is resolved against the caller's accessible agent set, and a caller whose set is **empty** sees **nothing**. That is worth stating plainly because the alternative is the dangerous one: an unscoped aggregate would quietly return every agent in the company. A viewer opening a shared dashboard pinned to agents they hold no grant for gets empty charts, not someone else's numbers. ## The Getting started checklist The first time a newly provisioned company's **admin** signs in, Neens shows a **Getting started** checklist (also reachable from the left nav; members and viewers never see it — tenant setup is the admin's job). It has five steps, in order: ### Name your workspace Rename the seeded default org and agent to names your team will recognize. The step completes on its own once you've added a second org or agent, or when you acknowledge it. ### Invite your team Send activation links to teammates ([Members & roles](/administration/members-and-roles)). Completes once the company has more than one member. ### Connect an LLM Confirm or add an [LLM connection](/administration/llm-connections) so judges can score your traces. The step shows **ready** as soon as any connection is visible to your default agent. ### Assign views Give teammates a default [view (persona lens)](/administration/personas) so each lands on the right home page. Completes once any member has an explicit lens assigned — or acknowledge it to keep the role-based defaults. ### Set up your first agent Hands off to the agent onboarding wizard, which mints your ingestion [API key](/administration/api-keys) and sets up automatic evaluation. Completes once the agent is provisioned or has received traces. Each step completes from a **live signal** (the checklist re-checks your workspace on every load) *or* from an explicit acknowledgement — so a step you completed out-of-band (say, you invited someone from Settings) still lights up. **Skip for now** dismisses the tracker; **Finish** marks it complete. Progress is per-company and admins share it. ## How it works - Every data row in Neens carries a `project_id`, stamped at write time, and each company's data lives in physically isolated storage — so org/agent scoping is enforced *inside* your company on top of hard cross-tenant isolation. - Step state for the checklist is computed live (`GET /onboarding/tenant`) from your orgs, agents, members, connections, and view assignments, merged with any explicit acknowledgements (`POST /onboarding/tenant/ack`). - Org and agent mutations are recorded to the [audit log](/administration/audit-log) (`org.create`, `project.delete`, …). ## Related - [Members & roles](/administration/members-and-roles) — invite teammates and assign orgs. - [API keys](/administration/api-keys) — mint a `nk_live_` key per agent. - [LLM connections](/administration/llm-connections) — power judges, insights, and the assistant. - [Send traces](/guides/send-traces) — start ingesting into your new agent. ================================================================================ # Members & roles Source: /docs/administration/members-and-roles/ ================================================================================ # Members & roles The people in your workspace are managed under **Settings → Members**. Every member holds one of three roles — **viewer**, **member**, or **admin** — and (for non-admins) a set of organization assignments that bound what they can see. Inviting, changing roles, and removing members is admin-only. ## At a glance | | | | --- | --- | | Where | **Settings → Members** (each row's **Manage** button opens a member detail drawer) | | Key API routes | `GET /members`, `GET /members/{id}/activity`, `POST /members/invite`, `PATCH /members/{id}`, `PUT /members/{id}/orgs`, `DELETE /members/{id}`, `POST /members/{id}/resend-invite`, `POST /members/bulk-orgs`, `POST /members/bulk-persona` | | Roles | `viewer` < `member` < `admin` (ordered — a higher role can do everything a lower one can) | | Who manages members | Admins (an admin can be scoped to specific orgs — see [Org-scoped admins](#org-scoped-admins)) | | Statuses | **active**, **invited** (not yet activated), **disabled** (suspended) | ## Roles | Role | Summary | | --- | --- | | **Admin** | Full control: sees every org and agent, manages members, company-wide LLM connections, the failure taxonomy, custom views, and the [audit log](/administration/audit-log). Can create any API key (including admin-role), and can revoke/delete any key or connection regardless of who created it. | | **Member** | The working role: sends traces, creates orgs/agents/datasets/dashboards, configures judges, schedules runs, reviews and labels sessions — within the orgs and agents they belong to. Can also **create API keys** (up to their own role) and **org/agent LLM connections**, and delete/revoke the ones **they created** (admin-created resources stay protected). | | **Viewer** | Read-only access to the orgs and agents they belong to. | A role can be granted at **org** scope (it applies to every agent in the org) or **agent** scope; when both apply, the higher role wins. Full permission matrix (minimum role per action) | Action | Minimum role | | --- | --- | | Read all data (traces, sessions, clusters, scores, judges, datasets, …) | viewer | | Send traces (ingest) | member | | Create/edit datasets, topics, annotations, labels | member | | Create/edit judge versions | member | | Enable/disable judge deployments | member | | Start scorer/enrichment runs | member | | Cancel an in-flight run | member | | Create/modify org- or agent-scoped LLM connections | member | | Create an API key (mint) — up to your own role | member | | Revoke an API key / delete a connection **you created** | member | | Create agents | member | | Create orgs | member | | Create a custom dashboard | member | | Share a dashboard org-wide | member | | Review: add ground-truth labels / critiques | member | | Delete judges/datasets/topics (destructive) | admin | | Create an **admin**-role API key | admin | | Revoke / delete an API key or connection **created by someone else** | admin | | Create/modify/delete **company-wide** LLM connections | admin | | Manage agent membership | admin | | Manage org membership (invite/remove members, change roles) | admin | | Share a dashboard company-wide | admin | | Delete a dashboard (non-owner) | admin | | Manage the failure taxonomy (merge/split/rename modes) | admin | | Confirm a candidate cluster as a failure mode | admin | | View the audit log | admin | | Author/manage custom views (personas) | admin | ## Invite a teammate ### Open the invite form In **Settings → Members**, click **Invite member** and enter their **email** (and optionally a display name). ### Choose a role `admin`, `member`, or `viewer`. ### Set organization access Pick the org assignment: - **All organizations** — every existing org in the company. For an admin this makes them a **company-wide admin** (the default); for a member or viewer they join every org. - **Specific organizations** — exactly the orgs you pick. For an admin this makes them an **org-scoped admin** ([restricted to those orgs](#org-scoped-admins)); for a member or viewer it bounds what they can see. - **No organizations** *(member/viewer only)* — the zero state: they start with an empty workspace and are prompted to create their own org (becoming its admin). If you don't specify anything, a member or viewer defaults to the company's primary org. ### Optionally pick a default view You can pre-assign a [view (persona lens)](/administration/personas) as the member's default. They still see the first-login picker — pre-selected on your recommendation — and can choose anything; a view is a layout default, never a permission. ### Send it Neens creates the member with status **invited** and a **one-time activation link** (`/activate?token=…`, valid for 14 days). The link is emailed to them, and also shown once in the UI as a fallback — copy it right away if your deployment has no email relay configured (in that case the mailer logs the message instead of sending). The invitee gets an email — subject `You're invited to on Neens` — with that link. They open it, set a password, and are signed in. If the link is lost or expires, use **Resend invite** on their row — it rotates the token and issues a fresh link (only possible while they're still **invited**; an activated account can't be re-invited). **Example.** To give a new analyst read-and-work access to just your **Support** org: click **Invite member**, enter `jordan@yourcompany.com`, choose the **member** role, pick **Specific organizations… → Support**, and optionally pre-select the **Support Ops** [view](/administration/personas) as their default. Send it. Jordan receives the invite email, clicks the link, sets a password, and lands in the Support org with the Support Ops layout — and no access to your other orgs. Invite them as **admin** with **All organizations** instead and they'd help you run the whole workspace. ### Didn't get the invite email? The invite email can take a minute, and can land in spam or be held by a corporate mail filter. If it hasn't arrived: - **Check spam/junk** and search for the sender, then mark it as safe so the reset and activation emails that follow aren't filtered too. - **Ask an admin to resend.** In **Settings → Members**, an admin opens the pending member's row and clicks **Resend invite**. That issues a *fresh* link and invalidates the previous one, so always use the most recent email. - **Copy the link directly.** Immediately after an invite (or a resend), Neens shows the one-time activation link once in the UI. An admin can copy it from there and pass it to you over your normal channel — useful when email delivery is slow. The link is single-use and expires after 14 days; once it's used or has expired, resend for a new one. Emails are globally unique across Neens. Inviting an address that's already registered returns a conflict — a generic one if it belongs to a different company, so membership elsewhere is never leaked. Your plan also caps the company's member count. ## Manage existing members Click **Manage** on a member row to open their detail drawer. It shows the member's role and status, when they **Joined**, their **Last sign-in**, their **Default view**, and an **Activity** timeline (see below). From the drawer, admins can: - **Role** — switch between admin, member, and viewer (`PATCH /members/{id}`). - **Suspend access** — set status to `disabled`. This revokes their active sessions and blocks sign-in immediately, but keeps the account, so you can **Reactivate** it any time. - **Organization access** — replace which orgs the member belongs to (`PUT /members/{id}/orgs`). For a member or viewer this scopes what they can see; for an admin it toggles between company-wide and org-scoped (see [Org-scoped admins](#org-scoped-admins)). Changing access never affects sign-in. - **Resend invite** — rotate the activation link for someone who hasn't activated (only shown while they're still **invited**). - **Remove member** — delete the member and revoke their sessions (`DELETE /members/{id}`). - **Reset two-factor authentication** — *operators only*, and only useful for someone who has lost both their authenticator and their recovery codes. It is a destructive confirm naming the user: it revokes their sessions, deletes their recovery codes and clears their enrollment, leaving the account password-only until they enroll again (`POST /admin/users/{user_id}/mfa/reset`). Verify who you are talking to through some channel other than the account you are about to unlock — see [Two-factor authentication](/administration/account#two-factor-authentication). You can also change a member's role and their default view directly from the row without opening the drawer. A member who has explicitly chosen their own view is never silently overridden — the UI reports the skip instead. ### Activity timeline The drawer's **Activity** section (`GET /members/{id}/activity`) is a lifecycle history for that one member: invites, role and status changes, org reassignments, resent invites, and removal — each with who did it and when. It's a focused, admin-only slice of the [audit log](/administration/audit-log) filtered to that member. **You can't lock yourself out.** Neens refuses to demote, suspend, or remove the company's only *usable* admin (an admin who is active and can sign in — an invited-but-never-activated admin doesn't count), and you can't remove yourself. ## Org-scoped admins By default an admin is **company-wide**: they see and manage every org and every member. You can instead make an admin **org-scoped** — restricted to specific orgs — from the drawer's **Organization access** section (or at invite time): pick **Specific organizations…** instead of **All organizations (company-wide admin)**. An org-scoped admin is a full admin *within their orgs only*: - They see and can manage only members who belong to one of their orgs (plus themselves). - They can only assign members to orgs they administer. - They can't create a company-wide admin — only more org-scoped admins confined to their own orgs. Company-wide admins and the operator are unaffected — they continue to see and manage the whole company. A member's or viewer's org list is a visibility scope only; an admin's org list is what they can *administer*. This is different from the operator control plane's own admin scoping (which governs the Neens operator's staff across customer companies). Everything on this page is about managing the members of **your** company. ## Bulk assignment Select multiple members on the Members page to act on them at once (admin-only): - **Assign orgs…** — replace the org access of every selected member in one step (`POST /members/bulk-orgs`). Each member keeps their own role. **Admins are skipped** (they're unrestricted, so their org scope isn't rewritten in bulk), and any member outside your own orgs is skipped too. - **Default view** — set (or clear) the selected members' default lens at once (`POST /members/bulk-persona`). This is available on every plan — assigning an existing view is member management, not view authoring. Members who chose their **own** view keep it — they come back as *skipped* — and leaving the view empty resets them to their role-based default. See [Views (personas)](/administration/personas) for what a view actually changes. ## How it works - Members live in a global directory keyed by email, so sign-in resolves a user to their company before any tenant data is touched. Org assignments are stored as membership grants in your company's own schema and drive [what non-admins can see](/administration/workspace#who-sees-what). - Activation and reset links are one-time tokens, stored hashed — Neens can show a link once but never recover it, which is why **Resend invite** rotates rather than re-displays. - Every membership mutation (`member.invite`, `member.update`, `member.set_orgs`, `member.bulk_set_orgs`, `member.remove`, `member.resend_invite`, `member.bulk_persona`) is recorded to the [audit log](/administration/audit-log) with before/after metadata. A member's own slice of that history powers the drawer's **Activity** timeline. ## Related - [Workspace, orgs & agents](/administration/workspace) — what org assignment scopes. - [Views (personas)](/administration/personas) — the default views you assign at invite time. - [Account settings](/administration/account) — what each member controls for themselves. ================================================================================ # Views (personas) Source: /docs/administration/personas/ ================================================================================ # Views (personas) A **view** (persona lens) tailors Neens to how you work: it changes which nav items are featured, which are tucked away, your landing page, and your default time range. It never changes what you're *allowed* to do — a view is a presentation default, and permissions come only from [roles](/administration/members-and-roles) and your plan. ## At a glance | | | | --- | --- | | Where | Topbar **View** switcher; first-login picker; authoring under **Settings → Views** | | Key API routes | `GET /personas`, `PUT /auth/persona`, `POST/PATCH/DELETE /personas` (authoring), `POST /members/bulk-persona` | | Switching views | Free for every user, every role, every plan | | Authoring custom views | Admins, on plans with the `custom_personas` feature (Silver and up) | | Permission impact | **None** — hidden nav items stay reachable by URL, search, and in-app links | **A view is never a permission.** Items a view hides are only hidden from the nav rail — every route stays reachable by direct URL, global search, and in-app links. Deleting the whole view layer would change nothing about security. ## The built-in views Neens ships nine platform views. Each has a home page, a default time range, and a featured/hidden nav split (everything else collapses under "All features"): | View | Focus | Lands on | | --- | --- | --- | | **Executive** | Fleet health at a glance — top issues and regressions | Executive digest dashboard (30-day window) | | **Developer** | The full build surface — traces, judges, enrichments, eval gates | Agent home (7-day window) | | **Product manager** | Quality trends, judge–human alignment, annotation throughput | PM quality dashboard (30-day window) | | **Business stakeholder** | Plain-language health for the orgs you care about | Org scorecard dashboard (7-day window) | | **Domain expert** | The review station — queue, disagreements, golden datasets | Review queue (7-day window) | | **Compliance & legal** | Eval-gate status, issues by severity, the audit trail | Compliance overview (30-day window) | | **Finance** | Spend and token usage by model, agent, and org | Cost explorer dashboard (30-day window) | | **Admin** | Run the workspace — members, connections, onboarding, audit | Getting started (7-day window) | | **Full workspace** | Everything featured, nothing hidden — the complete UI as one view | Overview | A user who has never picked or been assigned a view gets a **role-based default**: admins get **Admin**, members get **Developer**, viewers get **Business stakeholder**. ## Switch your view - **Topbar** — the **View** switcher (next to the language switcher) lists every view available in your workspace; pick one and the nav re-renders immediately. - **API** — `PUT /auth/persona` with a `personaKey`. An explicit pick is recorded as *your* choice: an admin-assigned default will never silently override it afterward. ### First-login picker The first time you sign in, Neens shows a one-time picker. It pre-selects your admin's recommendation (if one was set at invite time) or your role's suggested view. Pick one, or skip — skipping keeps the current default and the picker never blocks you again. ## View home dashboards Several views land on a **dashboard home**: choosing the view auto-pins its platform dashboards (e.g. the Executive digest, the Finance cost explorer) to your favorites, and its home page resolves to the first of them. Platform dashboards render on every plan but are immutable — **clone** one to customize it. See [Dashboards](/guides/dashboards). ## Assign views to your team Admins set a member's **default view** at invite time, from the member's row, or in bulk from the Members page (`POST /members/bulk-persona`) — see [Members & roles](/administration/members-and-roles#bulk-assignment). Assigning an existing view is free on every plan; only *authoring* views is gated. Members who explicitly chose their own view are never overridden. ## Author custom views Admins on plans with the `custom_personas` feature (Silver and up) can tailor views under **Settings → Views**: ### Create a view Name it, write a short description and up to four focus areas, pick a home route and default time range, choose the featured and hidden nav items, and optionally attach dashboards (the first becomes the view's home). `POST /personas`. ### Or customize a platform view Editing a built-in view **freezes it**: it becomes a custom view owned by your workspace and stops receiving platform upgrades. Untouched platform views keep improving automatically with each release. `PATCH /personas/{key}`. ### Delete when done Deleting a *customized platform* view resets it to the shipped default. Deleting a *purely custom* view also clears it from any member who had it as their default — they fall back to their role-based view, so nobody is stranded. `DELETE /personas/{key}`. The **Full workspace** view is the permanent escape hatch — it can be neither customized nor deleted, so every user can always get the complete UI. ## How it works - Views are stored per workspace. Platform views are re-synced from the product catalogue on upgrade unless you've customized them (customizing sets `custom` provenance, which freezes the row). - Your current view and how it was set (your own pick, an admin default, or role-derived) are stored on your user profile; the app resolves the effective view at sign-in and after every switch. - View authoring is double-gated: the admin role **and** the `custom_personas` plan feature. Switching and assigning are gated by neither. - Authoring events (`persona.create`, `persona.update`, `persona.delete`) are recorded to the [audit log](/administration/audit-log). ## Related - [Members & roles](/administration/members-and-roles) — assigning default views to members. - [Account settings](/administration/account) — your own view, language, and digest email. - [Dashboards](/guides/dashboards) — the persona home dashboards and cloning. ================================================================================ # API keys Source: /docs/administration/api-keys/ ================================================================================ # API keys An API key is the credential your agents and collectors use to send traces into a Neens agent. Every key is a `nk_live_…` bearer token minted for **one agent**: presenting it resolves the request to that agent (and its company) unambiguously, so a trace can never land in the wrong workspace. ## At a glance | | | | --- | --- | | Where | **Settings → API keys** | | Key API routes | `GET /settings/api-keys`, `POST /settings/api-keys`, `POST /settings/api-keys/revoke` | | Format | `nk_live_` + 32 hex characters | | Who can create | **Members and admins** (viewers cannot); a member may create a `writer`/`reader` key but only an **admin** may create an `admin`-role key | | Who can revoke | The person who **created** the key, or any **admin** — you can only revoke a key you created (admins can revoke any) | | Used by | The ingest endpoints — see [Send traces](/guides/send-traces) | | Secret handling | Shown **once** at creation; Neens stores only a hash and the display prefix | ## The three key prefixes Every bearer credential in the product announces what it is in its first few characters. The prefix is how you tell at a glance which credential you're holding — and the server dispatches on it too, so pasting the wrong kind into a config never silently half-works. | Prefix | What it is | Where it comes from | What it authenticates | | --- | --- | --- | --- | | `nk_live_` | **Agent (ingest) API key** — the machine credential | **Settings → API keys** (`POST /settings/api-keys`), shown once | Ingest (`/v1/traces`, `/ingest/*`) and agent-scoped programmatic calls: the [eval CLI](/guides/preprod-evals), the [MCP server](/guides/mcp), read APIs | | `nk_sess_` | **Human login session token** — opaque, short-lived | Minted when a person signs in to the app | The app's own requests on behalf of that user, with that user's role and org scope | | `nk_admin_` | **Control-plane admin key** — operator / company-admin | Issued by the control plane (a company's initial admin key is created when the company is provisioned) | The operator `/admin/*` control plane. It carries no agent scope, so it must never be used for ingest | Details you rarely need: an agent key is `nk_live_` plus 32 hex characters, and only its first 11 characters are stored for display alongside a SHA-256 hash. A session token carries a longer random tail and is bound to the session's absolute lifetime (**7 days**) and its sliding idle window (**1440 minutes**). An admin key's prefix also encodes its scope — `nk_admin_op_…` for an operator key, `nk_admin_co_…` for a company-admin key. **Only a `nk_live_` agent key pins the destination agent.** Ingest resolves the agent from the authenticated agent key first; a credential of any other kind leaves that unresolved, so the trace falls through to the `X-Neens-Project-Id` header, then a `project_id` in the body, then the bootstrap agent — which is how a trace ends up in a workspace you weren't looking at. Send traces with the agent key and that can't happen. A credential that doesn't resolve at all is a `401`, never a silent redirect. ## Create a key ### Open Settings → API keys Make sure the agent you want to ingest into is the active agent — a key is created under, and scoped to, the current agent. ### Name it and generate Click **New key** in the top right to open the create dialog. Give the key a name that identifies the caller (e.g. *CI pipeline*, *Production ingest*), pick a role (`writer` is the default for ingest), and click **Generate key** (`POST /settings/api-keys`). ### Copy the key immediately The full `nk_live_…` value is returned **once**, in the create response, with the banner *"New key created — copy it now, it won't be shown again."* Neens stores only a SHA-256 hash plus the first few characters for display — the full key is **not retrievable later**, by anyone. If you lose it, revoke it and mint a new one. ### Worked example: mint a key, then use it The UI button calls the same endpoint you can call yourself. Minting takes an authenticated caller with permission to manage API keys (your session, or an admin key) plus a `name` and an optional `role`: ```bash curl -X POST https://your-neens-host/settings/api-keys \ -H "Authorization: Bearer nk_sess_..." \ -H "Content-Type: application/json" \ -d '{"name": "CI pipeline", "role": "writer"}' ``` **Minting an operator-scoped key needs a fresh second factor.** A key is non-interactive: it can never be challenged for a code, which would make key issuance a quiet way to hold operator access that [two-factor authentication](/administration/account#two-factor-authentication) no longer covers. So when the caller is an operator with MFA enabled, the mint requires a session that proved a second factor in the last 15 minutes — or a current code sent as `"mfaCode"` alongside `name` and `role`. Without either, the response is `403 {"detail": {"reason": "mfa_reauth_required"}}`. Ordinary agent keys minted by a company member are unaffected. The response carries the secret exactly once, as `rawKey`: ```json { "key": { "id": "key_9f2c1a7b4e05", "name": "CI pipeline", "prefix": "nk_live_9f2", "role": "writer", "createdAt": "2026-01-14T09:31:02.118431+00:00", "lastUsedAt": null }, "rawKey": "nk_live_9f2c1a7b4e05d8..." } ``` Copy `rawKey` into your secret store — every later read (`GET /settings/api-keys`) returns only the `prefix`. Then present it as a bearer token: ```bash curl -X POST https://your-neens-host/v1/traces \ -H "Authorization: Bearer nk_live_9f2c1a7b4e05d8..." \ -H "Content-Type: application/json" \ -d @traces.json ``` Any OpenTelemetry exporter works the same way — set the endpoint to your host's `/v1/traces` and the `Authorization` header to the key. No `X-Neens-Project-Id` is needed: the key already resolves to one agent. ```bash neens eval run --create \ --dataset ds_golden \ --version-label "$GIT_SHA" \ -- python -m myagent ``` The CLI (`pip install neens-eval`, or `npx neens-eval run …` for the Node build) reads `NEENS_API_KEY` and authenticates every call with it. See [Pre-prod evaluations](/guides/preprod-evals). ```bash claude mcp add --transport http neens \ https://your-neens-host/mcp \ --header "Authorization: Bearer nk_live_9f2c1a7b4e05d8..." ``` The [MCP server](/guides/mcp) scopes every tool call to the key's agent. See [Send traces](/guides/send-traces) for the full set of ingest endpoints and SDK/collector setups. ## What a key resolves to A `nk_live_…` key is registered in two places when it's minted: the agent's own key list, and a global registry that maps the key's hash to its **company + agent**. On every ingest request, Neens resolves the key to exactly that pair and stamps the traces accordingly: - The key **is** the scope — a caller holding a key cannot read or write any other agent or company, and headers can't override what the key resolves to. ### Reads are scoped too, including by id The scope is not only a write-side rule. A `nk_live_…` key reads **only its own agent**, and that holds whether you ask for a list or name a single object by id. Fetching another agent's session, topic, enrichment run or eval run by id is refused even when you know the id exactly: ```bash # Allowed — a session in the key's own project. curl -s -o /dev/null -w '%{http_code}\n' \ -H "Authorization: Bearer nk_live_9f2c1a7b4e05d8..." \ https://your-neens-host/api/sessions/ # 200 # Denied — a session in a SIBLING project of the same company. curl -s -w '\n%{http_code}\n' \ -H "Authorization: Bearer nk_live_9f2c1a7b4e05d8..." \ https://your-neens-host/api/sessions/ # {"detail":"You do not have access to this project."} # 403 ``` Routes that filter the agent into the lookup itself answer `404` rather than `403` — an out-of-scope id is deliberately indistinguishable from one that does not exist, so the response cannot be used to probe which ids are real in an agent you cannot see. Either way no data crosses the boundary. This applies identically to the [MCP server](/guides/mcp), whose tools run against these same endpoints. - An **invalid, edited, or revoked** key is always rejected with `401` — it is never silently accepted or misrouted into a default agent. - A request with **no credential at all** is rejected with `401` — ingest on this deployment requires an agent key, never a silent redirect into a default agent. The key list (`GET /settings/api-keys`) shows each key's name, prefix (`nk_live_xxx…`), role, **who created it**, creation time, and last-used time — never the key itself. ## Revoke a key Click **Revoke** next to the key (`POST /settings/api-keys/revoke`). Revocation removes the key from the list and stops it resolving; subsequent requests with it get `401 Invalid or revoked API key`. You can revoke a key **you created**; an admin can revoke any key. The **Revoke** control only appears on keys you're allowed to revoke — a member never sees it on an admin's (or another member's) key, so admin-created keys are protected. **When does it actually stop working?** Neens caches key→identity resolutions briefly to keep the ingest path fast, and a revoke immediately invalidates that cache: - On deployments with a **shared (Redis) cache** — the recommended production setup — the invalidation reaches every server process at once, so the key stops working immediately. - On deployments using a **per-process in-memory cache** with multiple server processes, another process may still accept the key for up to the auth-cache TTL — **60 seconds**. - With caching disabled, revocation is immediate everywhere. Revocation cannot be undone, and a revoked key's value cannot be reused or recovered. Mint a replacement key first if the caller needs uninterrupted ingest, then revoke the old one. ## How it works - The raw key is generated server-side (`nk_live_` + 32 random hex characters) and hashed with SHA-256 before storage; the stored `prefix` (the first 11 characters) exists only so you can tell keys apart in the list. - The **Last used** column is updated as the key is used; because warm keys are served from the auth cache, it can lag by up to one cache TTL. - Key lifecycle events are recorded to the [audit log](/administration/audit-log) as `api_key.create` and `api_key.revoke`. ## Related - [Send traces](/guides/send-traces) — the ingest endpoints the key authenticates. - [Workspace, orgs & agents](/administration/workspace) — how agents scope your data. - [Audit log](/administration/audit-log) — who created or revoked which key, and when. ================================================================================ # LLM connections Source: /docs/administration/llm-connections/ ================================================================================ # LLM connections Neens reads **no global LLM API key**. Every LLM-powered feature — judge scoring, insight summaries, the assistant, topic and issue classification, enrichments, remediation generation, cluster labeling — resolves an **LLM connection** you configure per workspace, org, or agent. Your credentials stay yours: they are encrypted at rest and never echoed back by any API. ## At a glance | | | | --- | --- | | Where | **Settings → LLM providers** | | Key API routes | `GET/POST /llm-connections`, `PATCH/DELETE /llm-connections/{id}`, `POST /llm-connections/probe`, `POST /llm-connections/list-models` | | API formats | Anthropic, OpenAI-compatible, Google Gemini, OpenRouter, LiteLLM, Ollama, Custom, HTTP API | | Credentials | Write-only; encrypted at rest; never returned by any read | | Who can manage | Members and admins may create org/agent connections; **company-wide** connections are admin-only. You can delete a connection **you created**; admins can delete any | | Without one | LLM features skip or degrade gracefully — nothing else breaks | ## API formats The **API format** you choose describes the **API protocol the endpoint speaks**, not the model vendor. Most hosted and self-hosted model services speak the OpenAI chat-completions API, so for anything that isn't Anthropic, Google Gemini, or one of the named formats below, pick **OpenAI-compatible** and point it at the endpoint. | API format | What it is | Credential | | --- | --- | --- | | Anthropic | The Anthropic API (Claude models). | API key | | OpenAI-compatible | **Any endpoint speaking the OpenAI `/v1/chat/completions` API** — OpenAI itself, plus DeepSeek, Together, Fireworks, vLLM, LM Studio, and most hosted or self-hosted gateways. Set the Base URL to the endpoint's `/v1` URL. | API key | | Google Gemini | Google's native [Generative Language API](https://ai.google.dev/) (`generateContent`) for Gemini models. Leave the Base URL blank — it defaults to Google's endpoint (`https://generativelanguage.googleapis.com`) server-side. | Google API key | | OpenRouter | [OpenRouter](https://openrouter.ai) — one endpoint fronting hundreds of models from many providers. Base URL is filled in for you (`https://openrouter.ai/api/v1`). | API key | | LiteLLM | Your own [LiteLLM](https://docs.litellm.ai/) proxy — one OpenAI-compatible endpoint in front of many providers, with your keys and routing. Requires your proxy's base URL. | As required by your proxy | | Ollama | A local/self-hosted Ollama server. Base URL defaults to `http://localhost:11434/v1`. | None needed | | Custom | Any other OpenAI-compatible endpoint — an Amazon Bedrock–fronting gateway, an internal proxy. Requires a base URL. | As required by the endpoint | | HTTP API | **Outbound HTTP auth for [external-API](/guides/enrichments) judges and enrichments** — it holds the credential (bearer token, header, query parameter, or basic auth) Neens attaches when a judge or enrichment calls a third-party HTTP endpoint instead of an LLM. Not a model connection. | As required by the endpoint | To point Neens at **your own agent's HTTP endpoint** for [pre-prod evaluations](/guides/preprod-evals) — where Neens replays each golden prompt against your agent — use the separate **Add agent endpoint** button on the same page, not the API-format dropdown. See [Testing an agent endpoint](#testing-an-agent-endpoint) below. Local and self-hosted endpoints (Ollama, `localhost`-style base URLs) work without a credential. Remote managed providers require one — a run against a keyless remote connection fails fast with a clear error instead of making a doomed network call. ## Create a connection ### Open Settings → LLM providers Click **Add connection** and pick the **API format**. Fill in the model (e.g. `claude-sonnet-4-6`), the base URL where applicable, and the credential. For formats that publish a model catalogue (OpenAI-compatible, OpenRouter, LiteLLM, Ollama, and Custom endpoints) you can click **Load models** (`POST /llm-connections/list-models`) to pull the available model ids and pick from a searchable list instead of typing one. Listing is best-effort — if it can't be fetched, just enter the model id by hand. ### Choose its visibility A connection is visible at one of three levels: - **Company** — every org and agent in the workspace can use it (admin-only to create or edit). - **Org** — one or more specific orgs. - **Agent** — one or more specific agents. Members can only target orgs/agents they have access to, so a credential is never shared wider than its creator could see. Each connection records **who created it**, shown in the list. You can delete (disable) a connection you created; an admin can delete any. The delete control only appears on rows you're allowed to remove — so an admin-created (or another member's) connection is protected from deletion by a member. ### Test it Click **Test** (`POST /llm-connections/probe`) — Neens makes one real, tiny completion ("ping") through the provider and reports reachability. A reasoning model that spends the probe's 512-token budget without visible output still counts as reachable. Nothing from the probe is persisted. ### Save, optionally as the default Marking a connection **default** makes it the one runs fall back to when nothing more specific is chosen. There is one default at a time — setting it clears the previous one. ### Worked examples #### DeepSeek (an OpenAI-compatible vendor) DeepSeek, Together, Fireworks, vLLM, LM Studio, and most hosted or self-hosted gateways speak the OpenAI chat-completions API. You pick **OpenAI-compatible** — the API format, not a vendor name — and tell Neens which endpoint to call. ### Add connection → API format = OpenAI-compatible That's the key choice: **DeepSeek has no format of its own — it speaks the OpenAI API**, so you select OpenAI-compatible rather than looking for a "DeepSeek" entry. ### Base URL = the vendor's `/v1` URL For DeepSeek that's `https://api.deepseek.com/v1`. (Use whatever `/v1` URL your vendor publishes.) ### Model = the vendor's model id For example `deepseek-chat` or `deepseek-reasoner`. ### Paste the API key, then Test and save Paste the key the vendor issued you, click **Test** to confirm it's reachable, and save. #### Google Gemini ### Add connection → API format = Google Gemini This is the native Google format — no base URL needed. ### Model = `gemini-2.5-flash` Or `gemini-2.5-pro` for a higher-quality, slower model. ### Leave Base URL blank It defaults to Google's endpoint (`https://generativelanguage.googleapis.com`). ### Paste your Google AI Studio API key, then Test and save Create a key in [Google AI Studio](https://aistudio.google.com/), paste it into the credential field, click **Test**, and save. ### Testing an agent endpoint The **Test** action for an `agent_http` connection calls your endpoint exactly the way the pre-prod worker will, with a trivial ping prompt. This probe is SSRF-guarded: a base URL whose host resolves to a **private, loopback, or link-local** address is refused unless an operator has explicitly allowlisted that host. Redirects are never followed. That allowlist is what lets a deliberately internal endpoint — a Docker service, a VPC host, `localhost` — be probed and called on purpose. ## Credentials and secrets - Credentials are **write-only**: you can set or replace one, but no read endpoint ever returns it. The UI shows only whether a usable credential is present. - Stored credentials are **encrypted at rest** using a symmetric key. If the deployment has no encryption key configured, Neens refuses to store the plaintext — the save succeeds but returns a warning that the credential was discarded, so nothing sensitive is ever written unprotected. - Instead of pasting a key, you can supply a **secret reference** (`secretRef`, e.g. `secret://my-key`), which Neens resolves at use time from the environment variable `NEENS_SECRET_` on the server — useful when secrets are injected by your infrastructure rather than stored in Neens. ## How runs pick a connection When a judge run, enrichment, insight summary, or clustering pass needs an LLM, Neens resolves the connection in this order, always restricted to connections **visible to the run's agent**: 1. The **explicit connection** chosen in the feature's configuration (e.g. the judge editor). 2. The agent-visible connection marked **default**. 3. The first agent-visible connection. A run can never pick up a connection scoped exclusively to another org or agent — the same visibility rule the Settings page applies on read also governs the workers, so a credential never leaks across an internal boundary (and company-level physical isolation keeps it from ever crossing tenants). Judges can additionally select **multiple** connections as a pool: an eval run distributes its scoring calls across the pool round-robin, multiplying throughput across several hosts or API keys. See [Judges](/guides/judges). ## What consumes the connection | Feature | Uses the connection for | | --- | --- | | [Judges & scoring](/guides/judges) | Every judge call in eval runs, including continuous (on-ingest) evaluation | | [Pre-prod evaluations](/guides/preprod-evals) | Scoring replayed golden datasets; `agent_http` connections are what Neens calls when it invokes your agent | | [Insights](/guides/insights#the-fleet-briefing) | The AI-written fleet briefing, generated in the background and swapped in when ready | | Assistant | The in-app chat assistant | | [Topics](/guides/topics) & [Issues](/guides/issues-and-failure-modes) | Topic and issue classification | | [Enrichments](/guides/enrichments) | LLM-prompt enrichments over traces | | [Remediations](/guides/remediations) | Generating fixes and simulating them (counterfactual replay) | | [Clustering](/guides/clustering) | Labeling failure clusters and drafting root-cause hypotheses | ### Without a connection An agent with no visible connection **degrades gracefully, never errors the platform**: - Clustering still runs — clusters simply stay **unlabeled** until a connection appears. - Judge runs, enrichments, and remediation generation report a clear configuration error for that run instead of failing silently. - Anomaly/regression detectors, retention, and ingest make **no LLM calls at all**, so observability keeps working end to end. - The [Insights fleet briefing](/guides/insights#the-fleet-briefing) reports `unavailable` and serves its computed, real-number briefing instead — permanently, with nothing queued and nothing retried. That is a supported end state, not an error. ## Reference Connection fields | Field | Meaning | | --- | --- | | `name` | Display name (defaults to `provider/model`). | | `provider` | The API format. In the UI this field is labelled **API format**; on the API the field name stays `provider`. Its value is the API format the endpoint speaks — e.g. `anthropic`, `openai` (any OpenAI-compatible endpoint), `gemini`, `openrouter`, `litellm`, `ollama`, `custom`, `http_api` (outbound auth for external-API judges/enrichments). A pre-prod agent endpoint is the separate `agent_http` value, added via **Add agent endpoint**. | | `model` | Model id sent on completions (optional for `agent_http`). | | `baseUrl` | Endpoint base URL (required for `custom` and `agent_http`; defaults to the local server for `ollama`). | | `credential` | API key / bearer token — write-only, encrypted at rest. | | `secretRef` | Reference resolved from the server environment instead of a stored credential. | | `scopeType` + `scopeIds` | Visibility: `company`, or `org`/`project` with explicit targets. | | `isDefault` | The fallback connection for runs (one per workspace at a time). | | `requestsPerMinute` | Informational rate figure shown with the connection. | | `options` | Provider-specific knobs passed through on every completion (e.g. Ollama `num_ctx`, `top_p`, a default `timeout`; request shape and auth mode for `agent_http`). | Connection changes are recorded to the [audit log](/administration/audit-log) (`llm_connection.create`, `llm_connection.update`, `llm_connection.delete`). ## Troubleshooting | Symptom | Likely cause | Fix | | --- | --- | --- | | Save returns a warning that the credential was discarded | The deployment has no encryption key, so encryption at rest is impossible | Ask an operator to configure the encryption key, or use a keyless local endpoint / a `secretRef` | | Test fails for an internal `agent_http` URL | The host resolves to a private/loopback address and isn't allowlisted | Ask an operator to allowlist the host or CIDR | | Judge run errors with "no usable credential" | Remote provider with no stored key or resolvable secret reference | Add a credential, or switch to a local/Ollama endpoint | | Clusters have no labels | No connection is visible to the agent | Create one at agent, org, or company scope | ## Related - [Judges](/guides/judges) — the heaviest consumer of connections, including pools. - [Pre-prod evaluations](/guides/preprod-evals) — golden-dataset replays and `agent_http` agent endpoints. - [Audit log](/administration/audit-log) — who changed which connection, and when. ================================================================================ # Signing in & passwords Source: /docs/administration/signing-in/ ================================================================================ # Signing in, invites & password reset How you get into Neens, start to finish. Neens accounts are **provisioned, not self-serve** — there is no public sign-up form. You arrive either through an **invite** from an admin on your team, or, for the first admin of a brand-new company, through an activation email sent when the workspace is created. After that, you sign in at **`/login`** with your email and password. This page is the front door. For managing your account once you're in — changing your password, turning on two-factor authentication, ending other devices' sessions — see [Account settings](/administration/account). For inviting people, see [Members & roles](/administration/members-and-roles). If your company uses [single sign-on](/administration/single-sign-on), typing your work email on the sign-in page shows a **Continue with SSO** button that takes you to your identity provider instead of asking for a password. Admins configure it in **Settings → SSO**. ## At a glance | Surface | Where | You need | | --- | --- | --- | | **Activate your invite** | Emailed link → `/activate?token=…` | Your invitation email | | **Sign in** | `/login` | Your email + password (+ a code if MFA is on) | | **Forgot password** | `/forgot-password` | Your account email | | **Reset password** | Emailed link → `/reset-password?token=…` | The reset email | Every activation and reset email depends on your deployment having an email relay configured. If one isn't, the message — link included — is written to the server logs instead of being dropped, so an operator can still hand you the link. See the [FAQ](/faq#emails-arent-arriving). ## Activate an invite (your first sign-in) A new teammate never picks their own password from a sign-up form — they're invited, and they set their password from a one-time link. Here's both sides of that. ### Admins: invite someone ### Open the invite form In **Settings → Members**, click **Invite member** and enter their **email** (a display name is optional). ### Choose a role and organization access Pick **admin**, **member**, or **viewer**, and which organizations they can reach. The full meaning of each is in [Members & roles](/administration/members-and-roles#invite-a-teammate). ### Send it Neens creates the account with status **invited** and emails a **one-time activation link** (`/activate?token=…`, valid for **14 days**). The link is also shown once in the UI as a fallback — copy it right away if your deployment has no email relay. If the link is lost or expires, use **Resend invite** on the member's row; it issues a fresh link and invalidates the old one. An account that has already activated can't be re-invited — send them to [Forgot password](#reset-a-forgotten-password) instead. ### Invited users: activate and set your password ### Open the link in your email The email from your admin contains your activation link. Opening it takes you to `/activate`, which checks the token before showing the form — an expired or already-used link tells you so immediately, and the fix is to ask your admin to resend it. ### Set a password Type a password that clears the rules shown live beneath the field. The requirements are published by the server and update as you type, so you never have to guess: - **At least 12 characters** by default. Length is what matters — a memorable passphrase of a few words beats `Password1!`. (Your operator can raise this minimum; the form always shows the real number.) - **Not a known-breached or predictable password** — anything from public breach lists, keyboard walks like `qwertyuiop`, or a single word with a year on the end is refused. - **Not your own name, email, or company name.** If a password is rejected, the message names the rule it hit and what to do instead. ### Optionally set a display name, then finish Add a display name if you'd like (you can change it later), and submit (`POST /auth/accept-invite`). You're **signed in immediately** — no separate trip to `/login` the first time. If you're the **first admin** of a new company, activation drops you on the **Getting started** checklist — name your workspace, invite your team, connect an LLM, and send your first trace. See [Getting started](/getting-started). Everyone else lands in the app scoped to the orgs they were given. ## Sign in Once you have a password, sign in at **`/login`**. ### Enter your email and password Submit the sign-in form (`POST /auth/login`). If two-factor authentication isn't enrolled on your account, you're in. ### If two-factor is on, enter your code When you have [TOTP two-factor authentication](/administration/account#two-factor-authentication) enrolled, Neens answers with a code challenge rather than a session — nothing is signed in yet. Enter the current 6-digit code from your authenticator app (`POST /auth/mfa/challenge`), or use one of your one-time **recovery codes** if your phone isn't to hand. On success you land where you were heading. **Two-factor is required for operator accounts** and may be required for company admins on some deployments; it's optional for everyone else. A required account signs in on its password as usual and is then taken straight to the enrollment screen before it can reach anything else. ### If sign-in gets paused After several wrong passwords in a row, sign-in pauses for a short, **growing** wait — the page shows a countdown and re-enables itself when it's up. Nothing is broken and no one got in; this is the brute-force guard doing its job. - **You don't have to do anything but wait.** The pause is short at first and never long. - **The pause follows your account, not the browser tab** — reloading or trying a different browser won't clear it. - **Password reset is never blocked by this.** If you can't wait, run [Forgot password](#reset-a-forgotten-password) — completing a reset gets you back in, because the new password clears the failure streak. A separate "**Failed sign-in attempts on your account**" email may arrive after a run of failures. It's a heads-up that someone was guessing — **not** a sign that anyone succeeded. If it wasn't you, reset your password and consider turning on two-factor authentication. ## Reset a forgotten password Forgetting your password is a self-service fix — you never need an admin for it. ### Request a link On the sign-in page, follow **Forgot password** (`/forgot-password`) and enter your account email (`POST /auth/forgot-password`). The confirmation you see is **identical whether or not that email has an account** — Neens never reveals which addresses are registered, so the form can't be used to fish for who's on the platform. A reset email actually goes out only when the address belongs to an active account that already has a password. (An invited teammate who never set one uses their [activation link](#invited-users-activate-and-set-your-password) instead.) ### Open the reset link The emailed link is **single-use** and expires after **60 minutes** by default. `/reset-password` validates the token before it shows the form, so a stale link tells you straight away — just request a new one. ### Choose a new password Pick a new password against the same live rules as activation, and submit (`POST /auth/reset-password`). This **spends the link**, signs out every other session on your account, and signs you in fresh. ### If two-factor is on, complete the code challenge With MFA enrolled, your new password is saved but you're **not** signed in until you answer a code challenge. A reset recovers your password — it is deliberately not a way past your second factor, so access to your mailbox alone can't get someone in. The security and reset emails Neens sends never contain a reset link or token themselves — they point you at the **Forgot password** page, where you request a fresh one. So reading someone's mailbox doesn't hand over a working link on its own. ## Troubleshooting | Symptom | Likely cause | Fix | | --- | --- | --- | | Activation link says it's expired or invalid | The 14-day window lapsed, or the link was already used | Ask your admin for **Resend invite** | | "Too many attempts" on sign-in | Repeated wrong passwords paused your account | Wait out the countdown, or run **Forgot password** — reset is never blocked | | The reset or activation email never arrived | No email relay on your deployment, or a typo'd address | Ask your operator (they can read the link from the server logs); confirm the address. See the [FAQ](/faq#emails-arent-arriving) | | Sign-in accepts the password then asks for a code | Two-factor authentication is enrolled | Enter your authenticator's 6-digit code, or a recovery code | | Lost your authenticator **and** recovery codes | Nothing can be recovered on your own | An operator can reset your second factor — see [Account settings](/administration/account#if-you-lose-your-authenticator) | ## Related - [Account settings](/administration/account) — change your password, manage two-factor authentication and recovery codes, and end other devices' sessions. - [Members & roles](/administration/members-and-roles) — inviting teammates, roles, and org access. - [Getting started](/getting-started) — the first-admin path from activation to your first trace. ================================================================================ # Single sign-on (SSO) Source: /docs/administration/single-sign-on/ ================================================================================ # Single sign-on (SSO) Let your team sign in to Neens with the identity provider (IdP) you already run — Okta, Azure AD / Microsoft Entra, Google Workspace, Auth0, or anything that speaks **OIDC** or **SAML 2.0**. SSO is **per-tenant and bring-your-own-IdP**: you add a connection for your company, point one or more of your email domains at it, and members with those domains get a **Continue with SSO** button on the sign-in page. There is no shared Neens identity provider to trust and nothing to install — Neens is the service provider, your IdP is the source of truth for who your people are. ## At a glance | | | | --- | --- | | **Where** | **Settings → SSO** (company admins only) | | **Protocols** | **OIDC** (OpenID Connect) and **SAML 2.0**, one connection each or several | | **Routing** | By **email domain** — an address's domain decides which connection it uses | | **New users** | Auto-provisioned on first SSO login (just-in-time) at a role and agent you choose | | **Second factor** | Optional **Also require Neens MFA after SSO** per connection | | **Who can configure** | Company **admins**. Members just sign in | SSO **federates into** your existing Neens accounts — it does not replace them. A member's role, org access, and agent scope are still governed by Neens exactly as with password sign-in; SSO only changes how they *prove* who they are. Password sign-in keeps working for anyone not routed to a connection. ## Configure an OIDC connection OIDC is the simplest path if your IdP supports it (most do). You will move four values from your IdP into Neens, and one value — the **callback URL** — from Neens into your IdP. ### The fields Neens asks for Open **Settings → SSO → Add connection** and choose **Protocol: OIDC**. The form asks for: | Field | What it is | Where it comes from | | --- | --- | --- | | **Display name** | A label for this connection (e.g. *Acme Okta*) | You pick it | | **Issuer URL** | Your IdP's OIDC issuer / discovery base — Neens reads its `/.well-known/openid-configuration` | Your IdP's app / tenant settings | | **Client ID** | The public identifier of the app you create in the IdP | The IdP app you create below | | **Client secret** | The confidential secret for that app | The IdP app you create below | | **Scopes** | What Neens requests — `openid email profile` covers everything Neens needs | Leave as the default unless your IdP needs more | | **Email domains** | The domains routed to this connection (see [Email-domain routing](#email-domain-routing)) | Your company's email domains | The **callback URL** you register in the IdP is always: ``` https://YOUR_NEENS_HOST/auth/sso/oidc/callback ``` The client secret is **write-only**. After you save, the form shows *A client secret is configured* rather than the value — editing the connection later, leave the secret blank to keep it. Neens never displays a stored secret back to you. ### Okta walkthrough (OIDC) ### Create an OIDC web app in Okta In the Okta Admin console, go to **Applications → Create App Integration**, choose **OIDC - OpenID Connect** and **Web Application**, and continue. ### Set the redirect URI Under **Sign-in redirect URIs**, add: ``` https://YOUR_NEENS_HOST/auth/sso/oidc/callback ``` Assign the app to the users or groups who should be able to sign in to Neens. ### Copy the credentials From the app's **General** tab, copy the **Client ID** and **Client secret**. Your **Issuer URL** is your Okta org URL (for example `https://acme.okta.com`, or a custom authorization-server issuer if you use one) — Okta shows it under **Security → API → Authorization Servers**. ### Create the connection in Neens Back in **Settings → SSO → Add connection**, choose **OIDC**, paste the **Issuer URL**, **Client ID** and **Client secret**, keep **Scopes** as `openid email profile`, add your **Email domains**, tick **Enabled**, and save. **Azure AD / Microsoft Entra, Google Workspace, and Auth0** follow the exact same shape: create an OIDC / OpenID Connect web app, register the same redirect URI (`https://YOUR_NEENS_HOST/auth/sso/oidc/callback`), and copy the issuer, client ID, and client secret into the Neens form. The issuer for Azure AD is `https://login.microsoftonline.com//v2.0`; for Google it is `https://accounts.google.com`; for Auth0 it is your tenant domain (`https://acme.eu.auth0.com/`). ## Configure a SAML connection Use SAML if your IdP standardizes on it. Here the exchange runs the other way for two values: you give the IdP two Neens **service-provider (SP)** URLs, and you copy three values back from the IdP. ### The fields Neens asks for Open **Settings → SSO → Add connection** and choose **Protocol: SAML**. The form asks for: | Field | What it is | | --- | --- | | **IdP entity ID** | The issuer/entity ID of your IdP for this app | | **IdP sign-on URL** | The IdP's SAML SSO endpoint (where Neens sends the sign-in request) | | **IdP X.509 certificate** | The IdP's signing certificate, used to verify assertions | | **Email domains** | The domains routed to this connection | After you save, the connection's **Service provider details** panel shows the two URLs to hand your IdP admin: | Neens value | URL | | --- | --- | | **SP metadata URL** | `https://YOUR_NEENS_HOST/auth/sso/saml//metadata` | | **Assertion consumer service (ACS) URL** | `https://YOUR_NEENS_HOST/auth/sso/saml/acs` | The metadata URL returns a standard SAML SP metadata document, so an IdP that can import metadata can configure itself from that one URL. The ACS URL is where the IdP posts its signed assertion. ### Okta / Azure AD walkthrough (SAML) ### Create a SAML app in the IdP In Okta: **Applications → Create App Integration → SAML 2.0**. In Azure AD / Entra: **Enterprise applications → New application → Create your own → non-gallery**, then **Single sign-on → SAML**. ### Enter the Neens SP URLs Set the app's **Single sign-on URL / Reply URL (ACS)** to `https://YOUR_NEENS_HOST/auth/sso/saml/acs`, and its **Audience / SP entity ID / metadata** to the Neens **SP metadata URL** shown on the connection. Configure the **email address** as the SAML `NameID` / subject so Neens can match the user. ### Copy the IdP values into Neens From the IdP's SAML settings, copy the **IdP entity ID (issuer)**, the **IdP sign-on / SSO URL**, and the **X.509 signing certificate**. Paste all three into the Neens SAML form. ### Assign users and enable Assign the app to the right users or groups in the IdP, tick **Enabled** on the Neens connection, add your **Email domains**, and save. Neens verifies every SAML assertion against the certificate you provide — signature, audience, and validity window. If sign-in fails right after setup, the mismatch is almost always the **X.509 certificate** (re-copy it, with no stray whitespace) or the **ACS URL** registered in the IdP. ## Email-domain routing A connection only does anything once you route email domains to it. In the connection's **Email domains** field, list the domains your members sign in with (for example `acme.com`, `acme.co.uk`) — separate multiple domains with commas. - When someone types a work email on the [sign-in page](/administration/signing-in), Neens looks up the domain and, if it maps to an enabled connection, shows a **Continue with SSO** button for it. - A domain can belong to **one** connection across all of Neens. If a domain is already claimed elsewhere, saving is rejected rather than silently moving it. - An address whose domain isn't routed simply signs in with a password as before — SSO is additive, not a lockout. ## New users are provisioned on first sign-in The first time someone signs in through a connection and doesn't yet have a Neens account, Neens creates one **just-in-time** — no invite needed. Two per-connection settings control what that new account looks like: | Setting | Effect | | --- | --- | | **Role for new users** | The role a JIT-provisioned user gets — **admin**, **member**, or **viewer**. *Workspace default* uses the workspace's default role | | **Default agent for new users** | The agent a new user starts scoped to (optional) | A JIT-provisioned account has no password — it exists to be signed into through your IdP. If an email already belongs to a Neens account **in another company**, the SSO login is rejected rather than moving the user across tenants; identity in Neens is one account per person. **Also require Neens MFA after SSO.** By default a connection trusts the factor your IdP already enforced, so a successful IdP login signs the user straight in. Turn on **Also require Neens MFA after SSO** on a connection to additionally require the user's Neens [two-factor code](/administration/account#two-factor-authentication) after the IdP step — useful when you want a second factor Neens controls regardless of the IdP's policy. ## What the end-user experience looks like For a member, SSO is three steps and no password: ### Enter your work email On the sign-in page, start typing your work email. If your domain is routed to a connection, a **Continue with SSO** button (labelled with your provider) appears beneath the password form. ### Continue to your provider Click it and Neens sends you to your IdP. Sign in there the way you always do — including whatever MFA your IdP enforces. ### You're signed in Your IdP returns you to Neens, which signs you in (creating your account on the first visit). If the connection also requires a Neens second factor, you enter that code before you land in the app. The password sign-in and its [account-protection and reset flows](/administration/signing-in) stay exactly as they are for anyone not using SSO. ## Troubleshooting | Symptom | Likely cause | Fix | | --- | --- | --- | | No **Continue with SSO** button appears | The email's domain isn't routed, or the connection is disabled | Add the domain to the connection's **Email domains** and tick **Enabled** | | OIDC login returns to the sign-in page with an error | Redirect URI mismatch, or wrong issuer / client credentials | Confirm the IdP redirect URI is exactly `https://YOUR_NEENS_HOST/auth/sso/oidc/callback` and re-check the issuer, client ID and secret | | SAML login is rejected | Certificate or ACS URL mismatch | Re-copy the **IdP X.509 certificate**; confirm the IdP's ACS is `https://YOUR_NEENS_HOST/auth/sso/saml/acs` | | Can't save a domain | It's already claimed by another connection | A domain maps to one connection only — remove it from the other first | | Sign-in asks for a code after the IdP step | **Also require Neens MFA after SSO** is on for that connection | Enter your Neens authenticator code — this is expected | ## Related - [Signing in & passwords](/administration/signing-in) — the password, invite, and reset flows SSO sits alongside. - [Members & roles](/administration/members-and-roles) — what the role a new SSO user is given can do. - [Your account](/administration/account) — managing your own two-factor authentication. - [Connect Claude Code (MCP)](/guides/connect-claude-code) — the browser sign-in a developer does to connect Claude Code runs through this same SSO when your domain is routed. ================================================================================ # Account settings Source: /docs/administration/account/ ================================================================================ # Account settings Everything on this page is **personal to you** — it changes how Neens looks, speaks, and emails *you*, never what you or anyone else is allowed to do. Most of it lives under **Settings → Account**; your view and language are also switchable from the topbar. ## At a glance | | | | --- | --- | | Where | **Settings → Account**; topbar switchers for language and view | | Key API routes | `POST /auth/change-password`, `POST /auth/forgot-password`, `GET /auth/mfa`, `POST /auth/mfa/enroll`, `POST /auth/mfa/verify`, `POST /auth/mfa/challenge`, `POST /auth/mfa/recovery-codes`, `POST /auth/mfa/disable`, `GET /auth/sessions`, `DELETE /auth/sessions/{id}`, `POST /auth/sessions/revoke-all`, `PUT /auth/locale`, `PUT /auth/digest`, `POST /auth/digest/test`, `PUT /auth/persona`, `PUT /scores/prefs` | | Scope | Per user (score favorites are per user *and* per agent) | | Permission impact | None — these are presentation, security and notification settings for your own account | ## Change your password Under **Settings → Account**, the **Password** card asks for your **current password** even though you're already signed in — a hijacked-but-idle session can't silently rotate your credential. The new password has to be different from the current one and has to clear the password rules below (`POST /auth/change-password`). On success, **every other session is signed out** (other devices, and anyone who shouldn't have been there) and your current device gets a fresh session, so you stay signed in. Neens also emails you that your password changed — see [Security emails](#security-emails-you-may-receive). ## Password rules Every password field shows the live rules underneath it, read from the server, so you never have to guess. The defaults: - **At least 12 characters.** Length is the rule that matters. There is deliberately no "one uppercase, one number, one symbol" requirement — those push everyone toward `Password1!`, which is one of the most-guessed passwords in existence, while rejecting a genuinely strong passphrase for having no digit. A memorable phrase of several words is both stronger and easier to type. - **Not a known-breached password.** Your candidate is checked against a list of passwords that appear in public breach dumps and cracking lists. This is a local check — your password is never sent anywhere to be checked. - **Not your own details.** It can't contain your name, email address, or company name. - **Not a predictable shape.** Keyboard walks (`qwertyuiop`), long repeated or sequential runs, and a single word with a year on the end (`Sunflower2026`) are refused — cracking tools generate those exhaustively. If a password is refused, the message says which rule it hit and what to do instead. These rules apply when you **set** a password — activating an invite, changing it, or completing a reset. They are never applied when you **sign in**, so an older password that no longer meets the minimum keeps working until you next change it. Your operator can raise the minimum; it can't be lowered below 8. ## Two-factor authentication A second factor is what makes a stolen password insufficient on its own. Neens supports **TOTP** — the 6-digit code that rolls over every 30 seconds in an authenticator app — plus a set of one-time **recovery codes** for the day your phone isn't with you. It lives under **Settings → Account → Two-factor authentication**. Any RFC 6238 authenticator works. Neens hands your app a standard `otpauth://totp/…` secret and never contacts the app's vendor; codes are computed on your device and verified on the server. There is no SMS anywhere in this flow, and nothing to install on your phone that Neens has to approve. ### Turn it on ### Open Settings → Account → Two-factor authentication **Set up two-factor authentication** issues you a secret (`POST /auth/mfa/enroll`). Issuing it changes nothing yet: your account keeps signing in exactly as it does today until you finish the last step below. An enrollment you abandon halfway — a dead phone battery, a closed tab — leaves you fully able to get in, and starting again simply replaces the half-set-up secret. ### Add the secret to your authenticator Scan the QR code with your authenticator app. If the thing you're enrolling can't scan — a desktop authenticator, a password manager, a hardware token app — use the base32 **setup key** shown beside the QR (it looks like `JBSWY3DPEHPK3PXP`) and type it in. Both encode the same thing: a 6-digit code on a 30-second period. ### Enter your password and the current 6-digit code Type your **current sign-in password** and the code your app is showing, then confirm (`POST /auth/mfa/verify`). **This** is the step that enables MFA — Neens will never switch it on for a secret it has not yet seen a working code from, because "enabled on issue" is the classic way a mistyped setup locks somebody out of their own account. ```bash curl -X POST https://your-neens-host/auth/mfa/verify \ -H "Authorization: Bearer $NEENS_SESSION_TOKEN" \ -H "Content-Type: application/json" \ -d '{"password": "your-current-password", "code": "123456"}' ``` ```json { "enabled": true, "enrolledAt": "2026-07-31T14:02:11.417329+00:00", "recoveryCodes": ["k7m2q-9xf4t", "b3n8p-2wd6r", "…"] } ``` **Why it asks for your password again, when you just signed in with it.** Turning a second factor *on* decides who owns the account from that moment. Without this step, anybody who had your password could enroll *their* authenticator on *your* account — and you would then be the one locked out, with no way to remove a factor you never set up. It is the same password prompt that turning the feature off and regenerating your recovery codes have always asked for; it was missing from the one step that establishes the factor rather than removing it. It is asked **here** and not when you press *Set up two-factor authentication*, so an abandoned setup still costs nothing. A wrong password — or a request that omits the field — returns `401 {"detail": "Current password is incorrect."}`. The two answers are deliberately identical, so the response never reveals which of the two you got wrong. Nothing is enabled and no recovery codes are issued. Too many wrong passwords in a row pause the account the same way too many wrong codes do (see **"Too many attempts"** below). ### Save your 10 recovery codes You are then shown **10 recovery codes**, each of the form `k7m2q-9xf4t`. They are displayed **exactly once, on that screen** — Neens stores only hashes of them, the same way it stores your password, so nobody (including your operator) can show them to you a second time. **Each code works once.** Copy them somewhere you can reach *without* your phone: a password manager on another device, or printed and kept where you keep your passport. Leaving that screen without saving the codes is the one irreversible mistake in this flow. If you did, you're not locked out — sign in with your authenticator and use **Regenerate recovery codes** below, which issues a fresh ten and invalidates the ones you lost. ### Signing in once it's on ### Enter your email and password as usual Neens answers with a challenge rather than a session — nothing is signed in yet (`POST /auth/login` returns a short-lived challenge instead of a session token). The page asks for your code. ### Enter the 6-digit code from your authenticator Confirm (`POST /auth/mfa/challenge`). On success your session is issued and you land where you were heading. ### Or use a recovery code instead **Use a recovery code** on the same screen takes one of your ten. Paste it with or without the hyphen — `k7m2q-9xf4t` and `k7m2q9xf4t` are both accepted. The code is spent the moment it works, and the response tells you how many you have left. At two or three remaining, regenerate. **The code screen lasts 5 minutes and allows a few tries.** A mistyped code just tells you it was wrong — type the next one. But that screen expires after **5 minutes** and gives up after a handful of wrong codes. When it does, you go back to the email-and-password screen and sign in again for a fresh one; nothing is wrong with your account. The message is the same either way, so don't read anything into which of the two you hit. **"Too many attempts" on the code screen.** After a few wrong codes in a row the code screen pauses before it will take another, and the wait grows if you keep going — the page shows a countdown and re-enables itself. Signing in again does **not** clear it: the pause follows your account, not the code screen, which is the whole point of it. Three things end it, in the order worth trying: 1. **Wait it out.** The countdown is short at first (a couple of seconds) and never longer than a few minutes. 2. **Enter a correct code.** One code that actually works clears the pause immediately. If you have run out of patience with the authenticator, **Use a recovery code** counts too. 3. **Try from another network.** After enough wrong codes from one place, that *place* is blocked for a while rather than your account — so your phone's mobile data, or home instead of the office, usually just works. If none of those apply, your authenticator has probably drifted or been lost — see [If you lose your authenticator](#if-you-lose-your-authenticator). An operator can reset the second factor for you, which clears the pause with it. ### Rules that catch people out | Rule | What it means for you | | --- | --- | | **A password reset does not bypass MFA.** | Completing **Forgot password** changes your password and then still asks for your code before it signs you in. Mailbox access alone is not a way past the second factor. | | **A code is valid exactly once.** | Re-submitting the same 6 digits fails even while your app is still showing them — for example after a flaky network retry. Wait for the next code. | | **The accepted window is ±30 seconds, and no wider.** | Neens accepts the current code and one step either side. If your codes are *always* rejected, your phone's clock has drifted: turn on automatic/network time and try again. | | **API keys carry no second factor.** | A `nk_live_…` or `nk_admin_…` key is used by machines, so it can't be challenged. Minting one with operator scope therefore asks for a current code (or a session that entered one in the last 15 minutes) — otherwise key issuance would be a quiet way back to exactly the access MFA is protecting. See [API keys](/administration/api-keys). | ### Regenerate your recovery codes **Regenerate recovery codes** asks for your password and a current code, then issues a fresh set of ten (`POST /auth/mfa/recovery-codes`). It replaces **all** outstanding codes — used and unused — so any old printout stops working immediately. Like enrollment, the new codes are shown once. ### Turn it off **Turn off two-factor authentication** asks for your password *and* a current code (`POST /auth/mfa/disable`); a recovery code is accepted in place of the code. Both are required on purpose — an attacker sitting in a session you left open should not be able to strip the control that would have stopped them. **Operator accounts can't turn it off.** On a deployment where MFA is required for the platform operator, that request is refused (`mfa_required_for_operator`). Turning it off is also not the tool for "I got a new phone" on any account: sign in with a recovery code, regenerate your codes, and enroll the new device. ### Emails Neens sends when your second factor changes Three things can change your second factor, and Neens emails you about each one — **exactly one message per event**, to your account address. They are the only signal that reaches you *outside* the account, which matters precisely when the account is the thing that has been taken. | You'll get | When it fires | What's in it | What to do if you weren't expecting it | | --- | --- | --- | --- | | **"Two-factor authentication was turned on"** | Someone completed `POST /auth/mfa/verify` on your account | The time, and the IP address it was set up from when one is known | Somebody knows your password and has enrolled their own authenticator. **Change your password immediately**, then ask an administrator to reset two-factor authentication for you — you cannot remove a factor you did not enroll | | **"Two-factor authentication was turned off"** | Someone completed `POST /auth/mfa/disable` | The time, the IP address when known, and a reminder that your password is now the only thing protecting the account | Change your password immediately and turn two-factor authentication back on | | **"Two-factor authentication was reset"** | An operator ran the reset from **Settings → Members** | The time and **who did it** — the acting operator's address. You are signed out everywhere | You are now password-only and somebody else arranged that. Change your password immediately and contact your administrator | Every one of these: - **Never contains a secret.** No setup key, no recovery code, no session or challenge token. The link in them points at the token-free **Forgot password** page, so an attacker reading your mailbox gains nothing from the message itself. - **Names an IP address only when one is known and is genuinely an address** — a missing or unparseable value is left out rather than printed as "unknown". Depending on how your deployment is fronted, that address may be one the caller supplied, so treat it as a lead, not as proof. The reset notice carries no address at all: the person who acted was an operator, not you, so it names *them* instead. - **Is sent in your language**, from your account's locale preference. - **Is best-effort.** A mail outage never blocks the change itself, so "I got no email" is not proof that nothing happened. ### If you lose your authenticator | You still have | What to do | | --- | --- | | A recovery code | Sign in with it, then **Regenerate recovery codes** and set up your new device (turn MFA off and on again, or re-enroll from the new phone). | | Neither your authenticator nor a recovery code | Nothing can be *recovered* — the codes are hashed and the secret is encrypted — but your **operator can reset** two-factor authentication on your account, from **Settings → Members**. Expect them to verify who you are through some channel other than the account itself. The reset signs you out everywhere, then your next sign-in works on your password alone and asks you to enroll again. | ### If your account is *required* to have MFA Most deployments require MFA for the **platform operator** account, and may require it for company admins. "Required" here does not mean your password stops working. You sign in normally, and then the app takes you straight to the enrollment screen: until you finish it, your session can reach your own account page and nothing else, and any other request answers `403 mfa_enrollment_required`. Finishing enrollment lifts that on every device you are signed in on. ## Forgot your password? Use **Forgot password** on the sign-in page (`POST /auth/forgot-password`): ### Enter your email The response is identical whether or not the email is registered — Neens deliberately never confirms which addresses exist, so the form can't be used to enumerate accounts. A reset email is sent only if the address belongs to an active account that already has a password. ### Open the reset link The link is single-use and expires after **60 minutes**. The reset page checks the token before showing the form, so an expired link tells you immediately. ### Set a new password Completing the reset (`POST /auth/reset-password`) burns the token, revokes **all** prior sessions, and signs you in fresh. If you have [two-factor authentication](#two-factor-authentication) on, the last part changes: your new password is saved, but no session is issued until you answer a code challenge. A reset is a password recovery, not an MFA bypass. Invited teammates who never set a password use their **activation link** instead — the reset flow is only for accounts that already have one. If your deployment has no email relay configured, the reset message (including the link) is written to the server logs, so an operator can still hand it to you. **"Too many sign-in attempts"?** After several wrong passwords in a row, sign-in pauses for a short, growing wait — the page shows a countdown and re-enables itself. You don't have to do anything but wait. If you can't wait, **Forgot password** still works: password reset is never blocked by this, and completing one clears the pause immediately. ## Active sessions **Settings → Account → Active sessions** lists every device currently signed in to your account, so you can spot one you don't recognise and end it. Each entry shows: | | | | --- | --- | | **Device** | Browser and platform, read from the session's user agent — "Chrome · macOS". Hover for the full string. | | **IP address** | The address that session connected from, with a badge saying how much that address can be trusted. | | **Signed in / Last used / Expires** | When the session started, when it was last used, and when it runs out. | | **This device** | A green pill on the session you're using right now. | The IP badge is the same one used on the sign-in activity trail: **Edge header**, **Proxy hop** and **Socket peer** are addresses your deployment's own infrastructure observed, while an amber **Unverified** means the address was supplied by the client and shouldn't be read as a fact. Sessions created before this feature existed show no address at all rather than a placeholder. **No location lookup is performed.** Neens shows the IP address and how trustworthy it is — never a city or country. Turning an address into a place means shipping a geolocation database or calling a third party from your account page, and the answer is wrong often enough (VPNs, mobile carriers, cloud egress) that it would send you chasing a session that was your own all along. ### Sign a device out ### Revoke one session **Revoke** on any row ends that session immediately (`DELETE /auth/sessions/{id}`). The device is signed out the next time it makes a request. Revoking the row marked **This device** signs *you* out and returns you to the sign-in page. ### Or sign out everywhere **Sign out other devices** ends every session except the one you're using (`POST /auth/sessions/revoke-all`), after a confirmation. Your current session is deliberately kept so you don't lose the tab you're reacting in. If you think someone else has your password, do both: **change your password** (which by itself signs out every other session) and check this list afterwards. Signing out other devices alone evicts whoever is there; it does not stop them signing back in with a credential they still know. Then turn on [two-factor authentication](#two-factor-authentication), which is the only one of these that makes the leaked password insufficient rather than merely stale. ## Security emails you may receive Neens emails you — the account owner — about things only you can judge. You never opt in to these; they are the way a real compromise usually gets noticed. | Email | Sent when | | --- | --- | | **Your password was changed** | Someone completed a password change from a signed-in session. Includes when, and the IP address it came from when that address is known. | | **Your password reset is complete** | A password reset finished against your account, and the reset link that did it is now spent. | | **Failed sign-in attempts on your account** | A run of failed sign-ins hit your account. **No one got in** — this is a heads-up, not a breach notice. | If any of them was not you, use **Forgot password** to reset immediately and then review your [active sessions](#active-sessions). The security emails never contain a reset link or token — they point you at the reset *page*, where you request a fresh link. A repeated attack sends **one** "failed sign-in attempts" email per hour at most, so a sustained attempt can't be turned into a way of flooding your inbox. ## Interface language Pick your language from the topbar switcher or **Settings → Account** (`PUT /auth/locale`). The choice is stored on your profile and applies to the UI, API error messages, and the emails Neens sends you. Supported languages: **English (`en`), Spanish (`es`), French (`fr`), German (`de`), Italian (`it`), Portuguese (`pt`), Arabic (`ar`), and Japanese (`ja`)**. Arabic sets the interface to right-to-left text direction (the layout itself is not mirrored yet). Every one of them is fully translated — the interface never falls back to English behind your back. Before you sign in there is no switcher: Neens guesses from your browser's `Accept-Language`. Your first sign-in back-fills whatever language you were reading onto your profile, and from then on the stored value wins on every device until you change it here. See [Languages](/administration/languages) for the full picture: asking for a language from an API client with `Accept-Language`, which surfaces are translated, and the short list of strings that stay in English on purpose. ## Digest email An opt-in summary of your fleet — health tiles plus the insights routed to your [view](/administration/personas) — delivered by email at most **once a day**. - Toggle it on under **Settings → Account** and pick a cadence: **daily** or **weekly** (`PUT /auth/digest`). It's off by default. - The scheduled job sends daily digests at around 13:00 UTC; weekly digests go out roughly every seven days. - **Send test digest** (`POST /auth/digest/test`) composes and sends yours right now, without consuming the day's scheduled send — handy for checking what it will look like. - The digest only ever contains data from agents **you** can see, and opting out remembers your cadence for the next opt-in. Sending requires the deployment to have an email transport configured; the test button returns an error otherwise. On no-relay installs the scheduled digest is logged server-side instead of silently dropped. ## Your view (persona) Your **view** decides which nav items are featured, your landing page, and your default time range — never your permissions. Switch it from the topbar **View** switcher or `PUT /auth/persona`; an explicit pick is yours and won't be overridden by an admin default later. The first time you sign in, a one-time picker offers your admin's recommendation or a role-based suggestion — picking *or* skipping both dismiss it for good. See [Views (personas)](/administration/personas) for the built-in views and custom authoring. ## Score favorites On the [Scores](/guides/scores) page you can **star** score types you watch closely and **hide** ones that are noise to you (`PUT /scores/prefs`). Favorites float to the top; hidden metrics collapse out of the way (unhide before re-favoriting). These preferences are yours alone, kept **per agent**, and don't enable or disable the underlying judges — though the page offers the related judge controls inline for those with permission. ## Related - [Views (personas)](/administration/personas) — what each view emphasizes and how defaults are assigned. - [Members & roles](/administration/members-and-roles) — the permissions your account actually has. - [API keys](/administration/api-keys) — the machine credentials that carry no second factor, and what that means for minting one. - [Scores](/guides/scores) — the score catalogue your favorites organize. ================================================================================ # Languages Source: /docs/administration/languages/ ================================================================================ # Languages Neens ships its interface in **eight languages**, and every one of them is *fully* translated — not "mostly". This page covers how to choose yours, how an API client asks for a language, and what stays in English on purpose. ## At a glance | | | | --- | --- | | Where | Topbar **language switcher**, or **Settings → Account → Language** | | Key API routes | `PUT /auth/locale` (persist your choice), `Accept-Language` header (per request) | | Languages | `en` `es` `fr` `de` `it` `pt` `ar` `ja` | | Scope | Per user. Your language never changes what anyone else sees | | Falls back to | English — for content Neens does not own, and for the handful of API messages that are not in the translated set (see [What stays in English](#what-stays-in-english)) | ## Supported languages | Code | Language | Direction | | --- | --- | --- | | `en` | English | left-to-right | | `es` | Español | left-to-right | | `fr` | Français | left-to-right | | `de` | Deutsch | left-to-right | | `it` | Italiano | left-to-right | | `pt` | Português | left-to-right | | `ar` | العربية | **right-to-left** | | `ja` | 日本語 | left-to-right | Picking Arabic sets the document direction to RTL, so text, input fields, and the browser's own scrollbars and text selection follow Arabic reading order. **Arabic layout is not yet mirrored.** The direction flips, but the app's spacing and alignment still use physical `left`/`right` utilities rather than CSS logical properties, so paddings, margins, and column alignment stay in their left-to-right positions. Arabic is fully *translated* and usable; the layout pass is outstanding. ## Choose your language ### Before you sign in The sign-in page has no language switcher. Neens guesses from your browser's `Accept-Language`, so the sign-in form, error messages, and the password rules usually already speak your language — but it is a guess, not a choice, and there is no way to override it until you are signed in. ### After you sign in Use the topbar **language switcher**, or **Settings → Account → Language → Display language**. The switch is immediate — no reload — and is saved to your profile (`PUT /auth/locale`), so it follows you to any browser or device you sign in from. ### Your first sign-in stores whatever you were reading If your profile has no language stored yet, Neens back-fills the one the interface is currently using — your explicit pick if you made one, otherwise the browser guess. From then on the stored value wins everywhere, on every device, until you change it in Settings. Because the first sign-in promotes a browser guess into a stored preference, a user whose browser is set to a language they do not want should change it in **Settings → Account**; clearing browser state will not do it. ## Ask for a language from an API client Every JSON API route honours the standard **`Accept-Language`** request header. The web app stamps your active language on every call it makes; your own scripts and integrations can do the same. ```bash # German error copy from a deliberately unauthenticated call curl -s https://app.neens.ai/api/auth/me \ -H 'Accept-Language: de' # {"detail":"Nicht authentifiziert."} # Same call, Japanese curl -s https://app.neens.ai/api/auth/me \ -H 'Accept-Language: ja' # {"detail":"認証されていません。"} ``` ```python client = httpx.Client( base_url="https://app.neens.ai", headers={ "Authorization": "Bearer nk_live_…", "Accept-Language": "fr", }, ) r = client.get("/api/sessions", params={"limit": 20}) ``` ```ts const res = await fetch('https://app.neens.ai/api/sessions?limit=20', { headers: { Authorization: 'Bearer nk_live_…', 'Accept-Language': 'pt', }, }) ``` Header handling follows the spec, so you can pass a browser's header through unchanged: | You send | Neens uses | Why | | --- | --- | --- | | `de` | `de` | exact match | | `pt-BR` | `pt` | region is dropped when the base language is supported | | `fr-CA, fr;q=0.9, en;q=0.5` | `fr` | highest-quality supported entry wins | | `nl` | `en` | unsupported → English, never an error | | *(no header)* | `en` | — | `Accept-Language` affects the **response copy**, never the data. It does not filter, translate, or reorder your traces, scores, or any other stored record. ## What gets translated | Surface | Translated? | Notes | | --- | --- | --- | | The whole web interface | **Yes, completely** | Every navigation item, button, table header, tooltip, empty state, and error banner | | API error messages | High-traffic messages | The curated catalogue covers authentication, permission, and validation errors. A message outside it stays readable English rather than blank | | Transactional emails | **Yes** | Invitations, activation, password changed/reset, and failed-sign-in alerts, sent in the recipient's **stored** language | | The digest email | No — English | Localizing the digest body is deferred; only the greeting uses your name | | Right-to-left text direction | **Yes** (`ar`) | The document direction flips. Layout mirroring is still outstanding — see the note above | | Your own data | No — it's yours | Trace content, session text, agent and judge names, dataset rows, annotations | | LLM-generated text | Follows your LLM | Cluster labels, remediation drafts, and narrative summaries are written by the [LLM connection](/administration/llm-connections) you configured, from your own trace data | ## What stays in English Some strings are identical in every language on purpose, and translating them would make the product *harder* to use: - **The product name** — `Neens`. - **Acronyms borrowed unchanged** across all eight languages — `API`, `SDK`, `LLM`, `URL`, `JSON`, `CSV`, `HTTP`, `IP`, `PR`, `CI`, `MCP`, `SSO`, `TOTP`. - **Metric and verdict tokens** that must match what the API returns and the CI runners print — `pass^k`, `PASS`, `FAIL`, `p50`/`p95`/`p99`. - **Code you would copy-paste** — field names, file names, env var names, header names (`X-Neens-Project-Id`), span attributes (`neens.*`), key prefixes (`nk_live_…`), and model ids such as `claude-opus-5`. - **Vendor names** — GitHub, Slack, ClickHouse, Postgres, Langfuse, Zendesk, and the rest. The judgement calls — a word that happens to be identical in one language, a label that stays an acronym — are recorded one by one with a written reason. The purely mechanical cases (a bare number, a URL, an identifier like `user_id`) are recognised automatically and deliberately are **not** listed, so the recorded list stays short enough to actually read. ## How completeness is guaranteed A missing translation is easy to catch. A translation that is *silently the English string* is not: the language file has the key, everything loads, and only a reader who speaks the language notices. That is exactly what happened to one of our pre-production screens, whose sign-in hints sat in English across all seven non-English languages while every automated check reported green. Neens now blocks both failure modes before a change can ship: 1. **Every language carries every key.** A key that exists in English and is missing elsewhere fails the build. 2. **No language may carry the English string.** If a value is byte-identical to the English one, it fails the build — unless it is on the short, reasoned list above, or it is structurally untranslatable (a bare number, a URL, an identifier like `user_id`, a `{{placeholder}}`). 3. **Every language carries every plural form it needs.** English has two (`one`/`other`); Arabic has six and the Romance languages have three. A language missing one of its own forms falls back to English for exactly the counts in that form — for Arabic that was 0 and 2–99 — so the build now checks each language against its own CLDR rules, not against English's. 4. **Nothing can be deferred quietly.** The list of accepted exceptions can only shrink automatically; adding to it takes a deliberate, reviewed edit. So "eight languages" means the same thing for the eighth language as for the first. ## Troubleshooting | Symptom | Cause | Fix | | --- | --- | --- | | Interface is English after signing in on a new device | Your profile preference is set, but you had picked a *different* language on that browser before signing in | Signing in applies your stored profile language; switch from the topbar if you want to change it | | A **specific** error message is English while the UI is translated | The message is outside the curated API catalogue, or it embeds live values (an id, a limit) | Nothing to fix — the message stays readable rather than blank. Report it if it is a common one | | Emails arrive in English but the UI is translated | Emails use your **stored** profile language; a browser-only choice never reached the server | Sign in and pick your language from **Settings → Account** so it is persisted | | Cluster labels and remediation text are English | They are generated by your configured LLM from your trace data, not shipped copy | Nothing to fix in Neens; the language follows the model and the data | | Arabic reads right-to-left but the layout is not mirrored | Known limitation — the interface still uses physical `left`/`right` spacing | Nothing to do yet; the text direction is correct and the layout pass is tracked separately | Related: [Account settings](/administration/account) · [LLM connections](/administration/llm-connections) ================================================================================ # Audit log Source: /docs/administration/audit-log/ ================================================================================ # Audit log The audit log is your workspace's append-only answer to *"who did what, and when?"* Every security-relevant or destructive action a person (or automation) takes — inviting a member, rotating an LLM credential, deleting an agent, revoking an API key — is recorded as an event that **only admins can read**. ## At a glance | | | | --- | --- | | Where | **Audit Log** in the left nav (admin group) | | Key API route | `GET /audit` | | Who can read | Company **admins** only — members and viewers get `403` | | Scope | Company-wide: covers events across all your orgs and agents | | Retention shape | Append-only; events are never edited or deleted by the application | ## What gets recorded Events are named `.` and grouped by the surface they touch: | Area | Actions | | --- | --- | | Members | `member.invite`, `member.update`, `member.remove`, `member.resend_invite`, `member.bulk_persona` | | Sign-in security | `auth.invite_accepted`, `auth.password_change`, `auth.password_reset_requested`, `auth.password_reset` | | API keys | `api_key.create`, `api_key.revoke` | | LLM connections | `llm_connection.create`, `llm_connection.update`, `llm_connection.delete` | | Judges & evals | `judge.create`, `judge.delete`, `judge_version.create`, `judge_deployment.create`, `judge_deployment.update`, `eval_run.start`, `eval_run.cancel` | | Datasets | `dataset.create`, `dataset.update`, `dataset.delete`, `dataset.version_create` | | Enrichments | `enrichment.create`, `enrichment.update`, `enrichment.delete`, `enrichment_run.start` | | Topics | `topic_space.create`, `topic.update`, `topic.merge`, `topic.split` | | Orgs & agents | `org.create`, `org.update`, `project.create`, `project.update`, `project.delete` | | Views (personas) | `persona.create`, `persona.update`, `persona.delete` | | Onboarding | `tenant.onboarding_step_ack`, `tenant.onboarding_complete` | A failed password-reset *request* for an unknown email is deliberately **not** recorded — logging it would let an audit reader enumerate which addresses exist. Only resets issued for real, active accounts appear. **Sign-ins live on their own tab.** Logins, failed logins, throttles, logouts, rejected sessions, and operator impersonation are recorded as security events and shown under **Sign-in activity** on this same page, not in the workspace activity list below. The split is: this log answers *who did what inside your company*; security events answer *who authenticated, and who tried*. Authentication rejections happen before a company is even resolved, so they cannot be filed under one. ## What each entry contains | Field | Meaning | | --- | --- | | **When** | `createdAt` — UTC timestamp of the event | | **Who** | `actorId`, `actorName`, and `actorType` (`user`, `operator`, `service`, `scheduler`, `automated`, `system`) | | **What** | `action` (the dotted verb) and a human-readable `summary` | | **Target** | `targetType`, `targetId`, `targetName` — the resource that was affected | | **Where** | `orgId` / `projectId` when the event is scoped below the company | | **Outcome** | `status`: `success`, `failure`, or `denied` | | **Context** | `metadata` (structured extra detail, e.g. before/after values), plus the caller's `ipAddress` and `userAgent` | ## Filtering and search The **Audit Log** page (and `GET /audit`) supports: - **Time range** — the standard picker; defaults to the last **30 days**. - **Facet filters** — action, actor type, and target type dropdowns, populated from the values actually present in your log (plus `actorId`, `orgId`, `projectId`, and `status` as API parameters). - **Free-text search** — `q` matches against the summary, target name, and action. - **Sorting and paging** — sortable by time, action, actor, target, or status; pages of up to 500 events (100 by default). Picking a filter never collapses the other dropdowns — facets are computed over the full time-scoped log, so you can pivot between filters freely. ## Audit log vs. Activity feed These answer different questions: | | **Audit log** (this page) | [**Activity feed**](/guides/activity) | | --- | --- | --- | | Question | *Who changed what?* | *What background work ran?* | | Contents | Security-relevant mutations: members, keys, connections, judges, datasets, orgs/agents, auth | Eval, clustering, and enrichment runs — durations, target counts, success rates | | Audience | Admins only | Members | | Shape | Append-only event trail | Operational run history | (Platform operators additionally have their own separate log covering company provisioning and lifecycle — that never appears in a tenant's audit log.) ## How it works - Events are written **after** the primary action commits, on a separate connection, and the write is best-effort — an audit hiccup can never roll back or block the action itself. - The log lives inside your company's physically isolated storage, and reads are additionally scoped to your company, so no other tenant's events can ever appear. - Reading is gated by an admin-only permission; there is no API to modify or delete entries. ## Related - [Activity](/guides/activity) — the background-run feed for evals, clustering, and enrichments. - [Members & roles](/administration/members-and-roles) — the admin role that grants audit access. - [API keys](/administration/api-keys) and [LLM connections](/administration/llm-connections) — the credential events recorded here. ================================================================================ # Usage metering Source: /docs/administration/usage-metering/ ================================================================================ # Usage metering Neens records what every tenant is doing, feature by feature, as durable integer counters. This powers a per-workspace **Usage** view for admins. Counters are integer aggregates only — a usage row is `(company, project, feature, day, connection, count)`. **No trace, prompt, or response content is ever stored or transmitted by metering.** That invariant is what lets the aggregates roll up for billing and fleet operations while every byte of content stays inside the tenant. ## At a glance | | | | --- | --- | | Where (tenant) | **Usage** in the left nav — per-agent, per-feature counts and trends | | Where (operator) | **Usage** in the operator console — the same counts, fleet-wide | | Who can use it | Any workspace member sees their workspace's usage; the operator sees every tenant | | Key API routes | `GET /usage`, `GET /usage/summary`; operator: `GET /admin/usage`, `GET /admin/usage/companies` | | What's stored | Integer counts per `(company, project, feature, day, connection)` — never content | | Rolls up | Nightly into the cross-tenant rollup table | Metering is **on by default** and needs no configuration. The full tenant and operator Usage views work with nothing to set up. ## What's metered Every metered feature belongs to one category. Ingestion and LLM counters are recorded on the hot path as traces arrive and models are called; compute, curation, and storage counters are recorded when the corresponding job runs or artifact is created. | Category | Features | | --- | --- | | **Ingestion** | Traces, sessions, spans, and tool calls ingested | | **LLM** | LLM calls, input tokens, output tokens, and spend — sliced per **connection** | | **Compute** | Clustering runs, issue-classification jobs, eval runs, eval targets scored, pre-prod eval runs, enrichment runs, scenario generations, remediations generated, playground runs, chat messages, MCP calls | | **Curation** | Annotations, judges, datasets, dashboards, prompt versions, and scores created | | **Storage** | Storage used (a point-in-time **level**, not a running total) | **Spend is stored as integer micro-dollars** (1 USD = 1,000,000) so it stays an integer aggregate like every other counter and can roll up and cross the operator boundary without ever becoming a floating-point or free-text field. The Usage view renders it back as dollars. Most counters **accumulate** — each event adds to that day's running total. **Storage used** is the exception: it's a *level* metric, so each reading **overwrites** the day's value (the latest reading is the truth, not the sum). ## Tenant usage view An admin (or any member) opens **Usage** to see their workspace's activity. Behind it: - `GET /usage` returns per-feature (or per-day, per-agent, per-connection) integer totals for a time window, scoped to the caller's workspace. - `GET /usage/summary` returns the per-feature totals plus a daily timeseries for the headline features (traces ingested, LLM calls, LLM spend). The window defaults to the last 30 days; pass `since`/`until` (`YYYY-MM-DD`) to change it. Because the store is scoped by tenant, one workspace can never see another's counters. ## Operator fleet view An operator opens **Usage** in the operator console to see the same counts across **every** tenant — the usage-based cost basis for the fleet. - `GET /admin/usage` reads the cross-tenant rollup, grouped by `company`, `feature`, `day`, or `project` (operator-gated; an agent key is always denied). - `GET /admin/usage/companies` lists per-company totals with each company's name, tier, and status. The operator view reads the **rollup table**, which the nightly job keeps in sync (below) — so it's fast and doesn't fan out live queries across every tenant schema. ## How it works ### Counters are recorded locally As traces are ingested, LLM calls run, jobs execute, and artifacts are created, Neens increments the matching counter in a durable per-tenant table. Recording is best-effort and never blocks or breaks the operation it's counting. ### The nightly rollup folds tenants into the operator view Once a day, a leader-elected job reads each tenant's counters and upserts them into the operator's cross-tenant rollup table. The upsert is idempotent — re-running the same day's rollup overwrites the same rows, so it never double-counts. This is what the operator fleet view reads. The rollup runs under **leader election**, so a scaled deployment with several app replicas elects exactly one replica to run it — you never get N× rollups. ## Troubleshooting | Symptom | Cause | Fix | | --- | --- | --- | | Operator Usage view lags the tenant view | The rollup runs nightly | Wait for the next nightly rollup, or re-check after it fires | | Usage view is empty on a brand-new workspace | No activity has been metered yet | Ingest traces / run evals; counters accrue as you use the product | See also: [Cost & model pricing](/guides/cost-and-model-pricing) (the per-model prices behind every dollar figure — a separate signal from these feature counters) and [Data retention](/administration/data-retention) and [Erasure](/administration/data-erasure). ================================================================================ # Data retention Source: /docs/administration/data-retention/ ================================================================================ # Data retention **Data Governance** is where a company admin controls how long trace data lives, how PII is handled, and how a data subject is permanently erased on request. The page is organized into four tabs — **Overview**, **Retention**, **PII Redaction**, and **Erasure** — and this page covers the **Overview** and **Retention** tabs. See [PII redaction](/administration/pii-redaction) and [Erasure](/administration/data-erasure) for the other two. ## At a glance | | | | --- | --- | | Where | **Data Governance** in the left nav (**Admin** group) → **Retention** tab | | Who can use it | Company **admins** only — members and viewers get `403` | | Key API routes | `GET`/`PUT` `/retention/policies`, `DELETE /retention/policies/{id}` | | Scope | Company-wide, with optional per-agent overrides | | Runs | A nightly purge job deletes data older than each agent's effective window | ## Overview tab The **Overview** tab is the tab you land on — a posture snapshot across all three other tabs, so an admin can tell at a glance whether anything needs attention without opening each one: - **Retention** — the company-wide default window and how many agents currently carry an override. - **PII redaction** — the redaction mode and whether **Enforced** is on, so it's obvious whether a client-supplied header could still weaken it. - **Erasure** — recent erasure requests and their status, so an in-flight or failed request is visible without opening the **Erasure** tab. Each tile links straight into the tab it summarizes. ## Configure retention Retention decides how long an agent's traces are kept before Neens purges them. You set **one company-wide default**, then optionally **override it per agent**. ### Set the company-wide default Under **Retention → Company-wide default**, enter a number of days and **Save default** (`PUT /retention/policies` with `scope_type: "company"`). This applies to every agent that doesn't have its own override. For example, entering **365** keeps a year of trace data for every agent with no override. A value of **`0` means "keep forever"** — it's the safe default, so existing tenants never lose data until an admin opts in. Any negative or malformed value is coerced to `0` (keep forever), so a bad policy can never silently delete data with a surprise short window. ### Add a per-agent override Under **Retention → Add override**, pick an agent and a window, then **Add override** (`PUT /retention/policies` with `scope_type: "project"` and the agent's id). For example, to keep only 90 days of data for a high-volume **Order Tracking** agent while the rest of the company keeps a year, select **Order Tracking**, enter **90**, and save. An override always wins over the company default. Setting an agent's override to `0` explicitly exempts it from a shorter company policy. ### Read the Retention by agent table The **Retention by agent** table is the single merged view of what Neens will actually apply, per agent — there's no separate "overrides" table and "effective window" table to cross-reference: | Agent | Effective window | Source | | --- | --- | --- | | Order Tracking | 90 days | Override | | Support Agent | 365 days | Company default | | Billing Assistant | 365 days | Company default | - **Effective window** is what the nightly purge actually uses for that agent. - **Source** is **Override** when the row came from that agent's own policy, or **Company default** when it's falling through to the company-wide setting. A row never shows a bare number without telling you which one produced it. ### Reset an override To go back to the company default for an agent, remove its row (**Remove policy**, `DELETE /retention/policies/{id}`) from the override list. This deletes the *policy*, not any data — the agent's **Source** in the table immediately flips from **Override** to **Company default** and its effective window becomes whatever the company-wide setting is. ### How the effective window resolves For any agent, Neens resolves the window **most-specific-first**: 1. a **per-agent override**, else 2. the **company-wide policy**, else 3. the platform's built-in default (**keep forever**). `0` at any level is a real configured value ("keep forever"), not a fall-through — an admin who sets an agent to `0` is deliberately exempting it. ### What the nightly purge does A scheduled job runs once a night and, for every company, resolves each agent's effective window and deletes data older than `now − window` across **both** stores — the analytics store and the operational store (traces, spans, tool calls, scores, and derived records). Because it uses the same delete cascade as [erasure](/administration/data-erasure), retention coverage stays complete. Retention runs even when an agent has no override and no company policy — a true no-op only happens when both resolve to "keep forever." ## Permissions Retention requires the company **admin** role (roles order `viewer` < `member` < `admin`). Members and viewers can't open **Data Governance**, and the API returns `403` for the retention endpoints. A company's data lives in its own physically isolated storage, so a policy can never reach another tenant. Every policy change is recorded in the [audit log](/administration/audit-log). ## Related - [PII redaction](/administration/pii-redaction) — the **PII Redaction** tab: scrub sensitive values out of traces before they're stored. - [Erasure](/administration/data-erasure) — the **Erasure** tab: permanently delete one data subject's data on request (GDPR Art. 17). - [Audit log](/administration/audit-log) — the immutable record of retention-policy changes. - [Workspace, orgs & agents](/administration/workspace) — how agents scope your data and per-agent overrides. ================================================================================ # PII redaction Source: /docs/administration/pii-redaction/ ================================================================================ # PII redaction PII redaction scrubs personally identifiable information — emails, phone numbers, card numbers, government IDs, secrets — out of your traces **before Neens stores anything**. Turn it on per agent, choose how aggressive it is (tag, mask, or tokenize), and the raw values never land in any store, queue, or log. The same engine also sanitizes what leaves Neens: prompts sent to your configured LLM connection for judging, clustering, and enrichment are scrubbed too. This is the **PII Redaction** tab on **Data Governance** — one of four tabs alongside **Overview**, **Retention**, and **Erasure**. See [Data retention](/administration/data-retention) and [Erasure](/administration/data-erasure) for the others. ## At a glance | | | | --- | --- | | Where | **Data Governance** in the left nav (**Admin** group) → **PII Redaction** tab | | Key API routes | `GET`/`PUT` `/redaction/settings`, `POST /redaction/preview` | | Who can use it | Company **admins** only — the same role that manages retention. Reading, changing, and previewing are all admin-gated | | Scope | Per agent, with an optional per-request header override (`X-Neens-Pii-Redaction`) | | Modes | `off` · `tag` · `mask` · `tokenize` | | When it runs | At the ingest edge, before anything is durably written | ## Choose a mode Given this span input: ``` Customer jane.doe@example.com called from 415-555-0187 about card 4111 1111 1111 1111. ``` | Mode | What is stored | Use it for | | --- | --- | --- | | `off` | The text, unchanged. Nothing is scanned. | The default — zero cost when you don't need redaction. | | `tag` | The text, unchanged — but Neens counts what it *would* redact and stamps the counts on the session. | Measuring your PII exposure before enforcing anything. | | `mask` | `Customer [PII:email] called from [PII:phone] about card [PII:credit_card].` | Maximum privacy: the value is gone, only its type remains. | | `tokenize` | `Customer [PII:email:9f2ab7c4d1e0] called from [PII:phone:3c1d0e5f7a2b] about card [PII:credit_card:b04e88a1c72f].` | Privacy **plus** correlation — the same value always produces the same token (see below). | Placeholders are stable formats: `[PII:]` for mask, `[PII::<12-hex>]` for tokenize. Redaction is idempotent — a placeholder that is already in the text is never re-matched or double-redacted, so re-sending an already-redacted payload is safe. ### Tokenize: correlation without the raw value In `tokenize` mode each detected value is replaced by a short, deterministic token derived from a keyed hash that is unique to your agent. That means: - **The same customer email always maps to the same token** within an agent — you can still filter, group, and cluster by "this same person showed up in 14 failing sessions" without Neens ever storing who that person is. - **The raw value is never stored** and cannot be recovered from the token. - **Tokens don't correlate across agents** — the same email produces a different token in a different agent, so two agents' data can't be joined on a token. ## Enable redaction ### Open the PII Redaction tab Go to **Data Governance** in the left nav and open the **PII Redaction** tab. (If you don't see the page, you need the company **admin** role.) ### Pick a mode Select `tag`, `mask`, or `tokenize`. Redaction is **off by default** — nothing is scanned until you choose a mode. A common rollout: start with `tag` to see counts appear on new sessions, then switch to `mask` or `tokenize` once the preview looks right. ### Try it in the live preview Paste a representative payload into the **live preview** box on the tab. It shows the redacted result and a count per entity type, using your current allowlist and custom patterns — before you save anything. The preview text is **never persisted**. ### Optionally check Enforced **Enforced** makes your agent setting final: the per-request `X-Neens-Pii-Redaction` header is ignored, so no client can weaken (or change) redaction on individual requests. See [What "Enforced" means](#what-enforced-means). ### Save Changes take effect on new ingest within about half a minute. Redaction applies **from this point forward** — data ingested before you enabled it is not retroactively redacted (use [erasure](/administration/data-erasure) for data that must be removed). ### The same, over the API ```bash curl -X PUT https://your-neens-host/api/redaction/settings \ -H "Authorization: Bearer $NEENS_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "mode": "tokenize", "enforced": true, "allowlist": ["support@yourco.com"], "customPatterns": [ {"type": "employee_id", "pattern": "EMP-\\d{6}"} ] }' ``` `GET /redaction/settings` returns the saved settings plus the **effective** policy for the agent. Settings changes are recorded in the [audit log](/administration/audit-log). Whether you save from the tab or over the API, a policy change converges across every server in the fleet within about **30 seconds** — each process caches the resolved policy briefly. ## Preview an input The preview endpoint runs the agent's full policy (allowlist and custom patterns included) over a text you supply and returns what would happen — without storing the text: ```bash curl -X POST https://your-neens-host/api/redaction/preview \ -H "Authorization: Bearer $NEENS_API_KEY" \ -H "Content-Type: application/json" \ -d '{"text": "Reach me at jane.doe@example.com", "mode": "mask"}' ``` ```json { "mode": "mask", "redacted": "Reach me at [PII:email]", "entities": [{"type": "email", "start": 12, "end": 32}], "counts": {"email": 1} } ``` `mode` in the request is optional — omit it to preview with the agent's saved mode; the response's `mode` echoes the mode that was actually applied. Input is capped at **64 KB**. This is the same endpoint the tab's live preview box uses, and like the settings routes it requires the company **admin** role. ## Per-request override A client can request a different mode for a single ingest call with the `X-Neens-Pii-Redaction` header (`off`, `tag`, `mask`, or `tokenize`): ```bash curl -X POST "https:///ingest/raw" \ -H "Authorization: Bearer nk_live_your_key_here" \ -H "X-Neens-Pii-Redaction: mask" \ -H "Content-Type: application/json" \ -d '{ "session": {"id": "trace-001", "agent_name": "support-agent"}, "spans": [ {"id": "span-1", "name": "llm.call", "kind": "llm", "input": "Customer email is jane.doe@example.com", "output": "Got it, I will follow up."} ] }' ``` This is useful when one producer (say, a staging environment replaying real tickets) needs stricter redaction than the agent default, or when you want to trial a mode from one client before flipping the agent setting. An unrecognized header value is ignored, never an error. ### What "Enforced" means Precedence for any single ingest request: 1. **Enforced agent setting** — the agent mode applies; the header is ignored entirely. 2. **Valid `X-Neens-Pii-Redaction` header** — the header's mode applies. One refinement: when the agent has **no saved policy** at all, the header is honored only if it does not *weaken* the deployment-wide default mode (on the ordering `off` < `tag` < `mask`/`tokenize`, the last two equally strong) — a client cannot switch off a fleet-wide default with a header. A saved but un-enforced agent setting can still be weakened by the header; check **Enforced** to close that. 3. **Agent setting** — the saved mode applies. 4. Otherwise — the deployment's default mode applies (`off` unless your operator changed it). If redaction is a compliance requirement, check **Enforced**. Without it, any client holding an ingest key can send `X-Neens-Pii-Redaction: off` and land raw values. Enforced closes that door: the header becomes inert and the agent policy is the only policy. ## Allowlist and custom patterns Both live in the **Advanced** panel on the **PII Redaction** tab (and in the same `PUT /redaction/settings` body) — kept collapsed by default since most agents never need them: - **Allowlist** — values that must *never* be treated as PII, even when they match a pattern. An entry matches exactly, or as a **case-insensitive substring** of the detected value — which cuts both ways: a short or generic entry (say, `an`) can act as a kill switch that suppresses every detection it happens to appear in, so keep entries as specific as the value you mean to allow. Entries shorter than **3 characters** are rejected for exactly that reason. Typical entries: your own support address (`support@yourco.com`), a documented test card number, your office phone number. Up to **200** entries of **3–256** characters each. - **Custom patterns** — extra *deny* patterns of your own, each a `{type, pattern}` pair. The type names the placeholder (`[PII:employee_id]`) and must match `[a-z0-9_]{1,64}` — lowercase letters, digits, and underscores, up to 64 characters; the pattern is a regular expression. Up to **50** patterns of up to **512** characters each; a pattern that doesn't compile — or that uses constructs prone to catastrophic backtracking, like nested quantifiers — is rejected when you save, not at ingest time. ## What gets detected Detection is layered: format patterns with **checksum validation** where a checksum exists (so a random 16-digit order number isn't mistaken for a card), plus **field-name context** for values whose format alone is too ambiguous. | Entity type | Detected by | | --- | --- | | `email` | Pattern | | `phone` | Pattern — international and US formats; never matched inside a longer digit run | | `ssn`, `itin` | Pattern, with known-invalid ranges excluded | | `credit_card` | Pattern **+ Luhn checksum** — 13–19 digits with or without separators | | `iban` | Pattern **+ mod-97 checksum** | | `us_routing` | Pattern **+ ABA checksum** | | `us_bank_account` | Field name only (an account number has no distinguishing format) | | `ipv4`, `ipv6`, `mac_address` | Pattern — real octet ranges, so version strings don't match | | `date_of_birth` | Date pattern **only** when nearby text or the field name mentions birth/DOB | | `person_name` | Field name (`name`, `first_name`, `customer_name`, `patient_name`, …) — plus free-text names when the NER capability is active | | `street_address`, `zip` | Street-address pattern; ZIP only adjacent to address context | | `geo_coord` | Latitude/longitude pairs in `lat`/`lng`-named fields | | `passport`, `drivers_license` | Field name only (formats vary too much for a safe bare pattern) | | Secrets & API keys | Patterns for JWTs, bearer tokens, common API-key prefixes, URL-embedded credentials, and `password=`-style assignments | Structured payloads are scanned **recursively** — every string inside nested objects and arrays. A value under a sensitive field name (`email`, `ssn`, `address`, …) is redacted wholesale as that type, regardless of its format. Pseudonymous identifiers your integration depends on — `session_id`, `trace_id`, `span_id`, `end_user_id` — are deliberately **not** treated as PII. ## What egress sanitization covers Ingest redaction protects data **at rest** — what Neens stores. *Egress* sanitization protects data **in motion** — anything Neens sends out of your deployment. The two are configured separately: many teams keep ingest `off` (full-fidelity traces for their own debugging) but set egress to `mask` or `tokenize`, so raw values still never leave for a third party. You set this in **Data Governance → PII redaction** as the **egress mode** — leave it blank to reuse your ingest mode, or pick a distinct `off` · `tag` · `mask` · `tokenize`. Egress sanitization runs on **every** channel that carries your trace content off the deployment: - **LLM prompts** — everything sent to your [LLM connection](/administration/llm-connections): judge prompts, cluster labeling, root-cause analysis, enrichments, and the assistant. A model never sees a raw value, so it can't echo one back into a score or summary. - **Fix bundles / coding-agent handoff** — the failing-example excerpts *and* every heading, label, and diagnosis Neens wrote from those traces (the fix title, the failure-mode summary and root-cause narrative, the cluster label and description, and the drafted proof-eval checks) embedded in a [fix bundle](/guides/fix-bundles) are scrubbed **before the bundle leaves Neens**, through every channel it can leave by: the **Copy fix bundle** button, a webhook handoff to your coding agent, and the headless CLI driver. This closes a gap where an agent relying on egress-only sanitization (raw ingest, scrubbed LLM prompts) could previously have had raw excerpts ride out inside a fix bundle. The typed **code fix** itself — the before/after diff Neens proposes for *your* repository — is intentionally **not** scrubbed. That is your own code and must apply cleanly; redacting it would corrupt the patch. Everything else that Neens derived from your traces — failing-example inputs, outputs, and errors, and the narrative Neens wrote about them (the fix title, the failure summary and root-cause text, the cluster label and description, and the drafted proof checks) — passes through the egress engine. ### Example: a failing example inside a fix bundle A failing-example excerpt that was ingested raw looks like this: ``` Input: Refund request from jane.doe@example.com for card 4111 1111 1111 1111 Error: card on file for jane.doe@example.com did not match ``` With egress `mask`, the excerpt embedded in the bundle carries only the entity types: ``` Input: Refund request from [PII:email] for card [PII:credit_card] Error: card on file for [PII:email] did not match ``` With egress `tokenize`, the same values become stable per-agent tokens — so both mentions of the same email are still visibly the *same* person to whoever reads the bundle, without the address itself ever leaving Neens: ``` Input: Refund request from [PII:email:9f2ab7c4d1e0] for card [PII:credit_card:b04e88a1c72f] Error: card on file for [PII:email:9f2ab7c4d1e0] did not match ``` Egress sanitization fails **closed**. If Neens can't resolve your agent's redaction policy, it **refuses to emit** the fix bundle (and refuses to hand content to an LLM) rather than send raw content on a guess — you'll see the bundle fail to build instead of leaking. ## How it works - **Redaction happens at the edge, before anything durable.** The payload is scrubbed before it reaches the ingest queue, the durable staging buffer, or the dead-letter queue — so a raw value never exists in any store, not even transiently in a failed batch. This covers every ingest path: `/ingest/*`, OTLP `/v1/traces`, pulled inlet connectors, and pre-production run capture. - **Counts are stamped on the session.** A session whose payload had anything detected carries a compact per-type count in its metadata, and the session detail view shows a **PII redacted (N)** badge — in `tag` mode this is the whole effect, which is what makes it a safe dry run. - **Egress is sanitized too — on every outbound channel.** The same engine scrubs everything that leaves your deployment: LLM prompts *and* [fix bundles](/guides/fix-bundles) handed to a coding agent. By default egress follows your agent's redaction mode; you can set a separate `egressMode` via `PUT /redaction/settings` (for example, ingest `off` but egress `mask`, keeping stored traces raw while guaranteeing raw values still never leave for a provider or a bundle). See [What egress sanitization covers](#what-egress-sanitization-covers). - **It's fast.** Detection is a small number of precompiled pattern passes over the text, and the agent policy is cached in-process — no per-request database read on the ingest hot path. That cache is also why a settings change takes up to ~30 seconds to apply. ## Detection limitations Be honest with your DPO about what pattern-based detection can and cannot do: - **It's pattern- and context-based, not semantic.** Formats with checksums (cards, IBANs, routing numbers) are detected with very few false positives. Formatless identifiers (bank accounts, passports, driver's licenses) are caught **only** when they sit under a telling field name — a passport number pasted into free prose will pass through. - **Free-text person names and addresses are the hard case.** `customer_name: "Jane Doe"` is caught by field context; *"I spoke with Jane Doe yesterday"* is not — a name is just words. The optional **NER capability** adds a named-entity-recognition model that detects person names and locations in free text, closing most of that gap at some latency cost. If free-text name detection matters for your deployment, ask your Neens contact about enabling it. - **Redaction is forward-only.** It applies at ingest; data stored before you enabled it stays as it was. That includes work already queued: envelopes staged in the durable ingest buffer or the dead-letter queue before the change replay with the redaction that was in effect when they were captured — there is no retroactive scrub. Use [erasure](/administration/data-erasure) to remove historical data. - **A custom pattern is only as good as its regex.** Use the preview endpoint (or the tab's live preview) against real payload shapes before enforcing. For a defense-in-depth posture, prefer redacting at the source too — the less PII your agent puts into its own traces, the less there is to detect. Redaction here is the backstop that makes the guarantee, not a reason to log carelessly. ## Permissions All three routes — `GET` and `PUT /redaction/settings` (including the Enforced bit, allowlist, and custom patterns) and `POST /redaction/preview` — require the company **admin** role: the same privacy-admin role that manages [data retention](/administration/data-retention) and [erasure](/administration/data-erasure). Every settings change is recorded in the [audit log](/administration/audit-log). Settings are per-agent and, like all tenant data, physically isolated per company — one agent's policy never affects another's data. ## Related - [Data retention](/administration/data-retention) — the scheduled purge, and the rest of the Data Governance page. - [Erasure](/administration/data-erasure) — removing individual subject data that predates redaction. - [Audit log](/administration/audit-log) — the record of who changed redaction settings. - [Send traces](/guides/send-traces) — the ingest endpoints the `X-Neens-Pii-Redaction` header rides on. - [LLM connections](/administration/llm-connections) — the provider connection whose egress is sanitized. - [Fix bundles](/guides/fix-bundles) — the coding-agent handoff whose embedded trace excerpts egress sanitization also scrubs. ================================================================================ # Erasure Source: /docs/administration/data-erasure/ ================================================================================ # Erasure The **Erasure** tab on **Data Governance** permanently deletes one data subject's data across every store — for a GDPR **Article 17 "right to erasure"** request, an account deletion, or a data subject access request (DSAR). **It cannot be undone.** ## At a glance | | | | --- | --- | | Where | **Data Governance** in the left nav (**Admin** group) → **Erasure** tab | | Who can use it | Company **admins** only — members and viewers get `403` | | Key API routes | `POST`/`GET` `/retention/erasure`, `GET /retention/erasure/{id}` | | Scope | Company-wide, with optional per-agent narrowing | | Runs | Asynchronously on a dedicated cleanup queue, tracked to a terminal status | Erasure shares the same delete cascade as [retention](/administration/data-retention)'s nightly purge, so what an erasure removes and what retention ages out never drift apart. ## Run a subject erasure ### Identify the subject On the **Erasure** tab, describe *who* to erase using any combination of three selectors: | Selector | Use it when | Field(s) | | --- | --- | --- | | **Metadata match** | A person maps to a value carried in the traces (the common case), e.g. `user_id = u_123` | **Metadata key** + **Metadata value** (both required together) | | **Session IDs** | You already know the exact traces to remove | **Session IDs** (one per line or comma-separated) | | **Conversation IDs** | You want every session in specific conversations | **Conversation IDs** (one per line or comma-separated) | You can combine them — Neens unions the resolved sessions. The selector **cannot be empty**: a request with no metadata match and no ids is rejected, so an erasure can never fan out into "erase everything." Optionally **Limit to agent** to narrow the search to one agent. ### Record a reason Enter a **Reason** — a legal basis or ticket reference, for example `DSAR SUP-1043`. It's stored on the erasure register and the audit log. You're warned if you leave it blank; we strongly recommend always recording one for evidence. ### Confirm and submit **Erase subject data…** opens a confirmation dialog spelling out that this permanently deletes the subject's data across all stores and cannot be undone. Confirming calls `POST /retention/erasure`, which creates a tracked request and runs the cascade **asynchronously** on a dedicated cleanup queue — so a large erasure never blocks live scoring. Erasure is **irreversible**. It hard-deletes the subject — there is no restore, and the deleted data is not recoverable. Double-check the selector (especially a broad metadata match) before confirming. ```bash curl -X POST https://your-neens-host/retention/erasure \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{ "subject_key": "user_id", "subject_value": "u_123", "reason": "DSAR SUP-1043" }' ``` ```bash curl -X POST https://your-neens-host/retention/erasure \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{ "session_ids": ["trace_abc", "trace_def"], "conversation_ids": ["conv_9"], "reason": "Account deletion" }' ``` ## What gets deleted An erasure resolves the selector to a concrete set of sessions, then hard-deletes them everywhere: - **Operational store** — the sessions and their spans and tool calls, plus every derived record keyed to them: scores, enrichment outputs, session labels, cluster memberships, root causes, topic assignments, dataset items, annotations, and ground-truth labels. - **Analytics store** — the same sessions, spans, tool calls, scores, and enrichment outputs. The operational delete is the authoritative, transactional one; the analytics-store erase is best-effort so a hiccup there can't strand a completed operational delete (a subsequent retention pass is a backstop). **"Is everything really gone?"** One rollup, the hourly session-stats table, holds only **aggregate counts and quantiles — no PII, no trace content**. It is not touched by an erasure and ages out naturally with its own retention window. Individual subject data (anything identifying) is removed by the cascade above. ## Read the erasure register Every erasure produces durable compliance evidence directly on the **Erasure** tab: - **Erasure register** — lists every request, newest first (`GET /retention/erasure`). Each row shows the subject, reason, **sessions affected**, and status (`pending` → `running` → `completed` / `failed`). The status polls to a terminal state on its own. - **Deletion manifest** — expand a row (or `GET /retention/erasure/{id}`) to see the normalized **selector**, the **resolved session IDs**, and a per-store, per-table count of exactly what was deleted. - **Audit log** — every erasure request also lands an immutable entry in the [audit log](/administration/audit-log), independent of the register, so the *who / when / why* is preserved even if a register row were ever disputed. ## Permissions Erasure requires the company **admin** role (roles order `viewer` < `member` < `admin`). Members and viewers can't open **Data Governance**, and the API returns `403` for the erasure endpoints. A company's data lives in its own physically isolated storage — an erasure can never reach another tenant. ## Related - [Data retention](/administration/data-retention) — the scheduled purge that shares this same delete cascade. - [PII redaction](/administration/pii-redaction) — scrubbing sensitive values before they're ever stored, so there's less to erase later. - [Audit log](/administration/audit-log) — the immutable record of erasure requests. - [Members & roles](/administration/members-and-roles) — the admin role that grants access. ================================================================================ # FAQ Source: /docs/faq/ ================================================================================ # FAQ Common questions and troubleshooting for Neens. If something isn't covered here, ask the in-app [Assistant](/guides/assistant) or your workspace admin. ## Getting started ### How do I send my first trace? Create an API key in **Settings → API keys**, then point your OpenTelemetry exporter at Neens or POST a trace to `/ingest/raw`. See [Getting started](/getting-started) and [Send traces](/guides/send-traces). ### Which names do I have to spell exactly? The ones a typo makes silent rather than loud: environment variables (`NEENS_*`), the pre-prod correlation attributes (`neens.eval_run_id` / `neens.dataset_item_id` / `neens.version_label`), the request headers (`X-Neens-Project-Id` / `X-Neens-Company-Id`), and the API key prefixes (`nk_live_` for an agent key). Everything else — trace attributes, endpoints, field names — is either standard OpenTelemetry or discoverable from the UI and the API responses. ### Do I need to change my agent's code to use Neens? If your agent already emits OpenTelemetry traces, no — point the OTLP exporter at Neens and add your API key as a bearer token. Otherwise you can POST traces in the Neens-native raw JSON format. See [Send traces](/guides/send-traces). ### What's the difference between a trace and a session? A **trace** is one end-to-end run of your agent. A **session** groups related traces (a multi-turn conversation) by their shared conversation id. **Traces** shows individual runs; **Sessions** shows the conversation-level rollup. See [Traces & sessions](/guides/traces-and-sessions). ## Ingestion ### My traces aren't showing up Work through these: - **Wrong key or agent.** A trace lands in the agent its API key belongs to. Make sure you're viewing that agent (use the agent switcher in the sidebar). - **Auth header.** The key must be sent as `Authorization: Bearer nk_live_…`. A `401` means the credential was rejected — see [the next question](#why-am-i-getting-a-401-on-ingest). - **Give it a few seconds.** Ingestion is asynchronous — Neens returns `202 Accepted` immediately and the trace appears shortly after, once the background worker has processed it. Under load this lag can grow. To confirm a single trace synchronously while testing, add `?sync=true` to the ingest URL and you'll get the stored session id back. - **Payload shape.** For raw JSON, `session.id` is required. A malformed or unparseable body returns `422` with a message saying what's wrong. - **Size limits.** A single trace payload over 1 MiB (or a batch over 8 MiB) is rejected with `413`. - **Backpressure.** A `429` with a `Retry-After` header means Neens is deliberately throttling you — see [below](#why-am-i-getting-429-responses-on-ingest). ### Why am I getting a 401 on ingest? Neens never silently accepts a bad credential. If you supply a key and it doesn't resolve, the request is always rejected with `401` rather than being misfiled: - **`Invalid or revoked API key.`** — the `nk_live_…` key you sent doesn't exist, was mistyped/truncated, or has been revoked. Revocation takes effect immediately. Create a new key in **Settings → API keys** and make sure you copy it exactly. - **`Unrecognized API credential.`** — you sent a bearer token that isn't a Neens key at all (for example, a provider API key by mistake). - **`Ingest requires an API key. Provide 'Authorization: Bearer nk_live_…'.`** — you sent *no* credential, and this deployment requires ingest authentication. Mint a key in **Settings → API keys** and send it as `Authorization: Bearer nk_live_…` — see [API keys](/administration/api-keys). ### Why am I getting 429 responses on ingest? Your agent's ingest queue is backed up and Neens is applying backpressure instead of accepting work it can't process yet. The response includes a `Retry-After` header (5 seconds by default) — honor it and retry with backoff; nothing is lost as long as you retry. ### What formats does Neens accept? OpenTelemetry (OTLP, protobuf or JSON) at `/v1/traces`, OpenInference at `/ingest/openinference`, and Neens-native raw JSON at `/ingest/raw` (or `/ingest/batch` for many at once). All accept gzip-compressed bodies. See [Send traces](/guides/send-traces). ### Is there a size limit? Yes — a single trace payload can be up to 1 MiB and a batch up to 8 MiB (measured after decompression). Larger payloads are rejected with `413`. ### Can one API key write to multiple agents? No. A key is bound to a single agent, which is why you don't send an agent ID with your traces — the key determines the destination and can't be used to write elsewhere. ### Why do my Sessions and Traces counts differ? They count different things. **Traces** counts individual runs; **Sessions** counts conversations. Traces that share a conversation id collapse into one session, and a trace without one is its own single-trace session — so the session count is always less than or equal to the trace count. If everything shows as single-trace sessions, your instrumentation isn't emitting a conversation id (`gen_ai.conversation.id`, `session.id`, `conversation.id`, or `thread.id`). See [Traces & sessions](/guides/traces-and-sessions). ## AI features & LLM connections ### AI features aren't working / my clusters are unlabeled [#ai-features-arent-working] Neens ships with **no built-in model access** — every AI feature (judge scoring, cluster labels, insight summaries, topic classification, enrichments, remediation drafts, the assistant) runs on an LLM connection your workspace configures. If none is configured, or none is *visible* to your agent (connections can be scoped to specific orgs/agents), those features degrade gracefully rather than erroring: - traces still ingest and display normally, - failure clusters are still detected but stay **unlabeled**, - judge runs fail with a clear error instead of producing scores, - the assistant can't answer. Fix: an admin adds a connection in **Settings → LLM providers** (Anthropic, OpenAI or any OpenAI-compatible endpoint including local Ollama, or AWS Bedrock) and, if it's scoped, makes sure your agent is included. Use the connection test — it makes one real model call — to confirm the credential works. See [Getting started](/getting-started#connect-an-llm). ### Can the assistant change my data? Only with your explicit approval, and only within your own permissions. The in-app [Assistant](/guides/assistant) can curate datasets, record ground-truth labels, create and deploy judges, start eval runs, and move suggested fixes along — but it always shows you the change first and runs it only when you press **Approve**. The change is then made on your own credentials, so it can never do something your role doesn't already allow, and it is attributed to you — recorded against your name, and shown as having come from the Assistant wherever that resource keeps an audit trail. Nothing deletes: there is no delete tool. See [Changes it can make](/guides/assistant#changes-it-can-make). ## Scoring & evaluation ### Are my traces scored automatically? Yes, once an LLM connection is configured. New agents come with a **Primary Score** composite judge that continuously scores a sample of incoming traces. You can add more judges or run them on demand. See [Continuous evaluation](/guides/continuous-evaluation) and [Judges](/guides/judges). ### Why are only some of my traces scored? Continuous (on-ingest) scoring runs on a sample of traffic to control cost — 5% by default for the Primary Score, plus a per-day budget per judge deployment. To score a specific set, run a judge on demand against the traces or a dataset you choose — manual runs score every selected target, with no sampling. See [Continuous evaluation](/guides/continuous-evaluation). ### I ran a judge but no scores appeared Check the run itself before the scores: 1. **Open the run in Activity.** Every judge run appears in the **Activity** feed with its status and an error summary. A run that errored on every item produces no scores. See [Activity](/guides/activity). 2. **Understand the status.** A run's status reflects its per-item success rate: `completed` (at least 90% of attempted items scored successfully), `completed_with_failures` (in between), `failed` (under 10% succeeded). 3. **Per-item errors mean per-item gaps.** When the judge's model call fails on one item (after up to 3 automatic retries for transient errors), that item is marked failed and simply has no score — the rest of the run continues. 4. **Common root causes:** no LLM connection visible to the agent ([above](#ai-features-arent-working)), a revoked or exhausted provider credential, or provider rate limits — the Activity error summary tells you which. ### How do I trust a judge's scores? Use **judge alignment**: Neens compares judge scores against human labels, shows how closely they converge, and lets you drill into disagreements and relabel. See [Annotations & review](/guides/annotations-and-review). ### What's the difference between a judge and an enrichment? A **judge** produces a score (a quality measurement). An **enrichment** adds structured metadata (a category, sentiment, extracted fields) to make traces easier to filter and segment. See [Judges](/guides/judges) and [Enrichments](/guides/enrichments). ## Pre-prod evaluations ### How do I test a new agent version before shipping it? Run a **pre-prod evaluation**: replay a golden dataset against a candidate version, score the results with your existing judges, and compare against a baseline (a previous run or a production time window). Regressions gate the release. You can run your agent yourself and let Neens correlate the traces, or register your agent's HTTP endpoint and let Neens call it for every golden prompt. See [Pre-prod evaluations](/guides/preprod-evals). ### Neens refuses to call my agent endpoint That's the SSRF guard. When Neens calls your agent it makes outbound HTTP calls to the endpoint you registered, so by default it refuses any URL that resolves to a private, loopback, or link-local address — the item fails with an error like `agent endpoint url is not allowed: host '…' resolves to a disallowed (private/loopback/link-local) address`. If your agent genuinely lives on an internal address (a Docker service, a VPC host, `localhost` in development), an operator can allowlist it. Only expose endpoints you control. Also note each endpoint call is capped at 60 seconds — a hung endpoint fails that one item honestly rather than fabricating output. ### Do pre-prod evals affect my production metrics? They never slow production down — pre-prod runs execute on an isolated worker fleet. They are **not automatically excluded from your dashboards**, though: pre-prod scores carry `source: preprod`, which makes them their own score type and easy to filter, but a score-grain widget counts every source unless you say otherwise. Slice or filter by **Score source** (`score_source`) when you want production-only numbers. See [Pre-prod evaluations](/guides/preprod-evals). ## Diagnosing failures ### How does Neens decide what's a failure? By default it treats a trace as failing when its primary score falls below 0.5 — so it catches "silent" failures, not just runs that errored. Admins can change the selection criteria per agent (score-based with a chosen metric and threshold, all traces, or errors only). See [Clustering](/guides/clustering). ### What's the difference between a failure mode and an issue? A **failure mode** is a category in your failure taxonomy — Neens ships platform defaults and you can add your own. An **issue** is a failure mode being actively tracked as work, with a lifecycle (open → acknowledged → investigating → mitigating → resolved, plus mute). See [Issues & failure modes](/guides/issues-and-failure-modes). ### Why does a fix say "this is a downstream service issue, not your agent"? Because the evidence says so. When Neens generates a [remediation](/guides/remediations) it classifies **where** the failure lives from the traces — the HTTP status codes in the errors, the error classes, the span kinds. If your agent's tool is getting `5xx`/`503`/timeouts from its own backend (or from a service *it* calls), that's a **service** failure, not an agent defect — so Neens generates an **advisory** that names the owning component and recommends a resilience fix (backoff, circuit breaker, idempotency key) instead of a pointless prompt edit. See [Where the failure lives](/guides/remediations#where-the-failure-lives-the-failure-locus). ### Why is my remediation held as "Needs grounding" and not in the backlog? A remediation only enters the actionable backlog if it carries a **concrete change**. If the proposal came back empty or templated (e.g. generic "clarify the agent's instructions" filler), or there wasn't enough evidence to classify the failure, Neens parks it as **needs grounding** rather than pass off a placeholder as work. It's not lost — switch to the **Needs grounding** view (or add `include_ungrounded=true` to the API call) to see and work it. See [Needs grounding](/guides/remediations#needs-grounding-held-out-of-the-backlog). ### Why can't I mark a fix as applied? Applying (or verifying) an agent-fixable remediation requires a **bound proof** — a verification run or a proof-eval gate — so "we shipped it" and "we proved it works" stay distinct claims. Simulate the fix or run an [eval-verified PR](/guides/eval-verified-fix) to bind one; if you've verified it another way you can override with a recorded reason. See [Applying a fix requires a bound proof](/guides/remediations#applying-a-fix-requires-a-bound-proof). ## Emails & notifications ### Emails aren't arriving Neens sends transactional email for password resets, member invites, first-admin activation, and the optional daily digest. If none of them arrive: - **Check with your admin that an email provider is configured.** If none is, no email is sent. - **No provider ⇒ emails are logged, not sent.** With no transport configured, Neens never silently drops a message — it writes the full email, *including the activation or reset link*, to the server logs. An admin can copy the link from the logs to unblock you; invite flows also surface the raw link in the UI as a fallback. - **Expired link?** Password-reset links are valid for 60 minutes; invitation/activation links for 14 days. Request a fresh one. ## Access & administration ### Who can do what? Admins manage the workspace — team, agents, LLM connections, and keys. Members work within the orgs and agents they've been assigned; viewers are read-only. See the Administration section of these docs. ### How do teammates get access? An admin invites them by email from **Settings → Members**; the invite is a one-time activation link (valid 14 days) where they set a password. New members can be assigned to specific orgs on invite. ### I lost my API key — can I recover it? No. Neens stores only a hash of each key, so the raw value is shown only once at creation. Revoke the old key and create a new one — revocation takes effect immediately. ### What is a persona? Does it limit what I can do? A persona is a *lens*: it tailors which navigation items are emphasized, your landing page, and defaults to how you work (Executive, Developer, Product manager, …). It never grants or removes access — everything stays reachable, and you can switch or reset your lens any time. ## Data & privacy ### Is my data isolated from other companies? Yes. Every trace is scoped to your agent and company, and each company's data is kept physically separate. You only see data in the agents and orgs you have access to. ### How long is my data kept, and can I delete a specific person? An admin sets a **retention window** (a company-wide default plus optional per-agent overrides; `0` = keep forever), and Neens purges data older than the effective window nightly. To satisfy a GDPR Art. 17 request, an admin can run an irreversible **subject erasure** that hard-deletes one subject across every store, with a deletion manifest and audit trail. See [Data retention](/administration/data-retention) and [Erasure](/administration/data-erasure). ### How is spend calculated? From the token usage on your model spans — Neens reads input/output tokens from standard attributes and prices them per model. See [Metrics](/guides/metrics). ### Why does my spend figure say *partial*, and how do I fix it? Some traces in that window ran on a model with **no configured rate**. Neens has no fallback rate, so those tokens are **excluded** from the total rather than priced at a guess — meaning the number shown is a floor, and real spend is higher. Hover the badge's **ⓘ** to see which models are unpriced (**click** it to pin the popover open so you can follow the link inside; `Escape` or a click outside closes it), then have an admin set a rate for each in **Settings → Model pricing**. Rates are per 1M tokens, and `0` is a valid rate for a model you self-host. Full walkthrough: [Fix a spend figure that says *partial*](/guides/cost-and-model-pricing#fix-a-spend-figure-that-says-partial). ### Can I prove a cheaper model still hits our quality bar? Yes. Every score records which model produced the answer it graded, so `eval_pass_rate` slices by model — and by agent × model, since a cheaper model is often fine for one agent and not another. A trace that used two models lands in an explicit **Mixed** bucket and a score with no derivable model in an **Unknown** one; neither is folded into a real model's rate. See [Model comparison](/guides/model-comparison), and [Cost & model pricing](/guides/cost-and-model-pricing) for the cost half.