# Neens

> Neens turns your agent's failures into regression tests, so the ones you fix can't come back.

Neens is an agent observability and evaluation platform: ingest agent traces, cluster failures, score with automated judges, and close the loop with gated fixes. This file indexes the product documentation; append `-full` to the filename for the full text.

## Getting started
- [Getting started](/docs/getting-started/): From activating your Neens account to sending and scoring your first trace, including workspace setup, an LLM connection, and an ingest API key.

## Core concepts
- [Core concepts](/docs/concepts/): The vocabulary Neens uses — traces, sessions, spans, judges, scores, clusters, issues, datasets, remediations, eval gates, and the workspace hierarchy.

## Guides

#### Ingest & observe

- [Send traces](/docs/guides/send-traces/): How to send agent traces to Neens over OTLP, OpenInference, or raw JSON — authentication, endpoints, size limits, semantic-convention attributes, and response codes.
- [Framework quickstarts](/docs/guides/quickstarts/): Copy-paste OpenTelemetry config to send traces to Neens from popular agent frameworks — LangGraph, CrewAI, OpenAI Agents SDK, Pydantic AI, Claude Agent SDK, and the Vercel AI SDK. No Neens SDK, no fork.
  - [LangGraph](/docs/guides/quickstarts/langgraph/): Send LangGraph agent traces to Neens over OpenTelemetry — copy-paste exporter config, no Neens SDK.
  - [CrewAI](/docs/guides/quickstarts/crewai/): Send CrewAI agent traces to Neens over OpenTelemetry — copy-paste exporter config, no Neens SDK.
  - [OpenAI Agents SDK](/docs/guides/quickstarts/openai-agents/): Send OpenAI Agents SDK traces to Neens over OpenTelemetry — copy-paste exporter config, no Neens SDK.
  - [Pydantic AI](/docs/guides/quickstarts/pydantic-ai/): Send Pydantic AI agent traces to Neens over OpenTelemetry — copy-paste exporter config, no Neens SDK.
  - [Claude Agent SDK](/docs/guides/quickstarts/claude-agent-sdk/): Send Claude Agent SDK traces to Neens over OpenTelemetry — copy-paste exporter config, no Neens SDK.
  - [Vercel AI SDK](/docs/guides/quickstarts/vercel-ai-sdk/): Send Vercel AI SDK traces to Neens over OpenTelemetry — copy-paste exporter config, no Neens SDK.
- [Connectors](/docs/guides/connectors/)
  - [Trace inlets](/docs/guides/connectors/trace-inlets/): Pull agent traces into Neens from an existing tracing tool — Langfuse or Braintrust — on a schedule, so the whole failure-to-fix loop runs on top of your current install with zero migration.
  - [Connect your helpdesk](/docs/guides/connectors/outcome-inlets/): Pull real case outcomes — resolved, escalated, CSAT, resolution time — from Zendesk, Intercom, Salesforce or Jira Service Management into Neens on a schedule, keyed to the agent conversations that produced them, with the match rate visible on every connector.
- [Traces & Sessions](/docs/guides/traces-and-sessions/): How to browse, filter, and inspect agent traces in Neens — the trend header (volume, token usage, and cost over time), the Traces list, the trace inspector (spans, conversation, graph, system prompts, cost), and the Sessions conversation rollup.
- [Conversation transcript](/docs/guides/conversation-transcript/): How Neens reconstructs the user/assistant transcript from raw trace spans — which spans contribute turns, which payload shapes are understood, and how to instrument your agent so the Conversation tab, golden datasets, judge comparisons, and pre-prod baselines all capture the right question and the right answer.
- [Business outcomes](/docs/guides/business-outcomes/): Send Neens the real business result of each case — resolved, contained, escalated, CSAT, handle time, refund amount — key it to the agent traces that produced it, restate it when your system of record changes its mind, and see the honest match rate behind every number.
- [Emitted attributes](/docs/guides/emitted-attributes/): The neens.* span-attribute convention your agent uses to assert business facts about a conversation — an end-user id, whether it escalated, a handoff reason, in-conversation feedback and open outcomes — how Neens lifts them onto the session as emitted-provenance signal, and how to read them back as measures.
- [Agent Map](/docs/guides/agent-map/): How the Neens Agent Map views turn a run — or a whole cohort of runs — into a weighted agent/tool topology with the cost and latency bottleneck flagged.
- [Activity](/docs/guides/activity/): The Activity feed shows every background run Neens executes over your data — scoring, enrichment, and clustering — with a colour-coded succeeded/failed/running progress breakdown, statuses, and who-triggered-it attribution.

#### Diagnose

- [Failure clustering](/docs/guides/clustering/): How Neens groups failing sessions into named failure-mode clusters — what counts as a failure, when analysis runs, how clusters are labeled, and how new sessions are assigned online.
- [Issues & failure modes](/docs/guides/issues-and-failure-modes/): How the Neens failure taxonomy works — curating named failure modes, classifying failing traces onto them, and managing the resulting Issues through an operational lifecycle to a fix.
- [Topics](/docs/guides/topics/): How the Neens topic atlas organizes conversations by subject — spaces, topic curation, the volume and coverage-gap metrics, and using topics to slice dashboards and datasets.

#### Evaluate

- [Prompt management](/docs/guides/prompt-management/): Store agent and judge prompts as versioned, deploy-tagged entries in Neens, try them in the playground against your own LLM connection, and turn a prompt-change remediation into a new version with one click.
- [Judges](/docs/guides/judges/): Create, configure, and run LLM and rule-based judges that score your traces and sessions, including composite judges, eval runs, and run outcomes.
- [Scores](/docs/guides/scores/): How Neens stores quality scores, how to browse the score catalogue with donut tiles and per-user favorites, how to configure the display scale and color bands, and what the Primary Score composite is.
- [Scorer lifecycle](/docs/guides/scorer-lifecycle/): Curate your scorers with lifecycle stages (experimental → finalized → archived), keep the Scores page focused on finalized judges, and compare judge versions or export a run's scores with input and output.
- [Run Targets](/docs/guides/run-targets/): Define what population a scorer runs on — a versioned dataset, a live sample of new traffic, or a filtered historical sweep — and reuse one Run Target across many scorers.
- [Continuous evaluation](/docs/guides/continuous-evaluation/): How Neens automatically scores new traces as they arrive, including sampling, cold-start boost, daily cost caps, and pause/resume controls.
- [Enrichments](/docs/guides/enrichments/): Use LLM enrichments to extract structured fields from every trace or session, filter on them in Traces, and turn them into datasets for judges.
- [External API scoring](/docs/guides/external-api-scoring/): Score with a judge or enrich a trace by calling any HTTP API — a toxicity, sentiment, moderation, or grammar service, or your own scoring microservice — and mapping a value out of the JSON response, with no LLM.
- [Datasets](/docs/guides/datasets/): Curate trace and session examples into datasets — filter-based, cluster-based, or streaming — and cut immutable golden versions that feed judges and pre-prod evaluations.
- [Annotations & review](/docs/guides/annotations-and-review/): Label sessions in the prioritized Review queue, browse the shared ground-truth label store, and use judge-versus-human alignment metrics to iterate your judges.

#### Fix & ship

- [Remediations](/docs/guides/remediations/): Generate typed, evidence-grounded fix proposals for recurring agent failures — classified by where each failure lives (your agent, a tool contract, a downstream/upstream service, or a control working as intended) — then work each one through a Root cause → Fix → Proof → Ship loop, simulate its effect before shipping, apply it (only with a bound proof), and verify efficacy after.
- [Regenerate remediations](/docs/guides/regenerate-remediations/): Re-run the current fix engine on an existing remediation's failure to get a fresh, better-grounded proposal — archiving the old one as a restorable, superseded predecessor so nothing is lost — one fix at a time or across your whole open backlog.
- [Fix bundles](/docs/guides/fix-bundles/): Export a remediation as one coding-agent-ready pack — root cause, a typed fix, anonymized failing examples, and a ready-to-run proof-eval gate command — that you paste into your own coding agent to implement and gate the fix in your repo.
- [Eval-verified PR](/docs/guides/eval-verified-fix/): Neens proposes a fix for a confirmed failure, applies it on a branch, verifies it with pass^k pre-prod eval runs against the failure's own evals plus your accumulated regression set, and opens a pull request only when every run is green — a human always merges.
- [Autonomy levels](/docs/guides/autonomy/): Choose per agent how much of the failure→fix loop runs without a human — from propose-only to auto-accepting an eligible remediation and starting the eval-verified fix run unattended — with an auto-accept policy, protected blast radius, a spend budget that refuses an unpriceable run, a hold on any remediation that keeps failing, fail-safe suspension, and a decision log for every autonomous action.
- [Prompt optimization](/docs/guides/prompt-optimization/): Neens writes its own better system prompt for a confirmed failure mode by replaying that failure's historical traces against candidate prompts, grading them with your judges, and reflecting on the graded trajectories — then proves the winner on a held-out split and emits it as a normal remediation that still goes through the eval-verified PR gate.
- [Fix outcomes](/docs/guides/post-merge-efficacy/): After a Neens-opened fix PR merges, Neens auto-measures whether the fix actually reduced the failure in production and shows a per-agent Outcomes report — failure volume before vs after each merged fix, verified/regressed verdicts, headline reduction %, and median cluster-to-merge MTTR.
- [Eval gates](/docs/guides/eval-gates/): Turn a confirmed failure mode into a standing eval gate — a regression test built from your own failures that tracks a baseline pass rate and guards your releases.
- [What changed](/docs/guides/what-changed/): Record deploy events for your agent and correlate score drops, regressions, and failure spikes to the prompt, model, or tool change that landed just before them.
- [Pre-prod evaluations](/docs/guides/preprod-evals/): Replay a frozen golden dataset against a candidate agent version before it ships, score it with your existing judges, compare it to a baseline, and gate the release on regressions.
- [Model sweeps](/docs/guides/model-sweeps/): Replay one frozen golden dataset against several models at once under identical conditions, require each model to pass k independent runs, and get a cost estimate before you spend anything — so "we switched to the cheaper model" is a decision you can defend.
- [Sweep decisions](/docs/guides/sweep-decisions/): Turn a finished model sweep into one defensible sentence — which model to ship, at what quality bar, at what cost per case, and per agent — with the sample size, confidence interval and unpriced arms shown rather than rounded away.
- [Stress tests](/docs/guides/stress-testing/): Generate synthetic adversarial test scenarios from your failure-mode taxonomy — grounded in the traces where your agent actually failed — and run them through the pre-prod evaluation gate before you ship the next version.

#### Analyze

- [Dashboards](/docs/guides/dashboards/): Build custom dashboards from the metrics catalogue, scope and share them safely, and use the platform persona dashboards Neens ships out of the box.
- [Metrics catalogue](/docs/guides/metrics/): The Measure × Dimension model behind the Neens dashboards, digest tiles, and overview stats — every measure, every slice, and the dynamic metadata dimensions.
- [Business KPIs](/docs/guides/business-kpis/): Containment rate, resolution time and cost per case — measured per case rather than per trace, from the definition your agent declares, always shown with the provenance and the coverage behind the number, on one Business KPIs page — and how to promote any measure to a KPI with a target, a direction, an owner and a review cadence.
- [Derived measures](/docs/guides/derived-measures/): Zero-setup business measures Neens computes straight from the traces you already send — turns and traces per case, case duration, cost per turn, tool success and retry rates, span error rate and first-response latency — what each one tells you, when to watch it, and how to promote one to a KPI with a target and daily history.
- [Custom measures](/docs/guides/custom-measures/): Define your own KPI — your name, your source, your denominator — and have it behave like a measure Neens ships, in the gallery, on a dashboard, behind an alert rule, with its provenance and coverage attached.
- [Cost & model pricing](/docs/guides/cost-and-model-pricing/): How Neens turns token counts into dollars — the dated per-model price table, tenant rate overrides, the deliberate "unpriced" state, and how to set your own rates from Settings or the API.
- [Cost & Quality](/docs/guides/cost-and-quality/): Read the Cost & Quality page to see what your agent and your LLM judges cost, which cheaper models still clear a quality bar you can measure, and how to test a model switch against your golden suite before it reaches production.
- [Cost Optimization](/docs/guides/cost-optimization/): A per-agent view of what your agent's LLM traffic and your LLM judges cost over a window, how much of your traffic gets scored, and how to add cheaper scorer models like OpenRouter or a LiteLLM proxy.
- [Cost–quality frontier](/docs/guides/cost-quality-frontier/): Read the Cost & Quality frontier to cut agent and LLM-judge cost without measurably losing quality — what the axes mean, when a model is on the frontier or overpaying, the acceptance bar and the noise band, and how Neens states "no measurable quality loss" honestly.
- [Model comparison](/docs/guides/model-comparison/): Slice eval pass rate by the model that produced the answer — per agent, with an explicit Mixed and Unknown bucket — so "the cheaper model still hits our bar" is a number you can defend rather than a hunch.
- [Insights](/docs/guides/insights/): How the Neens detectors find anomalies, regressions, and issue spikes, how the Observe/Diagnose/Fix feeds and insight lifecycle work, how the fleet briefing loads instantly and upgrades itself to an AI-written narrative in the background, and how to get notified in-app, in Slack, by webhook, or by digest email.
- [Alert rules](/docs/guides/alerts/): Define your own metric-threshold alerts ("page me when eval pass-rate < 90%") that fire through the Neens in-app, Slack, and webhook notification sinks.
- [Assistant](/docs/guides/assistant/): The in-app Neens Assistant answers questions about your agent's traces, scores, issues, and fixes in plain language, and can curate datasets, record labels, set up judges and move fixes along — every change proposed for your approval first, under your own permissions.

#### Integrate & automate

- [MCP server](/docs/guides/mcp/): Connect any MCP-compatible agent to Neens to read the failure→fix loop (traces, failure modes, clusters, scores, eval runs, remediations) and write back to it (datasets, golden versions, annotations, judges, pre-prod evaluations, remediation lifecycle) — from a failure cluster all the way to a pre-prod comparison, including pulling a fix, verifying it against your preview deploy, and reporting the pull request back — using your agent API key.
- [Connect Claude Code](/docs/guides/connect-claude-code/): Connect Claude Code to the Neens MCP server with a per-user browser sign-in instead of a shared key, so every action Claude Code takes is attributable to you and governed by your role — including how to pick an agent and how to view and revoke your MCP authorizations.

## API reference
- [Ingest & observe — API](/docs/api-reference/ingest-observe/): Send traces and read back sessions, conversations, agents, and the agent topology.
- [Diagnose — API](/docs/api-reference/diagnose/): Failure clustering, the failure-mode taxonomy and Issues, topics, and insights.
- [Evaluate — API](/docs/api-reference/evaluate/): Judges, scores, enrichments, datasets, human annotations & review, and pre-prod eval runs.
- [Fix & ship — API](/docs/api-reference/fix/): Typed remediations, the evals-from-failures flywheel, and deploy-event correlation.
- [Analyze — API](/docs/api-reference/analyze/): Custom dashboards, the metrics catalogue, and saved analyses.
- [Assistant & prompts — API](/docs/api-reference/assistant/): The in-app chat assistant — questions plus approval-gated writes — and the versioned prompt registry.
- [Administration — API](/docs/api-reference/administration/): Tenancy, members & auth, personas, settings & LLM connections, retention, and the audit log.
- [More endpoints — API](/docs/api-reference/more/): Additional endpoints not grouped above.

## Administration
- [Workspace, orgs & agents](/docs/administration/workspace/): How the Neens company → organization → agent hierarchy works, who can see and create what, and the Getting started checklist a new workspace admin walks through.
- [Members & roles](/docs/administration/members-and-roles/): Inviting teammates, assigning organizations and roles, removing members, and the full role-based permission matrix that governs who can do what in a Neens workspace.
- [Views (personas)](/docs/administration/personas/): Persona lenses tailor the Neens navigation and landing page per user without ever changing permissions — including the built-in profiles, switching views, and authoring custom views.
- [API keys](/docs/administration/api-keys/): How to create, use, and revoke the Neens per-agent ingest API keys, what the key resolves to, and exactly when a revoked key stops working.
- [LLM connections](/docs/administration/llm-connections/): Configure the LLM providers Neens uses for judges, insights, the assistant, enrichments, remediations, and cluster labeling — including credential encryption, scoping, the connection test, and how features degrade without one.
- [Signing in & passwords](/docs/administration/signing-in/): The everyday access flows for a Neens account — activating an invite and setting your first password, signing in with email and password (and a code when two-factor is on), what to do when repeated failures pause sign-in, and resetting a forgotten password.
- [Single sign-on (SSO)](/docs/administration/single-sign-on/): Let your team sign in to Neens through your own identity provider — configure a per-tenant OIDC or SAML connection, route email domains to it, and have first-time users provisioned automatically, with an optional Neens second factor layered on top.
- [Account settings](/docs/administration/account/): Your personal Neens settings — password changes and resets, the password rules, two-factor authentication with TOTP and recovery codes, your active sessions and how to sign out other devices, the security emails Neens sends you, interface language, the digest email, your view (persona), and score favorites.
- [Languages](/docs/administration/languages/): The eight interface languages Neens ships, how to pick one for yourself or for an API client, exactly which surfaces are translated and which deliberately stay in English, and how translation completeness is guaranteed.
- [Audit log](/docs/administration/audit-log/): The admin-only, append-only record of who did what in your workspace — every security-relevant mutation, with actor, target, timestamp, and filters.
- [Usage metering](/docs/administration/usage-metering/): Neens meters every tenant's feature-level usage as durable integer counters — a tenant-admin Usage view and a nightly rollup — recording only integer aggregates, never trace, prompt, or response content.
- [Data retention](/docs/administration/data-retention/): Set a company-wide default retention window and per-agent overrides, and read the merged Retention by agent table that shows the effective window and its source for every agent.
- [PII redaction](/docs/administration/pii-redaction/): Detect and redact personally identifiable information (emails, phone numbers, card numbers, secrets, and more) at ingest time — with tag, mask, and tokenize modes, per-request overrides, and egress sanitization that also covers LLM prompts and fix bundles.
- [Erasure](/docs/administration/data-erasure/): Run an irreversible GDPR Article 17 subject erasure by metadata match or explicit session/conversation IDs, and read the erasure register's deletion manifest and audit trail.

## API reference
- [Ingest & observe — API](/docs/api-reference/ingest-observe/): Send traces and read back sessions, conversations, agents, and the agent topology.
- [Diagnose — API](/docs/api-reference/diagnose/): Failure clustering, the failure-mode taxonomy and Issues, topics, and insights.
- [Evaluate — API](/docs/api-reference/evaluate/): Judges, scores, enrichments, datasets, human annotations & review, and pre-prod eval runs.
- [Fix & ship — API](/docs/api-reference/fix/): Typed remediations, the evals-from-failures flywheel, and deploy-event correlation.
- [Analyze — API](/docs/api-reference/analyze/): Custom dashboards, the metrics catalogue, and saved analyses.
- [Assistant & prompts — API](/docs/api-reference/assistant/): The in-app chat assistant — questions plus approval-gated writes — and the versioned prompt registry.
- [Administration — API](/docs/api-reference/administration/): Tenancy, members & auth, personas, settings & LLM connections, retention, and the audit log.
- [More endpoints — API](/docs/api-reference/more/): Additional endpoints not grouped above.

## Optional
- [FAQ](/docs/faq/): Frequently asked questions and troubleshooting for Neens — ingestion errors, missing traces, unscored runs, LLM connection issues, pre-prod evals, and emails.