Core concepts

A short glossary of the terms Neens uses. Each entry is a definition plus a link to the guide that covers it in depth — everything else in the docs builds on these.

Traces & the data model

Trace — A single end-to-end run of your agent, made of spans. One trace is one interaction: a request comes in, your agent works, a response goes out. This is the raw unit you send to Neens; the Traces page lists them individually. See Traces & sessions.

Session — A conversation-level rollup of traces. Traces that share a conversation id (captured at ingest from standard attributes like gen_ai.conversation.id, session.id, conversation.id, or thread.id) are grouped into one session, so a multi-turn chat reads as one unit on the Sessions page. A trace with no conversation id is its own single-trace session. See Traces & sessions.

Span — One step inside a trace: an LLM call, a tool call, a retrieval, a guardrail check, an agent step, or a custom operation. Spans nest to form the shape of the run, and each carries inputs, outputs, timing, a status (ok / error), and — for model calls — the model name and token usage that Neens turns into cost (tokens × that model’s price — see Cost & model pricing). See Send traces.

Tool call — A tool or function invocation your agent made, extracted from its span. Neens captures the tool name, the arguments, the result, the duration, and any error. See Traces & sessions.

Agent — The named producer of a trace, read off each incoming trace. A single Agent workspace (see the workspace hierarchy below) can hold traces from several such agent names (or several versions of one) and slice metrics by name. See Metrics.

Evaluation

Judge — An automated evaluator that scores traces against a metric such as faithfulness, answer relevancy, or coherence. Judges are usually LLM-based (they run on your configured LLM connection) and can also be composed into composites. A judge has versions and deployments, is visible at platform, org, or agent level, and runs on demand, on a schedule, or continuously on incoming traffic. See Judges.

Score — The output of a judge for one target (a trace or a span): a metric key, a numeric value, an optional pass/fail label against a threshold, and a rationale explaining the verdict. The Scores page is the catalogue of every metric being produced. See Scores.

Primary Score — Your agent’s headline quality metric: a composite judge set up during agent onboarding that continuously scores a sample of incoming traces (5% by default), so you have quality signal from day one without configuring anything. See Continuous evaluation.

Enrichment — An LLM step that adds structured metadata to traces — a category, a sentiment, extracted fields — rather than a quality score. Enrichments run manually or automatically on ingest (capped per day), and their outputs become filters and dataset criteria. See Enrichments.

Topic — A subject-matter grouping of what your users actually ask about. Topics are proposed automatically and curatable by hand, and each shows volume and failure rate so you can see where quality problems concentrate. See Topics.

Dataset — A curated collection of examples (inputs and expected outputs) built manually, from a filter, from a cluster, or synced continuously from live traffic. A dataset can be snapshotted into immutable versions, and a version can be marked golden — the frozen reference set that eval gates and pre-prod evaluations replay. See Datasets.

Annotation — A human label on a trace: a pass/fail verdict with an optional critique. Annotations are the ground truth Neens aligns judges against — the review queue, judge-alignment reports, and golden datasets are all built from them. See Annotations & review.

Diagnosis

Cluster — An automatically discovered group of similar failing traces. Neens selects the failure set (by default, traces whose primary score falls below 0.5 — configurable to all traces or errors only), embeds them, clusters them, and labels each cluster using your LLM connection. Clusters are active until resolved. See Clustering.

Failure mode — A named category in your failure taxonomy — the durable “kind of failure” that clusters and classified traces roll up into. Neens ships a platform taxonomy and you can add custom modes. See Issues & failure modes.

Issue — A failure mode being actively tracked as work: it has a lifecycle (open → acknowledged → investigating → mitigating → resolved, plus muted), carries evidence sessions, and can own remediations. Issues are how you manage failures over time instead of rediscovering them. See Issues & failure modes.

Insight — A detected signal worth your attention: an anomaly (error-rate or latency jump), an evaluation regression, or an issue volume spike. Insights are deduplicated, tracked through new → recurred → resolved, routable to Slack/webhooks, and mutable with feedback. See Insights.

Fixing

Remediation — A typed, tracked fix for a failure — for example a prompt change or a tool fix — generated from real trace evidence. It moves proposed → accepted → applied → verified (or regressed), separate from your own triage lane (to do / in progress / done). You can simulate a remediation to preview its effect on real failing traces before shipping anything, and after a Neens-opened fix PR merges, Neens measures whether the failure actually dropped in production to set the verified / regressed verdict. Applying a fix requires a bound proof (a verification run or an eval gate) — Neens won’t mark a fix shipped on a hunch. See Remediations and Fix outcomes.

Failure locus — Where a failure actually lives, decided from the trace evidence rather than the cluster label: your agent’s reasoning, a tool contract, a downstream/upstream service, a quality issue, a control working as intended, or an incoherent/unknown grouping. The locus sets a remediation’s actionability — an in-repo fix, an advisory you route to a service owner (e.g. “this is a downstream 503, not your agent”), or no action when a guardrail fired correctly. A proposal with no concrete change is held as needs grounding and kept out of the actionable backlog. See Remediations.

Prompt optimization run — An offline search for a better system prompt for one confirmed failure mode. Neens replays that failure’s historical traces against candidate prompts (tool results are served from each recorded trace), grades every replay with your judges, and reflects on the graded results to write the next candidate — keeping the candidates that are best on at least one trace rather than only the best average. The winner has to beat the original prompt on a held-out set of traces it was never optimized against; only then does it become a prompt_change remediation, which still goes through the same eval-verified PR gate and human merge as any other fix. See Prompt optimization.

Eval gate — An evaluation built from your own failures and used as a regression guard. A gate ties a failure mode to a dataset of its real examples and a judge, and tracks the pass rate against a baseline — turning “we fixed this” into “this can’t quietly come back”. See Eval gates.

Pre-prod eval run — A whole-release gate. Before a candidate agent version ships, it replays a golden dataset against that version, scores the results with your existing judges, and compares against a baseline (a prior run or your live production window), flagging regressions. Two ways to run it: you run your agent and Neens correlates the emitted traces, or Neens calls your agent at an HTTP endpoint you register, once for every golden prompt. See Pre-prod evaluations.

Workspace & configuration

Company / Org / Agent — The workspace hierarchy. Agents were formerly called Projects — same concept, new name. Your company is the isolated tenant; it contains orgs (team- or product-level groupings), which contain agents. The agent is the working scope: traces, judges, datasets, and dashboards all live in an agent, and ingest API keys are bound to exactly one agent. Admins see everything; members see the orgs and agents they’ve been given. See the Administration section.

Persona — A lens, not a permission. A persona (Executive, Developer, Product manager, Compliance, Finance, …) changes which navigation items are emphasized, your landing page, and default time ranges — it never grants or removes access, and everything stays reachable. Admins can assign personas and author custom ones. See the Administration section.

LLM connection — Your configured model provider: Anthropic, OpenAI (or any OpenAI-compatible endpoint, including local Ollama), or AWS Bedrock, with a model and an encrypted credential. Every AI feature in Neens — judges, cluster labels, insights, topics, enrichments, remediations, the assistant — resolves the agent’s visible connection; there is no built-in model access. Connections are scoped company-wide or to specific orgs/agents, with one default per scope. Set up in Settings → LLM providers — see Getting started.

Putting it together

You send traces, Neens rolls them into sessions, scores them with judges, clusters failures into issues, and gives you remediations, eval gates, and pre-prod evaluations to fix problems and keep them fixed — with datasets and annotations keeping the loop grounded in real, human-verified data.