Metrics catalogue
Everything you can chart in Neens comes from one semantic catalogue: a registry of measures (the things you can compute — trace counts, error rate, latency percentiles, spend, eval pass rate, …) and dimensions (the columns you can slice them by — time, org, project, agent, model, …). Widgets, digest tiles, and overview stats all resolve through it, so the same number means the same thing everywhere.
At a glance
| Model | Measure × Dimension: a query is a registered measure, optionally grouped by dimensions valid for it |
| Key API routes | GET /measures/catalogue (the full menu), GET /measures/fields (your agent’s dynamic metadata dimensions) |
| Safety | Only valid measure × dimension combinations can be requested — never arbitrary SQL; filter values are always parameterized |
| Scope | Every query is constrained to the caller’s accessible agents |
| Used by | Dashboard widgets, the platform persona dashboards, the daily digest email tiles, and the Overview / Insights fleet stats |
Naming: the measure keyed sessions is labeled Traces (individual agent runs), and conversations is labeled Sessions (traces rolled up by conversation) — matching how the rest of the app names them. See Core concepts.
Measures
Twenty measures ship, organized into the seven categories the widget gallery uses (Volume · Quality · Latency · Errors · Cost · Topics · Business). The headline set:
- Volume —
sessions(Traces),conversations(Sessions),span_count(Spans) - Quality —
eval_pass_rate,avg_score,score_count,cluster_count - Latency —
avg_latency,p50_latency,p99_latency - Errors —
error_rate - Cost —
spend_usd,tokens_in,tokens_out,avg_tokens_per_trace - Topics —
topic_size - Business —
containment_rate,resolution_time_p50,resolution_time_p90,cost_per_case
Full measure reference — key, label, unit, and what each computes
| Key | Label | Unit | Grain | What it computes |
|---|---|---|---|---|
sessions | Traces | count | traces | Number of individual traces (agent runs) |
conversations | Sessions | count | traces | Distinct conversations — traces rolled up by conversation id |
span_count | Spans | count | spans | Number of spans (steps) recorded |
error_rate | Error rate | ratio | traces | Share of traces whose status is error |
avg_latency | Average latency | ms | traces | Mean end-to-end trace duration |
p50_latency | p50 latency | ms | traces | Median trace duration |
p99_latency | p99 latency | ms | traces | 99th-percentile trace duration |
spend_usd | Spend | usd | model calls (spans) | LLM cost — tokens × the model’s price from the price table, computed at query time (not a stored column). Unpriced models contribute $0 and the result is labelled partial |
tokens_in | Input tokens | count | model calls (spans) | Sum of prompt tokens sent to models |
tokens_out | Output tokens | count | model calls (spans) | Sum of completion tokens generated |
avg_tokens_per_trace | Avg tokens / trace | count | traces | Average total tokens (in + out) per trace |
eval_pass_rate | Eval pass rate | ratio | scores | Share of scores at or above their threshold |
avg_score | Average score | score (0–1) | scores | Mean score value |
score_count | Total scores | count | scores | Number of scores recorded |
cluster_count | Failure clusters | count | clusters | Number of failure clusters found |
topic_size | Topic size | count | topic assignments | Traces assigned to each topic |
containment_rate | Containment rate | ratio | cases | Share of decided cases the agent handled with no escalation — see Business KPIs |
resolution_time_p50 | Resolution time (p50) | ms | cases | Median time to resolve a case, from your measured outcomes |
resolution_time_p90 | Resolution time (p90) | ms | cases | The slow tail — the 10% of cases that take longest to resolve |
cost_per_case | Cost per case | usd | cases | LLM spend divided by the cases handled in the range |
Notes:
- Grain matters for slicing. Trace-grain measures slice by trace attributes (agent, status, your metadata); span-sourced cost measures slice by
model; score-grain measures slice by judge, metric, and — because the model and agent behind each score are recorded on the score itself — bymodel,agent, andmodel_source; case-grain measures slice by the attributes of a case’s first trace. spend_usd,tokens_in, andtokens_outshare one source and one price table, so tokens and dollars always stay mutually consistent. A model with no price contributes tokens but no dollars, and the spend result carries a partial flag naming the unpriced models — see Cost & model pricing.topic_sizehas no timestamp, so thetimedimension and time-range filters don’t apply to it.- The four Business measures are computed per case (
conversation_id, falling back to the trace id), andcontainment_rate/resolution_time_*need the agent to have declared what an escalation and a resolution mean. They render with the provenance (measured vs inferred) and the coverage behind the number — see Business KPIs.
Dimensions
A dimension only applies to measures whose grain carries it — the builder (and the API) enforce this, so you can’t ask for “spend by judge” or “error rate by metric”.
timeand the scope dimensionsorg/projectapply to (almost) everything —timebuckets by hour or day.- Trace-grain slices:
agent,status,source,conversation, plus the derived binslatency_bucketandtoken_bucket. modelhas two homes. On the span-sourced cost measures (spend_usd,tokens_in,tokens_out) it’s spend and token volume per model; those three also slice byagent, so spend by agent — which agent is burning the budget — is a first-class breakdown. On the score-grain measures (eval_pass_rate,avg_score,score_count) it’s quality per model — the model that produced the answer each score graded. See Model comparison.- Case-grain slices (the Business measures):
agent,status,source,conversation— read off the case’s first trace — plusqueueandoutcome_meta:…from the matched outcome’s metadata. - Score-grain slices:
judge,score_source,score_label,target_type,model,agent, andmodel_source(Model attribution — how the score’s model was determined), plusmetric_key(pass rate only) and the derivedscore_bucketdistribution. span_kind/span_statusslicespan_count;topicslicestopic_size.
Full dimension reference — key, label, and which measures it applies to
| Key | Label | Applies to |
|---|---|---|
time | Time | Every measure except topic_size (bucketed by hour or day) |
org | Org | All measures |
project | Project | All measures |
agent | Agent | Trace-grain: sessions, conversations, error_rate, avg_latency, p50_latency, p99_latency, avg_tokens_per_trace; the cost measures spend_usd, tokens_in, tokens_out; the score measures eval_pass_rate, avg_score, score_count; and every Business measure |
model | Model | spend_usd, tokens_in, tokens_out (the model that ran) and eval_pass_rate, avg_score, score_count (the model that produced the graded answer) |
model_source | Model attribution | eval_pass_rate, avg_score, score_count — how the score’s model was determined (preprod_run / span / session_uniform / mixed / unknown) |
status | Status | sessions, error_rate, avg_latency, p50_latency, p99_latency, and the Business measures |
source | Source | sessions, error_rate, avg_latency, p50_latency, p99_latency, and the Business measures (ingest source: otlp / openinference / raw) |
conversation | Conversation | Trace-grain measures and the Business measures |
queue | Queue | containment_rate, resolution_time_p50, resolution_time_p90 — the queue field of the matched outcome’s metadata |
outcome_meta:… | your outcome metadata key | containment_rate, resolution_time_p50, resolution_time_p90 — any other key in the outcome’s metadata |
latency_bucket | Latency range | Trace-grain measures (derived bins of duration; group-by only) |
token_bucket | Token range | Trace-grain measures (derived bins of in + out tokens; group-by only) |
metric_key | Metric | eval_pass_rate |
judge | Judge | eval_pass_rate, avg_score, score_count |
score_source | Score source | eval_pass_rate, avg_score, score_count |
score_label | Score label | eval_pass_rate, avg_score, score_count |
target_type | Target type | eval_pass_rate, avg_score, score_count (session vs span scores) |
score_bucket | Score range | score_count (derived bins of the score value; group-by only) |
span_kind | Span kind | span_count |
span_status | Span status | span_count |
topic | Topic | topic_size (grouped by topic name) |
meta:… | your metadata key | Trace-grain measures — see below |
Derived-bin dimensions (latency_bucket, token_bucket, score_bucket) can only be used as group-bys, not as filters. Every other dimension works as both a group-by and a filter.
Dynamic metadata dimensions (meta: keys)
Any top-level scalar key you send in a trace’s metadata becomes a sliceable dimension of the form meta: + the key — for example, traces tagged with a region field can be grouped or filtered by meta:region on any trace-grain measure. There’s nothing to register:
GET /measures/fieldsdiscovers the metadata keys present in your accessible traces (optionally for one agent with?projectId=), and the widget builder offers them automatically.- Keys must look like identifiers (letters, digits,
_,.,-; up to 64 characters) — anything else is rejected. - Metadata dimensions apply to trace-grain measures only (the metadata lives on the trace).
Where measures are used
| Surface | How |
|---|---|
| Custom dashboards | Every widget is a measure + dimensions + filters; the builder’s menu is this catalogue (GET /measures/catalogue), including ~45 ready-made presets across the seven categories |
| Platform persona dashboards | The Executive digest, Product quality board, Finance cost explorer, and Org scorecard are built entirely from catalogue measures — nothing bespoke |
| Digest email | The fleet-health tiles (Traces, Error rate, Eval pass rate, Spend) resolve through the catalogue, so the email can never drift from the in-app numbers |
| Overview & Insights | The workspace Overview KPIs/trends/segments and the Insights fleet line resolve through the same layer |
How it works
A request names a measure, dimensions, filters, and a time range. The catalogue validates the combination and emits a neutral query description; a resolver turns that into SQL for whichever backend stores your traces, always constrained to the agents you can access. Two guarantees fall out of this design:
- No invention: an unknown measure, an inapplicable dimension, or a filter on a derived bucket is rejected with a clear error — nothing is silently approximated.
- No injection: filter values are bound as parameters, and metadata keys are strictly validated, so no widget or API call can smuggle SQL.
Related
- Dashboards — build widgets from these measures.
- Business KPIs — the case-grain measures (containment, resolution time, cost per case), and how to declare what they mean in your data.
- Custom measures — add measures of your own to this catalogue; they behave like the ones above everywhere.
- Cost & model pricing — where the prices behind
spend_usdcome from, and how to set your own. - Model comparison — slicing the quality measures by model and by agent × model, and what the Mixed / Unknown buckets mean.
- Insights — detectors that watch some of these signals for you.
- Core concepts — traces vs sessions, scores, and thresholds.