Agent versions

When more than one build of the same agent sends traces to one Neens agent — a canary next to the current release, a rollout that is still draining, a staging build pointed at production data — you want to see each build separately. Neens reads an agent version off every trace and lets you group the Traces and Sessions charts, dashboard widgets and failure modes by it, so you can answer “is this failure only in v1.3?” or “how far has v2.4 rolled out?” without guessing.

At a glance

Tag your traces with a version

Use any string that names a build: a semantic version (v2.4.0), a release name, or a short commit SHA. Neens takes the version for a trace from the first of these that is present:

1. a span attribute neens.version_label (the first span in the trace that carries one),
2. the OTLP resource attribute neens.version_label,
3. the OTLP resource attribute service.version (the standard OpenTelemetry one, so an agent that already sets it is versioned with no change).

An empty value counts as no version. The resource fallbacks apply to OTLP (POST /v1/traces); on POST /ingest/openinference and POST /ingest/raw, put neens.version_label on a span.

import os
 
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
 
resource = Resource.create({
    "service.name": "support-agent",
    # The build that is running — e.g. injected by your deploy pipeline.
    "neens.version_label": os.environ.get("AGENT_VERSION", "v2.4.0"),
})
provider = TracerProvider(resource=resource)
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(
    endpoint="https://<your-neens-host>/v1/traces",
    headers={"Authorization": "Bearer nk_live_your_key_here"},
)))
trace.set_tracer_provider(provider)

with tracer.start_as_current_span("agent.handle_request") as span:
    span.set_attribute("neens.version_label", variant.version)
    ...

import { NodeSDK } from '@opentelemetry/sdk-node'
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http'
import { resourceFromAttributes } from '@opentelemetry/resources'
 
const sdk = new NodeSDK({
  resource: resourceFromAttributes({
    'service.name': 'support-agent',
    // The build that is running — e.g. injected by your deploy pipeline.
    'neens.version_label': process.env.AGENT_VERSION ?? 'v2.4.0',
  }),
  traceExporter: new OTLPTraceExporter({
    url: 'https://<your-neens-host>/v1/traces',
    headers: { Authorization: 'Bearer nk_live_your_key_here' },
  }),
})
sdk.start()

tracer.startActiveSpan('agent.handleRequest', (span) => {
  span.setAttribute('neens.version_label', variant.version)
  // ...
  span.end()
})

export OTEL_RESOURCE_ATTRIBUTES="neens.version_label=v2.4.0"
# or, if your agent already reports its version the standard way:
export OTEL_RESOURCE_ATTRIBUTES="service.version=2.4.0"

{
  "session": { "id": "trace-001", "agent_name": "support-agent" },
  "spans": [
    {
      "id": "span-1", "name": "agent.handle_request", "kind": "agent",
      "attributes": { "neens.version_label": "v2.4.0" }
    }
  ]
}

Break the Traces chart down by version

A rollout reads directly off the chart: the new version’s band grows while the old one shrinks, and anything still on the old build after the cutover is easy to spot.

On Sessions, a session made of several traces is counted once. If its traces carry different versions, it is counted under the lowest version label; it is Unversioned only when none of its traces carries a version.

Through the API:

curl -H "Authorization: Bearer nk_live_your_key_here" \
  "https://<your-neens-host>/api/sessions/trend?range=30d&group_by=version"

{
  "group_by": "version",
  "series": [
    { "key": "v2.4.0", "other": false, "total": 752, "counts": { "2026-09-28": 33, "...": 0 } },
    { "key": "v2.3.0", "other": false, "total": 62,  "counts": { "2026-09-28": 3 } },
    { "key": null,     "other": false, "total": 47,  "counts": { "2026-09-28": 2 } }
  ]
}

A null key is the unversioned group. Add version=v2.3.0 to scope the chart and its tiles to one version, or version=__unversioned__ for the traces without one.

Add a version breakdown to a dashboard

In a dashboard, click Add widget, open the Custom tab, pick a trace-grain measure such as Traces, Error rate, or p50 latency, and set Dimension to Agent version. A bar, donut, or table works best. Traces without a version show as Unversioned.

curl -X POST -H "Authorization: Bearer nk_live_your_key_here" \
  -H "Content-Type: application/json" \
  "https://<your-neens-host>/api/dashboards/<dashboard-id>/widgets" \
  -d '{"type": "bar", "measure": "error_rate", "dimensions": ["version"], "range": "7d"}'

version also works as a widget filter (for example "filters": {"version": ["v2.4.0"]}) and on the per-case business measures, where a case takes the version of its first trace. It does not apply to the cost, token, span, or score measures. The builder doesn’t offer those combinations, and the API rejects them with 422. See the full matrix in the metrics catalogue.

See which versions a failure mode hits

Open a failure mode from Failure Modes. The Agent versions card on its Overview tab lists, for each version:

Compare hit rates, not shares. A version that serves most of the traffic also owns most of every failure. For example, if v2.3.0 hits a failure mode at 26% and v2.4.0, serving four times the traffic over the same weeks, hits it at 0%, the failure is specific to v2.3.0 (here, v2.4.0 fixed it). A version with traffic in that period but no traces in the failure mode is listed with a 0% hit rate, not left out. Pre-prod replays are not counted.

The same breakdown is version_distribution on GET /clusters/{id}, and the get_failure_mode MCP tool returns it as versionDistribution. list_traces accepts a version filter.

Troubleshooting

Related

• Traces & sessions — the list, filters and the trend chart.
• Failure modes & clustering — how failure modes are formed.
• Cost & Quality — cost and quality per agent version.
• Pre-prod evaluations — test a candidate version before it ships.