Datasets
A dataset is a curated collection of trace/session examples — the failures you want to fix, the golden interactions you want to protect, the slice of traffic you want a judge to score. Datasets can fill themselves from a filter or a cluster, keep syncing as new traffic matches, and be frozen into immutable versions that anchor pre-prod evaluations.
At a glance
| Where | Datasets in the sidebar; New dataset to create |
| Sources | manual, filter (Traces-style filters), cluster (a discovered failure cluster) |
| Items | Sessions, traces, or imported JSON rows; captured fields: Input, Output, All raw data |
| Sync | On demand (Sync now), or hourly and automatic for streaming datasets |
| Versions | Immutable snapshots; one can be marked golden per dataset |
Create a dataset
Pick a source
In New dataset, choose where items come from:
- Manual — start empty (optionally upload entries as CSV or JSON), then add sessions or traces by hand or in bulk. You can also create a manual dataset directly from a selection of sessions in one step.
- From filter — define a filter with the same vocabulary as the Traces page (status, agent, tags, scores, topics, even enrichment field values). Every matching session becomes an item.
- From cluster — pick a discovered failure cluster; its member sessions become the items.
Scope and sample
For filter/cluster sources:
- Sampling — All matches (default), a fixed Count, or a Percentage.
- Add entries from — All history, Only new (from now), or Since date (a backfill lower bound on session start time).
- Fields to capture — which parts of each session are stored on the item: Input, Output (both by default), and/or All raw data.
A live preview shows how many sessions match and how many will be sampled before you create anything.
Keep it fresh (optional)
Turn on Keep syncing new matches (streaming) and Neens re-runs the source every hour on the hour, adding newly matching sessions (never duplicating existing items, and never removing any). Non-streaming filter/cluster datasets can still be synced on demand with Sync now.
Work with items
Inside a dataset you can search items, filter by kind and captured fields, and sort. Each item carries its captured input, output, and an optional expected output:
- Add individual traces/sessions, bulk-add a selection, or import JSON rows.
- Edit an item’s input and expected output — the expected output is what correctness-style judges and pre-prod evaluations compare against. The captured output is read-only (it’s what actually happened).
The captured output is the agent’s derived final answer — the last assistant turn of the trace’s reconstructed transcript, not “the last span that produced output”. That keeps a trailing guardrail verdict or tool result from being frozen into a golden set, but it also means a trace whose Conversation tab is empty or wrong captures an empty or wrong output, with no error. Check the Conversation tab on a few traces before you cut a golden version — see Conversation transcript.
- Export the live dataset — or any version — as JSON or CSV.
Golden datasets and versions
The live item list of a filter/cluster dataset is a moving target — syncs keep adding to it. When you need a fixed reference set, cut a version:
- A version snapshots the current items into a frozen list with an auto-incrementing number (v1, v2, …), an optional name and notes, and the item count at snapshot time.
- Version items are immutable. Only the metadata (name, notes, golden flag) can change later; the frozen items never do.
- Mark a version as golden to designate it the reference set — at most one version per dataset is golden at a time (marking a new one automatically unmarks the previous).
Golden versions are what pre-prod evaluations replay: a pre-prod run takes a dataset and a version (defaulting to the golden one if you don’t pick), freezes those items as its work list, and scores the candidate agent against them. Because the version is immutable, every candidate is measured against exactly the same examples.
Judges target the live dataset; pre-prod runs target a version. When you point a judge’s eval run at a dataset, it scores the dataset’s current members — useful for continuously scoring a curated slice. Pre-prod evaluations pin a frozen version so comparisons stay apples-to-apples.
The review → annotation → golden loop
Datasets are the middle of the Neens improvement loop:
- Find failures — clusters, low scores, and issues surface what’s going wrong (see Traces & sessions).
- Label them — humans confirm pass/fail with rationale in the Review queue, building ground truth.
- Curate — build a dataset from the confirmed failure cluster or a filter capturing the pattern, fill in expected outputs, and cut a golden version.
- Guard — judges score the live dataset continuously, and every candidate release replays the golden version in a pre-prod evaluation before it ships.
Curation is a deliberate human step: labels tell you what belongs in the golden set, but you choose what to include — there’s no automatic promotion from label to dataset item.
How it works
Creating a filter/cluster dataset materializes it immediately: the source is evaluated once and matching sessions are captured as items. Sync re-evaluates the same source and adds only new matches. Streaming datasets use a high-water mark — each hourly sync picks up from where the last one ended, so a busy agent isn’t rescanned from scratch every hour. Deleting a dataset removes it and its items; sessions and traces themselves are never touched.
API reference
| Route | Purpose |
|---|---|
GET /datasets / POST /datasets | List / create (source_kind ∈ manual|filter|cluster, sampling, fields, streaming, historical_from). |
POST /datasets/preview | Preview match + sample counts without creating. |
PATCH /datasets/{id} / DELETE /datasets/{id} | Rename/retag / delete. |
POST /datasets/{id}/sync | Re-run the source; returns {added, total}. |
GET /datasets/{id}/items | List/search items; POST …/items, POST …/items/bulk, PATCH …/items/{item_id}, DELETE …/items manage them. |
POST /datasets/from-selection | Create a manual dataset from selected sessions in one call (optionally snapshotting golden v1). |
POST /datasets/{id}/versions | Cut an immutable version ({name?, notes?, golden?}). |
GET /datasets/{id}/versions / GET …/versions/{version}/items | List versions / a version’s frozen items. |
PATCH /datasets/{id}/versions/{version} | Edit version metadata only (name, notes, golden flag). |
GET /datasets/{id}/export, GET …/versions/{version}/export | Export as ?format=json or csv. |
Datasets are addressed by id or by name in these routes; there is no single-dataset detail endpoint — the list plus the items/versions routes cover it.
Troubleshooting
| Symptom | Cause → fix |
|---|---|
| Sync now isn’t available | The dataset is manual — only filter/cluster datasets have a source to re-run. |
| Streaming dataset isn’t growing | Nothing new matches the source since the last sync (check the filter), or traffic stopped. Syncs run hourly on the hour. |
| Pre-prod run can’t find a version | The dataset has no golden version and none was specified — cut a version and mark it golden. |
| Items look stale after editing the filter | The stored source config is fixed at creation; create a new dataset for a different filter. |
See also the FAQ.