Evaluate — API reference

Judges, scores, enrichments, datasets, human annotations & review, and pre-prod eval runs.

judges

POST /eval-runs

Create Eval Run

GET /eval-runs/export

Export Eval Run Scores

Export a judge run’s (or several runs’) scores WITH each target’s input + output.

GET /eval-runs/{run_id}

Get Eval Run

POST /eval-runs/{run_id}/cancel

Cancel Eval Run

POST /eval-targets/estimate

Estimate Targets

GET /judges

List Judges

POST /judges

Create Judge

POST /judges/preview

Preview Judge Prompt

Render the EXACT prompt a judge would send to the LLM, hydrated on a real sample session.

GET /judges/run-status

List Run Status

Poll-friendly, status-ONLY projection of every enabled deployment.

POST /judges/test-run

Test Run Judge Prompt

Execute the in-progress judge prompt against a real sample session ON THE FLY and return the model’s verdict — WITHOUT persisting anything.

PATCH /judges/{judge_id}

Patch Judge

DELETE /judges/{judge_id}

Delete Judge

GET /judges/{judge_id}/compare

Compare Judge Runs

Compare two eval runs of the SAME judge — typically two different versions.

POST /judges/{judge_id}/deployments

Create Deployment

PATCH /judges/{judge_id}/deployments/{deployment_id}

Patch Deployment

DELETE /judges/{judge_id}/deployments/{deployment_id}

Delete Deployment

PATCH /judges/{judge_id}/lifecycle

Patch Judge Lifecycle

Promote/demote a judge through its lifecycle: experimental → finalized → archived.

GET /judges/{judge_id}/runs

List Judge Runs

All eval runs for a judge (current + historical), in the Activity-run shape so the UI can reuse the same row layout. Joins eval_runs → judge_deployments and left-joins activity_runs (act_eval_{run_id}) for name / duration / errors.

GET /judges/{judge_id}/targets

List Judge Targets

Per-binding aggregate for the scorer’s “results by target” table: one row per enabled deployment with its Run Target summary and the latest scored/avg/passRate over that target’s population. Project-scoped; NULL-not-zero on an empty population. A legacy inline binding (no Run Target) still appears, with runTarget: null.

POST /judges/{judge_id}/versions

Create Version

scores

GET /scores

List Scores

GET /scores/display-settings

Get Display Settings

Resolved score-display config for the current project (scale + green/yellow/red bands + per-metric overrides). Readable by anyone who can read the project — the UI needs it to render every score. Falls back to platform defaults when nothing is persisted.

PATCH /scores/display-settings

Patch Display Settings

Update the project’s score-display config. Admin-only (MANAGE_SCORE_DISPLAY) — it changes what every viewer sees. Presentation only: it never alters stored scores or grading. Values are clamped to safe ranges and bands kept non-inverted (warn <= pass).

GET /scores/distribution

Distribution

The label breakdown for one score type — the drill-down’s answer to /scores/timeseries.

GET /scores/metrics

List Metrics

PUT /scores/prefs

Upsert Pref

GET /scores/timeseries

Timeseries

enrichments

GET /enrichment-catalog

Enrichment Catalog

GET /enrichment-fields

Enrichment Fields

The enrichment output fields that actually have values in this project, with the distinct values observed (capped). Drives the Traces/Sessions enrichment filter and the “Create dataset from enrichment” picker. Only fields with at least one output are returned, so the UI never offers a filter that can’t match anything.

POST /user-enrichment-runs

Create Run

GET /user-enrichment-runs/{run_id}

Get Run

GET /user-enrichments

List Enrichments

POST /user-enrichments

Create Enrichment

PATCH /user-enrichments/{enrichment_id}

Update Enrichment

Edit an enrichment. Metadata (name / output fields / prerequisites / status) mutates in place; a changed definition (prompt or LLM connection) creates a new immutable version and promotes it (version history stays append-only, matching POST.../versions). Disable = status='draft' (also drops the on-ingest trigger back to manual); re-enable = status='enabled'.

DELETE /user-enrichments/{enrichment_id}

Delete Enrichment

Hard-delete an enrichment and everything it owns — versions, runs, computed outputs, and its Activity-feed reflections — project-scoped so a scoped caller can’t delete another project’s enrichment.

POST /user-enrichments/{enrichment_id}/dataset

Create Dataset From Enrichment

The enrichment→dataset→judge bridge: materialize a dataset whose membership is every session whose enrichment output field == value. The dataset carries an enrichment_filter source config, so (when streaming) hourly sync keeps the cohort fresh as new traces are enriched — then any judge eval-run can target it via dataset_id.

POST /user-enrichments/{enrichment_id}/deployments

Create Deployment

POST /user-enrichments/{enrichment_id}/versions

Create Version

datasets

GET /datasets

List Datasets

POST /datasets

Create Dataset

POST /datasets/from-selection

Create From Selection

One-call “create dataset from selected rows”: create a manual dataset, bulk-add the selected sessions/traces, and — when golden — snapshot an immutable v1 marked golden. Returns the dataset dict with a versions summary (newest first).

POST /datasets/preview

Preview Dataset

Mirror /eval-targets/estimate for dataset sources: count eligible + sampled sessions for a source config, without persisting anything. Project-scoped.

PATCH /datasets/{id_or_name}

Update Dataset

DELETE /datasets/{id_or_name}

Delete Dataset

GET /datasets/{id_or_name}/export

Export Dataset

GET /datasets/{id_or_name}/items

List Items

List a dataset’s items with optional text search (input/output/expected), kind + captured-field-presence filters, and whitelisted server-side sort. All params are additive: with none supplied the historical ORDER BY created_at ASC is preserved.

POST /datasets/{id_or_name}/items

Add Item

DELETE /datasets/{id_or_name}/items

Delete Item

POST /datasets/{id_or_name}/items/bulk

Bulk Add Items

Bulk-add sessions/traces to an existing dataset, de-duped against what’s already in it. Sessions are field-captured (respecting the dataset’s fields unless the body overrides fields); traces are added as reference rows. Returns {added, skipped}. Project-scoped.

PATCH /datasets/{id_or_name}/items/{item_id}

Update Item

POST /datasets/{id_or_name}/sync

Sync Dataset

Re-run a filter/cluster dataset’s stored source config, adding newly-matching sessions not already present. Respects sampling and (for streaming) the last_synced_at high-water mark. Returns {added, total}. 404 unless the dataset is a filter/cluster source. Project-scoped.

GET /datasets/{id_or_name}/versions

List Versions

POST /datasets/{id_or_name}/versions

Create Version

Snapshot the dataset’s current items into a new immutable version (version = max+1). Body {name?, notes?, golden?}. When golden is set, sibling versions are un-goldened (one golden per dataset). Returns the version row.

PATCH /datasets/{id_or_name}/versions/{version}

Update Version

Mutate version METADATA only — {name?, notes?, golden?}. Snapshot items are immutable. Setting golden:true un-goldens sibling versions (one golden per dataset).

GET /datasets/{id_or_name}/versions/{version}/export

Export Version

Export an immutable version’s snapshot (json|csv), same shape as the dataset export (plus a version summary in the JSON body).

GET /datasets/{id_or_name}/versions/{version}/items

List Version Items

List an immutable version’s snapshot items — paged/searchable/sortable exactly like the live /items endpoint. Read-only.

annotations

GET /annotations

List Annotations

POST /annotations

Create Annotation

GET /annotations/reviewers

List Reviewers

GET /annotations/trend

Annotations Trend

Annotation-throughput trend for the Review Queue page: per-day (or per-hour) count of annotations created over the selected window, plus a period-over-period total. Scoped to the current project — the review loop’s output volume over time.

review

POST /review

Create Review

Write a ground-truth label. Reviewer + role resolved from the request identity.

GET /review-queue

Review Queue

Active-learning-ranked sessions a human should review next.

POST /review/adjudicate

Adjudicate

Principal override: supersede every conflicting active label on a target and write the authoritative one.

GET /review/alignment

List Alignment

Persisted judge↔expert alignment rows (the trend), newest first. Project-scoped.

GET /review/alignment/disagreements

Alignment Disagreements

Sessions where this judge disagreed with expert ground truth, split FN/FP. Project-scoped.

POST /review/alignment/refresh

Refresh Alignment

Recompute + persist judge↔expert alignment for every judge that has scores.

GET /review/gold

List Gold

List the active gold-standard labels (is_gold = 1), newest first. Project-scoped.

GET /review/label-options

Label Options

Grouped options a reviewer can tag a verdict with: confirmed taxonomy failure modes and discovered clusters (each carrying its finite L1 bucket/bucketLabel + session count).

GET /review/labels

List Labels

Unified human-label history — the source the Annotations tab renders.

GET /review/leaderboard

Leaderboard

Per-reviewer QUALITY = agreement with the expert/consensus on gold items, not volume.

GET /review/queue-config

Get Queue Config

Current review-queue config for the project (defaults synthesized when unset).

PUT /review/queue-config

Put Queue Config

Upsert the project’s review-queue config (gated require_action('review')).

GET /review/reviewers

List Label Reviewers

Distinct reviewers that have written a label in this project. Project-scoped.

preprod-evals

GET /preprod-compare

Compare Runs

Aligned N-run comparison over a shared golden version.

GET /preprod-compare/trajectory

Trajectory Diff

Diff the candidate’s captured trajectory against the baseline run’s captured trajectory for the SAME golden prompt.

GET /preprod-evals

List Preprod Evals

List pre-prod eval runs (project-scoped) with progress counts.

POST /preprod-evals

Create Preprod Eval

Create a pre-prod eval run + snapshot the golden version’s items into pending item rows.

GET /preprod-evals/trend

Preprod Trend

Run-cadence and pass-rate trend for the Pre-prod Evals page. Primary series: per-day (or per-hour) count of runs created over the window. Secondary: the pass-rate across scored items in the same window. A window with no scored items reports a null pass-rate, never a fabricated 0%. Scoped to the current project.

POST /preprod-evals/verify

Verify Fix

Verification webhook — PROVE a fix against a candidate endpoint/branch.

GET /preprod-evals/{run_id}

Get Preprod Eval

Run detail: the run header + per-item summary + an aggregate rollup.

POST /preprod-evals/{run_id}/cancel

Cancel Preprod Eval

Cancel a run (terminal). A completed run cannot be cancelled.

GET /preprod-evals/{run_id}/comparison

Comparison

Candidate-vs-baseline comparison.

GET /preprod-evals/{run_id}/gate

Gate

Evaluate the run’s gate_json against the candidate results → pass/fail + reasons.

GET /preprod-evals/{run_id}/items

Get Preprod Items

The frozen prompt list for the harness/runner: [{item_id, input, expected_output}].

GET /preprod-evals/{run_id}/metrics

Metrics

Per-metric score rollup for a run’s captured+scored sessions — the CI-gate surface.

POST /preprod-evals/{run_id}/run

Run Preprod Eval Route

Trigger a PUSH run: Neens HTTP-calls the run’s agent endpoint for every frozen golden prompt, captures + links each response, then scores it — no user harness.

POST /preprod-evals/{run_id}/start

Start Preprod Eval

Transition awaiting_traces → running (idempotent).