Watch mode: run your evals on every save

neens eval watch brings the pre-prod evaluation loop into your editor. It watches your agent’s source files, and every time you save it replays your golden dataset against your local agent, gates the result, and prints what changed since the previous run: which items newly fail (▼), which newly pass (▲), which still fail, the gate verdict, and how the headline numbers moved. You find a regression while you are still editing the prompt, not in CI.

At a glance

Install

Watch mode ships in both neens-eval SDKs from version 0.4.0, with the same flags and exit codes.

pip install "neens-eval>=0.4.0"   # installs the `neens` console script

npm install --save-dev "neens-eval@>=0.4.0"   # installs the `neens-eval` bin

Start watching

export NEENS_BASE_URL="https://neens.example.com"   # your Neens origin
export NEENS_API_KEY="nk_live_..."                   # your agent API key

export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="$NEENS_BASE_URL/v1/traces"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer $NEENS_API_KEY"

curl -s -H "Authorization: Bearer $NEENS_API_KEY" "$NEENS_BASE_URL/api/datasets"

neens eval watch --dataset ds_golden --version-label feat-refunds \
  --include "*.py" --include "prompts/*" \
  -- python -m myagent

npx neens-eval watch --dataset ds_golden --version-label feat-refunds \
  --include "*.ts" --include "prompts/*" \
  -- node dist/myagent.js

Reading the output

Every iteration prints a header, the normal run log, and then the diff against the previous iteration that produced results:

── iteration 3 · feat-refunds-watch.20261003T091530.3 ────
  changed: src/agent.py
  created run ppr_7c1e09a4b2d3f5e6
Running 12 item(s) with concurrency=1 (version='feat-refunds-watch.20261003T091530.3')
  …
  gate: FAILED (was PASSED at #2) · exit 1
  pass rate 83.3% (-8.3 pts) · avg score 0.812 (-0.041) · passed 10 (-1) · failed 2 (+1) · errored 0 (+0) · regressions 1 (+1)
  vs iteration #2:
    ▼ newly failing: 1
        ▼ dvi_4f1a9c  pass → fail  "What's the refund window for opened items?"
    ▲ newly passing: 0
    ✗ still failing: 1
        ✗ dvi_9c2e07  fail → fail  "Can I return a gift without a receipt?"
    = unchanged passing: 10

Items are matched by their golden-item id, so the diff is item by item, not just a pass-rate delta. An item that errored or was never scored counts as not passing. It is never counted as a pass, and it is never silently dropped from the diff.

What triggers a run

The watcher polls the watched files for changes. It needs no file-system notification support, so it behaves the same on macOS, Linux and in containers.

These directories are always skipped, at any depth: .git, .hg, .svn, node_modules, __pycache__, .venv, venv, dist, build, .mypy_cache, .pytest_cache, .ruff_cache, .tox, .next. A tree with more than 5,000 watched files prints a warning. The scan stops at 50,000 files, so narrow a large repository with --path or --include.

Runs never overlap. If you save while an iteration is running, the watcher finishes that iteration first and then runs exactly one follow-up, however many saves you made meanwhile.

Files your agent writes don’t loop. If your agent writes into the watched tree on every run (a log file, a local database, a trace file), the watcher notices that the same file changed during two runs in a row, prints a warning, and stops re-running for that file’s changes made during a run. A save you make to it while the watcher is idle still starts a run. To be explicit, exclude it: --exclude "*.log".

Version labels

Each iteration is stamped with its own version label, <label>-watch.<session>.<n>, for example feat-refunds-watch.20261003T091530.3. <label> is --version-label (default local), <session> is the UTC time the watcher started, and <n> is the iteration number. Every run on the Pre-prod Evals page, and every trace it produced, can be traced back to the save that caused it.

Use it in scripts and git hooks: --once

--once runs a single iteration, without watching anything, and exits with the gate code. That is the same contract as neens eval run, so it works in a pre-commit or pre-push hook:

# .pre-commit-config.yaml
repos:
  - repo: local
    hooks:
      - id: neens-eval
        name: Neens pre-prod gate
        entry: neens eval watch --once --dataset ds_golden --version-label pre-commit -- python -m myagent
        language: system
        pass_filenames: false
        stages: [pre-push]

Install the hook type with pre-commit install --hook-type pre-push. Without pre-commit, a plain .git/hooks/pre-push (or a husky pre-push) works the same way:

#!/bin/sh
npx neens-eval watch --once --dataset ds_golden --version-label pre-push -- node dist/myagent.js

A hook runs on every push, and every push spends judge tokens. Most teams put it on pre-push rather than pre-commit.

Machine-readable output: --json

With --json, the watcher writes one JSON object per iteration on its own line to stdout. Human logs still go to stderr. Each object carries iteration, runId, versionLabel, exitCode, gatePassed, policyBlocked, error, trigger (the changed files), metrics, comparedTo, deltas, and diff (newlyFailing, newlyPassing, stillFailing, unchangedPassing, onlyInCurrent, onlyInPrevious). diff is null on the first iteration.

Stopping, errors and exit codes

Troubleshooting