Classify failures with a decision model

A classifier judge gives each trace exactly one label. By default the labels are the failure modes in your agent’s taxonomy, read fresh on every run. You can also supply your own inline label set. A classifier can run on an ordinary chat model or on a decision model, such as typesafe/jev-1.13 on OpenRouter. A decision model returns a choice, a confidence and a probability for every label, quickly and cheaply.

At a glance

Which engine to use

Auto is the default engine. It picks the engine from the connection each run uses: a model whose id starts with typesafe/ runs as a decision model, and any other model runs as a chat model. Choose LLM (chat model) or Decision model only when you want to fix the engine regardless of the connection.

Set it up

Example: create one over the API

The same judge, created with POST /api/judges:

curl -X POST "$NEENS_BASE_URL/api/judges" \
  -H "Authorization: Bearer $NEENS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Issue class (Jev)",
    "type": "classifier",
    "scope": "trace",
    "level": "project",
    "definition": {
      "method": "classification",
      "target_type": "trace",
      "taxonomy_source": "failure_modes",
      "metric_key": "issue_class_jev",
      "engine": "decisions",
      "required_data": ["input", "output", "messages", "metadata"]
    }
  }'

To let the connection’s model choose the engine (the Auto setting), leave engine out of the definition: a connection whose model id starts with typesafe/ then runs it as a decision model.

For your own labels, set "taxonomy_source": "inline" and add a complete label set:

"labels": [
  { "name": "refund_issued", "definition": "The agent confirmed a refund was issued to the customer." },
  { "name": "refund_denied", "definition": "The agent told the customer the refund request was declined." },
  { "name": "escalated", "definition": "The agent handed the conversation to a human agent." }
]

The request is rejected with a 422 and a message saying what to fix when an inline classifier has no labels, engine is present but isn’t llm or decisions, the metric key is issue_class, the metric key is already used by another judge in the agent, or the metric key isn’t a valid slug.

To try a definition before saving it, send it to POST /api/judges/test-run with "type": "classifier" and an optional connection_id. Without a connection_id, the test uses the agent’s default connection.

How it works

• The taxonomy is read at run time. With Project failure modes (live), each run reads your agent’s current failure modes. When you add, rename or reword a failure mode, the next run uses the change, with no need to edit the judge.
• Definitions become the decision criteria. Each failure mode’s (or inline label’s) name and definition is what the model chooses between. The definitions shape the labels more than any other setting.
• One label per trace. The classifier writes a single categorical score per trace under its metric key, alongside the confidence. With a decision model it also keeps the per-label probabilities.
• No match. With the live taxonomy, a healthy trace is labelled none. With inline labels, a trace that fits no label is skipped and gets no score.

Tips

• Compare it with the built-in classifier before you rely on it. Run your decision-model classifier on the same traces as the built-in Issue Classification judge and compare the two metric keys on the Scores page. Look into the traces where they disagree before you switch over.
• Write concrete, behavioural definitions. “The agent called a search tool for a question that needed the orders tool” classifies far better than “tool problems”. Describe what can be observed in the trace, not the cause you suspect.
• Keep the labels distinct. Two failure modes with overlapping definitions split the probability between them and lower the confidence on both. Merge or sharpen them in your taxonomy.
• Pin the model version on the connection (see the warning above), and create a new connection when you deliberately move to a newer model.

Troubleshooting