External API scoring

An external API judge or enrichment does not call an LLM. Instead it makes an HTTP request to a third-party API — a toxicity, sentiment, moderation, grammar, or PII service, or your own scoring microservice — templates the trace’s evidence into the request, and maps a value out of the JSON response. A judge turns that value into a normalized 0–1 score or a category label; an enrichment turns it into typed output fields.

Reach for this whenever a signal you want already exists behind an API and doesn’t need a model prompt: a moderation score, a language detector, a readability grade, a regex-service verdict, or a scoring endpoint your own team owns.

At a glance

Create an external API judge

Worked example — sentiment (no auth)

This judge calls the public text-processing.com sentiment API — no key required — and grades the agent’s answer as a category. It is seeded, enabled, in the demo as Answer Sentiment (external API), so you can open it and click Run.

{
  "url": "https://text-processing.com/api/sentiment/",
  "method": "POST",
  "content_type": "form",
  "body_template": { "text": "{{output}}" },
  "response": { "label_path": "label" },
  "output_spec": {
    "type": "categorical",
    "labels": ["neg", "neutral", "pos"],
    "passing": ["neutral", "pos"],
    "higher_is_better": true
  },
  "target_type": "session"
}

The API replies with {"label": "pos", "probability": {...}}; label_path: "label" selects "pos", and the categorical output spec marks neutral/pos as passing.

Worked example — toxicity (with auth)

This judge calls Google’s Perspective API, which returns a 0–1 toxicity score and needs an API key. It is seeded, enabled, in the demo as Response Toxicity (Perspective), wired to the Perspective API (toxicity) HTTP API connection — add a key to that connection and it runs live.

{
  "url": "https://commentanalyzer.googleapis.com/v1alpha1/comments:analyze",
  "method": "POST",
  "content_type": "json",
  "connection_id": "conn-httpapi-perspective-…",
  "body_template": {
    "comment": { "text": "{{output}}" },
    "requestedAttributes": { "TOXICITY": {} }
  },
  "response": {
    "score_path": "attributeScores.TOXICITY.summaryScore.value",
    "invert": true
  },
  "target_type": "session"
}

score_path plucks the nested toxicity value; invert: true stores 1 − toxicity, so a high score means a clean answer.

Create an external API enrichment

An enrichment works the same way but writes typed output fields instead of a score. In place of response, an enrichment maps each output field to a response path via field_paths:

{
  "url": "https://text-processing.com/api/sentiment/",
  "method": "POST",
  "content_type": "form",
  "body_template": { "text": "{{output}}" },
  "field_paths": {
    "sentiment": "label",
    "positive_score": "probability.pos"
  }
}

Each key in field_paths is an output-field key you declared on the enrichment; its value is the response path. A field whose path resolves to null/absent is simply not written for that target (an honest miss, never a fabricated value). This example is seeded, enabled, in the demo as Response Sentiment (external API) with a sentiment enum field and a positive_score number field.

Template variables

String values in the URL, headers, and body template may embed these {{vars}}, resolved from the target trace/session at run time:

An unknown variable renders as an empty string. Non-string evidence (messages, metadata) is JSON-encoded so it drops cleanly into a body. Each variable is truncated to 50,000 characters before templating, so an enormous transcript can’t blow past an endpoint’s input limit.

Response paths

A response path selects a value out of the parsed JSON body using dots for object keys and brackets for list indices:

A missing path resolves to nothing — the judge records that one target as an honest failure and the enrichment simply omits that field, rather than inventing a value.

Authenticate with an HTTP API connection

For an endpoint that needs a credential, create an HTTP API connection under Settings → Connections and reference it from the definition by connection_id. The connection holds the encrypted credential and the auth style; the definition never contains the secret. Supported auth types (set on the connection):

How it works

• On each run Neens builds the request per target, applies the connection’s auth, and makes one HTTP call per target. follow_redirects is off, so a redirect can’t bounce the call to an internal host after the safety check.
• The response body is buffered up to 4 MiB and parsed as JSON; a non-JSON success is handed back as raw text so a path against a string still works.
• Each call is bounded to 30 seconds. A slow, oversized, blocked, or erroring endpoint fails that one target with a classified error — never a fabricated score or output.
• Judge results flow through the same persistence path as any judge, so scores appear in Scores, on the Traces page, and in dashboards. Enrichment values become filterable columns like any enrichment.

Troubleshooting

See also the Judges and Enrichments guides for the shared run, trigger, and scoring mechanics.