Code judges (custom Python)

A code judge is a judge whose verdict comes from a Python function you write, not from an LLM. Use one when the check is a rule rather than a matter of taste: a word budget, a forbidden pattern, a JSON schema, a fuzzy match against the reference answer. It is fast, cheap, and gives the same answer every time it sees the same trace.

At a glance

What your code can and cannot do

Your code runs in an isolated sandbox, built so that code from one company can never reach another company’s data, Neens itself, or the internet. Read this before you write a judge that needs anything beyond the trace in front of it.

More limitations to plan for:

• State between traces. Traces are graded in batches of up to 25. Your module is loaded fresh for each batch, so a module-level variable can carry over between traces in the same batch, but never between batches. Don’t count anything across traces: each trace’s result must depend only on that trace.
• Output type must match what you return. A judge left on the default Score 0–1 scale must return {"score": …}. Code that returns {"label": "pass"} fails every trace with Invalid return value until you set Output type to Binary pass/fail (or Categorical labels).
• Capacity. Code runs share sandbox capacity. When it is busy, traces wait; a trace that waits too long is skipped with Execution capacity is busy, never failed or given a made-up score. Run the judge again later to grade it.
• Daily budget. Your company’s code runs share a daily compute budget.
• Plan. Code judges are part of the Gold plan.

Need data from an API?

Keep the call outside the sandbox and the logic inside it:

• The service returns the verdict itself. Use an External API judge (Judge types). Neens calls your endpoint for each trace and maps the response to a score.
• You need the service’s answer and some Python on top. Make the External API judge a component of a composite judge whose Method is Custom Python. The API is called outside the sandbox, and your combine() reasons over its score next to the other components.
• The question is a judgment call about text. Use ctx.llm, which asks the agent’s own LLM connection on your code’s behalf.

This is what happens if code tries the network directly:

import requests
 
 
def judge(target):
    # Fails on every trace: there is no network in the sandbox.
    r = requests.post("https://api.example.com/check", json={"text": target.get("output", "")})
    return {"score": r.json()["score"]}

Every trace fails with Your code raised an error (ConnectionError). The same check, split so the call happens outside the sandbox:

# combine.py on a composite judge whose components are:
#   policy_check: an External API judge that calls https://api.example.com/check
#   reply_length_budget: a code judge
def combine(components, ctx):
    policy = components.get("policy_check")       # the API's verdict, normalized to 0-1
    length = components.get("reply_length_budget")
    if policy is None:
        ctx.skip("no_policy_score")                # the API judge has not scored this trace yet
    if policy < 0.5:
        return {"score": 0.0, "reason": "the policy service flagged this reply"}
    return {"score": min(policy, length if length is not None else 1.0)}

Write your first code judge

BUDGET = 120  # words
 
 
def judge(target, ctx):
    reply = (target.get("output") or "").strip()
    if not reply:
        ctx.skip("no_reply")          # nothing to grade: skipped, not failed
    words = len(reply.split())
    if words <= BUDGET:
        return {"score": 1.0, "reason": f"{words} words"}
    over = words - BUDGET
    return {"score": max(0.0, 1 - over / BUDGET), "reason": f"{over} words over budget"}

The authoring contract

The function

def judge(target):          # the simplest form
    ...
 
def judge(target, ctx):     # with the context object
    ...

The function must be a top-level def judge taking one or two positional arguments. Imports, constants and helper functions at module level are fine. Each trace is graded by a fresh call; do not rely on state carried between calls (see State between traces).

target: the evidence

target is a dict holding the evidence keys you ticked. It is the same evidence an LLM judge gets, so a code judge and an LLM judge grade the same thing.

Read optional keys with target.get(...): a key the trace does not carry is left out, not set to None. Long text values are cut to a fixed length before your code sees them.

ctx: the context object

What to return

Always return a dict. What goes in it depends on the judge’s output scale:

"reason" is optional on every scale and is shown next to the score. A pass/fail judge also accepts True/False as the label.

Results are checked strictly. A score outside the scale, a label that is not on the list, NaN or infinity, or a value that is not a dict fails that trace with Invalid return value. Nothing is clamped or guessed, so a bug in your code never shows up as a plausible score.

Skipping a trace

Call ctx.skip("reason") when there is nothing to grade, for example a trace with no reply. The trace is marked skipped with the reason “The code judge chose to skip this target”, and it counts neither as a pass nor as a failure. That is the difference from returning 0: a skip does not drag the average down.

More examples

import re
 
STACK = re.compile(r"Traceback \(most recent call last\)")
PLACEHOLDER = re.compile(r"\{\{\s*\w+\s*\}\}")
 
 
def judge(target):
    reply = target.get("output") or ""
    if STACK.search(reply):
        return {"label": "fail", "reason": "the reply shows a stack trace"}
    if PLACEHOLDER.search(reply):
        return {"label": "fail", "reason": "unfilled template placeholder"}
    return {"label": "pass"}

def judge(target):
    n = len((target.get("output") or "").split())
    label = "short" if n < 10 else "long" if n > 300 else "ok"
    return {"label": label, "reason": f"{n} words"}

from rapidfuzz import fuzz
 
 
def judge(target, ctx):
    expected = target.get("expected_output")
    if not expected:
        ctx.skip("no_reference")
    similarity = fuzz.token_set_ratio(target.get("output") or "", expected) / 100
    return {"score": similarity, "reason": f"{similarity:.0%} similar to the reference"}

def judge(target):
    tools = [m["name"] for m in target.get("messages", []) if m.get("kind") == "tool"]
    if "lookup_order" in tools:
        return {"score": 1.0, "reason": "looked the order up"}
    return {"score": 0.0, "reason": f"answered without lookup_order (tools: {tools or 'none'})"}

Packages and runtimes

Your code runs on a runtime: a fixed Python version with a reviewed set of packages, such as numpy, pandas, scipy, scikit-learn, regex, rapidfuzz, jsonschema, pydantic and textstat. Nothing is installed when your code runs, and there is no network access.

• Runtime. Pick one in the form; the list also shows every package and its exact version. Leaving it on the current default pins that runtime when you save. GET /sandbox/runtimes returns the same list.
• requirements.txt. Optional. List what you import, with versions if you care (numpy>=2, pandas~=3.1). Each line must be met by the runtime. The form checks every line as you type and names a line it cannot meet, suggesting the closest package name; saving checks again. pip options (-r, --index-url) and direct URLs are not supported.
• The standard library is always available.
• Need another package? Build a custom environment.

Custom environments

When the runtime does not include a package you need, build a custom environment: your own set of packages, added on top of a runtime. It is built once, offline, and then every code judge, combine() and code enrichment of the agent (or of the company, for a shared one) can use it.

Build one

1. Open Settings → Code environments and choose New environment.

2. Name it, choose who can use it (This agent, or Shared with the company, which needs a company admin), and paste or upload a requirements.txt:

   textstat==0.7.4
   rapidfuzz>=3

3. Choose Build environment. The build runs in the background: the list shows Queued, then Building, then Ready or Failed. Open the environment to see its build log, its scan report and the locked list of every package it installs, each with the sha256 of its file.

Then pick the environment under Custom environment in the code form, above the runtime. Picking it selects the runtime it was built on. The form checks your requirements.txt against the runtime plus the environment, and a test run uses it, so you can try it before you save.

import textstat  # from the environment
 
def judge(target, ctx):
    grade = textstat.flesch_kincaid_grade(target.get("output") or "")
    return {"score": 1.0 if grade <= 8 else 0.0, "reason": f"grade {grade:.1f}"}

The API is POST /sandbox/environments (name, requirements, scope), then GET /sandbox/environments/{id} until status is ready. Save a version with "environment_id": "<id>" in its definition.

The rules

• Vetted packages only. Packages come only from the package index your Neens deployment vets, and only as binary wheels. A package published only as source is refused (no setup.py ever runs). pip options (-r, --index-url, -e) and direct URLs are refused.
• The runtime wins. A package the runtime already includes must match its version. Anything that needs a different version fails the build as a conflict with the runtime.
• Hashes. Every installed file is recorded with its sha256. You may pin hashes yourself with --hash=sha256:…; if you pin one line, pin every line.
• Scanned. A package is rejected if it ships a startup hook (.pth file), a native program, an installed script, a module named like the standard library, or a command named like one.
• Caps. Up to 50 requirements and 256 MB per environment, up to 10 environments and 20 builds a day per company. Your administrator may set different numbers; the Code environments page shows the current ones.
• Pinned builds. A version you save keeps the exact build it was saved with. Rebuild replays the recorded lock, so it produces the same build unless the index now serves different files, which fails the rebuild. To pick up new package versions, create a new environment and save a new version on it.
• Retiring. A retired environment cannot be picked for new versions. Runs of versions saved on it fail with Runtime retired until you save a new version on another environment. Retiring cannot be undone.
• Isolation. Environments belong to your company. Another company can never see, use or run one of yours, and a project environment is not visible from your other agents.
• Plan. Custom environments come with code judges (the Gold plan). If the page says they are not available, your deployment has no vetted package index; ask your administrator.

When a build fails

A failed environment shows the reason, the build log and, for a rejected package, the scan report. Fix the requirements.txt and create the environment again, or choose Rebuild when the cause was temporary.

A build that is still Queued or Building long after it started can be rebuilt; the earlier build’s result is then ignored.

To use the environment from a code enrichment, pick it the same way in the enrichment’s code form, or send "environment_id" in the enrichment definition.

Limits

Set the time and memory limits in the form; leave them blank for the defaults. A limit applies to each trace, so one slow trace never fails the others.

Daily compute budget

Your company also has a daily compute budget: the CPU time that code judges, code composites and code enrichments may use together in one UTC day. By default, Gold plans include 3,600 CPU-seconds a day. Test runs count toward it too. Each batch of traces is charged its CPU time or its running time, whichever is larger, so code that waits (for example, sleeps) still uses budget.

When the budget runs out, the traces that have not run yet are skipped, not failed, with the reason daily code-execution budget used up. The run itself still completes, and the budget resets at 00:00 UTC. A test run started after that point does not execute and says so. Run the scorer again the next day to grade the skipped traces.

Usage (admins) shows how much of today’s budget is used and when it resets.

Test run

Test run executes the code in the form, unsaved, on up to three traces you choose, and shows for each one:

• the verdict (score or label, pass/fail and reason) exactly as a real run would store it,
• everything the code printed (print and ctx.log), with anything that looks like a secret masked,
• the error class, message and line number when something went wrong,
• the time taken and peak memory.

A test run is capped at 10 seconds in total, persists nothing, and is limited to 20 runs per minute per person. It needs permission to configure judges; an API key cannot start one.

Printed output appears only in test runs. Real runs never store what your code prints, so do not use printing to record results: return them.

Running a code judge

Once saved, a code judge runs everywhere a judge runs: Run now, Run Targets, and continuous evaluation. Runs are split into batches that execute in parallel, so a large run finishes in roughly the time of its slowest batch. You can cancel a run at any time; traces not yet graded are marked cancelled.

Each graded trace becomes a normal score. Its details record which runtime ran it and a fingerprint of the exact code, so you can tell which version of your judge produced a score.

When some traces fail, the run says how many and why, for example 2 of 40 targets failed — user_exception: ZeroDivisionError: division by zero (line 7). Open the run to see the error on each failed trace.

Code judges across Neens

A code judge scores like any other judge, so you can use it wherever Neens asks for one.

• Pre-prod runs. Pick a code judge among the judges that score a pre-prod run. It grades each replayed trace in the sandbox, and its verdicts count toward the run’s pass rate and regressions exactly like an LLM judge’s.
• Eval gates. A gate that includes a code judge settles on its verdicts, the same way it does for an LLM judge.
• Fix verification. A code judge can prove a fix: the candidate build is replayed against the failure’s golden set and the code grades every trace.
• Scorer comparisons. A code judge has no scorer LLM to vary, so you pick no connections. It appears as a single Judge code column, scored by the judge’s own code. Its cost shows as —, not applicable: there is no model spend to report, and it is never shown as $0. A code judge that calls ctx.llm is the exception: its cells record the model and tokens of those calls.
• Continuous evaluation. A code judge enabled on ingest scores new traces as they arrive. It collects new traces for a few seconds and scores them in one batch, so a verdict can appear a few seconds after the trace does.

Composite judges with combine()

A composite judge normally merges its component scores with a weighted sum or an expression. Pick Custom Python as the Method to merge them with a Python function instead, for example to fail a reply outright when one component is too low.

WEIGHTS = {"helpfulness": 0.6, "correctness": 0.4}
 
 
def combine(components, ctx):
    safety = components.get("safety")
    if safety is not None and safety < 0.5:
        return {"score": 0.0, "reason": f"safety {safety:.2f} is below 0.5"}
    scored = {k: v for k, v in components.items() if k in WEIGHTS and v is not None}
    if not scored:
        ctx.skip("no_quality_scores")
    total = sum(WEIGHTS[k] for k in scored)
    return {"score": sum(WEIGHTS[k] * v for k, v in scored.items()) / total}

How combine() works:

• components is a dict keyed by each component judge’s metric key. Each value is that judge’s score on the trace, normalized to 0–1 whatever its native scale, or None when the component has no score for the trace.
• Return {"score": …} between 0 and 1. The composite’s Threshold decides pass or fail, as for the other methods.
• "reason" is optional. Neens always writes its own explanation of the score from the components; your reason is added beneath it as an author note, never in its place.
• The minimum number of components and the Missing policy still apply before your code runs: a trace that does not have enough component scores is skipped or failed by that policy, and combine() is not called for it.
• combine() reads the components’ existing scores. It never runs the component judges itself, so score the components first.
• ctx.skip, the limits, the runtimes and the errors are the same as for a code judge.

Code enrichments

An enrichment can also be a Python function. Use one to extract facts by rule rather than with an LLM: whether a reply contains an email address, how many order numbers it mentions, which product line a trace belongs to. Code enrichments need a plan that includes Python enrichments (Gold).

Write enrich.py

Click New enrichment, pick Custom Python as the Mode, and declare the Output fields your code fills in. For this example, declare has_pii with value type boolean and pii_count with value type number:

import re
 
EMAIL = re.compile(r"[\w.+-]+@[\w-]+\.[\w.]+")
PHONE = re.compile(r"\+?\d[\d -]{7,}\d")
 
 
def enrich(target, ctx):
    text = target.get("output") or ""
    if not text:
        ctx.skip("no_output")
    found = EMAIL.findall(text) + PHONE.findall(text)
    return {"has_pii": bool(found), "pii_count": len(found)}

enrich(target) or enrich(target, ctx) returns a dict whose keys are your output field keys. target always carries input, output, messages and metadata, with the same shapes as a code judge’s evidence; ctx is the same object as a code judge’s. Each returned value must match its field’s value type:

A very long string is cut to a fixed length. A field set to None counts as not returned.

Another example: route traces by topic

Declare one output field, topic, with value type enum and the values billing, shipping, returns and other. Then filter traces by topic, or build a dataset from one topic:

KEYWORDS = {
    "billing": ("invoice", "refund", "charge", "payment"),
    "shipping": ("delivery", "tracking", "courier", "shipped"),
    "returns": ("return", "exchange", "rma"),
}
 
 
def enrich(target, ctx):
    text = (target.get("input") or "").lower()
    for topic, words in KEYWORDS.items():
        if any(w in text for w in words):
            return {"topic": topic}
    return {"topic": "other"}

What is checked

Every value is checked against its output field before anything is stored:

Test it

The form’s right-hand panel has a Test run button. It runs the unsaved code on a recent trace, and after the first run you can tick up to three traces to run on. For each trace you see the value of every output field (a field your code left out shows not returned), anything it printed, the time taken and peak memory, and, when something went wrong, the error class and message with the failing line highlighted in the editor. Nothing is saved. Save the enrichment, then run it like any other enrichment, on demand or on every new trace.

Run it

Save the enrichment, then run it like any other enrichment: Run now over a filter, a dataset or recent traces, or turn on Run continuously on ingest to enrich new traces as they arrive (see Run it). The run lists how many traces were enriched, skipped and failed. When traces fail, the run names the most common error and its first message, for example 3 of 120 sessions failed — invalid_result: bad_return: …; open the run to see each failed trace. The values your code returned show up as filterable attributes on Traces and Sessions, exactly like an LLM enrichment’s.

Calling an LLM from code (ctx.llm)

Some checks are mostly rules with one judgment call in the middle: parse the reply, check the JSON, and only then ask whether the tone is right. ctx.llm lets a code judge, a combine() or a code enrichment make that call without leaving Python.

Turn on Allow LLM calls (ctx.llm) under the runtime settings in the form, then call it:

import json
 
def judge(target, ctx):
    reply = target.get("output") or ""
    if not reply:
        ctx.skip("no_reply")
    try:
        order = json.loads(reply)                     # the rule part: plain Python
    except ValueError:
        return {"score": 0, "reason": "reply is not JSON"}
    verdict = ctx.llm(                                # the judgment part: one LLM call
        "Answer yes or no. Is this apology sincere?\n" + order.get("message", ""),
        max_tokens=5,
    )
    return {"score": 1 if verdict.strip().lower().startswith("yes") else 0,
            "reason": "tone check: " + verdict.strip()}

It uses this agent’s LLM connection. Calls go through the LLM connection configured for the agent in Settings → LLM connections, the same one its LLM judges use, and the agent’s PII redaction policy is applied to the prompt before it is sent. Your API key never reaches your code or the sandbox: Neens makes the call on the code’s behalf and hands back only the reply. If the agent has no LLM connection, each trace that calls ctx.llm fails with Capability not available.

The call

ctx.llm(prompt, *, system=None, model=None, max_tokens=512, temperature=0.0, json=False)
ctx.llm(messages=[{"role": "user", "content": "..."}], system="...")

• Pass either a prompt string or a messages list (roles user and assistant, ending with user); put a system prompt in system.
• model overrides the connection’s default model for this call.
• json=True asks for JSON and returns the parsed value (a dict or list) instead of text. A reply that is not valid JSON raises LLMError.
• ctx.llm_batch([...]) sends up to 8 prompts in one round trip and returns the replies in order. Each item is a prompt string or a dict with the same keys as ctx.llm. A failed item comes back as an LLMError in its place instead of stopping the others.

The call blocks until the reply arrives. Time spent waiting for the LLM does not count toward Time per trace; each call has its own 30-second limit.

Limits

Errors

The three exceptions are available in your code without an import, and also as ctx.LLMError and so on. Catch them if you want a fallback; uncaught, the trace fails.

Saving a judge whose code calls ctx.llm while Allow LLM calls is off is refused, naming the line.

Cost and test runs

Each score produced with ctx.llm records the model and tokens it used, so the judge’s LLM cost shows up in Cost and quality like an LLM judge’s. A code judge that never calls ctx.llm records none. Test run makes real calls and shows, for each trace, how many calls it made, which model answered, the tokens in and out, and the cost (a dash when the model has no price).

Turning on ctx.llm makes a judge’s verdict depend on an LLM, so it is no longer guaranteed to give the same answer twice. Keep temperature=0.0 and short, closed questions.

Errors and what to do

Permissions and scope

Anyone who can configure judges can author a code judge for the agents they work on. Org-level and all-agents code judges need a company admin, because the code then runs on every agent in that scope. Code judges never read another agent’s traces: a run only ever sees the traces of the agent it runs in.

Troubleshooting

• The output is always empty. Check that Final output is ticked under Evidence to attach, and look at the evidence panel: it shows exactly what target holds.
• The score is right in a test run but the run fails. Test runs use the code in the form; a run uses the saved version. Save a new version after editing.
• ModuleNotFoundError. The package is not in the runtime. Open the runtime’s package list in the form; packages cannot be installed.
• Every trace fails with bad_return: expected {"score": number}. The code returns a label, but the judge is on the default Score 0–1 output type. Set Output type to Binary pass/fail (or Categorical labels) and test again.
• ConnectionError, URLError or a DNS error. The code tried to reach the network, which the sandbox does not have. See Need data from an API?.
• The score changes between runs. Code that uses randomness or the wall clock is not deterministic. Use ctx.now instead of datetime.now(), and seed any random generator.