Alert rules
The Neens detectors find anomalies and regressions automatically, but sometimes you know exactly what you want to watch: “page me when the toxicity pass-rate drops below 90%” or “tell me when the error rate goes over 5%.” Alert rules let you say that directly — a metric, a threshold, and a window — and route the result through the same notification sinks as insights.
At a glance
| What it is | A user-defined check: watch a metric over a window and fire when it crosses a threshold |
| Key API routes | GET/POST /alerts, GET/PATCH/DELETE /alerts/{id}, GET /alerts/catalogue, POST /alerts/{id}/test |
| When it runs | Every few minutes, per agent, on the server’s schedule |
| Where it fires | The in-app Insights feed, plus any configured Slack / webhook sink — the same plumbing insights use |
| Who can set them | Any member (viewers are read-only) |
Anatomy of a rule
Every rule is four choices plus a couple of options:
| Field | Meaning |
|---|---|
| Metric | Any catalogue measure — eval pass rate, error rate, p99 latency, spend, tokens, average score, the business KPIs (containment rate, resolution time, cost per case), and any custom measure your company has defined — those are company-wide definitions, so deleting one breaks every rule that names it |
| Comparator | <, ≤, >, or ≥ |
| Threshold | The number to compare against (a fraction like 0.9 for rate metrics — the builder shows it as a percentage) |
| Window | today, 24h, 7d, or 30d — the period the metric is measured over |
| Metric key (optional) | For eval pass rate, narrow to one judge’s metric (e.g. toxicity) so you alert on that scorer alone |
| Severity | high / medium / low — high-severity alerts are the ones the Slack sink posts |
| Sinks & tags (optional) | Restrict which sinks this rule uses, and attach routing tags for the webhook sink |
| Cooldown | Minimum minutes between repeat notifications while the metric stays breached (default 60) |
The rule fires when metric <comparator> threshold. An empty or idle window (no data) is never a breach — an alert fires on real signal, not on the absence of it.
Creating a rule
Create one with POST /alerts:
curl -X POST https://your-neens/api/alerts \
-H "Authorization: Bearer $NEENS_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"name": "Toxicity pass-rate floor",
"measureKey": "eval_pass_rate",
"metricKey": "toxicity",
"comparator": "lt",
"threshold": 0.9,
"window": "24h",
"severity": "high"
}'GET /alerts/catalogue returns the alertable measures, comparators, windows, and severities the rule-builder offers, so a UI never has to hardcode them.
“Page me when containment drops below 85%” is the same shape: "measureKey": "containment_rate", "comparator": "lt", "threshold": 0.85. Containment is measured over decided cases, so Test the rule first — it shows you the current value, and whether the agent’s coverage is large enough for the threshold to mean anything. See Business KPIs.
Test before you trust it
POST /alerts/{id}/test evaluates the rule right now and reports the current value and whether it would breach — without recording anything, notifying anyone, or touching the rule’s state. Use it to sanity-check a threshold against live data.
{ "value": 0.83, "display": "83%", "breached": true, "comparator": "lt", "threshold": 0.9, "window": "24h" }Trend alerts — page me when a KPI slips
A threshold rule compares a number to a fixed line. Sometimes what you care about is the direction of travel: not “is containment below 85%?” but “has containment fallen from where it was last week?”. A KPI trend rule watches a Business KPI’s own recent history and fires when it has regressed by at least a percentage you set, over a window you set.
A trend rule references a KPI (not a raw measure), because a trend needs the KPI’s recorded daily history and its declared direction — so a drop in containment and a rise in cost per case are both “regressed”, each oriented correctly.
curl -X POST https://your-neens/api/alerts \
-H "Authorization: Bearer $NEENS_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"name": "Containment is slipping",
"ruleType": "kpi_trend",
"config": { "kpiId": "a68e0cbb059f4b0ba38f409ee2145b19",
"lookbackDays": 7, "minDeltaPct": 5 },
"severity": "high"
}'| Field | Meaning |
|---|---|
ruleType | kpi_trend (the default is threshold). A trend rule needs no measureKey/comparator/threshold — it derives them from the KPI |
config.kpiId | The KPI to watch — from GET /kpis/summary or the Business KPIs page. Must be a KPI in your agent |
config.lookbackDays | How far back the comparison point sits — 7 compares this week to last |
config.minDeltaPct | The regression that pages you, as a percentage. 5 means “a 5% or larger adverse move” — small enough to catch a real slip, large enough not to fire on noise |
A trend rule only fires on a genuine regression — never on “we don’t know”. If the KPI has too
little recorded history to draw a trend, or its
definition changed inside the window,
the trend reads unknown — and an unknown trend is not a breach. An alert that paged you because
Neens couldn’t measure something would be worse than silence.
What happens on a breach
When the periodic evaluator finds a rule breaching, it:
- Records an alert in the Insights store (type
alert, in the Observe feed) — so it appears in-app alongside detector insights, with the same lifecycle and feedback. - Dispatches it to the sinks — the in-app feed always, plus Slack (for high-severity rules) and any webhook whose tags match. This is the exact notification-sink plumbing insights use; no new configuration is needed.
- Respects the cooldown — while the metric stays breached, the rule won’t re-notify until its cooldown elapses, so a persistent problem doesn’t spam you.
When the metric recovers (stops breaching), the rule’s open alert is automatically resolved and drops out of the feed — mirroring how detector insights self-resolve.
Each rule keeps a short history (GET /alerts/{id}/events) of its breaches and recoveries, and its last evaluated value and state on the rule itself.
Alert rules reuse the exact same notification sinks as insights, so Slack and webhook delivery is configured once, centrally — see Insights › Notification sinks. There is no separate alert egress path and no new credentials.
Troubleshooting
| Symptom | Cause | Fix |
|---|---|---|
| A rule never fires | The window is idle (no data ⇒ no breach), or the threshold direction is inverted | Use Test to see the current value and confirm the comparator |
A rule on a custom:… measure never fires, and Test returns nothing | Somebody deleted the custom measure — definitions are company-wide, so an admin in another agent can remove one your rule depends on. The rule stays enabled and resolves no value | Point the rule at an existing measure, or have the measure redefined. See Before you delete a shared measure |
| No Slack message on a breach | The Slack sink only posts high-severity insights, or isn’t configured | Set the rule’s severity to high and configure the Slack sink |
A metricKey is rejected | Only eval pass rate supports narrowing to a specific scorer metric | Drop metricKey, or switch the rule’s metric to eval pass rate |