GuidesAlert rules

Alert rules

The Neens detectors find anomalies and regressions automatically, but sometimes you know exactly what you want to watch: “page me when the toxicity pass-rate drops below 90%” or “tell me when the error rate goes over 5%.” Alert rules let you say that directly — a metric, a threshold, and a window — and route the result through the same notification sinks as insights.

At a glance

What it isA user-defined check: watch a metric over a window and fire when it crosses a threshold
Key API routesGET/POST /alerts, GET/PATCH/DELETE /alerts/{id}, GET /alerts/catalogue, POST /alerts/{id}/test
When it runsEvery few minutes, per agent, on the server’s schedule
Where it firesThe in-app Insights feed, plus any configured Slack / webhook sink — the same plumbing insights use
Who can set themAny member (viewers are read-only)

Anatomy of a rule

Every rule is four choices plus a couple of options:

FieldMeaning
MetricAny catalogue measure — eval pass rate, error rate, p99 latency, spend, tokens, average score, the business KPIs (containment rate, resolution time, cost per case), and any custom measure your company has defined — those are company-wide definitions, so deleting one breaks every rule that names it
Comparator<, ≤, >, or ≥
ThresholdThe number to compare against (a fraction like 0.9 for rate metrics — the builder shows it as a percentage)
Windowtoday, 24h, 7d, or 30d — the period the metric is measured over
Metric key (optional)For eval pass rate, narrow to one judge’s metric (e.g. toxicity) so you alert on that scorer alone
Severityhigh / medium / low — high-severity alerts are the ones the Slack sink posts
Sinks & tags (optional)Restrict which sinks this rule uses, and attach routing tags for the webhook sink
CooldownMinimum minutes between repeat notifications while the metric stays breached (default 60)

The rule fires when metric <comparator> threshold. An empty or idle window (no data) is never a breach — an alert fires on real signal, not on the absence of it.

Creating a rule

Create one with POST /alerts:

curl -X POST https://your-neens/api/alerts \
  -H "Authorization: Bearer $NEENS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Toxicity pass-rate floor",
    "measureKey": "eval_pass_rate",
    "metricKey": "toxicity",
    "comparator": "lt",
    "threshold": 0.9,
    "window": "24h",
    "severity": "high"
  }'

GET /alerts/catalogue returns the alertable measures, comparators, windows, and severities the rule-builder offers, so a UI never has to hardcode them.

“Page me when containment drops below 85%” is the same shape: "measureKey": "containment_rate", "comparator": "lt", "threshold": 0.85. Containment is measured over decided cases, so Test the rule first — it shows you the current value, and whether the agent’s coverage is large enough for the threshold to mean anything. See Business KPIs.

Test before you trust it

POST /alerts/{id}/test evaluates the rule right now and reports the current value and whether it would breach — without recording anything, notifying anyone, or touching the rule’s state. Use it to sanity-check a threshold against live data.

{ "value": 0.83, "display": "83%", "breached": true, "comparator": "lt", "threshold": 0.9, "window": "24h" }

Trend alerts — page me when a KPI slips

A threshold rule compares a number to a fixed line. Sometimes what you care about is the direction of travel: not “is containment below 85%?” but “has containment fallen from where it was last week?”. A KPI trend rule watches a Business KPI’s own recent history and fires when it has regressed by at least a percentage you set, over a window you set.

A trend rule references a KPI (not a raw measure), because a trend needs the KPI’s recorded daily history and its declared direction — so a drop in containment and a rise in cost per case are both “regressed”, each oriented correctly.

curl -X POST https://your-neens/api/alerts \
  -H "Authorization: Bearer $NEENS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Containment is slipping",
    "ruleType": "kpi_trend",
    "config": { "kpiId": "a68e0cbb059f4b0ba38f409ee2145b19",
                "lookbackDays": 7, "minDeltaPct": 5 },
    "severity": "high"
  }'
FieldMeaning
ruleTypekpi_trend (the default is threshold). A trend rule needs no measureKey/comparator/threshold — it derives them from the KPI
config.kpiIdThe KPI to watch — from GET /kpis/summary or the Business KPIs page. Must be a KPI in your agent
config.lookbackDaysHow far back the comparison point sits — 7 compares this week to last
config.minDeltaPctThe regression that pages you, as a percentage. 5 means “a 5% or larger adverse move” — small enough to catch a real slip, large enough not to fire on noise

A trend rule only fires on a genuine regression — never on “we don’t know”. If the KPI has too little recorded history to draw a trend, or its definition changed inside the window, the trend reads unknown — and an unknown trend is not a breach. An alert that paged you because Neens couldn’t measure something would be worse than silence.

What happens on a breach

When the periodic evaluator finds a rule breaching, it:

  1. Records an alert in the Insights store (type alert, in the Observe feed) — so it appears in-app alongside detector insights, with the same lifecycle and feedback.
  2. Dispatches it to the sinks — the in-app feed always, plus Slack (for high-severity rules) and any webhook whose tags match. This is the exact notification-sink plumbing insights use; no new configuration is needed.
  3. Respects the cooldown — while the metric stays breached, the rule won’t re-notify until its cooldown elapses, so a persistent problem doesn’t spam you.

When the metric recovers (stops breaching), the rule’s open alert is automatically resolved and drops out of the feed — mirroring how detector insights self-resolve.

Each rule keeps a short history (GET /alerts/{id}/events) of its breaches and recoveries, and its last evaluated value and state on the rule itself.

Alert rules reuse the exact same notification sinks as insights, so Slack and webhook delivery is configured once, centrally — see Insights › Notification sinks. There is no separate alert egress path and no new credentials.

Troubleshooting

SymptomCauseFix
A rule never firesThe window is idle (no data ⇒ no breach), or the threshold direction is invertedUse Test to see the current value and confirm the comparator
A rule on a custom:… measure never fires, and Test returns nothingSomebody deleted the custom measure — definitions are company-wide, so an admin in another agent can remove one your rule depends on. The rule stays enabled and resolves no valuePoint the rule at an existing measure, or have the measure redefined. See Before you delete a shared measure
No Slack message on a breachThe Slack sink only posts high-severity insights, or isn’t configuredSet the rule’s severity to high and configure the Slack sink
A metricKey is rejectedOnly eval pass rate supports narrowing to a specific scorer metricDrop metricKey, or switch the rule’s metric to eval pass rate