Assistant
Neens Assistant is an in-app chat that answers questions about your workspace’s data in plain language. Ask what’s happening with your agents — it looks up the traces and sessions, scores, issues, and fixes you already have and answers with real numbers, streaming the reply as it writes. It also answers how-to and “can Neens do X?” questions by searching this documentation, so its guidance stays grounded in features Neens actually has rather than guesswork.
It can also do a few things, not just describe them: curate a dataset from failing traces, record a ground-truth label, create and deploy a judge, start an eval run, or move a suggested fix along. Every one of those is shown to you as a proposal and runs only when you press Approve — see Changes it can make.
At a glance
| Where | The floating Sparkles launcher, bottom-right of every page |
| Key API routes | POST /chat/stream (streamed answers), GET/PUT /chat/settings (per-agent connection) |
| Needs | An LLM connection with tool-calling support (LLM connections) |
| Scope | Answers and changes stay within your own permissions and the currently selected agent |
| Changes | 8 write actions, each proposed for approval; nothing deletes |
Open it and ask
Click the Sparkles launcher in the bottom-right corner to open the panel. Use the header controls to Maximize it into a larger centered window (Esc restores) or Minimize it back to the launcher.
Type a question and press Enter (Shift+Enter for a new line). While it answers you can press Stop to end the turn and keep what it has written so far; if a turn fails, a Retry button re-sends your last question. An empty panel greets you with suggested questions you can click — tailored to whether an agent is selected.
As it works, the Assistant shows small activity chips (“Counting matching traces…”, “Reviewing judges and scorers…”) so you can see what it looked at on the way to an answer.
Questions it can answer
Each capability below is a tool the Assistant can call. Every tool reads live data from your agent — it never answers data questions from the model’s own memory, and if a lookup comes back empty or fails, it says so plainly. Questions about how Neens works are answered the same way: the Assistant searches this documentation and grounds its reply on what it finds, rather than inventing feature names, pages, or step-by-step flows. If the docs don’t cover something, it tells you it isn’t certain instead of guessing.
| Ask it… | What it looks at |
|---|---|
| ”Which scorers are enabled, and how are they trending?” | The score catalogue: metric, source, average, pass rate, and period-over-period delta |
| ”What failure modes exist? Any new ones?” | The latest failure-clustering run: top failure modes, session counts, NEW flags, root-cause hypotheses |
| ”Tell me more about this failure cluster” | One cluster’s label, description, root cause, suggested fix, and example sessions |
| ”What issues are open right now?” | The Issues taxonomy: name, lifecycle state, severity, session counts, last seen |
| ”How many traces failed the relevancy score this week?” | An exact trace count for any filter combination — status, agent, score, issue, enrichment value, time window |
| ”Show me examples of traces that errored” | Up to 10 matching traces with agent, status, start time, and failure mode |
| ”What agents / models / tools show up in my traces?” | The live filter values present in your agent’s traces |
| ”Which enrichments run here, and what values do they produce?” | The agent’s enrichments and their observed output fields |
| ”How is the agent doing?” | KPI snapshot: trace/session/span counts, error rate, p50/p99 latency, plus a 14-day daily volume/error series |
| ”How are our business KPIs doing?” | Your Business KPIs — each commitment’s current value, whether the target is met, and the recent trend; ask about one KPI to also see the failure clusters eroding it. A missing number stays unknown, never a fabricated zero |
| ”What judges exist? When did X last run?” | Each judge’s type, scope, latest version, and deployment status/last run |
| ”What should we fix first?” | Suggested remediations ranked by priority, with status and confidence |
| ”How is everything doing across agents?” | Org-level KPIs and a per-agent health table (works with no agent selected) |
| “Anything unusual lately?” | The Observe / Diagnose / Fix insight feeds (works with no agent selected) |
| “Which agents and orgs do I have?” | The agents and orgs your account can access |
| ”How do I set up a daily digest? Can Neens do X?” | This product documentation — feature and how-to answers are grounded on the docs, with a link, and it says so plainly when the docs don’t cover something (works with no agent selected) |
Changes it can make
The Assistant has eight write actions. It never runs one on its own initiative: it proposes the change, the panel shows you exactly what would happen, and nothing is written until you press Approve.
| Ask it… | What it does | Permission it needs |
|---|---|---|
| ”Put these failing traces in a golden dataset” | Create dataset — creates a dataset, optionally populated from the traces you were just looking at, optionally as an immutable golden version 1 | Member |
| ”Add these to the Golden Q&A set” | Add to dataset — adds traces/sessions to an existing dataset by id or name; reports how many were added vs skipped as duplicates | Member |
| ”Mark this session as a fail — it looped on refunds” | Add annotation — records a pass/fail ground-truth label with an optional critique, optionally golden, optionally tied to a failure mode | Member |
| ”Create a judge that checks refund correctness” | Create judge — creates an LLM-prompt judge and its version 1 (experimental) | Member |
| ”Turn that judge on” | Deploy judge — deploys a judge version with a manual or on_new_trace trigger. You don’t have to name a version: it deploys the latest unless you ask for a specific one | Member |
| ”Score the recent traces with it” | Run eval — starts an evaluation run for a deployed judge. Naming the judge is enough when it has one deployment | Member |
| ”Suggest a fix for this cluster” | Generate remediation — generates a grounded suggested fix for a failure cluster or failure mode | Member |
| ”Accept that fix and mark it in progress” | Update remediation — transitions a remediation’s status, work state, or labels | Member |
Nothing deletes. There is no tool that deletes a dataset, judge, label, remediation, agent, or trace — on this surface or on the MCP server. Deleting is an admin action you take yourself in the UI.
Approve is the only confirmation
The approve/decline card is how the Assistant asks. It will not ask you to type a confirmation first — no “reply Yes, deploy it and I’ll proceed”, no plan restated for your sign-off before the card appears. Once you’ve asked for a change, the next thing you see is the card, and one click on Approve runs it.
It asks a question only when a genuinely required detail is missing — what to call a new dataset, say. Optional details it leaves out are filled in for you: ask it to turn a judge on without naming a version and it deploys the latest one; ask it to score with a judge that has a single deployment and it finds that deployment itself. So a two-action request costs two clicks, not two clicks plus four typed replies.
A worked example
Ask for the change
You: Add the failing traces from the “Refund loop” failure mode to a golden dataset.
The Assistant looks things up first — you’ll see its usual activity chips (“Reviewing latest clustering results…”, “Sampling matching traces…”) — and then stops.
Read the proposal
A card appears in the conversation:
✎ The assistant wants to make a change Create dataset “Refund loop failures” and add 6 traces to it as an immutable golden version 1 [ Approve ] [ Decline ] Runs as you, with your permissions
The sentence is built from the actual arguments the Assistant chose, so it always describes the call that would run — not a generic “perform a write”. Nothing has happened yet.
Approve — or decline
Press Approve and the card turns into a running, then completed, activity chip, and the Assistant continues its answer with the result:
✓ Created Refund loop failures with 6 items (golden v1). You can open it under Datasets.
Press Decline and the card reads “Declined — nothing ran.” The Assistant is told you declined and answers around it, usually by offering an alternative. Typing a new question instead of answering also leaves the change unmade.
Worked example: create, deploy and run a judge
Standing up a judge is three changes, so it is three cards — and three clicks, with nothing to type in between.
Ask for all three at once
You: Create a judge that checks refunds quote the real amount, turn it on, and score the last week of traces with it.
Approve the judge
The Assistant proposes the first change immediately — it does not ask you to confirm the plan first:
✎ The assistant wants to make a change Create judge “Refund amount accuracy” scoped to each trace, criteria: “The reply must state the refund amount returned by the refund tool, and must not…” [ Approve ] [ Decline ] Runs as you, with your permissions
The criteria it drafted are on the card, because the rubric is the judge — you’re approving the substance, not just the name. Press Approve once. The judge and its version 1 are created.
Approve the deployment
The next card follows straight on. Notice it names no version id:
✎ The assistant wants to make a change Deploy judge “judge_a41f” at its latest version with the manual trigger [ Approve ] [ Decline ]
You never had to look a version id up in the UI, and the Assistant never had to guess one — it reads the judge’s ids straight out of its own lookup, and omitting the version means “the latest one”, resolved when you approve. Press Approve.
Approve the run
✎ The assistant wants to make a change Start an eval run for the enabled deployment of judge “judge_a41f” [ Approve ] [ Decline ]
Same story for the deployment: the judge has exactly one enabled deployment, so naming the judge is enough. Press Approve and the run starts:
✓ Created Refund amount accuracy, deployed version 1 with a manual trigger, and started an eval run over 128 traces. It’s running now — I’ll have scores under Scores when it finishes.
Three changes, three Approve clicks, nothing typed. If you’d rather stop after the judge is created, Decline the next card — each change is approved on its own, and declining one doesn’t undo the ones you already approved.
If a judge has more than one enabled deployment, “score with it” is ambiguous, and the Assistant says so and asks which one rather than picking for you. If it has none, it offers to deploy it first.
What a viewer sees
The Assistant offers the same tools to everyone, because the permission check happens where it always does — inside the Neens API, on your own credentials. If your role can’t make the change, approving it returns a plain permission error rather than a failure:
✕ The current user’s role does not permit this action on this project.
Which is the same answer the button in the UI would give you. Ask an admin if you need a broader role; see Roles and permissions.
Scope and safety guarantees
No silent changes. A change is always proposed, described in one line, and made only after you approve it — and it is made as you, on your own credentials, so it can never do something your role doesn’t already allow. Every change is attributed to you: the row it creates or updates records you as the author, and where the resource keeps an audit-trail entry (datasets and judges) that entry names you and records that the change came from the Assistant. Deletes are not available at all.
The Assistant sees exactly what you can see and does exactly what you could do, and nothing more:
- Your credentials, re-checked per call. Each tool call re-enters the regular Neens APIs in-process, forwarding only your own authorization and agent/tenant headers. Tenancy resolution, agent scoping, and permission checks run again on every tool call — read or write — exactly as if you had made the request from your browser. There is no separate data path the Assistant could use to bypass them, and approving a change is not what authorizes it: a call you aren’t permitted to make returns a permission error to the model, which reports it.
- One agent per conversation. A chip in the header shows the active scope — the agent name, or All agents when none is selected. With an agent selected, agent tools are available; without one, only the org-level tools are. Switching agents clears the conversation, so answers never mix scopes.
- Data is data, not instructions. Tool results are treated as workspace data; directives embedded in trace content are ignored by instruction.
How it works
POST /chat/stream runs an agent loop over your configured LLM: the model streams its answer,
optionally requesting tool calls; Neens executes each tool, feeds the compacted result back,
and repeats until the model answers without tools — up to 8 rounds per question, after which
it closes out honestly with what it found. Responses stream over Server-Sent Events, so text
appears as it’s generated and tool activity shows in real time.
A write breaks that loop. When the model asks for one, Neens does not run it: it emits the proposal and ends the turn. Your answer travels on the next request — the panel re-sends the conversation together with the exact call you approved, Neens re-checks that call against the tool’s own schema, runs it, and the loop resumes with the result. That means there is no pending change stored anywhere: a proposal you never answer simply never happened.
Tool results are aggressively compacted before reaching the model (lists capped, long strings truncated, verbose blobs dropped; at most 6,000 characters per result and 30,000 per question) — the Assistant is built for summaries and pointers, not for dumping a full trace into chat.
When a tool fails in a way nobody wrote a message for — a database that isn’t reachable, an
upstream that refuses the connection — the Assistant says “the tool failed unexpectedly” and
shows a short reference id instead of the underlying error. That is deliberate: the raw message
tends to carry server paths and internal hostnames, and it would end up in the model’s context and
the conversation transcript. The full detail, with the traceback and the same reference id, goes to
the server log (the neens.agent_tools logger) — quote the id when you report the problem. It is
the same contract the MCP server
uses, because both surfaces run the same tool substrate.
Only user and assistant turns are accepted from the client; the system prompt and tool
results are always server-authored. A request may carry at most 40 messages; the server keeps
the most recent 30 (up to 24,000 characters) as context.
Configuration
The Assistant uses your agent’s LLM connection to generate answers — like every LLM-powered
feature in Neens, it makes no calls of its own without one. By default it uses the agent’s
default connection; an agent admin can pin a specific one under Settings → Assistant
(the Connection field, defaulting to Agent default connection). Programmatically this
is GET/PUT /chat/settings; updating it requires the manage-LLM-connections permission, and
only connections visible to the agent can be selected.
The model must support tool calling. The Assistant answers data questions by calling tools. If the configured connection’s model rejects tools, the Assistant reports it and asks you to pick a different connection under Settings → Assistant.
No connection yet? If an agent has no usable LLM connection, the Assistant doesn’t error — it tells you and points you to Settings → LLM connections. See LLM connections to set one up.
Troubleshooting
| Symptom | Cause | Fix |
|---|---|---|
| ”No LLM connection is configured for this project” | The agent has no LLM connection, or the Assistant override points at a removed one | Add a connection in Settings → LLM connections, or reset the override in Settings → Assistant |
| ”…doesn’t support tool calling” | The connection’s model rejects the tools parameter | Pick a tool-capable model/connection in Settings → Assistant |
| ”…does not permit this action on this project” in an answer | A tool call hit your real permission boundary | Expected — the Assistant can’t see past your access, or make a change your role doesn’t allow; ask an admin if you need broader scope |
| ”the tool failed unexpectedly (ref: …)” | A tool hit an unanticipated server-side error; the detail is withheld from chat on purpose | Give an operator the ref — the full exception and traceback are in the server log under that id (neens.agent_tools) |
| The Assistant says it will change something but nothing happens | The proposal card is still waiting on you | Press Approve on the card; a change is never made without it |
| It offers to delete something | It shouldn’t — there is no delete tool | Nothing was deleted; do it yourself under the relevant page |
| Answers say data is missing that you expect | It hasn’t been ingested, scored, or clustered yet in this agent | Check the source page directly (Traces, Judges) — the Assistant only reports what exists |