Agent Observability

Every answer your agent produces is the result of a sequence of steps: retrieving schema, looking up golden examples, generating SQL, executing it, building a chart, and writing the narrative. Agent Observability gives you the full trace of that process — plus the automated evaluations that graded the result — so you can see exactly why an answer came out the way it did.

​

Reaching a trace

From a live chat — after the agent responds, click Observability → beneath the answer. The trace opens in the right-hand panel and follows the conversation as it continues. From Chat History — open Chat History, click a row, and switch the sidebar to the Observability tab. This works for every conversation, including deployed end-user chats and MCP calls. The tab is hidden for conversations that made no tool calls, because there is nothing to trace.

​

What the trace shows

​

The summary bar

At the top of the trace, a row of chips summarises the whole conversation:

• steps — total number of tool calls the agent made.
• duration — total time spent inside those calls, in milliseconds.
• rag — how many golden queries and golden charts were retrieved as references.
• evals — how many automated evaluations ran. The chip is green when all of them passed and amber when any did not.
• errors — present only when at least one step failed.

Expand and Collapse open or close every step at once.

​

Per-exchange sections

Below the summary, the trace is broken into one section per user question, each with up to four blocks: User Query — the question as the user asked it. Agent Pipeline — the ordered list of steps the agent took. Every step shows its own duration and success/error state, and expands to reveal its inputs and outputs. The steps you’ll see include: Tools you connected through MCP appear here too, under their own names. RAG Sources — wherever a step used golden assets, the trace lists them as cards showing the retrieved question, its SQL, and a preview of its chart, each with the similarity score that earned it a place. When a trace shows the agent leaning on a bad example, the card names the Golden Asset to go and fix. Agent Response — the narrative the user actually read. Evaluations — the automated grades for this exchange. See below.

​

Evaluations

Upsolve runs continuous, LLM-based evaluations against production traces. They run automatically in the background — you don’t trigger them, and they don’t slow the agent’s answer down — and their verdicts flow back into the Evals column in Chat History and into the Evaluations block of the trace.

​

Reading a verdict

Each evaluation resolves to one of three states:

• Pass — the evaluation’s criterion was met.
• Fail — the criterion was not met, or the evaluation itself errored.
• Pending — the evaluation hasn’t returned a verdict yet. Evaluations run asynchronously, so a very recent conversation will often show pending for a short while.

The row-level verdict in Chat History is the worst state present: any failure makes the row fail, otherwise any pass makes it pass, otherwise it’s pending. Hover the pill for the per-evaluation breakdown. Expand an evaluation in the trace to see:

• Score — the numeric grade the evaluator returned, when it produced one.
• Description — what this evaluation checks.
• Annotation — the evaluator’s written reasoning for the verdict it gave. This is the part worth reading: it usually names the specific thing that went wrong.

​

What gets evaluated

Evaluations fall into two broad families. The exact set is configured per deployment, so talk to your Upsolve contact to add one or tune an existing one for your agent. Answer quality — did the agent actually answer the question, and is the answer right? Answer Correctness Judge is the workhorse here: it grades the response against the question that was asked and the data the agent retrieved to answer it. Conversation health — how the exchange is going across turns, rather than within a single answer. Multi-Turn Forgetfulness Detector catches the agent losing context established earlier in the conversation — a filter the user set three turns ago that silently stopped being applied. User Frustration Detector catches the user re-asking, correcting, or pushing back, which is the earliest reliable signal that something is wrong even when no step errored and no thumbs down was given.

​

Using traces to improve your agent

Observability earns its keep when an answer is wrong. The common patterns: Wrong SQL, right-sounding answer — the agent generated plausible SQL that doesn’t implement your business logic. Open SQL Generation and read the query. The fix is usually a more precise System Prompt rule, or a Golden Query demonstrating the correct approach. No RAG sources retrieved for a common question — the agent answered from scratch instead of referencing a validated example. Check whether a Golden Query covers this pattern, and whether the phrasing is semantically close enough to trigger retrieval. The wrong golden asset was retrieved — the trace shows a RAG source that doesn’t fit the question. Note which asset the card names, then sharpen its question text in Golden Assets, or retire it. Correct SQL, wrong chart — the data was right but the visualization misled. Add a Golden Chart for this question type. Errors in the pipeline — a red step names the failure directly. A failed SQL execution usually points at a data model that promises a column the warehouse doesn’t have; a failed validation usually points at generated SQL the dialect rejects. High cost or many steps — the trace shows repeated SQL attempts before a correct result. That normally means the data model annotations or system prompt need more specificity, so the agent stops guessing. A thumbs down with no obvious error — read the Annotation on the failing evaluation, then the Agent Response block. Framing problems (over-claiming, missing caveats, too much jargon) don’t surface as errors anywhere else.

​

Next steps