> ## Documentation Index > Fetch the complete documentation index at: https://docs.upsolve.ai/llms.txt > Use this file to discover all available pages before exploring further. # Tests > Write the questions your agent must answer correctly, run them as a batch whenever the agent changes, and block versions that break them. > [!NOTE] > > Tests are a **Beta** feature. Every change to a workspace (a new skill, a reworded prompt, a column description) can fix one answer and quietly break another. Tests catch that. A **test case** is a question your agent must answer correctly, plus the SQL that answers it. Run all your test cases against the draft before you save a version, and an **evaluator** grades every answer against your reference so you see exactly what passed, what failed and why. Turn on **CI** and Upsolve runs the tests for you every time someone saves a version, and won't let a failing version reach production. ![The Tests page with Test cases, Run history and Evaluator tabs, showing eight test cases with their run counts and latest result](https://mintcdn.com/upsolve/6hKReQ_w6B1jt5Gu/images/ai-agent-builder/improve/tests-list.png?fit=max\&auto=format\&n=6hKReQ_w6B1jt5Gu\&q=85\&s=440f777b958a527530c185e5c969e33e) ## Where tests live Open your workspace and click **Tests** in the **Test & Ship** section of the sidebar. The page has three tabs: * **Test cases**: your questions and their reference SQL. * **Run history**: every run, and every case's result in it. * **Evaluator**: how answers are graded. ## Anatomy of a test case Click **Add test case** to open the editor. A test case has: | Field | What it is | | - | - | | **Question** (required) | The question, phrased the way your users would ask it. This is what the agent receives. | | **SQL query** (required) | The reference SQL: a query whose result is the correct answer. It must run successfully before the case can be saved. Use the **Preview as** picker to run it as a particular user while you write it. | | **Chart** (optional) | A reference chart. Add one when the right answer is a particular visualization; it makes the case a chart case. | | **Use as example question** | Shows this question as a quick-question button in the chat. | | **Use as golden asset** | The agent retrieves this question, its SQL and its chart as a worked example when it answers similar questions. See [Golden Assets](https://docs.upsolve.ai/ai-agent-builder/golden-assets.md). | The two switches change how the agent behaves for your users. They don't change how the case is run or graded. ![The Edit test case panel with Question, SQL query and optional Chart fields, and switches for Use as example question and Use as golden asset](https://mintcdn.com/upsolve/6hKReQ_w6B1jt5Gu/images/ai-agent-builder/improve/tests-case-form.png?fit=max\&auto=format\&n=6hKReQ_w6B1jt5Gu\&q=85\&s=8c0bdffb45c3798584329f3785d70d4b) Test cases belong to the workspace **draft**, like the rest of its context. Adding or editing one is saved with **Add test case** or **Save to draft**, and becomes part of the next version you save. ### Ways to create a test case * **From scratch**: click **Add test case** on the Tests page. * **From a real conversation**: in [Chat History](https://docs.upsolve.ai/ai-agent-builder/chat-history.md) or the Playground, open a trace and click **Save as test case**. The question, the last SQL that succeeded and any chart are filled in for you, so turning a good answer into a test takes a few seconds. This is the fastest way to build a suite that reflects what your users actually ask. * **With the Builder agent**: ask the [Builder agent](https://docs.upsolve.ai/ai-agent-builder/builder-agent.md) to "add a test case for…". It writes the question and reference SQL and asks you to approve them. * **Through the API or CLI**, for teams that keep their test suite in code. Click any case in the list to see its question and reference SQL, run its SQL (**Get data**), **Edit** it, or check its **Recent test runs**. ![A test case's Recent test runs tab, listing each run with its result, how long ago it ran, its duration and the version it ran against](https://mintcdn.com/upsolve/6hKReQ_w6B1jt5Gu/images/ai-agent-builder/improve/tests-case-recent-runs.png?fit=max\&auto=format\&n=6hKReQ_w6B1jt5Gu\&q=85\&s=a7b4e3ddb82e2bbea0f8acd93a1dd67e) > [!TIP] > > Write reference SQL that reads from the same tables and views you want the agent to use. The evaluator compares the agent's answer to what your SQL returns, so a reference that takes a shortcut the agent can't see will fail answers that are actually right. ## Run your tests The green button on the **Test cases** tab runs a batch. Its label tells you what it will run: * **Run all**: every test case. * **Run N filtered**: the cases matching the status filter, for example only the ones that failed last time. * **Run N selected**: the cases you've ticked. The ▶ on a row runs that one case. While a run is going, the button becomes **Stop**. Tests run against the version shown in the header: normally the **Draft**, so you are checking the changes you're about to save. Pick a saved version there to test it instead. A single run replays at most **50 cases**. With more than that, the button reads "Run 50 of N". ### What happens in a run For each case, Upsolve: 1. Asks the agent the question, exactly as a user would, and records the SQL it ran, the chart it drew and what it said. 2. Runs your reference SQL. 3. Hands both to the **evaluator**, which grades the agent's answer against your reference and writes a report. A run you start by hand answers as no particular user, so row-level security doesn't filter it. The same run also checks the draft's row-level security rules. ## Read the results The **Run history** tab lists every run: when it started, what started it (**Manual**, or **Save vN** for a run triggered by saving a version), the agent version it ran against, how many cases passed, how long it took and its pass rate. Switch to **Cases** to see one row per case result instead, with its cost. ![The Run history tab in Runs view, listing runs with their start time, trigger, agent version and how many test cases passed or failed](https://mintcdn.com/upsolve/6hKReQ_w6B1jt5Gu/images/ai-agent-builder/improve/tests-run-history.png?fit=max\&auto=format\&n=6hKReQ_w6B1jt5Gu\&q=85\&s=59abcb109c7f1b4d5844d1fdff1519c0) Expand a run to see its cases, and click a case to see how it was graded. The evaluator writes one section per criterion, saying whether it was met and why: ![A failed test case result showing the evaluator's report with sections for 'Answers the question', 'Consistent with the reference' and 'Complete', each marked met or not met with an explanation](https://mintcdn.com/upsolve/6hKReQ_w6B1jt5Gu/images/ai-agent-builder/improve/tests-result-report.png?fit=max\&auto=format\&n=6hKReQ_w6B1jt5Gu\&q=85\&s=82deb1ef8bee3cec2e7ce8abbcb26fc7) Below the report, **Expected** (the reference query and chart) sits side by side with **Actual** (the query the agent ran, the chart it produced and what it said), so you can see exactly where they differ. ![Expected versus actual: the test case's reference query on the left, and the query the agent ran and the table it produced on the right](https://mintcdn.com/upsolve/6hKReQ_w6B1jt5Gu/images/ai-agent-builder/improve/tests-result-expected-actual.png?fit=max\&auto=format\&n=6hKReQ_w6B1jt5Gu\&q=85\&s=6215d55fb2d3fbc5d7fd782b5a8cbb09) When a case fails, ask the [Builder agent](https://docs.upsolve.ai/ai-agent-builder/builder-agent.md) to "fix the context based on the last test run". It reads the failures and the evaluator's reports, and fixes the skill, prompt or description that caused them. Then run the tests again. ## Set up the evaluator The **Evaluator** tab controls how answers are graded. Its settings apply to every test case in the workspace. ![The Evaluator tab with Output type, What the evaluator sees (SQL, Chart config, Sample data) and Evaluator model settings](https://mintcdn.com/upsolve/6hKReQ_w6B1jt5Gu/images/ai-agent-builder/improve/tests-evaluator.png?fit=max\&auto=format\&n=6hKReQ_w6B1jt5Gu\&q=85\&s=40aeced1acb2da1bdfa3bdff6fb06ff0) ### Output type * **Pass / fail** (the default): the evaluator decides whether the answer passes. * **Numeric score**: the evaluator gives a score in a range you set (for example 0 to 10), and a case passes when its score is at least your threshold. The evaluator never sees the threshold, so it can't grade towards it. Changing the threshold re-grades past scored runs without running them again. ### What the evaluator sees * **SQL** (always on): the question, your reference query, the query the agent ran, and the descriptions of the tables and columns they read. Never any rows. * **Chart config**: the chart the test case asks for and the one the agent produced. Turn it on when most of your cases are chart questions. * **Sample data**: rows from running both queries, so the evaluator can compare what they return rather than only what they say. This sends data from your warehouse to the model. ### Evaluator model By default the evaluator follows the workspace's AI models (**Workspace settings → AI models**), or Upsolve's default model when none are set. You can pin a specific provider and model instead. Each result names the model that graded it, and **Use the default** switches back. ### Grading instructions The grading instructions are the evaluator's rubric, in your own words. Name your criteria (the default uses **Answers the question**, **Consistent with the reference** and **Complete**), and the evaluator writes one section of its report per criterion, which is what you read on a failing case. Upsolve adds the reply format for you; write the judgment, not the format. ![The Grading instructions editor showing the default rubric with three named criteria](https://mintcdn.com/upsolve/6hKReQ_w6B1jt5Gu/images/ai-agent-builder/improve/tests-evaluator-instructions.png?fit=max\&auto=format\&n=6hKReQ_w6B1jt5Gu\&q=85\&s=8c9a9f4a346b0f8de14c93ebcf05e0f6) Some rules always hold, whatever the instructions say: * Matching SQL is a strong hint to the evaluator, never an automatic pass. Two different queries can both be right. * An empty answer fails without being graded. * If the evaluator itself can't run, the case fails and is marked **Evaluator could not run**, so a grading outage never looks like a pass. ## CI: check tests every time a version is saved Running tests by hand relies on someone remembering. CI makes it automatic. Go to **Workspace settings → CI** and turn on **Check test cases when saving a version**. From then on, saving a version first: 1. Runs every test case against the draft. 2. Checks the draft's row-level security. A version that fails either check is marked as failed and **can't be promoted to production**. A version passes only when every case and the RLS check pass; there is no partial pass rate. Runs you started yourself never count towards this check, so every version is tested as it is saved. With CI on, the save dialog lists what will be checked under **Before this version ships** and the button reads **Run checks & save version**. With CI off, the button reads **Save version without CI**, and you must tick **I understand this ships unverified** to continue. > [!NOTE] > > * If CI is on and the workspace has no test cases, saving a version is blocked until you add one. > * Your organization can require CI for every workspace. The switch is then locked and shows **Required by your organization**. ### With environments If your project uses [environments](https://docs.upsolve.ai/ai-agent-builder/workspaces.md) (production, staging and so on), saving a version only freezes it, and tests run when you **deploy** the version to an environment. Each environment has its own **Require tests** switch and a **Test as** user, so deploy-time tests answer as a real user of that environment, against that environment's database. A deployment that fails its tests leaves the environment on its previous version. ## Cost Every test case is a full agent turn plus one evaluator call, and both are billed to your **AI credits**. Each case result shows its cost. Cases run three at a time. > [!NOTE] > > For organizations that handle confidential data, grading is based on SQL only: the agent's written answer and any rows are never sent to the evaluator. Results show a **Graded on SQL only** badge, and the Sample data option is unavailable. ## Next steps > **[Builder Agent](https://docs.upsolve.ai/ai-agent-builder/builder-agent.md)** > > Let the Builder agent fix the context behind failing tests. > **[Self-improvement](https://docs.upsolve.ai/ai-agent-builder/self-improvement.md)** > > Scheduled reviews that add tests and check their own changes against yours. > **[Golden Assets](https://docs.upsolve.ai/ai-agent-builder/golden-assets.md)** > > Turn trusted test cases into worked examples for the agent. > **[Chat History](https://docs.upsolve.ai/ai-agent-builder/chat-history.md)** > > Save real questions as test cases. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.