Tests

Every change to a workspace (a new skill, a reworded prompt, a column description) can fix one answer and quietly break another. Tests catch that. A test case is a question your agent must answer correctly, plus the SQL that answers it. Run all your test cases against the draft before you save a version, and an evaluator grades every answer against your reference so you see exactly what passed, what failed and why. Turn on CI and Upsolve runs the tests for you every time someone saves a version, and won’t let a failing version reach production.

​

Where tests live

Open your workspace and click Tests in the Test & Ship section of the sidebar. The page has three tabs:

• Test cases: your questions and their reference SQL.
• Run history: every run, and every case’s result in it.
• Evaluator: how answers are graded.

​

Anatomy of a test case

Click Add test case to open the editor. A test case has: The two switches change how the agent behaves for your users. They don’t change how the case is run or graded. Test cases belong to the workspace draft, like the rest of its context. Adding or editing one is saved with Add test case or Save to draft, and becomes part of the next version you save.

​

Ways to create a test case

• From scratch: click Add test case on the Tests page.
• From a real conversation: in Chat History or the Playground, open a trace and click Save as test case. The question, the last SQL that succeeded and any chart are filled in for you, so turning a good answer into a test takes a few seconds. This is the fastest way to build a suite that reflects what your users actually ask.
• With the Builder agent: ask the Builder agent to “add a test case for…”. It writes the question and reference SQL and asks you to approve them.
• Through the API or CLI, for teams that keep their test suite in code.

Click any case in the list to see its question and reference SQL, run its SQL (Get data), Edit it, or check its Recent test runs.

​

Run your tests

The green button on the Test cases tab runs a batch. Its label tells you what it will run:

• Run all: every test case.
• Run N filtered: the cases matching the status filter, for example only the ones that failed last time.
• Run N selected: the cases you’ve ticked.

The ▶ on a row runs that one case. While a run is going, the button becomes Stop. Tests run against the version shown in the header: normally the Draft, so you are checking the changes you’re about to save. Pick a saved version there to test it instead. A single run replays at most 50 cases. With more than that, the button reads “Run 50 of N”.

​

What happens in a run

For each case, Upsolve:

1. Asks the agent the question, exactly as a user would, and records the SQL it ran, the chart it drew and what it said.
2. Runs your reference SQL.
3. Hands both to the evaluator, which grades the agent’s answer against your reference and writes a report.

A run you start by hand answers as no particular user, so row-level security doesn’t filter it. The same run also checks the draft’s row-level security rules.

​

Read the results

The Run history tab lists every run: when it started, what started it (Manual, or Save vN for a run triggered by saving a version), the agent version it ran against, how many cases passed, how long it took and its pass rate. Switch to Cases to see one row per case result instead, with its cost. Expand a run to see its cases, and click a case to see how it was graded. The evaluator writes one section per criterion, saying whether it was met and why: Below the report, Expected (the reference query and chart) sits side by side with Actual (the query the agent ran, the chart it produced and what it said), so you can see exactly where they differ. When a case fails, ask the Builder agent to “fix the context based on the last test run”. It reads the failures and the evaluator’s reports, and fixes the skill, prompt or description that caused them. Then run the tests again.

​

Set up the evaluator

The Evaluator tab controls how answers are graded. Its settings apply to every test case in the workspace.

​

Output type

• Pass / fail (the default): the evaluator decides whether the answer passes.
• Numeric score: the evaluator gives a score in a range you set (for example 0 to 10), and a case passes when its score is at least your threshold. The evaluator never sees the threshold, so it can’t grade towards it. Changing the threshold re-grades past scored runs without running them again.

​

What the evaluator sees

• SQL (always on): the question, your reference query, the query the agent ran, and the descriptions of the tables and columns they read. Never any rows.
• Chart config: the chart the test case asks for and the one the agent produced. Turn it on when most of your cases are chart questions.
• Sample data: rows from running both queries, so the evaluator can compare what they return rather than only what they say. This sends data from your warehouse to the model.

​

Evaluator model

By default the evaluator follows the workspace’s AI models (Workspace settings → AI models), or Upsolve’s default model when none are set. You can pin a specific provider and model instead. Each result names the model that graded it, and Use the default switches back.

​

Grading instructions

The grading instructions are the evaluator’s rubric, in your own words. Name your criteria (the default uses Answers the question, Consistent with the reference and Complete), and the evaluator writes one section of its report per criterion, which is what you read on a failing case. Upsolve adds the reply format for you; write the judgment, not the format. Some rules always hold, whatever the instructions say:

• Matching SQL is a strong hint to the evaluator, never an automatic pass. Two different queries can both be right.
• An empty answer fails without being graded.
• If the evaluator itself can’t run, the case fails and is marked Evaluator could not run, so a grading outage never looks like a pass.

​

CI: check tests every time a version is saved

Running tests by hand relies on someone remembering. CI makes it automatic. Go to Workspace settings → CI and turn on Check test cases when saving a version. From then on, saving a version first:

1. Runs every test case against the draft.
2. Checks the draft’s row-level security.

A version that fails either check is marked as failed and can’t be promoted to production. A version passes only when every case and the RLS check pass; there is no partial pass rate. Runs you started yourself never count towards this check, so every version is tested as it is saved. With CI on, the save dialog lists what will be checked under Before this version ships and the button reads Run checks & save version. With CI off, the button reads Save version without CI, and you must tick I understand this ships unverified to continue.

​

With environments

If your project uses environments (production, staging and so on), saving a version only freezes it, and tests run when you deploy the version to an environment. Each environment has its own Require tests switch and a Test as user, so deploy-time tests answer as a real user of that environment, against that environment’s database. A deployment that fails its tests leaves the environment on its previous version.

​

Cost

Every test case is a full agent turn plus one evaluator call, and both are billed to your AI credits. Each case result shows its cost. Cases run three at a time.

​

Next steps