> ## Documentation Index > Fetch the complete documentation index at: https://docs.upsolve.ai/llms.txt > Use this file to discover all available pages before exploring further. # Get Test Run > Get one test run: its status while it works, its progress, and its verdict once it settles. Poll this to wait for a run to finish. ## OpenAPI ````yaml get /v2/api/projects/{projectId}/workspaces/{workspaceId}/test-runs/{jobId} openapi: 3.1.0 info: title: Upsolve API version: 1.5.0 servers: - url: https://api.upsolve.ai security: [] tags: [] paths: /v2/api/projects/{projectId}/workspaces/{workspaceId}/test-runs/{jobId}: get: summary: 'Get one test run: its status while it works, its …' description: >- Get one test run: its status while it works, its progress, and its verdict once it settles. Poll this to wait for a run to finish. operationId: GetV2ApiProjectsProjectIdWorkspacesWorkspaceIdTestRunsJobId parameters: - name: apiKey in: query required: false description: >- Upsolve API key, generated under Account → Developer settings in the Upsolve Hub. Keys are user-scoped: pass organizationId alongside the key to select which org to act on. schema: type: string minLength: 1 description: >- Upsolve API key, generated under Account → Developer settings in the Upsolve Hub. Keys are user-scoped: pass organizationId alongside the key to select which org to act on. examples: - up_embed_************ examples: example1: value: up_embed_************ - name: organizationId in: query required: false description: >- Your Upsolve organization's UUID — the org that owns the API key and project. This is NOT a project organization: do not pass a value registered via /projects/register-organization, and do not pass an `externalId` (only `projectOrganizationId` accepts those). Omit it when using an org-bound API key — the org is read from the key and any value sent here is ignored. Required only for user-scoped API keys (created under Account → Developer settings) and for the Supabase JWT path, where the org is ambiguous and membership is verified against it. schema: description: >- Your Upsolve organization's UUID — the org that owns the API key and project. This is NOT a project organization: do not pass a value registered via /projects/register-organization, and do not pass an `externalId` (only `projectOrganizationId` accepts those). Omit it when using an org-bound API key — the org is read from the key and any value sent here is ignored. Required only for user-scoped API keys (created under Account → Developer settings) and for the Supabase JWT path, where the org is ambiguous and membership is verified against it. examples: - 123e4567-e89b-12d3-a456-426614174000 type: string format: uuid pattern: >- ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})$ examples: example1: value: 123e4567-e89b-12d3-a456-426614174000 - name: projectId in: path required: true description: >- GET /v2/api/projects/:projectId/workspaces/:workspaceId/test-runs/:jobId Parameter schema: type: string format: uuid pattern: >- ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})$ - name: workspaceId in: path required: true description: >- GET /v2/api/projects/:projectId/workspaces/:workspaceId/test-runs/:jobId Parameter schema: type: string format: uuid pattern: >- ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})$ - name: jobId in: path required: true description: >- GET /v2/api/projects/:projectId/workspaces/:workspaceId/test-runs/:jobId Parameter schema: type: string format: uuid pattern: >- ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})$ responses: '200': description: >- GET /v2/api/projects/:projectId/workspaces/:workspaceId/test-runs/:jobId Positive response content: application/json: schema: type: object properties: status: type: string const: success data: type: object properties: jobId: type: string workspaceId: type: string status: type: string enum: - pending - processing - completed - failed - cancelled description: >- `cancelled` is terminal and is NOT a kind of failure: a builder stopped the run, so nothing was learned about the workspace and `outcome` is null. outcome: type: - string - 'null' enum: - passed - failed draftFingerprint: type: string description: >- Identifies what this run tested: the working copy AND the test cases it replayed. Editing either changes it, which is what makes the publish gate exact and what resets the verdict when a case is rewritten. partial: type: boolean description: >- True when the run replayed only some of the cases. A partial run never satisfies the publish gate. publishVersion: type: - number - 'null' environment: type: string description: >- The environment whose database the run read: `production`, or the environment of the project user it answered as, such as `staging`. progressMessage: type: - string - 'null' errorMessage: type: - string - 'null' totalCases: type: number completedCases: type: number result: type: - object - 'null' properties: cases: type: array items: type: object properties: caseId: type: string description: >- The Golden Question this case was built from. question: type: string status: type: string enum: - passed - failed - errored description: >- 'failed' means the agent answered and scored below the workspace's threshold; 'errored' means the agent never answered. Derived from `score` on every read, so it follows the threshold rather than being frozen at run time. score: description: >- The evaluator's score as a WHOLE NUMBER on this workspace's configured scale. It is both the number shown to a builder and the number compared against the threshold, so a client must never round it again or compare `rawScore` instead. Absent on an errored case (nothing was graded) and on runs recorded before scoring existed — which is NOT the same as 0, and must not be defaulted to it. type: number actualAnswer: type: string expectedSql: type: string description: >- The golden reference the judge graded against. expectedChart: default: null description: >- The test case's reference chart definition, as authored. Null when the case is a query rather than a chart. type: - string - 'null' actualChart: default: null description: >- The chart the agent produced for this case, or null. type: - object - 'null' properties: config: type: object properties: title: type: string description: >- Descriptive title for the chart that clearly explains what data is being visualized. Should be concise but informative, e.g., 'Monthly Sales Revenue by Region', 'User Engagement Trends Over Time' type: type: string enum: - line - bar - area - scatter - pie - donut - advancedPie - table - basicTable - statCard - kpiCard - heatMap - map - combo description: >- Visualization type that best represents the data pattern. Choose based on data characteristics: LINE (trends over time), BAR (categorical comparisons), AREA (cumulative trends), SCATTER (correlations between 2+ metrics), PIE/DONUT (part-to-whole with 1 dimension + 1 measure), TABLE (detailed data view), STATCARD/KPICARD (single key metrics), HEATMAP (2D patterns), MAP (geographical data with lat/lng), COMBO (multiple chart types). STRICT REQUIREMENTS: pie/donut need exactly 1 dimension (categories) + 1 measure (count/sum only), scatter needs 2+ measures, map needs latitude + longitude + 1+ dimensions, kpi/stat cards need 1+ measures dimensions: type: array items: type: object properties: name: type: string label: type: string field: type: string type: type: string enum: - string - number - date - boolean - mixed - json aggregation: type: string enum: - count - distinct_count - sum - avg - min - max - first - collect_list bucketing: type: string enum: - minute - hour - day - week - month - quarter - year - yearToDate format: type: object properties: label: description: >- Custom label for the field (overrides default field name) type: string formatOptions: description: >- JSON string containing formatting options for Intl.NumberFormat or Intl.DateTimeFormat. Must be a valid JSON object string. The object must be compatible with Intl.NumberFormat options for numeric fields or Intl.DateTimeFormat options for date fields. Example for numbers (not required format): '{ "style": "currency", "currency": "USD" }'. Example for dates (not required format): '{ "year": "numeric", "month": "long" }'. You can use any valid options from these Intl APIs, not just the examples shown. type: string formatLocale: description: >- Locale string for formatting (e.g., 'en-US', 'de-DE'). Used with both Intl.NumberFormat for numeric fields and Intl.DateTimeFormat for date fields. type: string labelSuffix: description: >- Text to append after the formatted value (e.g., '%', ' units') type: string labelPrefix: description: >- Text to prepend before the formatted value (e.g., '$', 'Total: ') type: string dateFormat: description: >- Date format string for date/time fields (e.g., 'YYYY-MM-DD', 'MM/DD/YYYY') type: string additionalProperties: false required: - name - field - type additionalProperties: false description: >- Categorical or time-based fields used to group and segment data. These create the x-axis for most charts and define categories for pie/donut charts. Common examples: date fields (for time series), categorical fields (regions, products, users), or any field that creates meaningful groupings. For pie/donut charts, use exactly 1 dimension representing the main categories you want to compare. Dimensions can have date bucketing (hour/day/week/month/quarter/year) for time-based aggregations measures: type: array items: type: object properties: name: type: string label: type: string field: type: string aggregation: type: string enum: - count - distinct_count - sum - avg - min - max - first - collect_list format: type: object properties: label: description: >- Custom label for the field (overrides default field name) type: string formatOptions: description: >- JSON string containing formatting options for Intl.NumberFormat or Intl.DateTimeFormat. Must be a valid JSON object string. The object must be compatible with Intl.NumberFormat options for numeric fields or Intl.DateTimeFormat options for date fields. Example for numbers (not required format): '{ "style": "currency", "currency": "USD" }'. Example for dates (not required format): '{ "year": "numeric", "month": "long" }'. You can use any valid options from these Intl APIs, not just the examples shown. type: string formatLocale: description: >- Locale string for formatting (e.g., 'en-US', 'de-DE'). Used with both Intl.NumberFormat for numeric fields and Intl.DateTimeFormat for date fields. type: string labelSuffix: description: >- Text to append after the formatted value (e.g., '%', ' units') type: string labelPrefix: description: >- Text to prepend before the formatted value (e.g., '$', 'Total: ') type: string dateFormat: description: >- Date format string for date/time fields (e.g., 'YYYY-MM-DD', 'MM/DD/YYYY') type: string additionalProperties: false required: - name - field - aggregation additionalProperties: false description: >- Numeric fields that get aggregated to create the y-axis values or chart data points. Each measure must specify an aggregation type: COUNT (number of records), SUM (total values), AVG (average), MIN/MAX (extreme values), DISTINCT_COUNT (unique values). For pie/donut charts, use exactly 1 measure with COUNT or SUM aggregation. For scatter plots, use 2+ measures for x/y coordinates. Measures support formatting options like currency, percentages, and decimal places comparisons: description: >- Period-over-period comparisons to show growth/decline trends. Options: 'previous_period' (compare to previous time bucket), 'year_over_year' (compare same period last year), 'custom' (user-defined period). Each comparison references a measure and applies the comparison logic. Useful for showing percentage changes, growth rates, and trend analysis in business metrics type: array items: type: object properties: name: type: string measure: type: string compareBy: type: string enum: - previous_period - year_over_year - custom customPeriod: type: number required: - name - measure - compareBy additionalProperties: false mapConfig: description: >- Required only for MAP chart types. Specifies which fields contain latitude and longitude coordinates. Both fields must be numeric and contain valid coordinate values. Used in combination with dimensions to create geographical visualizations showing data distribution across locations type: object properties: latitude: type: string longitude: type: string additionalProperties: false filters: description: >- Pre-aggregation filters applied to raw data before grouping and calculations. Supports operators: LIKE (text pattern matching), IN (value inclusion), BETWEEN (range filtering). Use to exclude irrelevant data, focus on specific segments, or handle data quality issues. Examples: filter by date ranges, specific categories, or exclude null values. Filters help create focused, relevant visualizations type: array items: type: object properties: field: type: string operator: type: string enum: - LIKE - IN - BETWEEN - '=' - '!=' - '>' - '>=' - < - <= - NOT LIKE - NOT IN value: anyOf: - type: string - type: number - type: boolean - type: array items: anyOf: - type: string - type: number - type: boolean required: - field - operator - value additionalProperties: false havingFilters: description: >- Post-aggregation filters applied after grouping and measure calculations. Used to filter out aggregated results that don't meet criteria. Common use cases: show only categories with count > 10, exclude regions with sales < $1000, focus on top performers. Essential for creating clean visualizations by removing noise or low-value data points type: array items: type: object properties: field: type: string aggregation: type: string operator: type: string value: anyOf: - type: string - type: number - type: boolean - type: array items: anyOf: - type: string - type: number - type: boolean required: - field - operator - value additionalProperties: false settings: description: >- Visual customization and presentation options. Controls: layout (horizontal/vertical), colors (custom color palettes), legend (show/hide and position), stacking (for bar/area charts), data labels (show values on chart), axis labels, debug mode. Use to enhance readability, match brand colors, or optimize for different screen sizes and use cases type: object properties: layout: description: >- Chart orientation: 'horizontal' renders bars/categories horizontally with values extending right, 'vertical' renders bars/categories vertically with values extending upward. Primarily affects BAR charts. Use horizontal for long category names or many categories (easier to read labels), vertical for typical column charts or time series. Default is usually horizontal type: string enum: - horizontal - vertical secondYAxis: description: >- Enables the second Y-axis for charts with exactly 2 measures. When enabled, the second measure will be rendered on the right Y-axis with independent scaling. This is useful when comparing two metrics with different scales or units (e.g., revenue in dollars vs. quantity in units). Only applies to line, area, bar, and combo chart types with exactly 2 measures. Default is false type: boolean colors: description: >- Custom color palette for chart series and categories. Provide an array of color strings in any valid CSS format (hex: '#FF5733', rgb: 'rgb(255,87,51)', named: 'blue'). Colors are applied sequentially to measures/series. Use to match brand guidelines, improve accessibility, or create visual hierarchy. If not specified, default color scheme is applied. Minimum 1 color recommended, but provide enough colors to cover all series to avoid repetition type: array items: type: string legend: description: >- Controls legend visibility on the chart. Set to true to show the legend (which displays labels for different series, measures, or categories), false to hide it. Legends help users identify what each color/line/bar represents. Hide legend for single-series charts or when labels are self-explanatory. Show legend for multi-series charts, pie/donut charts, or when colors represent different data categories. Default is typically true type: boolean legendPosition: description: >- Legend placement relative to chart area. Options: 'bottom' (common default, good for wide charts, also use this when generating PIE or DONUT charts), 'right' (doesn't affect chart width), 'left' (uncommon), 'top' (good for horizontal space-constrained displays). Choose based on available space and chart aspect ratio. Left/right work well for tall charts, top/bottom for wide charts. Only applies when legend is set to true default: bottom type: string enum: - left - right - top - bottom stacking: description: >- Enables stacked mode for BAR and AREA charts with multiple series. When true, values are stacked on top of each other showing cumulative totals and part-to-whole relationships. When false, series are displayed side-by-side (bars) or overlapping (areas). Use stacking to show total values and individual contributions simultaneously, or to emphasize cumulative trends. Not applicable for pie, scatter, or single-series charts. Default is false type: boolean showDataLabels: description: >- Controls whether numeric values are displayed directly on chart elements (on bars, lines, pie slices, etc.). When true, shows the actual data values as labels on or near the visual elements. Improves readability and eliminates need to reference axes, but can clutter charts with many data points. Best for charts with few data points (< 20), important for presentations or when precise values matter. Default is false to maintain clean appearance type: boolean enableLegendSlider: description: >- Adds interactive slider/scroll functionality to the legend when there are many series or categories. Useful for charts with 5+ series where a static legend would consume too much space. When true, legend becomes scrollable or paginable. Improves usability for complex multi-series visualizations without overwhelming the interface. Only relevant when legend is enabled and there are multiple items. Default is false type: boolean showAxisLabels: description: >- Controls visibility of axis labels (x-axis and y-axis labels/titles). When true, displays axis labels that describe what each axis represents (e.g., 'Revenue ($)', 'Month', 'Sales Count'). When false, hides these labels for a cleaner, more compact chart. Show labels for clarity and context, especially for presentations or standalone charts. Hide for dashboards where context is clear or space is limited. Default is typically true type: boolean debug: description: >- Enables debug/development mode for troubleshooting. When true, may display additional information like raw data, generated SQL queries, configuration details, or console logs. Intended for developers and debugging purposes only. Should always be false in production environments. Use to diagnose chart configuration issues, verify data transformations, or understand chart behavior during development type: boolean pageSize: description: >- Number of rows to display per page in BAR chart types. Common values: 10 (compact), 15 (default balance), 25-50 (data-heavy tables). Larger values reduce pagination clicks but increase initial load time and scrolling. Default is 15 rows per page default: 15 type: number sliceLimit: description: >- Number of categories to display in PIE and DONUT chart types. Default is 7 categories default: 7 type: number maxSeries: description: >- Number of series to draw in a LINE, BAR or AREA chart whose second dimension splits it into one series per value (e.g. a stacked bar per year, split by athlete). The series with the largest totals are drawn and the rest are summed into one 'Other' series, so stacked totals stay correct. Default is 20; every series is drawn on every bar, so a much higher value slows the page default: 20 type: number designHint: description: >- The visual form the user asked for when `type` cannot express it. The designed (themed) rendering of the chart draws this form when the data fits; the standard renderer ignores it. Set 'funnel' when the user asks for a funnel or conversion stages (keep `type` as 'bar' with one dimension for the stages and one measure), 'gauge' for a single value against a target, 'treemap' for a part-to-whole across many categories, 'radar' for comparing several measures across a few categories. Leave unset otherwise type: string enum: - funnel - gauge - treemap - radar - horizontalBar - stackedBar - donut cornerRadius: description: >- Corner rounding, in pixels, of this chart's bars, slices and card in the designed (themed) rendering: 0 is square, 4-8 is softly rounded, 12-16 is very round. Set it only when the user asks for rounded or square corners; unset follows the workspace theme. The standard renderer ignores it type: integer minimum: 0 maximum: 16 requestedStyle: description: >- Which of this chart's style settings the USER explicitly asked for: 'colors' ('make it green' — set `colors` too), 'legend' ('hide the legend' — set `legend` / `legendPosition` too), 'dataLabels' ('show the values' — set `showDataLabels` too). The designed (themed) rendering draws everything else in the workspace theme, so list a setting only when the user asked for it, never for colours or labels chosen on their behalf. Keep the list when modifying a chart type: array items: type: string enum: - colors - legend - dataLabels additionalProperties: false orderBy: description: >- Sorting configuration for chart data. Specify field name and sort direction (ascending/descending). Supports multiple sort fields for complex ordering. Common use cases: sort bars by value (descending for top performers), sort time series by date (ascending), sort categories alphabetically. Helps create meaningful data hierarchy and improves chart readability type: array items: type: object properties: field: type: string direction: type: string enum: - ascending - descending required: - field - direction additionalProperties: false sqlOverride: description: >- Advanced: Custom SQL query to override auto-generated chart query. Use only when standard schema cannot express complex data requirements. Must return data compatible with specified dimensions and measures. Warning: bypasses schema validation and may cause errors if not properly formatted type: string required: - title - type - dimensions - measures additionalProperties: false description: >- KPI Card charts require at least one measure. chart: description: >- The agent's chart as v3 (query, shape, render), when it drew one; absent on runs recorded before v3 type: object properties: v: type: number const: 3 required: - v additionalProperties: {} data: type: array items: type: object propertyNames: type: string additionalProperties: {} displayed: type: boolean description: >- False when the agent built the chart but never displayed it. truncated: type: boolean description: >- True when the row set was capped on the way out of the agent. required: - config - data - displayed - truncated additionalProperties: false actualSql: default: [] description: >- The SQL the agent actually ran for this case, in call order. Empty when the turn ran no query. type: array items: type: string errorMessage: description: >- Why the AGENT failed to answer, when it never produced one — a transport error, a timeout, an aborted run. Present only on an `errored` case, and never a judgement about the answer. type: string evaluatorError: description: >- Why the EVALUATOR could not run, as a stable code (EVALUATOR_NOT_CONFIGURED, EVALUATOR_UNAUTHORIZED, EVALUATOR_RATE_LIMITED, EVALUATOR_UNREACHABLE, EVALUATOR_REQUEST_FAILED, EVALUATOR_MALFORMED_RESPONSE, EVALUATOR_MODEL_INVALID, EVALUATOR_MODEL_UNAVAILABLE, EVALUATOR_MODEL_NOT_ENTITLED, EVALUATOR_SCORE_OUT_OF_RANGE, EVALUATOR_REFERENCE_SQL_MISSING). When present the case FAILED because NOTHING GRADED the answer — not because the answer was bad — and it carries NO score, so moving the threshold can never turn an outage into a verdict. type: string report: description: >- The evaluator's markdown report: one section per criterion the workspace's instructions named, saying whether the answer met it and what the evidence was. Present in both output modes — a score or a boolean without a reason is not something a builder can act on. On a failed evaluator it carries the reason instead. Absent on an errored case and on runs recorded before it existed. type: string confidence: description: >- LEGACY. How peaked the previous evaluator's distribution was, 0-1. Nothing writes it any more and it drives no verdict; kept so a stored run still parses. type: number rawScore: description: >- LEGACY. The previous evaluator's own float across its five fixed quality levels (0-4), before rescaling onto this workspace's range. Nothing writes it any more, but it is still READ so a run recorded before the rework re-derives its verdict exactly as it always did when the threshold moves. Never displayed. type: number threadId: type: string description: >- The agent thread this case ran on, for finding its Arthur traces. durationMs: type: number costCredits: description: >- What this case cost in AI credits — the agent's turn plus the evaluator that graded it, at the same fixed rate the turn is billed at. Absent when the provider reported no usage; absent is NOT zero. type: number gradingScope: description: >- 'sql_only' when the evaluator graded this case without seeing the agent's answer or any rows — what a confidential-data organization's cases are graded on. Weaker evidence than a full grade: it shows the right query ran, not that the answer reported it correctly. Absent on a full grade, on an errored case and on a case decided without the evaluator. type: string const: sql_only evaluatorModel: description: >- The model that graded this case, as 'provider/model' (for example 'gemini/gemini-3.8-flash'). Absent when no evaluator reply was accepted, and on runs recorded before it was stored. type: string required: - caseId - question - status - actualAnswer - expectedSql - expectedChart - actualChart - actualSql - threadId - durationMs additionalProperties: false rls: type: array items: type: object properties: table: type: string description: The data model key for the table. status: type: string enum: - governed - ungoverned - failed description: >- `governed` passes. `ungoverned` means no row-level security policy is applied to the table. `failed` means it has one that cannot be turned into a valid query. Both of the latter fail the check. passed: type: boolean description: >- Derived from `status`: only `governed` passes. error: type: - string - 'null' required: - table - status - passed - error additionalProperties: false rlsReport: type: object properties: warnings: type: array items: type: object properties: table: type: string property: type: string description: >- The placeholder that could not be resolved, e.g. `user.region`. affectedUsers: type: number description: >- How many of the SCANNED users lacked it -- not how many exist. required: - table - property - affectedUsers additionalProperties: false scannedUsers: type: number description: >- How many project users the policies were substituted for. truncated: type: boolean description: >- True when the project has more users than the scan cap, so the warnings describe a sample rather than everybody. required: - warnings - scannedUsers - truncated additionalProperties: false evaluatorOutputType: description: >- What the evaluator was asked for when this run was graded. Read it to decide whether to show a score at all — a 'pass_fail' run has none. Absent on runs recorded before it existed; read an absent one as 'score', since every such run was graded by the previous evaluator, which always produced one. type: string enum: - pass_fail - score gradingScope: description: >- 'sql_only' when this run's cases were graded on their SQL only, without the agent's answer or any rows — what a confidential-data organization's runs are graded on. Absent on a full-scope run and on runs recorded before it existed. type: string const: sql_only omittedCases: description: >- Cases the run left out because one run replays at most 50. Absent when it left none out, and on runs recorded before it existed. type: number failureSummary: type: - string - 'null' required: - cases - rls - failureSummary additionalProperties: false createdAt: type: string updatedAt: type: string completedAt: type: - string - 'null' required: - jobId - workspaceId - status - outcome - draftFingerprint - partial - publishVersion - environment - progressMessage - errorMessage - totalCases - completedCases - result - createdAt - updatedAt - completedAt additionalProperties: false required: - status - data additionalProperties: false '400': description: >- GET /v2/api/projects/:projectId/workspaces/:workspaceId/test-runs/:jobId Negative response content: application/json: schema: type: object properties: status: type: string const: error error: type: object properties: message: type: string code: type: string details: type: object propertyNames: type: string additionalProperties: {} required: - message additionalProperties: false required: - status - error additionalProperties: false security: - APIKEY_2: [] components: securitySchemes: APIKEY_2: type: apiKey in: query name: apiKey ```` This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.