Experiments

An experiment answers “is this prompt better than that one?” with your own test cases. It joins a dataset, a target (a prompt version or typed messages on one credential and model) and some evaluators. Each run sends every dataset item through the target, scores each answer, and records the answers, scores, time and cost. Then you compare two runs side by side.

Every run makes real provider calls and spends money. Members, admins and owners can create experiments and start runs; viewers can read them. Experiments live under Experiments in the project. To run the same comparison on every pull request, use the eval gate.

Before you start

• The member, admin or owner role in the project. Viewers can read
• A dataset with items. Evaluators are optional, and an admin or owner creates them
• A provider credential in your organization, a model with a known price, and CREDENTIALS_KEYS set on the server
• The worker running, because it executes the runs
• To use the API or the command line: a personal access token with the write scope

Create an experiment

Part	Meaning
Name	1 to 200 characters, unique in the project
Dataset	The items to replay
Target	What answers each item (below)
Cost limit (max_cost_usd)	The most one run may spend on the target’s calls. From above 0 up to 1000 dollars; 5 by default
max_items	How many dataset items a run reads. 1000 by default, and never more than the server’s EXPERIMENT_MAX_ITEMS
Evaluators	Up to 20 evaluators that score every answer

An experiment cannot be edited or deleted after it is created. Its dataset, target, evaluators and limits are fixed, so every run of it is comparable. To change them, make a new experiment.

Targets

A target names a credential of your organization (any provider), a model and optional parameters (temperature, max_tokens up to 4096, top_p). The model must have a price, because the cost limit could not be enforced without one (422 EXPERIMENT_MODEL_UNPRICED). There are two kinds:

• Prompt version. A stored prompt version, filled in with each item. This is how you test a prompt change.
• Messages. Typed system, user and assistant messages, sent unchanged for every item. The item’s input is not used. This tests a model or a parameter against fixed instructions.

How items fill the target

For a prompt target, the item’s input fills the template’s variables:

• an input that is a JSON object whose values are all strings supplies the variables by name ({"question": "..."});
• any other input becomes the single variable input, as text.

An item that cannot fill every variable the template uses gets the item error missing_variables: question, tier, and one whose filled-in prompt would exceed 256 KB gets prompt_too_large. That fails that item only, not the run.

Start a run

POST /api/v1/projects/{project_id}/experiments/{experiment_id}/runs queues a run and answers 202; a worker executes it. Send an Idempotency-Key so a retry cannot start a second run (the body is part of what a retry must repeat).

• One run at a time. While a run is queued or running, another start answers 409 RUN_IN_PROGRESS.
• Compare prompt versions with one experiment. The body may be {"target_override": {"prompt_version_id": "..."}}. For a prompt-version experiment, that runs another version of the same prompt, with the experiment’s credential, model and parameters. Anything else answers 422 TARGET_OVERRIDE_INVALID. Every run keeps the effective target it used, so one experiment can hold a run for version 1 and a run for version 2.
• Numbering. Runs are numbered 1, 2, 3 in the order they start, and a number never changes.

Run an experiment from code

The dashboard is one way to create and run an experiment. You can also use the command that the eval gate runs, or the API. Both authenticate with a personal access token that has the write scope, because a run spends money.

From the command line

The Python SDK includes a runner. It finds your dataset, prompt and evaluators by name, creates the experiment the first time, starts a run, waits for it to end and compares the mean scores with the thresholds you give. The token is read from the environment, and SPANLIGHT_HOST and SPANLIGHT_PROJECT_ID can replace --host and --project-id.

pip install spanlight

export SPANLIGHT_TOKEN=spl_pat_...
python -m spanlight._experiments \
  --host https://spanlight.example.com --project-id <project id> \
  --dataset support-questions --prompt support/answer:3 \
  --evaluators relevance,format --fail-below relevance=0.8,format=1 \
  --credential-id <credential id> --model claude-haiku-5-5 \
  --max-cost-usd 2

The module is spanlight._experiments. There is no python -m spanlight command. When the run ends, the runner prints a Markdown report like this one (the numbers are an example):

### Spanlight eval gate: support-questions / support/answer v3

Gate **passed**. [Open the run](https://spanlight.example.com/<organization id>/<project id>/experiments/<experiment id>/runs/<run id>)

Items 50, errors 0, cost 0.0412 USD, p50 812 ms, p95 1490 ms.

| Score | Mean | 95% CI | Items | Threshold | Result |
| --- | --- | --- | --- | --- | --- |
| format | 1 | 92–100% | 50 | >= 1 | pass |
| relevance | 0.84 | — | 50 | >= 0.8 | pass |

The 95% CI column is the interval on the pass rate of a yes/no score (see Intervals and significance), and “—” for other scores. When a gated yes/no score’s threshold lies inside its interval, the report adds a line such as format: Interval includes the threshold; the result is not conclusive at this sample size. The verdict does not change: the gate still compares the mean. When a mean is below its threshold the report says failed, adds a Below threshold: list such as relevance: 0.75 < 0.8, and the command exits with 1. The exit codes are listed in the eval gate page. If you leave out a required setting, the command exits with 2 and names what is missing:

error: --host or SPANLIGHT_HOST is required; --project-id or SPANLIGHT_PROJECT_ID is required; --credential-id is required; --model is required; SPANLIGHT_TOKEN is required

Through the API

Create the experiment, then start a run. This example tests a stored prompt version on one credential and model, with two evaluators attached:

export SPANLIGHT_HOST=https://spanlight.example.com
export PROJECT=<project id>

curl -sS -X POST "$SPANLIGHT_HOST/api/v1/projects/$PROJECT/experiments" \
  -H "Authorization: Bearer $SPANLIGHT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "answer-v3",
    "dataset_id": "<dataset id>",
    "config": {
      "target": {
        "kind": "prompt_version",
        "prompt_version_id": "<prompt version id>",
        "credential_id": "<credential id>",
        "model": "claude-haiku-5-5",
        "params": { "max_tokens": 512 }
      },
      "max_cost_usd": "2.00",
      "max_items": 100
    },
    "evaluator_ids": ["<evaluator id>", "<evaluator id>"]
  }'

The answer is 201 with the experiment. A target keeps every parameter you did not set as null:

{
  "id": "<experiment id>",
  "name": "answer-v3",
  "dataset_id": "<dataset id>",
  "config": {
    "target": {
      "kind": "prompt_version",
      "prompt_version_id": "<prompt version id>",
      "credential_id": "<credential id>",
      "model": "claude-haiku-5-5",
      "params": { "temperature": null, "max_tokens": 512, "top_p": null }
    },
    "max_cost_usd": "2.00",
    "max_items": 100
  },
  "evaluator_ids": ["<evaluator id>", "<evaluator id>"],
  "created_by": "<user id>",
  "created_at": "2026-10-13T09:30:00Z",
  "run_count": 0,
  "last_run": null
}

Start a run with the experiment’s id. Send an Idempotency-Key so that a retried request cannot start a second run:

curl -sS -X POST "$SPANLIGHT_HOST/api/v1/projects/$PROJECT/experiments/$EXPERIMENT/runs" \
  -H "Authorization: Bearer $SPANLIGHT_TOKEN" \
  -H "Idempotency-Key: $(uuidgen)" \
  -H "Content-Type: application/json" \
  -d '{}'

The answer is 202. The run is queued and a worker executes it:

{
  "id": "<run id>",
  "experiment_id": "<experiment id>",
  "number": 1,
  "target": {
    "kind": "prompt_version",
    "prompt_version_id": "<prompt version id>",
    "credential_id": "<credential id>",
    "model": "claude-haiku-5-5",
    "params": { "temperature": null, "max_tokens": 512, "top_p": null }
  },
  "status": "queued",
  "error": null,
  "started_at": "2026-10-13T09:30:00Z",
  "finished_at": null,
  "heartbeat_at": null,
  "started_by": "<user id>",
  "totals": null
}

Poll GET /api/v1/projects/$PROJECT/experiments/$EXPERIMENT/runs/$RUN until status is done or failed. The totals are computed live while the run is going. A 409 RUN_IN_PROGRESS means another run of the experiment is still queued or running.

What a run does

1. Takes the ids of the dataset’s first max_items items (oldest first) when it starts, and reads those items, with four calls in flight at a time.
2. Sends each item to the target. Each call is a trace in environment experiments, named experiment:<experiment name> and tagged spanlight:experiment:<run id>. A prompt target links the span to its prompt version.
3. Scores the answer with each attached evaluator. A heuristic evaluator and a judge receive the item’s expected_output, if it has one. Each score is stored on the item and also written on the item’s trace (source eval), under the evaluator’s name.
4. Stores the item: the answer (text and finish_reason), the duration, the cost, the scores, an error if there was one, and a copy of the dataset item it answered.

What a run keeps of the dataset

A run scores the items that exist when it starts. An item added to the dataset while the run is going is not scored, and an item deleted while it is going is skipped. An item edited while it is going is scored on the version the run read. To rule out changes completely, lock the dataset.

Every stored item keeps a copy of the dataset item it was scored against: its input, expected_output and metadata exactly as the run read them, on items that ended in an error too. The run’s items (GET .../runs/{run_id}/items) return the copy as input, expected_output and item_metadata, with position, the item’s number within the run (1, 2, 3 in dataset order). Editing or deleting a dataset item later does not change what an earlier run shows it was measured against, and does not renumber its items.

Items of a run made before runs kept a copy have input, expected_output and item_metadata set to null. They are not filled in from the dataset, because its items may have changed since. Run the experiment again to get a run with copies.

When the run ends, its totals are worked out from the stored items:

Total	Meaning
Items, errors	How many items ran and how many ended in an error
Cost	The sum of the items whose cost is known. If any call has no price or no usage, the cost is a lower bound and is flagged as such
Latency p50, p95	Nearest-rank percentiles over the answered items
Scores	Per score name: the mean and count of a numeric or yes/no score (yes counts 1), or the count per category. A yes/no score also has the number of items that passed and a 95 % interval on the pass rate

Unknown values are shown as “—”, never 0.

The cost limit

The cost limit protects you from a run that is larger than you thought. Before it sends an item, a run checks that what has been spent, plus the estimates of the calls in flight, plus this item’s estimate stays within max_cost_usd. The estimate is a ceiling made from the prompt, the model’s price and the output limit. When the next item would pass the limit, the in-flight calls finish, the check is repeated with their real costs, and if it still fails the run ends failed with cost_cap_reached. The UI says “Stopped: cost limit reached”. The items already stored stay, so a stopped run is still readable.

The limit covers the target’s calls only. Evaluators spend under their own budgets: a judge draws on its monthly budget, which this experiment shares with everything else that judge does that month. A judge that is out of budget leaves that item’s score out; it does not fail the item or the run. So the most an experiment can spend is its cost limit plus what its judges have left in their budgets.

Run states and errors

A run is queued, running, done or failed. A failed run says why in error:

Error	Meaning
cost_cap_reached	The cost limit was reached; stored items stay
abandoned	The worker died or stopped sending its heartbeat for 10 minutes
cancelled	The worker was shut down or timed out while the run was going
model_unpriced	The model lost its price, so the limit could not be enforced; nothing was called
credentials_not_configured	The worker has no CREDENTIALS_KEYS, so it cannot open the stored credential. Set it on the worker and run again; nothing was called
credential_missing, prompt_version_missing, experiment_config_invalid	The run could not start; nothing was called
internal_error, lease_lost	A failure on Spanlight’s side; the run stopped

An item has its own error: missing_variables: ..., BUDGET_EXCEEDED when a project budget set to block stopped the call, or the provider’s failure. It may also carry an evaluator_error when scoring itself broke; the answer and its cost are kept.

Compare two runs

Choose two runs of an experiment and open Compare. Both must have finished; comparing a run that is queued or running answers 409 RUN_IN_PROGRESS.

• Deltas are B minus A. For a numeric score (a yes/no score counts 1 and 0), the delta is b - a to four decimals, per item and for the whole run.
• Higher is better. Every score Spanlight produces, built-in judges included, is higher-is-better, so a positive delta is improved and a negative one regressed. A score where lower is better is not supported; the page says so. Improved and regressed always carry an icon and a word as well as a colour.
• Categories have no delta: equal values are unchanged, different ones count as neither improved nor regressed.
• Items match on the dataset item. Items present in both runs come first; items that only one run has are listed after them with the other side empty and no deltas.
• Items edited between the runs are left out. When both runs kept a copy of an item and its input or expected_output differs between them, the two scores were measured against different cases, so they are not compared. Such an item has changed: true and no deltas, and it is left out of the improved, regressed and unchanged counts and of the significance test; changed_items says how many there are. Each run’s own means, intervals, cost and latency still cover all of its items. Item metadata is not compared, because evaluators never see it. When the item is in one run only, or either run was made before runs kept a copy, changed is null; such an item is handled as before.
• Cost and p95 duration show the difference between the runs. When either side’s cost is unknown, the cost difference is “—” rather than a guess.

The page also states the trade-off in a sentence built from the differences, for example “B scores +0.08 on relevance, costs +$0.012, p95 latency +310 ms.”

Intervals and significance

Fifty items is a small sample. A pass rate of 78 % on 50 items could well be 65 % or 88 % on the next 50, so Spanlight shows how sure each number is.

• A yes/no score has a 95 % interval. The run’s Scores card shows, for example, “78% (95% CI 65–88%, n = 50)”, with the interval drawn as a thin range under the value. It is the Wilson score interval, which stays honest at small samples and at 0 % or 100 %. With no scored item there is no interval (“—”), never 0 to 100 %. Below 10 items the card adds “few items”. A run that finished before intervals existed shows “—”: its stored totals are not rewritten.
• Each score of a comparison has a test. The items scored in both runs are paired. For a yes/no score the test is the exact McNemar test, which looks only at the items that flipped: “Pass rate 72% → 78% on 50 items (10 flipped: 8 up, 2 down). Not significant (exact McNemar, p = 0.1094): the runs may not differ.” For a numeric score it is the paired sign test, which counts the items that went up and down and ignores the ties: “Mean 0.71 → 0.76 on 50 items (31 up, 12 down, 7 unchanged). Significant (sign test, p = 0.0054).” Categories have no test.
• What “significant” means. A difference is called significant when p is below 0.05: if the two runs truly behaved the same, a difference this large would happen by chance less than one time in twenty. “Not significant” does not mean the runs are the same; it means this sample cannot tell them apart. Add items, or treat the change as unproven. Both sentences use a neutral colour, never red or green.
• When there is nothing to test. When every paired item gave the same answer in both runs the comparison says “No items changed” (p = 1). When no item is scored in both runs it says “Nothing to compare: no item is scored in both runs.”

Both tests are exact (no approximation and no continuity correction), and 0.05 is not a setting. A yes/no score stored as the numbers 0 and 1, as an old SDK does, is treated as yes/no when its score config says so. In the API, a run’s totals.scores.<name> gains passed, ci_low and ci_high, and each score of a comparison gains ci_a, ci_b and test (kind mcnemar or sign, n, improved, regressed, unchanged, p_value, significant); a value that does not apply is null. Each compared item carries input and expected_output (run A’s copy, else run B’s), position (its number in run A, else in run B) and changed, and the comparison carries changed_items.

The demo seed

The live demo ships a ready-made experiment. spanlight demo seed-evals creates, in the demo project:

• the prompt support/answer with two versions (the demo bot’s own prompt, and a tightened rewrite), with production on version 2 and staging on version 1;
• the dataset support-questions, with 50 items taken from the demo’s own recorded answers;
• the judge relevance, on the built-in template, model claude-haiku-5-5, sampling 5 % of traces, with a monthly budget of $0.50;
• the experiment answer-v1-vs-v2, with a $0.50 cost limit per run and two runs: version 1, then version 2 through target_override.

It uses the demo organization’s Anthropic credential, so it needs ANTHROPIC_API_KEY. Because an experiment runs one run at a time, you run the command once to queue version 1 and again, after that run ends, to queue version 2. It is safe to repeat: what exists is kept. The steps, the prompts and how to re-seed are in the demo evaluations runbook.

Routes

Under /api/v1/projects/{project_id}/experiments.

Route	Needs	Does
GET, POST	project:read, experiments:run	List (with run count and latest run) and create (409 NAME_TAKEN)
GET /{experiment_id}	project:read	One experiment
POST /{experiment_id}/runs	experiments:run	Start a run
GET /{experiment_id}/runs	project:read	Runs, newest first, with the totals stored when each ended
GET /{experiment_id}/runs/{run_id}	project:read	One run; totals are computed live while it runs
GET /{experiment_id}/runs/{run_id}/items	project:read	The run’s items in dataset order, each with its copy of the dataset item
GET /{experiment_id}/runs/{run_a}/compare/{run_b}	project:read	The comparison, B minus A

A create body is {"name", "dataset_id", "config": {"target", "max_cost_usd"?, "max_items"?}, "evaluator_ids"?}. A 404 means the dataset, an evaluator or the prompt version is not in the project, or the credential is not in the organization.

Check it worked

Open the experiment and select Start run. The Runs card lists “Run #1”. When the worker has finished, the run is Done and the Scores card shows the mean of each evaluator across the items that were scored. Open Items to see the answer and the scores of each dataset item. If a run stops early, the run page says why in plain words, such as “Stopped: cost limit reached”.

Troubleshooting

• The run stays queued. The worker is not running. It executes every run. See The worker is not running.
• 409 RUN_IN_PROGRESS when you start a run. An experiment runs one run at a time. Wait for the current run to end, then start again.
• “Stopped: cost limit reached”. The next item could have passed max_cost_usd. The items already stored stay. Make a new experiment with a higher limit, or fewer items (max_items).
• “Not started: the model has no price, so the cost limit can’t be enforced”. The limit needs a price (422 EXPERIMENT_MODEL_UNPRICED at creation). Pick a priced model.
• A run is Failed with another reason. Every reason, and what each means, is in Run states and errors.

Next steps

Eval gate for CIFail a pull request when a score drops below a threshold.

EvaluatorsScore traces automatically with rules, a judge or your endpoint.

PromptsVersions and labels that you can change without a deploy.

DatasetsTest cases built from your traces or from a file.