Datasets

A dataset is a named list of test cases that you replay through a prompt in an experiment. Each item has an input, an optional expected_output and some metadata. Build one from real traces that went wrong or went well, or upload a file.

Anyone who can see the project can read datasets. Members, admins and owners can create, edit and delete datasets and their items, and admins and owners can lock a dataset so nobody changes it. Datasets live under Datasets in the project.

Before you start

• The member, admin or owner role in the project. Viewers can read
• To add items from traces: traces already stored in the project
• To upload items: a CSV or JSONL file

Datasets and items

• A dataset has a name (1 to 100 characters, unique in the project) and an optional description (up to 1000 characters). It holds up to 10 000 items; adding more answers 422 DATASET_FULL and stores nothing.
• input is required and can be any JSON value except null: a string, or an object whose string values fill a prompt’s variables (see experiments).
• expected_output is optional JSON. It is null when there is none, never an empty value. Some evaluators (exact_match, levenshtein, and the faithfulness judge) compare the answer with it.
• metadata is a JSON object of your own notes, for example a category or a source.
• input and expected_output are at most 64 KB each and metadata at most 8 KB, measured as compact JSON. A larger value is rejected, never cut. A NUL character is removed from every string, because the database cannot store it.
• Items also remember where they came from (source_trace_id, source_span_id) when they were copied from a trace.

You can change an item’s expected_output (send null to clear it) and replace its metadata as a whole. The input of an item cannot be changed; delete the item and add a new one.

Deleting a dataset removes its items. It is refused with 409 DATASET_IN_USE while an experiment uses it, and with 409 DATASET_LOCKED while the dataset is locked.

Add items from traces

The fastest way to a realistic dataset is to start from what your application actually received. On Traces, select traces and choose Add to dataset, or use the trace detail page.

For each trace Spanlight copies one item from a span:

• the span you name, or else the trace’s first LLM span by start time;
• its stored input becomes the item’s input, and its stored output becomes the item’s expected_output;
• the item’s metadata is {trace_id, span_id, model, environment}, and its source fields are set.

The recorded output is only a starting point. It is what your application said, not necessarily what it should have said. Edit the expected output of each item before you rely on it.

A trace that gives no item is listed in rejected with a reason and does not stop the others:

Reason	Meaning
trace not found	The id is not a trace of this project
no llm span	The trace has no LLM span to copy
span not found	You named a span the trace does not have
no payload stored	The span has no stored input, because the project does not capture payloads
already in this dataset	The dataset already has an item copied from this span of this trace. A second copy would count the same case twice in every run

You can add up to 200 traces in one request. The copied text is what was stored at ingestion, with secrets already redacted.

The same trace with a different span is a different case, so it is added. Items added as JSON or from a file have no source trace and are not checked for repeats.

Traces already in another dataset

A trace can reach two datasets of the project, for example a tuning set and a holdout set, which makes the holdout score look better than it is. Spanlight adds the item, and the answer says where else the trace already is: also_in has one entry per added trace that already has an item (from any span) in another dataset of the project.

{
  "accepted": 2,
  "rejected": [],
  "item_ids": ["<item id>", "<item id>"],
  "also_in": [
    {
      "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
      "datasets": [{ "id": "<dataset id>", "name": "support-holdout" }]
    }
  ]
}

also_in is [] when no added trace is in another dataset. If the item should not be in this dataset, delete it. Only datasets of the same project are checked.

Upload a file

Upload a .jsonl or .csv file from the dataset page, or POST it as multipart/form-data to /api/v1/projects/{project_id}/datasets/{dataset_id}/items/upload.

• Format. UTF-8 text (a byte order mark is removed). Any other extension or encoding answers 415 UNSUPPORTED_FILE.
• Size. At most 5 MB. A larger file answers 413 PAYLOAD_TOO_LARGE.
• A bad line never fails the file. Each line or record that cannot be used is reported with its line number and a reason, and everything else is accepted. The reason says what is wrong (“not valid JSON”, “input is required”) and never repeats the content of the line. At most the first 100 problems are listed.
• Parsing stops once the file holds more items than a dataset can, and the upload is then refused as DATASET_FULL.

JSONL

One JSON object per line. input is required, expected_output is optional, and every other key goes into metadata (an object under metadata is used as the base). Blank lines are skipped.

{"input": "How do refunds work?", "expected_output": "Refunds go back to the original payment method.", "topic": "billing"}
{"input": {"question": "Can I change my plan?", "tier": "pro"}}
{"input": "Where is my invoice?", "metadata": {"topic": "billing", "priority": 2}}

CSV

The first row is a header and must contain an input column. expected_output is optional, and every other column goes into metadata as a string. Quoted commas and line breaks work, and blank rows are skipped.

input,expected_output,topic
"How do refunds work?","Refunds go back to the original payment method.",billing
"{""question"": ""Can I change my plan?"", ""tier"": ""pro""}",,account

A cell in input or expected_output that holds a JSON object or array is stored as that value; anything else is stored as text. In the second row above, input becomes an object. A record that cannot be read at all (for example a field that is never closed) is reported and ends the file there, because everything after it cannot be trusted.

Add items by API

POST .../datasets/{dataset_id}/items takes {"items": [{"input", "expected_output"?, "metadata"?}, ...]} with 1 to 1000 items. POST .../datasets/{dataset_id}/items/from-traces takes {"trace_ids": [...], "span_id"?}. Both need datasets:write and answer 201 with accepted (and rejected for traces).

Route (under /api/v1/projects/{project_id})	Needs	Does
GET /datasets	project:read	List; name returns the dataset with exactly that name
POST /datasets	datasets:write	Create (409 NAME_TAKEN)
GET, PATCH, DELETE /datasets/{dataset_id}	read, datasets:write	Read, change the description, delete
GET /datasets/{dataset_id}/items	project:read	Items by id, oldest first
PATCH, DELETE /datasets/{dataset_id}/items/{item_id}	datasets:write	Edit expected_output and metadata, or delete an item
POST /datasets/{dataset_id}/lock, POST /datasets/{dataset_id}/unlock	datasets:lock	Lock or unlock the dataset (see Lock a dataset)

The dataset’s name is its handle in the command line and in the eval gate: they look it up with GET /datasets?name=....

Lock a dataset

Lock a dataset that must not change, such as a holdout set you report scores on. While it is locked, every change to its items is refused with 409 DATASET_LOCKED and the message “This dataset is locked. An admin can unlock it.”:

• adding items as JSON, from a file or from traces;
• editing or deleting an item;
• deleting the dataset.

Reading the dataset and its items, changing its description, and running experiments on it still work. A locked dataset cannot change while a run is reading it.

Only admins and owners can lock and unlock (the datasets:lock permission), so a member cannot undo the lock. POST /api/v1/projects/{project_id}/datasets/{dataset_id}/lock and .../unlock both answer 200 with the dataset, whose locked_at is the time it was locked, or null when it is unlocked. Both are safe to repeat: locking a locked dataset keeps its first locked_at. A personal access token with only the read scope cannot use them. Each lock and unlock is recorded in the audit log as dataset.lock or dataset.unlock.

Check it worked

After an upload, the dialog “Upload finished” shows two counts, Accepted and Rejected. Close it and open the dataset: the items card lists the accepted items instead of “This dataset has no items”, and a capacity bar shows how full the dataset is. After you add traces, the toast reads “Added to” followed by the dataset name.

Troubleshooting

• “This file type is not supported”. The dashboard checks the file before it sends anything. Only .jsonl and .csv files can be uploaded. Convert the file and try again. (The API answers 415 UNSUPPORTED_FILE for a wrong type or a file that is not UTF-8 text.)
• “This file is larger than 5 MB”. Split it into smaller files and upload each one. (The API answers 413 PAYLOAD_TOO_LARGE.)
• “This file is not UTF-8 text”. Save the file with UTF-8 encoding and try again.
• “This file can’t be read as a dataset”. The line below the title says why, for example “The header row needs an input column.” for a CSV or “The file has no items. Each line needs an input.” for an empty file.
• “This dataset is full.” A dataset holds up to 10,000 items (422 DATASET_FULL), and the whole request is refused. Delete items you no longer need, or start a new dataset.
• The Rejected tile is not zero. Each rejected line is listed with its line number and a reason, such as “input is required”. Fix those lines and upload them again. The rest was accepted.
• A trace is missing after “Add to dataset”. The trace is in rejected with a reason such as “No stored payload”. See the reasons in Add items from traces.
• 409 DATASET_LOCKED when you add, edit or delete. The dataset is locked. Ask an admin or owner to unlock it, or add the items to another dataset. See Lock a dataset.

Next steps

ExperimentsReplay a dataset through a prompt and compare two runs.

EvaluatorsScore traces automatically with rules, a judge or your endpoint.

Eval gate for CIFail a pull request when a score drops below a threshold.