Playground

The playground lets you try a prompt before you ship it. Pick a prompt version or write messages, choose a provider credential and a model, and run. You can put up to three configurations next to each other, for example the same prompt on two models, or two prompt versions on one model, and compare the answers, the tokens, the cost and the time they took.

Every run is a real provider call. It spends money on the credential you chose, and it is recorded as a trace like any other call. Nothing is simulated. Members, admins and owners can run the playground; viewers can read the traces it leaves.

Before you start

• The member, admin or owner role in the project. Viewers can read the traces a run leaves
• A provider credential in your organization, added by an owner under Gateway, Credentials
• CREDENTIALS_KEYS set on the server, and the gateway running in the API process (GATEWAY_MODE embedded, the default)

Set up a run

Each configuration has:

Part	Meaning
Credential	A provider credential of your organization (OpenAI, Anthropic or an OpenAI-compatible server). The key is used on the server and never shown.
Model	The model name sent to the provider.
Parameters	temperature (0 to 2), max_tokens (1 to 4096) and top_p (0 to 1). A parameter you leave unset is not sent, so the provider’s default applies. Anthropic requires max_tokens, so an unset value becomes 1024.
Source	Either a prompt version with a value for each of its variables, or typed messages (1 to 50, each with a role of system, user or assistant) that are sent as they are.

A text prompt is sent as one user message. Choosing a version fills the model from its model advice when the model field is empty; you can change it freely.

When a prompt needs variables, fill in their values for each configuration. A configuration whose variables are not all filled fails by itself (missing_variables: question) without calling the provider, and so does one whose filled-in prompt would exceed 256 KB (prompt_too_large).

Once you have a configuration you like, you can save it as a new version of a prompt. That is the ordinary “add a version” call, so a template identical to the latest version is refused.

What a run does

• The calls go out at the same time, one attempt each, with no retries and no fallbacks, so what you see is what that credential and model do.
• Answers are not streamed. The page shows each answer when it is complete, and time to first token is “—”.
• Each result shows the answer, the token usage, the duration and the cost. Cost and tokens are “—” when the model has no price or the provider returned no usage; unknown is never shown as 0.
• One configuration failing does not affect the others. Its column shows the error (a provider failure, a missing variable, or a project budget that blocks the call) and the other columns still show their answers.

Where the traces go

Every configuration leaves a trace in the project:

• environment playground, trace name playground, tag spanlight:playground;
• the trace’s user id is your account id;
• the span records the prompt version when you used one, so it appears in that version’s usage.

Open a result’s trace from the run to see the exact request and response. Evaluators never sample playground traces, so trying prompts does not trigger automatic scoring.

Cost and limits

• The playground has no spend cap of its own. It is a person typing, and each call is bounded by max_tokens, which is at most 4096. The cost still counts: it appears in the project’s spend and in any budget. A budget set to block stops the call, and the column shows BUDGET_EXCEEDED.
• 30 runs a minute per person. A run is one request of up to three configurations. Over the limit the API answers 429 RATE_LIMITED with Retry-After.
• It needs CREDENTIALS_KEYS to be set so credentials can be opened, and the gateway to be running in the API process; otherwise the API answers 409 NOT_CONFIGURED.
• The prompts and answers are never written to Spanlight’s logs. They exist as the trace’s spans, and follow the project’s payload capture and retention settings like any other trace.

From the API

POST /api/v1/projects/{project_id}/playground/runs needs the experiments:run permission (members and above) and accepts an Idempotency-Key.

{
  "configs": [
    {
      "credential_id": "0192f3a1-0000-7000-8000-000000000001",
      "model": "claude-haiku-5-5",
      "params": { "max_tokens": 300 },
      "source": {
        "prompt_version_id": "0192f3a1-5c3e-7b1a-9d2c-8e4f6a7b8c9d",
        "variables": { "question": "How do refunds work?" }
      }
    },
    {
      "credential_id": "0192f3a1-0000-7000-8000-000000000001",
      "model": "claude-haiku-5-5",
      "source": { "messages": [{ "role": "user", "content": "Say hello in French." }] }
    }
  ]
}

The answer has one result per configuration, in the same order:

{
  "results": [
    {
      "output": "Refunds are issued to the original payment method within five days.",
      "usage": { "input_tokens": 212, "output_tokens": 31, "cached_tokens": null },
      "cost_usd": "0.0000367",
      "latency_ms": 842.1,
      "ttft_ms": null,
      "trace_id": "5b8efff798038103d269b633813fc60c",
      "span_id": "eee19b7ec3c1b174",
      "error": null
    },
    {
      "output": null,
      "usage": null,
      "cost_usd": null,
      "latency_ms": null,
      "ttft_ms": null,
      "trace_id": null,
      "span_id": null,
      "error": "BUDGET_EXCEEDED"
    }
  ]
}

cost_usd is a decimal string, or null when unknown. A credential that is not your organization’s, or a prompt version that is not your project’s, answers 404 for the whole request.

Replay a stored request

A replay sends the request a span recorded to the provider once more, as it was, and shows the new answer beside the recorded one. Use it to reproduce a failure a reviewer found, or to check that a fixed provider or model now answers well. Like a run, a replay is a real provider call, recorded as a trace.

Open a span on a trace’s page and select Replay, or Replay failing span while you review a queue. The playground opens a replay column with:

• the stored request, read-only, with Edit to change it as JSON;
• a credential of the span’s provider, each shown with its base URL so you can see where the request goes;
• an optional model of the same provider, and the most output tokens (at most 4096, the stored value by default);
• the project’s replay budget for the month, for example “$1.20 of $5.00 this month”.

Run replay shows the Recorded output and the Replay output side by side, the tokens, cost and duration, and a line that says how faithful the replay was. Save as dataset item keeps the case. The span’s Replays tab lists its earlier replays with who ran them, the model, the fidelity and the cost.

What can be replayed exactly

The span came from	It stored	Replay
The Python SDK (wrap_openai, wrap_anthropic)	The provider request body, without transport options such as the API key or timeouts	Exact
The gateway, including the playground, judges, experiments and replays	The request body as the client sent it	Exact
OTLP	Only the messages: no tools and no parameters	Not exact: the messages, sent to the model the span names. Add tools and parameters yourself with Edit
POST /v1/traces from another client	Whatever it sent	Exact when the input is a request body with a model and messages or input

A span from Anthropic needs an Anthropic credential. A span from OpenAI or any server that speaks its API (azure, groq, ollama and others) needs an OpenAI or OpenAI-compatible credential. The request is never converted from one provider’s format to the other’s: that would rewrite the tools, the system prompt, the stop sequences and the response format, and the result would no longer be the request that failed. To try the messages on another provider, use a playground run instead.

What a replay changes

Every field of the stored request is sent untouched, the tools, tool_choice, response_format, system, stop, temperature, seed and metadata included, except:

Field	Change	Why
stream, stream_options	Removed	A replay is never streamed, so its usage is certain. First token is “—”
n	Set to 1 when above 1	It multiplies the bill
max_tokens, max_completion_tokens, max_output_tokens	Lowered to the smaller of the stored value, your value and 4096	A replay is bounded like a playground run
max_tokens on an Anthropic request that has none	Set to 1024, or your value when lower	Anthropic requires it
model	Replaced when you pick another model

The result says which fields changed. “Exact replay — identical request” appears only when the replayed span’s request hash equals the stored one’s, which holds when nothing but the streaming fields changed. Any other change, an edit included, makes the replay “Modified”, and an OTLP span’s replay is “Not exact”.

When a replay is refused

The Replay button says why a span cannot be replayed, and the API answers 422 with one of these codes. For a redacted or truncated request, Edit and replay lets you repair it first.

Code	The stored span	What to do
REPLAY_NOT_LLM	Is not an LLM call	Replay the LLM span inside it
REPLAY_NO_PAYLOAD	Has no stored request: payload capture was off, or retention removed it	Nothing to send. Turn on payload capture for future calls
REPLAY_PAYLOAD_TRUNCATED	Was cut at the 1 MB payload limit when it was stored	Edit the request into a complete one
REPLAY_PAYLOAD_REDACTED	Holds [REDACTED] or [MAX_DEPTH] where a secret or a too-deep value was removed at ingest	Edit those parts out. Run replay stays disabled while a marker remains
REPLAY_UNSUPPORTED_INPUT	Is not a provider request (a plain text prompt, a body without a model), or an edit is not one, or an OTLP span has no model and none was picked	Pick a model, or fix the edit
REPLAY_PROVIDER_MISMATCH	Needs a credential of the other provider	Pick a credential of the span’s provider
REPLAY_MODEL_UNPRICED	Uses a model with no price	Add a price override, or pick a priced model
REPLAY_BODY_TOO_LARGE	Your edited request is larger than 256 KB	Shorten it

The replay budget

Replays are a paid call anyone with the member role can repeat with one click, so each project has a monthly replay budget, REPLAY_MONTHLY_BUDGET_USD on the server (5 US dollars by default, per UTC month). Before each call Spanlight reserves the most it could cost (the request’s text counted as input tokens, plus every allowed output token) and refuses the replay when that would pass the budget: nothing is sent, the API answers 409 REPLAY_BUDGET_EXCEEDED with the month’s figures, and the column says “Skipped: this month’s replay budget is spent.” After the call the replay is charged its real cost. A call that broke after it was sent is charged the reservation, because the provider may have billed it. Setting the budget to 0 turns replay off (409 NOT_CONFIGURED).

The budget is the replay’s own. The project’s budgets still apply: one set to block stops the call, and the result shows BUDGET_EXCEEDED. Replays also share the limit of 30 playground runs a minute per person.

Where replay traces go

A replay leaves a trace in environment playground, named replay, tagged spanlight:replay, with your account id as its user and the release of the trace it replays. Its span carries spanlight.replay.source_trace_id, spanlight.replay.source_span_id, spanlight.replay.fidelity, spanlight.replay.changed and spanlight.replay.id. Replays are recorded in the audit log with the credential, the model and the cost, never the content. The answer is shown as it came back: a tool call in it is shown, never executed.

Replay from the API

Under /api/v1/projects/{project_id}:

Route	Needs	Does
GET /traces/{trace_id}/spans/{span_id}/replay-source	project:read	What a replay would send (fidelity, surface, provider_kind, body) or the refusal, and the span’s last 10 replays. Calls nothing
POST /playground/replays	experiments:run	Replay a span. Accepts an Idempotency-Key
GET /playground/replays?trace_id&span_id	project:read	Replays, newest first
GET /playground/replays/budget	project:read	monthly_budget_usd, spent_usd, reserved_usd, remaining_usd, month_start

{
  "trace_id": "5b8efff798038103d269b633813fc60c",
  "span_id": "eee19b7ec3c1b174",
  "credential_id": "0192f3a1-0000-7000-8000-000000000001",
  "max_tokens": 1024
}

model and edited_body (the whole request, as a JSON object) are optional too. The answer has replay_id, fidelity (exact, modified or messages_only), changed, same_request_hash, source (the replayed span and its recorded output text), and the output, usage, cost_usd, latency_ms, trace_id, span_id and error of a playground result. same_request_hash is null when the replayed span could not be stored, so the two requests could not be compared. A span outside the project or a credential outside the organization answers 404.

Check it worked

Choose a credential and a model, write a message and select Run. The column shows the answer and a row of figures labelled Tokens, Cost, Latency and First token. First token is “—” because answers are not streamed. Then open Traces and filter the environment to playground: the run left a trace named playground.

Troubleshooting

• A card says “Playground is not set up”. The server has no gateway runtime or no CREDENTIALS_KEYS (409 NOT_CONFIGURED). An operator sets them. See Configuration.
• A column says “Credential rejected” or “Model not found”. The provider refused the stored key (401 or 403), or does not know the model for that credential (404). Check the credential under Gateway, Credentials, and the model name.
• A column says “Rate limited”. The provider answered 429 for that configuration. The other columns are not affected. Run again shortly.
• The columns say “Nothing to run”. A credential or prompt version in the run no longer exists (404). Pick them again and run.
• A column says “Budget reached”. A budget set to block stops the call (BUDGET_EXCEEDED). Raise the budget or wait for the next period. See Budgets.
• A column says “Missing variables”. Every variable of the prompt needs a value in each configuration. Fill in the names it lists and run again.
• A replay is refused with a REPLAY_* code. The table in When a replay is refused says why and what to do.
• “Skipped: this month’s replay budget is spent.” The project’s replays this month have used REPLAY_MONTHLY_BUDGET_USD (409 REPLAY_BUDGET_EXCEEDED). Wait for the next UTC month, or ask an operator to raise it.
• A card says “Too many runs”. The limit is 30 runs a minute for each person (429 RATE_LIMITED). Wait for Retry-After. See Limits.

Next steps

PromptsVersions and labels that you can change without a deploy.

ExperimentsReplay a dataset through a prompt and compare two runs.

Provider credentialsStore provider keys encrypted and rotate the master keys.

Budgets and the weekly digestSpend limits that notify or block gateway calls.