Changelog

All notable changes to Spanlight are recorded here. The format follows Keep a Changelog, and the project is pre-1.0, so minor versions may change behaviour; breaking changes are called out.

[Unreleased]

Added

• Locked datasets. Admins and owners can lock a dataset that must not change, such as a holdout set: POST /api/v1/projects/{project_id}/datasets/{dataset_id}/lock and .../unlock, under the new datasets:lock permission (read-scoped tokens are refused). While a dataset is locked, adding items (as JSON, from a file or from traces), editing or deleting an item and deleting the dataset answer 409 DATASET_LOCKED; reading it, changing its description and running experiments on it still work. Both routes are idempotent, DatasetOut carries locked_at, and each change is audited as dataset.lock or dataset.unlock.
• Runs keep a copy of every item they score. Each experiment item stores the dataset item’s input, expected_output and metadata as the run read them, error items included, and the run items API returns them as input, expected_output and item_metadata, with position, the item’s number within the run. Editing or deleting a dataset item no longer changes what an earlier run shows it was measured against. Items of runs made before this release return null for the copy; they are not filled in from the live dataset.
• Repeated traces are caught when adding from traces. A trace whose span is already an item of the dataset is rejected as already in this dataset instead of being added twice, and the answer’s new also_in lists the other datasets of the project that already hold each added trace, so the same conversation reaching a tuning set and a holdout set is visible.

Changed

• A run scores a fixed set of items. It takes the ids of the dataset’s first max_items items when it starts: items added while it runs are not scored, and items deleted while it runs are skipped. Before, a run read the dataset page by page and could pick up items added mid-run.
• Compare leaves out items edited between the runs. When both runs kept a copy and an item’s input or expected_output differs, the item gets changed: true and no deltas, and is left out of the improved, regressed and unchanged counts and of the significance test; the comparison’s new changed_items counts them. Each run’s means, intervals, cost and latency still cover all of its items. Compared items also carry input, expected_output and position from the runs’ copies. Item metadata is not compared.

Fixed

• Item metadata is kept when adding items as JSON or uploading a file. Both paths stored {} instead of the metadata sent; editing an item’s metadata afterwards already worked.

Upgrading

Run spanlight migrate after upgrading. 1010 adds the copy columns to experiment_items and a CHECK on them (catalog only; the CHECK is then validated with one scan of the table that does not block writes); its downgrade drops the columns and loses every stored copy. 1011 adds datasets.locked_at (nullable, catalog only) and the dataset.lock and dataset.unlock audit actions; its downgrade deletes those audit events and forgets which datasets were locked. 1012 builds the partial index dataset_items_source_trace_idx CONCURRENTLY, which does not block reads or writes but waits for transactions already open on the table to finish; if the build is interrupted, running spanlight migrate again recovers (it drops the invalid leftover first). During a rolling deploy, a run finished by an older worker stores no copy and shows as an older run, and an older api replica can miss a new lock for the seconds both versions run.

[0.7.0] - 2026-10-20

Agent traces and long payloads: compare two traces step by step, search what the model was sent and said, keep inputs and outputs up to 1 MB, and trace Node.js apps with a TypeScript SDK. Dashboards load faster on large projects, and an overloaded api process answers quickly instead of stalling.

Added

• TypeScript SDK (sdks/typescript, package spanlight, not yet on npm). wrapOpenAI traces chat.completions.create and responses.create and wrapAnthropic traces messages.create, plain and streamed, including the providers’ .stream() helpers; observe, withSpan and updateTrace trace your own code, with the current span kept in AsyncLocalStorage. Spans go to POST /v1/traces with the Python SDK’s guarantees: background batching, gzip by default, retries with Retry-After, a bounded queue with loss counters, and errors from your code and the providers reaching it unchanged. Usage is recorded as the Python SDK records it: input_tokens includes cache reads and writes, OpenAI cache_write_tokens and Anthropic cache_creation_input_tokens (with the 1-hour part) are kept, the service tier is normalized with the shared table, and a stream without its usage leaves the output count unknown. Node.js 22 or newer.
• Agent framework guides with recorded fixtures. New integration pages for the OpenAI Agents SDK, LangGraph, Pydantic AI and LlamaIndex, each backed by a recorded OTLP fixture of a small agent run (one tool round trip, two model calls) through the framework’s own OpenTelemetry instrumentation: openinference-openai-agents 2.5.3, openinference-langgraph (LangGraph 1.2.16), pydantic-ai 2.57.0 and openinference-llama-index 4.6.0. The fixtures are on the OTLP compatibility page and in the weekly matrix.
• Search what the model said. The Traces search box (and q on GET /api/v1/projects/{id}/traces and exports) now also matches span content: with three or more characters it searches the end of each span’s input and the start of its output (2,000 characters of each) for spans in the selected time range. A pg_trgm index (migration 1007, built concurrently) answers it. A search that cannot finish within 5 seconds, usually a common word over a long range, answers 422 SEARCH_TOO_BROAD with a hint to narrow the range or the text, instead of a 503 the dashboard retried. The index is only usable under row-level security when the migration runs as a Postgres superuser (the bundled Compose and Railway setups do); otherwise content matching covers the last hour of the selected range, and names and ids still cover all of it.
• Trace compare. Select two traces and choose Compare to see them step by step: spans are paired by name and order, with the change in duration, tokens and cost, whether the input and output changed, and steps that exist or failed on one side only. Also GET /api/v1/projects/{project_id}/traces/compare?a=&b=.
• agent span kind. OpenTelemetry GenAI invoke_agent and create_agent, OpenInference AGENT and Traceloop agent spans are now agent instead of chain, with their own colour and icon in the waterfall. Native ingestion accepts "kind": "agent". Spans stored earlier keep chain. Migration 1006 adds the enum value; Postgres cannot remove it, so the downgrade sets agent spans back to chain instead.

Changed

• Two unusable indexes dropped (migration 1009). traces_tags_idx and insights_evidence_trace_ids_idx serve only @> and &&, which row-level security never lets Postgres turn into index conditions for the app’s role, so they were never read while every insert paid to maintain them. Tag filters keep running on the time window’s traces as before. Dropped concurrently, so ingestion keeps writing during the migration; downgrade rebuilds both.
• Inputs and outputs are kept up to 1 MB each (was 32 KB). Long RAG contexts and agent histories are stored and shown whole; only a payload over 1 MiB of JSON (after redaction) is still cut to its first 1 MiB and marked truncated. The Python SDK (0.2.1) keeps input and output strings up to 1,000,000 characters (was 32,000); attribute strings stay at 32,000, and a cut attribute now marks the span truncated too. Request limits are unchanged (5 MB per body, 1,000 spans per batch). Migration 1008 adds spans.payload_bytes, the stored size of a span’s payloads.
• Trace detail returns at most 8 MB of payloads. Spans past that point in a very large trace come back with payload_omitted: true and no input or output; the new GET /api/v1/projects/{project_id}/traces/{trace_id}/spans/{span_id} returns one span whole, and the dashboard loads it when the span is opened. Long texts open at their first 20,000 characters and large JSON at its first 200 entries, with Show all.
• Memory stays bounded with larger payloads. The gateway’s write backlog is also bounded by payload bytes (GATEWAY_RECORD_BACKLOG_MB, default 128), and JSONL exports cut a batch at 64 MiB of payloads as well as at 2,000 spans.
• Dashboard metrics read rollups for every window. The overview, time series and per-model views read whole, rolled-up hours from the hourly rollups and only the partial edges and the most recent hour or so from the spans, for windows of any length. A 24-hour dashboard on a project with 2 M spans a day loads about ten times faster. Counts, tokens and cost are exact (windows over 24 hours no longer count their first and last hour in full); p50/p95 are histogram estimates (approximate: true) whenever a rollup hour is read, so they stay exact only for the last hour. The rollup job now rewrites the last three hours every five minutes and the rest of the last 48 hours about once an hour, so a span that arrives more than two hours late can take up to an hour to show in charts.
• An overloaded api process answers at once instead of queueing. Past API_MAX_IN_FLIGHT requests in flight (default 64) on /api and /v1, a request is answered 503 SERVICE_UNAVAILABLE with Retry-After: 1 without touching the database; the SDKs back off on it. /gw and /health are not counted. API_POOL_TIMEOUT_SECONDS now defaults to 1 second (was 5).
• API keys are cached per api process for 10 seconds (API_KEY_CACHE_TTL_SECONDS). The secret, revocation and expiry are still checked on every request against the cached row; a key revoked through another api process keeps working on this one for up to that long. A revocation on the same process takes effect at once, and 0 turns the cache off.
• A lost database connection answers 503 instead of 500. The api no longer tests pooled connections before use, so after a Postgres restart the first request on each stale connection answers 503 SERVICE_UNAVAILABLE with Retry-After: 5 and logs db_connection_lost, once per restart.
• Pages follow your role. A role that can read a page but not change it sees one view-only note and no write controls; a page it cannot read shows a no-access page on a direct link. A viewer opening an annotation queue no longer claims items it cannot score.
• Secrets are redacted in span and trace names as well as in payloads. Ids, tags and the release stay as sent.
• Implausible token counts are not priced. A span with more than 100,000,000 input or output tokens is stored with a NULL cost, so one bad value cannot inflate spend, budgets and projections.
• LLM judges answer on thinking models. Claude models that think by default get output_config.effort: "low", the default judge max_tokens is 1024 (was 256), and an answer cut off at max_tokens fails as judge_truncated instead of judge_no_answer.

Fixed

• Python SDK: tracing no longer raises into your code or loses whole traces on bad values (Python SDK 0.2.1). with spanlight.span(...) re-raises an exception whose __str__ raises unchanged, instead of a RuntimeError from the SDK. set_usage ignores counts that are not whole numbers from 0 to the server’s limit, record_error accepts any value, and set_attribute(s) and span(attributes=) ignore what they cannot use, all instead of raising. Trace fields the server would reject, which made it drop every span of the trace, are made to fit: "" is unset, non-strings become text, name, user_id and session_id are cut to 256 characters, environment to 64 and release to 128, tags="beta" is one tag (it was four), and a trace keeps 20 tags of up to 64 characters, with one warning per kind of fix. A span that cannot be encoded as JSON is dropped alone and counted as failed, instead of losing its whole batch. wrap_anthropic and wrap_openai also trace the clients that with_options(...) and copy(...) return, and wrap_anthropic traces messages.parse and client.beta.messages (create, parse, stream and batch results), so fast mode, compaction and the tool runner record spans; they recorded none. A forked child that only calls flush() no longer waits on the parent’s in-flight batch, and an @observe async generator left by async for ... break is no longer recorded as failed with CancelledError when the event loop shuts down.
• LlamaIndex prompt-building spans no longer mark a trace as unpriced. LlamaIndex labels OpenAI._prepare_chat_with_tools an OpenInference LLM span although it makes no model call. An OpenInference LLM span with only input.value (no messages and no token counts) is now stored as a chain with no model.
• A session’s cost is marked as a lower bound when one of its traces has an unpriced call (has_unpriced on the session list and a user’s recent sessions), as trace costs already were.
• The release comparison no longer lists repeated provider errors as new. Long letter-and-digit ids, such as Anthropic request ids, are normalised like UUIDs before messages are compared.
• API edge cases. A pagination cursor holding a lone surrogate or NUL is a 422 (was a 500); OTLP JSON intValue that is not an integer string is 400 INVALID_OTLP (it was stored as 0); gateway paths with a trailing slash answer the provider-shaped 404 instead of a bare 307.
• Gateway cache. The cache key ignores user only on OpenAI surfaces, so a cached answer no longer serves an Anthropic request the provider would reject, and a non-200 answer reports bypass, not miss.
• Dashboard. User-supplied names and ids are isolated from right-to-left overrides; sessions named . or .. open; money over $1,000 has thousands separators; calls with no provider (cache hits, refused calls) are labelled “No provider”; empty thinking blocks are hidden; a Doctor finding resolved by hand no longer claims “No recurrence for 24 hours”.

Upgrading

Run spanlight migrate (or restart the stack) after upgrading, as a Postgres superuser (the bundled postgres user is one). Migration 1004 replaces the windowed span index with one that also carries usage_rollup, built CONCURRENTLY before the old one is dropped. 1005 sets fillfactor = 70 and tighter autovacuum on traces; it rewrites nothing and fails cleanly after 5 seconds if an autovacuum holds the table (run migrate again). 1006 adds the agent span kind (catalog only; earlier spans keep chain). 1007 adds the pg_trgm extension and the content search index, built CONCURRENTLY (about 24 minutes and 1.35 GB on a 4.2 M-span table); as a non-superuser it skips the LEAKPROOF marks and content search covers the last hour of a range only. 1008 adds spans.payload_bytes (nullable, catalog only). 1009 drops traces_tags_idx and insights_evidence_trace_ids_idx CONCURRENTLY; ingestion keeps writing, and the downgrade rebuilds them. Set SPANLIGHT_IMAGE and SPANLIGHT_WEB_IMAGE to the 0.7.0 images.

[0.6.0] - 2026-10-19

The first tagged server release: images on GHCR, and every change recorded under 0.2.0 to 0.5.0 below ships in it.

Prompts, datasets and evals: manage prompts as versions, score traces by hand or automatically, and compare a change on your own test cases before it ships.

Added

• Prompt registry. Named prompts, text or chat, with immutable numbered versions and movable labels such as production. Templates use {{variable}} and are checked identically by the server, the dashboard and the SDK. A prompt can get 100 versions a day, saving an identical version is refused, and a prompt that an evaluator or experiment uses cannot be deleted. Each version shows its use over the last 7 days (traces, LLM calls, cost, p95 duration).
• Prompts for your application. GET /v1/prompts/{name}?label= or ?version= with an API key that has the new prompts:read scope, answered with an ETag so an unchanged prompt is a 304. Spans can carry the id of the prompt version they used (prompt_version_id), Traces can be filtered by it, and trace detail links to it.
• Playground. Run a prompt version or typed messages on up to three credential and model configurations side by side. Every run is a real provider call, recorded as a trace in the playground environment; the playground has no spend cap of its own beyond the project’s blocking budgets, and is limited to 30 runs a minute per person.
• Scores. Numeric, boolean and categorical scores on a trace or a span, from the API (POST /v1/scores, scope scores:write, up to 1000 per request with per-score rejection), by hand on the trace page, or from evaluators. Score configs (type, range, categories) are optional: the first score with a name creates an implicit one. Traces can be filtered by score (score_name with score_min, score_max or score_value, and has_scores), and the trace list shows each trace’s latest scores.
• Annotation queues. Queue traces for review under Review. Reviewers claim the next item (an abandoned claim is freed after 30 minutes), give one score per config of the queue, skip or release it, and can work entirely from the keyboard.
• Datasets. Build a dataset from traces or upload .jsonl and .csv files (up to 5 MB and 10 000 items per dataset), with bad lines reported by line number instead of failing the file. Items can be edited, and the recorded output of a trace is offered only as a starting point for the expected output.
• Evaluators. Score traces automatically: heuristic rules (regex, json_valid, length, contains, exact_match, levenshtein), an LLM judge with four built-in templates (relevance, faithfulness, toxicity, format_compliance; every score is 0 to 1, higher is better) or your own chat prompt, and signed webhook evaluators. Online sampling runs every minute over traces that ended 30 seconds to 5 minutes ago, never samples Spanlight’s own traces (evals, playground, experiments and doctor environments, and judge calls), and queues each trace at most once.
• Judge budgets. A judge needs a monthly budget and reserves the most a call could cost before it makes the call, under a lock, so concurrent runs cannot both pass the limit; a run that does not fit is marked budget_exceeded and sends no request. The budget covers sampled runs, tests and experiment use, and a meter shows spent, reserved and remaining.
• Experiments. Replay a dataset through a prompt version or typed messages, score every answer, and compare two runs item by item with deltas, an improved and regressed count, cost and p95 duration. A run can override the prompt version, so one experiment compares versions of a prompt. A cost limit reserves the estimate of every call in flight, so a run never starts an item that could pass it.
• Eval gate. The eval-gate GitHub Action and python -m spanlight._experiments run a prompt over a dataset in CI and fail the build when a mean score falls below a threshold.
• Demo. spanlight demo seed-evals creates a prompt with two versions, a 50-item dataset from the demo’s recorded traffic, a budgeted judge and a two-run experiment.
• Screens and navigation. Prompts, Playground, Datasets, Review, Evaluators and Experiments under a new Improve group in the sidebar (G P, G L, G N, G Q, G E, G X), scores on trace and span detail, and a Scores column on Traces.
• Roles and audit. Members can create prompts, score configs, datasets, annotation queues and experiments, add scores and start experiment runs; evaluators are admin and owner only, because a judge spends money unattended. The main changes to them are recorded in the audit log (prompt.*, score_config.*, dataset.*, annotation_queue.*, evaluator.*, experiment.*).
• Operations. New Prometheus metrics (spanlight_scores_written_total, spanlight_prompt_reads_total, spanlight_evaluator_runs_total, spanlight_evaluator_cost_usd_total, spanlight_experiment_items_total) feed an Evaluations row on the Grafana dashboard. Prices include claude-haiku-5-5, the model the demo uses.
• Prompt-cache writes and prompt-size tiers are priced. Anthropic bills a cache write at 1.25 times the input rate for a 5-minute cache and 2 times for 1 hour, and bills Claude Haiku 5.5 prompts over 100,000 tokens at a higher row of rates; Spanlight priced both as plain input, up to 80 % under the invoice on real calls. A span’s usage now carries cache_write_tokens and cache_write_1h_tokens (the part written for 1 hour), both optional, with input_tokens still the whole prompt. The Python SDK (Span.set_usage(..., cache_write_tokens=, cache_write_1h_tokens=), filled by wrap_anthropic; the attribute usage.cache_creation_input_tokens is no longer set), the gateway (Anthropic answers and streams, and spanlight.cache.original_usage) and OTLP (gen_ai.usage.cache_creation.input_tokens, or the older gen_ai.usage.cache_creation_input_tokens; OpenTelemetry has no lifetime split, so OTLP writes are priced as 5-minute) all report them, and OTLP also reads cache reads from gen_ai.usage.cache_read.input_tokens, the name current instrumentations and LiteLLM use, before the older gen_ai.usage.cache_read_input_tokens. Price table 2026-10-15 adds the write rates to every Anthropic row and a second claude-haiku-5-5 row for prompts over 100,000 tokens (min_prompt_tokens 100001); the 2026-10-01 table is kept beside it so older spans price as before. GET /api/v1/prices and price overrides carry cache_write_5m_per_mtok, cache_write_1h_per_mtok and min_prompt_tokens; an override is one tier, so overriding the base row of a tiered model leaves the built-in higher tier in force. Invalid combinations reject the span (cached_tokens + cache_write_tokens over input_tokens, cache_write_1h_tokens over cache_write_tokens or sent alone). The span Usage card shows a Cache writes row with the 1-hour part. Migration 0508; its downgrade deletes the tier rows and drops the write rates, and an upgrade after it leaves the surviving built-in rows without write rates, so read the upgrade guide before rolling back below it.
• Service tiers are priced. A span records the tier the provider billed (standard, batch, priority or flex) in service_tier, and its token cost is multiplied by the tier’s factor from the price snapshot: batch at half price for Anthropic, OpenAI and Google, OpenAI flex at half, Anthropic priority at the standard rate and marked commitment billed. OpenAI priority has no single factor (OpenAI publishes a rate per model), so it is priced at the standard rate and marked unmodelled. Native JSON takes usage.service_tier and refuses any other value; the Python SDK, the gateway and OTLP (openai.response.service_tier, gen_ai.openai.response.service_tier, spanlight.usage.service_tier) map the provider’s own spelling and keep one they do not know in usage.service_tier_raw, priced as standard. Data residency (inference_geo) and fast mode are not priced yet: the SDK and the gateway price such a call at the standard rate and set usage.pricing_modifier_unknown.
• Web search fees are priced. Anthropic and OpenAI bill $10.00 per 1,000 web searches. A span counts them in web_search_requests (Anthropic server_tool_use.web_search_requests, the completed web_search_call items of an OpenAI Responses answer, usage.server_tool_use.web_search_requests in native JSON, spanlight.usage.web_search_requests over OTLP), and the fee is added to cost_usd and reported again in the new fee_usd. A fee is kept even when the token cost is unknown, without becoming the span’s cost.
• Streams cut before their usage are unknown, not cheap. A stream that ends without the provider’s final usage event, from the SDK or the gateway, keeps its input counts, stores output_tokens and the cost as unknown and says why in usage.unreported_reason (client_disconnected, stream_error, stream_cut_by_fault, include_usage_not_set or no_final_usage). The trace counts the span as unpriced.
• Cost explanation. Every model call’s span has a Cost breakdown on the trace page, and GET /api/v1/projects/{project_id}/traces/{trace_id}/spans/{span_id}/cost-explanation returns it: one line per token class and fee with its quantity, rate, tier factor and amount, the price row that was used, and a status (ok, mismatch, no_usage, no_price, price_version_missing, override_deleted, cache_hit) that compares the rebuilt figure with the stored one. It never prices the span again with today’s table. Readable by members and by API keys with traces:read.
• spanlight prices diff compares the bundled price snapshot with LiteLLM’s public price file and lists rates that differ, models priced only here, and chat models Spanlight does not price yet, with a proposed row for each. It changes nothing; every figure is checked against the provider’s page before it goes into a snapshot.
• Price table 2026-10-19: OpenAI coverage. Adds the GPT-6 and GPT-5.x families, the o-series, chat-latest, gpt-4.1-nano and the legacy OpenAI models, 56 OpenAI rows in all, with a second row for prompts over 272,000 tokens where OpenAI bills one. GPT-5.6 and later bill cache writes at 1.25 times the input rate: the Python SDK (wrap_openai) and the gateway now record OpenAI’s cache_write_tokens, so those writes are priced instead of counted as plain input. gpt-4o-2024-05-13 is priced by its own row; it was priced at the gpt-4o rate, half what OpenAI bills. The cost breakdown labels the default cache-write line “Cache write”.
• Price table 2026-10-16. Adds Gemini (google) prices, with a second row above 200,000 prompt tokens for the Pro models, Claude Fable 5.1 and Mythos 5.1, Claude Fable 5 and Mythos 5, Claude Opus 4 and 4.1, Claude Sonnet 4 and Claude 3.5 Haiku, and carries the service-tier factors and the web search fees. It corrects the Claude Sonnet 5.5 cache-read rate to $0.10 (it was $0.20); spans priced under 2026-10-15 keep their stored cost. The Prices page lists the tier factors and fees.
• Provider aliases for pricing. A span’s provider is folded onto the name the price table uses before the price is looked up: gemini, vertex_ai and the other Google spellings find the google prices, and claude the anthropic ones. A price override saved under any spelling (gemini, vertex_ai, google) matches spans sent under every spelling of that provider. azure and azure_openai are not folded onto openai, because Azure bills its own rates: an Azure span stays unpriced unless an override for its provider name prices it. The stored provider does not change.
• Batch results in the Python SDK. wrap_anthropic records the results you read with messages.batches.results, and the new spanlight.record_openai_batch_output(lines, batch_id=...) records an OpenAI Batch output or error file. Each request becomes an llm span at the batch rate in a trace named batch <batch_id>; the ids come from the batch and request ids, so reading the results again updates the same spans.
• Billing sources. Owners and admins can connect a provider account to an organization under /api/v1/orgs/{org_id}/billing-sources: an Anthropic Admin API key, an OpenAI admin key, or a CSV source, mapped to the projects it pays for (one account per provider per project). The admin key is encrypted with CREDENTIALS_KEYS, checked once with the provider when it is added or replaced, and never returned, logged or audited; spanlight reseal-credentials re-seals it with the provider credentials. A new permission, billing_sources:manage, covers the changes; any member can read the sources.
• Reconciliation with the provider’s bill. A source with an admin key is fetched every day from 03:30 UTC (the last 7 days, or 31 on its first fetch) and can be refreshed on demand once every 5 minutes (POST …/billing-sources/{id}/refresh). A CSV source takes uploads in a documented ten-column format (POST …/upload, at most 5 MB and 100,000 rows; a refused file lists up to 20 reasons by row and column and stores nothing; each upload replaces the provider’s figures for every day from its first to its last). GET …/billing-sources/{id}/reconciliation, also as .csv, compares the provider’s amount with Spanlight’s figures for the mapped projects per UTC day and model group, over up to 31 days: the signed residual, the explained lines (batch spend, web search fees, untraced traffic, Anthropic priority commitment, unpriced calls, unreported usage, unmodelled pricing modifiers, spans past retention), credits and charges Spanlight cannot see listed apart, and days marked provisional until 48 hours after they end and a later fetch. Spanlight’s side comes from new daily cost rollups, recomputed every 15 minutes for the last three days and nightly for the last eight, never from a scan of the spans. GET …/runs lists the fetches and uploads. The guide “Reconcile cost with your provider bill” covers keys, scopes, project mapping, reading the report and the CSV template.
• Provider health. A new project page, after the Doctor, tells a provider’s outage or limit from your own bug or the gateway’s limiter. Per provider: LLM calls, error rate, the share of failed calls the provider refused with a 429, and the most common kind of refusal; failed calls per hour or day by error class; the rate-limit kinds with how many carried a retry-after; what retries cost; the 20 newest failed calls with the provider’s error type, code and request id; and the incidents on the providers’ own status pages. The API is GET /api/v1/projects/{project_id}/providers/health, …/providers/errors/timeseries and …/providers/errors/samples, readable by every role.
• Provider error fields and rate-limit kinds. Every failed span records the provider’s error type and code (provider_error_type, provider_error_code) and, for a capacity refusal, which limit refused it (rate_limit_kind): requests, tokens, spend_cap, acceleration, overloaded, gateway (the gateway key’s own limit, never the provider’s) or unknown. The kind comes from the provider’s documented error types, codes and messages, with its rate-limit headers as a last resort; a successful span never gets one. The carriers are span attributes (provider.error.type, provider.error.code, provider.request_id, provider.retry_after_s, provider.ratelimit) that the gateway and the Python SDK set and any client may send; the server filters provider.ratelimit through an allowlist of 29 rate-limit headers, each value at most 64 printable characters, so no other header is ever stored. OTLP spans get a kind from the status message. Spans stored earlier keep these fields empty.
• Gateway attempt log and rate-limit headers. A gateway span records the status the gateway answered with (spanlight.gateway.status), its own error code when it made the error (spanlight.gateway.error_code), the provider’s request id and rate-limit headers (all of them on a failure, the limit and remaining ones on a success), and spanlight.gateway.attempt_log: one entry per attempt with the target, credential name, status, error, duration and Retry-After, at most 10 (the first 9 and the last). Nothing new is read from the database on the gateway’s path.
• Retry cost. Traces carry retry_calls and retry_cost_usd: within a trace, every call of the same request but the last is a retry, and its cost is retry cost. The cost is unknown, never zero, when a retried call has no price, and both are unknown for traces stored before this release until a span of theirs is sent again. Traces can be filtered with has_retries and sorted with sort=retry_cost_usd; the trace list has a Retries column and the trace header a retries line.
• Provider status page incidents. The worker polls the status pages of Anthropic, OpenAI and Google (Vertex AI and Gemini) every 5 minutes, from fixed addresses, and keeps their incidents for two years. GET /api/v1/projects/{project_id}/provider-incidents lists those that overlap a window, with each feed’s poll state; the Overview’s calls and errors chart can shade the incidents of the providers you used, and the Doctor’s provider_incident finding names the status page incident that overlaps it. An incident only explains: it never opens, closes or silences a finding or an alert. Air-gapped servers switch polling off with PROVIDER_STATUS_POLLING_ENABLED=false.
• Telemetry completeness. Spanlight counts the spans lost before they were stored: gateway spans its recorder could not keep (overflow, error, rejected), spans ingestion rejected, by reason, and spans the Python SDK dropped or could not deliver, which it now reports. GET /api/v1/projects/{project_id}/telemetry-completeness and a new Overview card give the share of spans that were stored, as a lower bound.
• “Was this right?” on Doctor insights. Members, admins and owners can mark an insight right or wrong, with an optional note (PUT and DELETE /api/v1/projects/{project_id}/insights/{insight_id}/feedback, new permission insights:feedback). The Doctor’s new Precision tab and GET …/insights/precision show, per detector, the votes, the precision and its Wilson 95 % interval, and whether it meets the target of 0.8 with a lower bound of 0.6 on 20 votes; spanlight doctor precision prints the same table across every project for operators.
• Alert rules on error classes and kinds. A new metric, error_calls, counts failed LLM calls, and error_rate and error_calls accept the filters error_class and rate_limit_kind, so a rule can watch, for example, any spend_cap refusal.
• Fall back on a rejected key, if you choose. A gateway route can list auth_error in fallback.on to move to the next target when the provider answers 401 or 403. It is off by default, because another credential answering for a rejected key hides a revoked key, and it needs two targets on different credentials.
• The Doctor’s accuracy harness is in the repository. backend/accuracy/ holds the scenarios and scoring that measure the provider-path detectors against planted problems; its offline tests run in CI, and a live run is manual because it makes paid provider calls.
• Open core. Code under directories named ee is under the new Spanlight Enterprise License and runs only with an offline-verified license key in EE_LICENSE_KEY; everything else stays MIT, and nothing that was MIT has moved. The directories hold no features yet. Without a key the application runs as the community edition; a key that does not verify stops startup, an expired one turns its features off (403 EE_LICENSE_REQUIRED), and spanlight license-status shows what a key grants. See ADR 16.
• Trace metadata. A trace can carry up to 32 key-value pairs, such as a customer, a feature or a team: trace.metadata in native JSON (strict: an invalid entry rejects the span with a reason naming the key), spanlight.metadata.<key> span or resource attributes over OTLP, the x-spanlight-metadata header on gateway calls, and update_trace(metadata=) in the Python SDK (the last three drop an invalid entry and keep the span). Keys are 1 to 64 characters of a-z, 0-9, _, . and -; values 1 to 256 characters, numbers and booleans stored as text. Values merge per key, the last one winning, a key the trace has can always be updated, new keys are added in code-point order while there is room, and nothing is ever deleted, so a replay changes nothing. Values are redacted like payloads. The trace list shows a chip per promoted key, filters by metadata_key and metadata_value (the key alone finds the traces without it) and by exact name, and the trace header lists every pair; GET .../filters lists the metadata_keys and trace_names of the last 7 days.
• Attribution. A new project page (G C) and GET /api/v1/projects/{project_id}/attribution group spend, LLM calls, errors, unpriced calls, tokens, retries and retry cost by a metadata key, trace name, end user or release, with the traces that lack the dimension (none) and the window’s totals on every page; the rows always add up to the totals. A window of 24 hours or less is read live from the traces for any key; a longer one comes from daily rollups, refreshed every 15 minutes, which keep up to 1,000 values per UTC day, environment and key and fold the rest into other. Admins promote up to five keys in project settings (promoted_metadata_keys); a new key is backfilled over the last 7 days, and an unpromoted key over more than 24 hours is 422 KEY_NOT_PROMOTED. Each row links to its traces. spanlight attribution backfill rebuilds older days.
• Attribution export. POST .../exports with "kind": "attribution" writes the attribution table of a window as CSV, through the same queued export flow as traces. Trace exports gain the metadata filter, and trace files a metadata column (CSV) or object (JSONL).
• Budgets on a metadata pair or a trace name. Budgets take the scopes metadata (scope_id is key=value, such as customer_id=acme) and trace_name, and scope_id may be 400 characters. A blocking budget matches a gateway call by its x-spanlight-metadata and new x-spanlight-trace-name headers, and for a call that joins a trace, by the stored trace when a header is missing (best effort until the trace’s first span is stored). The key does not need to be promoted.
• OTLP reads OpenInference, the Vercel AI SDK and Traceloop. Besides the OpenTelemetry GenAI attributes (gen_ai.*, also written by LiteLLM), POST /v1/otlp/traces now reads OpenInference (llm.*, input.value), the Vercel AI SDK’s own attributes (ai.*, from ai 5 and 6 and the legacy telemetry of 7) and Traceloop’s workflow spans (traceloop.*): provider, model, token counts, prompts and completions (flattened llm.input_messages.* and gen_ai.prompt.* messages are rebuilt into lists of at most 256 messages), kind, finish reason, time to first token, trace name and trace metadata. The convention is detected per span and stored as the attribute spanlight.convention, the convention’s own kind as spanlight.operation; a span that mixes conventions has its gaps filled from its gen_ai.* attributes. Every content attribute is removed from the stored attributes, read or not, so prompt text follows the payload capture setting. An embeddings call that reports only input tokens is priced with 0 output tokens and marked usage.output_tokens_implied; a model call whose exporter sent no token count is marked usage.unreported_reason not_exported. The Prometheus counter spanlight_otlp_convention_spans_total{convention} counts accepted spans per convention.
• Usage is counted once. When a span with token counts has a descendant in the same trace with token counts too, as an agent step over its model call or the Vercel AI SDK’s ai.generateText over ai.generateText.doGenerate does, the outer span is a usage rollup (usage_rollup on the span): it keeps everything it was sent, and every total of tokens and cost leaves it out, from trace totals to the Overview, alerts, budgets, the Doctor, Users, prompt usage, attribution and reconciliation. Counts of spans, calls and errors still include it, a rollup is never a retry, and its cost explanation has the status usage_rollup. The rule reads stored spans only, so it holds for the Python SDK, the gateway and OTLP alike and whatever order spans arrive in. The trace page marks such a span “Counted on child spans”, and JSONL trace exports carry usage_rollup on each span.
• OTLP compatibility page and exporter guides. The docs gain a page generated from the fixtures that lists, per library, version and scenario, which fields Spanlight captures and which the exporter does not send, and guides with the tested setup for the Vercel AI SDK, LiteLLM, OpenLLMetry (Traceloop) and OpenInference.
• spanlight recompute-usage-rollups --since 30d flags the usage rollups of traces stored before the upgrade, and retotals them. Without it, an older trace is corrected only when a new span of it arrives.
• Usage state. Spans in the trace API carry usage_state: priced, rollup, usage_not_reported, partial_usage, model_unknown, no_price or not_applicable. The span usage card uses it to say why a cost is unknown, for example “The exporter sent no token counts”.
• Recorded exporter fixtures. What each supported exporter library and version sends is recorded as a fixture under sdks/fixtures/otlp/ and replayed through the mapper by the test suite, against a local fake provider with no provider call or key. A weekly workflow records the latest releases, diffs them with the fixtures and keeps one issue labelled otlp-matrix current when they differ. Recorded so far: the OpenTelemetry OpenAI instrumentation 2.4b0, LiteLLM 1.105.1, OpenInference for OpenAI 0.1.65 and LangChain 0.1.79, Traceloop 0.62.4, and the Vercel AI SDK 5.0.0, 6.0.0 and 7.0.142.
• Daily review. A review policy (one per project, under Review) fills an annotation queue every UTC night from 01:00 with the traces of the previous day worth reading, each with its reason: an open Doctor finding, an error, a cost outlier (above the p99 cost of its trace name over 7 days, from 30 priced traces) or a random share of up to 5 %. A daily cap (1 to 500) keeps a quarter of its places for random picks, picks are deterministic and never duplicated, and a completed day can be selected by hand from yesterday back to 7 days. Queue items carry reason and reason_detail (numbers and ids, never payload text), queues show progress per reason and filter by it, and the annotate screen says why each trace is there, with R to replay its failing span and D to save it as a dataset item that records why it was kept. Routes: GET, PUT, DELETE /review-policy, POST /review-policy/runs, GET /review-policy/selections. New problem codes: QUEUE_LACKS_FAILURE_CATEGORY, NOTHING_SELECTED, DAY_NOT_COMPLETE, DAY_TOO_OLD, ALREADY_SELECTED, and QUEUE_IN_USE when a policy’s queue is deleted.
• Failure categories. A failure category is a categorical score named failure_category, created with POST /score-configs/failure-category (2 to 30 categories, none required) from an editable starting list. A review policy’s queue must ask for it. GET /releases/failure-categories and a card on Releases show each category’s rate per release over the reviewed traces, with Wilson 95 % intervals; a release nobody reviewed has no rate rather than 0 %. Readable with a project API key that has traces:read.
• Exact replay. Send a span’s stored request to the provider again from the span panel, the annotate screen or the playground, and compare the recorded and the new answer. SDK and gateway spans replay exactly; OTLP spans, which store only messages, replay as “not exact”. A replay is never streamed, sends n = 1 and at most 4096 output tokens, refuses another provider’s credential instead of converting the request, and refuses redacted, truncated or missing payloads with a reason (REPLAY_* codes); the request can be edited first. “Identical request” is shown only when the request hashes match. Replays run under a per-project monthly budget, reserved before each call, and are recorded as traces named replay in the playground environment. Routes: GET …/spans/{span_id}/replay-source, POST /playground/replays, GET /playground/replays, GET /playground/replays/budget. New problem codes: REPLAY_BUDGET_EXCEEDED and the eight replay refusals.
• Honest uncertainty. Yes/no scores of an experiment run carry the number of items that passed and a Wilson 95 % interval on the pass rate (“78% (95% CI 65–88%, n = 50)”). Comparing two runs adds an exact McNemar test for yes/no scores and a paired sign test for numeric scores, with a plain sentence saying whether the difference is significant at 0.05 and what “not significant” means. The eval gate’s report gains a 95% CI column; the gate itself still compares the mean.
• Review loop metrics. spanlight_review_selected_total{reason}, spanlight_replay_runs_total{outcome}, spanlight_replay_cost_usd_total and six new spanlight_feature_used_total values (review_selection, review_item_completed, failure_category, replay, dataset_item_from_review, experiment_interval) feed a Review loop row on the Grafana dashboard.

Changed

• A new error class, gateway. A call Spanlight’s gateway refused or failed by itself (a blocking budget, a blocked upstream address, a busy database, an internal error) is class gateway instead of being classed by its status, so Spanlight’s own 500 and 503 are no longer counted as the provider’s 5xx by the Overview, the Doctor or alerts. An error the gateway made takes its class from its code: a malformed request is client, a gateway key’s rate limit rate_limit. Anthropic’s 400 for a spend limit you set yourself is rate_limit, not client. Spans stored earlier keep their class.
• Calls refused by a gateway key’s rate limit are recorded as spans. They have no model or input, because the gateway never read the body, and the kind gateway. They are dropped first when the gateway’s span recorder is busy (and counted as lost), they do not count in the key’s per-minute usage, and the Doctor’s error_spike and rate_limit_pressure leave them out.
• The gateway passes Anthropic’s anthropic-priority-* response headers to the client, like the other rate-limit headers.
• The prompts:read and scores:write key scopes now do something. They were accepted but reserved. Existing keys keep their scopes, so a key made for sending traces cannot read prompts or send scores: create a new key with the scope.
• Native JSON ingestion accepts prompt_version_id on a span. Older servers ignore it. OTLP has no field for it.
• Trace summaries carry has_scores and a few latest scores, and GET .../traces/filters lists score_names.
• Dataset uploads may be 5 MB. Every other /api/v1 body is still limited to 1 MiB; the limit for the upload route is checked while the body streams.
• The landing page covers the whole product. A new “Cost you can audit” section presents the cost breakdown and usage states, reconciliation with the provider’s bill, and attribution with budgets; “Diagnose and improve” presents the Doctor and provider health, the gateway, prompts and replay, evals with confidence intervals, and the daily review. The hero shows a drawn trace beside its cost breakdown, computed from the real claude-sonnet-5-5 rates of the 2026-10-16 price table and captioned as an illustration, instead of an empty frame waiting for a screenshot; the screenshot pipeline (frontend/scripts/capture-landing.ts) is removed. The “works with” row lists the OTLP conventions Spanlight reads (OpenInference, Vercel AI SDK, OpenLLMetry, LiteLLM), the nav and footer link the new sections and the cost and attribution guides, and a test checks that every documentation link points at a page that exists.
• Possibly breaking: model names are priced case-sensitively, and surrounding whitespace is stripped at ingest. A span’s model is stored without the whitespace around it (" gpt-4o " becomes gpt-4o, a whitespace-only name becomes null), and its case is kept. Pricing now matches the stored name exactly, as alert rules, rollup filters and budgets already did, so a model sent in another case than the provider’s id (GPT-4o, Claude-Sonnet-4-5) costs null (“—”) and is listed among the project’s unpriced models instead of being priced while budgets and alerts on gpt-4o missed it. Send the provider’s own model id to keep costs.
• OTLP keeps a span whose resource or token values do not fit. A deployment.environment over 64 characters or a service.version over 128 is dropped instead of rejecting every span of the service; a token count sent as a whole double (12.0) is stored as 12, and one with a fraction, NaN or an infinity is dropped (the span is kept with the count unknown and its cost null). One otlp_values_dropped warning per batch logs how many values were dropped, never the values. Native /v1/traces still rejects such spans.
• SDK: usage.missing_reason is renamed usage.unreported_reason. A streamed OpenAI Chat Completions call without stream_options={"include_usage": True} now carries usage.unreported_reason set to include_usage_not_set, instead of usage.missing_reason set to stream_options.include_usage not set. Update any filter or dashboard that reads the old attribute.
• Possibly breaking: the Python SDK records Azure clients as azure_openai. wrap_openai used to record an AzureOpenAI or AsyncAzureOpenAI client as openai, so its spans were priced at OpenAI’s list rates. It now records azure_openai unless provider= is passed, and such spans have no cost until an organization adds a price override for azure_openai, because Azure bills its own rates.
• A blank provider is stored as unknown. The provider of a span is stored without surrounding whitespace, and an empty or whitespace-only provider as null, on every ingestion path; native JSON answered an empty one with 422.
• Spans carry service_tier, web_search_requests and fee_usd in GET .../traces/{trace_id}; the Usage card shows the tier, and the playground’s run usage may have an unknown output count.
• Blocking budgets say they are a soft limit. The gateway’s 402 BUDGET_EXCEEDED message now ends with (soft limit, measured every 60 s), the budgets page labels a blocking budget Blocking (soft), and the dialog explains why: spend is measured every 60 seconds, so calls keep going through until the next measurement. On 2026-10-14 a $0.002 budget started blocking after 85.5 s, with spend at $0.0122, six times its amount. Nothing about how blocking works has changed; the action is still block.
• Gateway traces are named after the model you asked for. A gateway call that starts a new trace names it "<surface> <model>" with the model in your request, before route aliases and before the provider’s answer, so chat_completions gpt-4.1-mini stays that even when the provider answers with a snapshot. The span keeps the answered model. Traces stored earlier keep their names.
• A repeated x-spanlight-* header counts by its last value, documented now for every header, the two new ones included.
• OTLP names dropped metadata in partialSuccess. A batch that dropped spanlight.metadata.* entries answers with rejectedSpans at 0 and a message counting them. Other dropped values are still only logged.
• 422 UNKNOWN_SCOPE has a new detail for every budget scope, The budget's scope_id does not name anything its scope can measure., and a field message that says what the scope expects.
• The project.update audit event records the promoted keys it replaced under previous, next to changes.
• Possibly breaking: OTLP gen_ai spans for tools and agents are no longer llm. A span whose gen_ai.operation.name is execute_tool is now kind tool, and one with invoke_agent, create_agent or invoke_workflow is chain, so they no longer count as LLM calls; retrieval is retrieval, and any other operation that is not a model call is other. A gen_ai span without the attribute is still llm. Spans stored earlier keep their kind.
• Trace totals leave usage rollups out. A trace’s tokens and cost are the sum of its billable spans, so a trace from an agent framework that reported a call twice now costs what the call cost, and may be lower than the sum of its spans’ costs.
• OTLP error classes read http.response.status_code and error.type. The stable status attribute is read before http.status_code; an error.type of one to three digits is a status, and an exception class name in it (RateLimitError, PermissionDeniedError, ContextWindowExceededError, BadRequestError) is matched before the status message. A BadRequestError still yields to a message that names a more specific cause.
• More finish reasons are recognized. Vercel AI SDK content-filter, tool-calls, error and unknown, Gemini safety, recitation, blocklist and malformed_function_call, and the OpenTelemetry GenAI singular tool_call that Traceloop sends, from ai.response.finishReason, llm.response.finish_reason, gen_ai.response.finish_reason, gen_ai.completion.0.finish_reason and llm.finish_reason. response.finish_reasons and gen_ai.response.finish_reasons sent as text holding a JSON array, as LiteLLM does (["stop"]), are read as the list.

Fixed

• A busy database answers 503, not 500, inside a gateway call. A pool or statement timeout while the gateway served a call (during the budget check, for example) answered 500 INTERNAL_ERROR and logged a traceback. It now answers 503 SERVICE_UNAVAILABLE with Retry-After, which the official SDKs retry, as the same failure while checking the key already did, and logs a warning.
• A gateway call with a body that cannot be forwarded as JSON (NaN) has a classified span. It is answered 400 INVALID_REQUEST, and its span, which had no status and the class unknown, now records the status and the class client.
• A gateway span whose write fails is retried instead of lost. A span the database refused was dropped after one try, so a paid call could leave no span and recorded spend (and the budgets evaluated on it) came out low. A write that fails for a passing reason (a lost connection, a timeout, a deadlock) is now retried after 0.1, 0.5 and 2 seconds, and the write backlog (GATEWAY_RECORD_BACKLOG) grows from 1000 to 5000; a waiting span holds its input and output already redacted and cut to 32 KB, so a waiting span’s payloads are at most 64 KB of JSON as stored (the memory it takes in the process is larger, so size the backlog by what the api can spare). A span still not stored is counted in spanlight_gateway_record_failures_total as before and in the new gateway_spans_lost field of /health/ready, which never makes the probe fail. A span whose input or output cannot be prepared for the queue is counted the same way instead of only being logged.
• A gateway call whose body holds NUL, a lone surrogate, NaN or Infinity keeps its span. Plain json.loads accepts all four and Postgres stores none of them, so the call was answered but its span was refused (and, for a lone surrogate, the call itself failed with a 500). The span now stores the body with NUL removed, each lone surrogate replaced by ? and each non-finite number as its name; a lone surrogate is forwarded as the \uXXXX escape the client sent; and a model holding NUL or a lone surrogate is 400 INVALID_REQUEST instead of a 500. A span write the database refuses outright is no longer retried for 2.6 seconds, and its log line names the error type and SQLSTATE, never the message.
• A Lab truncated_stream call is recorded as failed. The client still gets the first frames and a clean end of stream without the final event, but the span, which said ok, now has status error and the message 200 FAULT_TRUNCATED_STREAM: …, like a malformed_json call. A stream too short to cut passes whole and stays ok.
• A gateway call cancelled while the provider was answering now shows the attempt it made. Its span reported spanlight.gateway.attempts 0 and no provider, so a request the provider had received, and may bill, left no trace of the attempt. The span now lists it, and spanlight_gateway_attempts_total counts it as cancelled.
• A card number next to other digits is redacted. A card followed or preceded by a CVV, an expiry or a quantity (4242 4242 4242 4242 123, qty 2 4242424242424242) was read as one longer number that fails the Luhn check, and was stored whole. Card-shaped runs (groups of four or more digits, the last of two or more, separated by single spaces or dashes) are now searched for 13-19 digit cards made of whole groups; the neighbouring digits are kept, and lists of small numbers, dates and phone numbers are left alone. A card typed in groups of one to three digits is not detected.
• An ingestion client that disconnects mid-body gets 400 BAD_REQUEST (“The request body was not fully sent.”) on /v1/traces, /v1/otlp/traces and /v1/scores, and the gateway answers 400 INVALID_REQUEST, instead of being logged as an unhandled server error.
• A gzip or deflate body that is cut short is refused with 400 BAD_REQUEST (400 INVALID_REQUEST on the gateway) instead of being accepted when the part that arrived happened to parse. An empty body sent as gzip or deflate gets the same answer; on ingestion it was 400 INVALID_JSON.
• Evaluator run and test scores send value as a JSON number, as trace scores and the documented contract already did; it was a decimal string ("0.8").
• A dataset upload full of unparseable lines reports at most 100 rejected lines, like a file of invalid items already did; every bad JSON line used to be listed, so a 5 MB file could produce a response many times its size.
• Rolling back below migration 0505 with evaluator scores no longer blocks the next upgrade. The downgrade now clears scores.evaluator_run_id before it drops the evaluator runs.
• Migrations validate every row of a table under row-level security. Run as the table owner, a migration validated a foreign key with a query that row-level security filtered, so migration 0505 could mark scores.evaluator_run_id valid while a score pointed at a missing run. Every VALIDATE CONSTRAINT now runs with the worker bypass set for its own transaction, and such a dangling reference fails the upgrade.
• SDK: Prompt.compile accepts a variable value holding a lone surrogate (as os.fsdecode produces), and a fallback template holding one is an InvalidTemplateError (PromptUnavailableError from prompts.get) instead of a UnicodeEncodeError.
• SDK: a Retry-After: nan or inf header is ignored and the exporter waits its normal backoff instead of retrying at once.
• Starting a review no longer ends the session while items are pending. Opening work mode sent two claims at once (the page mounts twice while the app switches to its focus layout), the second found the first one’s item locked and the page said “All done”. The app now picks its layout from the route it is about to show, so the page mounts once, and a remounted page waits for the claim already on its way.
• The new-experiment form says when a list fails to load. A failed dataset list left the field loading forever and a failed credential list showed an empty select; datasets, credentials, prompts and evaluators now each show “Couldn’t load the …” with Try again.
• Release pickers stop loading when the release list fails: both selects are disabled and say “Couldn’t load releases”.
• No more Content-Security-Policy error in the console on every page. zod built schemas before its no-eval setting applied; the setting now loads with zod itself.
• The dataset upload preview counts only what the server will store. A bare carriage return outside quotes, or a cell over 65,536 characters, stops the server’s CSV reader; the preview now marks that line “unreadable record; the rest is skipped” and counts nothing after it.
• Email channels accept internationalized addresses such as ops@例え.jp, which the API already accepted.
• A free-text categorical score of up to 200 characters is accepted when it holds emoji or other characters outside the Basic Multilingual Plane; the form counted UTF-16 units instead of characters.
• Experiment comparisons never show “+0.0000” or round a pass rate up to “100%”. A score change below 0.0001 reads “+<0.0001”, and a yes/no pass rate rounds down, so “100%” means every item passed. An evaluator’s budget meter rounds down the same way, so it reads “100%” only once the budget is used up.
• Accessibility: the audit log’s filter pills, the Lab probability slider and the dataset file picker have accessible names.
• The landing page no longer captions a missing screenshot as “A real screen from the live demo”.
• Layout fixes at 375 px and on short desktop windows. On a phone the account pages’ “Back to app” becomes an icon button instead of sliding under the theme toggle, the span detail tabs wrap into two rows instead of overlapping, a trace’s start time wraps instead of being cut off, and gateway KPIs shrink together to fit instead of showing “100. …”. In the sidebar only the page list scrolls, the current page is scrolled into view, and fades show when more pages are above or below.
• Users figures count spans that arrive late. The per-user figures were recomputed for yesterday and today only, so a span that arrived later for an older trace never reached them. Every 15 minutes the job now recomputes the last three UTC days, and a nightly pass (from 02:00 UTC) the last eight. The Users pages show when the figures were last updated (“Updated 4 min ago”), and the API returns it as computed_at.

Security

• The web image’s Caddy is rebuilt from Caddy 2.11.7 with Go 1.27.2 and golang.org/x/net 0.60.0: the official caddy:2.11-alpine binary (Go 1.26.8, x/net 0.59.0) carries CVE-2026-78667, CVE-2026-78669 and CVE-2026-97031, which failed the release scan.
• A judge’s input is passed as marked data and the model is told to treat it as data, which reduces and does not remove prompt injection. Scores are advice, not proof; this is stated in the documentation.
• Judge spend is bounded by a reservation per evaluator rather than a check after the fact, and an unpriced model never runs.
• Webhook evaluators use the same vetted client as alert webhooks (https only, no private addresses unless WEBHOOK_ALLOW_PRIVATE_TARGETS, no redirects), are signed with the channel’s secret, make one attempt with a 30 second limit, and never put the URL or the secret in a run’s error or a log line.
• A regex rule runs with a 1 second time limit on at most 10 000 characters, so a pattern that backtracks without end fails its run instead of holding a worker.
• Dataset files are parsed with fixed limits (5 MB, 64 KB per value, 10 000 items), in a thread pool, and an error never repeats the content of a line.
• A malformed backup database URL (for example one without a host, where the password reads as the port) is refused with a fixed message; the error used to quote the bad port, which could be the password.
• “Explain with Claude” no longer turns off row-level security inside the request to check the organization’s budget. The sum is a narrow database function, and the reservation is written under the request’s project like any other row.
• The new tables are protected by row-level security like traces: child tables (prompt versions, labels, dataset items, queue items, evaluator runs, experiment runs and items) are filtered through their parent. Scores are keyed by project and id, so a client-chosen id cannot collide with, or hide behind, another project’s.

SDK

• spanlight.prompts.get(name, label="production", *, version=None, fallback=None) fetches a prompt with a revalidating cache. After the cache time it serves the stale copy if the server fails, then the fallback, and raises PromptUnavailableError, the SDK’s one deliberate exception, only with neither. Prompt.compile(**variables) follows the server’s template rules and raises MissingVariablesError or InvalidTemplateError.
• Span linking. spanlight_prompt=compiled on wrap_openai and wrap_anthropic calls, with compiled: for everything inside a block, and spanlight.no_prompt() to keep a task unlinked.
• spanlight.score(trace_id, name, value, *, span_id=None, comment=None) sends a score in the background with the spans, with an id per call so a retry cannot store it twice. It never raises.
• Experiment runner python -m spanlight._experiments, behind the eval-gate Action.
• Python SDK 0.2.0, the next release. Span.set_usage accepts service_tier and web_search_requests, and Span.clear_output_tokens() makes an output count unknown; wrap_anthropic and wrap_openai record the tier, web searches and the reason a stream had no usage; wrap_anthropic records batch results; record_openai_batch_output is new; usage.missing_reason is renamed usage.unreported_reason. A failed provider call records the provider’s error type and code, the retry-after it asked for and its rate-limit headers, filtered through the same 29-name allowlist as the server, so no other response header leaves your process; a call the Spanlight gateway refused itself records the gateway’s code instead. Every batch of spans carries a client object with the exporter’s running totals of dropped and failed spans, which older servers ignore.
• Python SDK 0.2.0 reports its own cuts. A span whose input or output the SDK shortened (a string over 32,000 characters, a list or mapping over 1,000 entries, nesting deeper than 12, or a generator over 1,000 yielded items) is sent with truncated: true, and the server stores the span as truncated. Before, such a span arrived under 32 KB and was stored as complete.
• Python SDK 0.2.0 compresses by default. gzip now defaults to True, so span and score batches are sent with Content-Encoding: gzip; every Spanlight server accepts it. Pass gzip=False only for a proxy that cannot forward a compressed body.
• Python SDK 0.2.0 also adds update_trace(metadata=...) (and Spanlight.update_trace). It applies the server’s rules on the client: an invalid entry is dropped with a DEBUG log naming the key, never the value, and a trace keeps at most 32 keys, so the server never rejects a span for its metadata. Older servers ignore the field.
• Python SDK 0.1.2. Span.set_usage accepts cache_write_tokens and cache_write_1h_tokens, and wrap_anthropic fills them from cache_creation_input_tokens and cache_creation.ephemeral_1h_input_tokens; the attribute usage.cache_creation_input_tokens is no longer set.
• Eval gate report. The score table gains a 95% CI column for yes/no scores (64–88%, the low bound rounded down and the high one up; — for other scores), and a gated score whose threshold lies inside its interval gets the line “Interval includes the threshold; the result is not conclusive at this sample size.” Exit codes are unchanged.
• Python SDK 0.1.1. The eval-gate report prints p50 and p95 latency in whole milliseconds, and its “Open the run” link and the Action’s run-url output open the run page instead of the experiment.

Settings

Variable	Default	Effect
EVALS_ENABLED	true	Schedule online evaluation. Off, evaluators run only in experiments and tests.
EVAL_SAMPLER_MAX_RUNS_PER_CYCLE	50	Most runs one evaluator gets per one-minute sampling cycle.
EXPERIMENT_MAX_ITEMS	1000	Ceiling for an experiment’s max_items.
GATEWAY_RECORD_BACKLOG	5000 (was 1000)	Most gateway spans waiting to be written, retries included.
PROVIDER_STATUS_POLLING_ENABLED	true	Poll the Anthropic, OpenAI and Google status pages every 5 minutes. Off, nothing is fetched and the incident list says polling is disabled; for air-gapped servers. Allow status.claude.com, status.openai.com and status.cloud.google.com on port 443 otherwise.
BILLING_SOURCES_ENABLED	true	Allow calls to the providers’ cost APIs. Off, adding or re-keying a billing source skips the key check, nothing is fetched and a refresh answers 409 NOT_CONFIGURED; CSV uploads and reports still work.
REVIEW_SELECTION_ENABLED	true	Schedule the nightly selection of traces into each project’s review queue. Off, a review policy can still be run by hand.
REPLAY_MONTHLY_BUDGET_USD	5.00	Most a project may spend on replays per UTC month, in US dollars. 0 turns replay off (409 NOT_CONFIGURED). Replays also need CREDENTIALS_KEYS.
ATTRIBUTION_ENABLED	true	Schedule the attribution refresh, every 15 minutes. Off, attribution windows longer than 24 hours stop updating; shorter windows stay live.

Migrations

Run spanlight migrate and then spanlight ensure-app-role (or restart the stack) after upgrading. Migrations 0500 to 0506 add prompts, versions and labels, the span-to-version link, scores and score configs, datasets, annotation queues, evaluators and their runs, and experiments, runs and items. Migration 0507 adds explain_month_to_date_cost, the function the explain budget sums with; ensure-app-role grants the app role permission to call it. Migrations 0501 and 0502 build their indexes on spans and traces CONCURRENTLY: writes continue, and on a large table the build takes a few minutes and extra disk. 0502 also adds traces.has_scores; a retry of a failed CONCURRENTLY build drops its half-built index first. Migration 0600 adds spans.service_tier, spans.web_search_requests and spans.fee_usd (empty on older spans, with no backfill) and the server_tool_fees table; 0601 adds billing sources, their project mappings, fetch runs and provider cost lines; 0602 adds the daily cost rollups and user_stats_daily.computed_at. Run spanlight sync-prices (the api and worker do it when they start) to load the 2026-10-16 prices and fees. Migrations 0700 to 0705 add the provider error columns to spans and the retry columns to traces (empty on older rows; the new checks are validated without blocking writes), the hourly error rollup, the provider incident tables, the telemetry loss tables, the insight feedback table and the error_calls alert metric. The error rollup is backfilled over each project’s retention by the rollup job. Rolling back below 0700 deletes the spans of class gateway, which the older check does not allow. Migrations 0800 to 0804 add traces.metadata (empty on older traces, with no backfill) and the merge_trace_metadata function (callable by every role: no ensure-app-role step is needed for it), projects.promoted_metadata_keys, the attribution_rollups_daily table, the attribution export kind and the metadata and trace_name budget scopes with a 400-character scope_id. They take short locks only. Rolling back below 0804 deletes the budgets of the two new scopes and any budget whose scope_id is longer than 200 characters; below 0803, the attribution exports (their files stay in the bucket); below 0800, all trace metadata. Migration 0900 adds spans.usage_rollup (false on older spans, with no backfill: run spanlight recompute-usage-rollups --since 30d to flag older traces); it changes the catalog only and takes a short lock. Rolling back below it drops the column. Migration 1000 adds reason (manual on every existing item) and reason_detail to annotation_queue_items, with an index built CONCURRENTLY; 1001 adds review_policies and review_selections and the review and replay audit actions; 1002 adds replay_runs; 1003 allows the job outcome already_selected. The three new tables use forced row-level security. Rolling back below 1000 drops the reasons; below 1003, the jobs that ended already_selected lose their outcome.

[0.5.0] - Not tagged; included in 0.6.0

The Doctor: Spanlight reads your traces, opens findings about how your application calls LLMs, and shows what changed between releases and who drives cost.

Added

• Error class, request hash and finish reason on LLM spans. Every failed span gets one class (auth, rate_limit, timeout, context_length, content_filter, provider_5xx, network, client or unknown) from its HTTP status or message, and Traces can be filtered by it. Spans carry a request_hash (the same model and request give the same hash, computed before payload capture so it exists even when inputs are not stored) and a canonical finish_reason (stop, length, tool_calls, content_filter or other) for OpenAI, Anthropic and OTLP spans. Gateway spans for faults that answer before the provider call now record the simulated status and Retry-After. Spans stored before this release have none of these and are not backfilled.
• Fifteen detectors. Deterministic rules over a project’s recent spans: error_spike, latency_regression, cost_spike, retry_storm, retry_after_ignored, rate_limit_pressure, truncated_outputs, context_growth, cache_opportunity, tool_loop, truncated_stream_accepted, unsupported_parameter_retried, client_timeout_misconfigured, unpriced_spend and provider_incident. They run every 15 minutes for each project with traffic in the last 24 hours, skip calls the Integration Lab faulted when they judge traffic or provider health, and return nothing below a minimum sample size. Each finding carries its evidence (up to 20 traces), the layer where it fails, whether it is measured or inferred, a suggested fix and a way to verify it. The catalogue, thresholds and fixes are on the new Doctor page of the documentation.
• Insights lifecycle. A problem is one insight however often it recurs: it counts occurrences, resolves itself after 24 hours without a sighting, and reopens if it returns. Admins and owners can acknowledge, resolve, mute (up to 90 days, with a reason) and unmute. A project’s insight notifications go to a list of its organization’s alert channels; only a critical insight that opens notifies, once, as an insight.opened webhook event (signed like alert events), a Slack message, an email or a PagerDuty trigger. The weekly digest gains an Insights section.
• Health score. Overview shows a 0 to 100 score: 100 minus penalties for open critical and warning insights, error rate, a slower p95 and higher spend than the previous window. It is “—”, never 0, when the window has no LLM calls.
• Explain with Claude. An opt-in button on an insight asks Claude to explain the finding from its evidence. Admins and owners only, capped per organization at EXPLAIN_MONTHLY_BUDGET_USD a month with the cost reserved up front, shown under an “Advisory” label as plain text. Only redacted excerpts of at most five example traces are sent, through the organization’s own Anthropic credential and traced into the project.
• Releases. Set a release with the SDK, the OTLP service.version attribute or the gateway header x-spanlight-release. The Releases page lists releases in a range and compares two: deltas, model mix, error classes and new error messages, with exact percentiles over a range of up to 30 days.
• Users. The Users page ranks end users (the SDK user_id, OTLP user.id or x-spanlight-user) by cost, errors or traces over whole UTC days, with a daily chart and recent sessions for each. Statistics are refreshed every 15 minutes. Releases and Users can also be read with a project API key that has the traces:read scope.
• Lab example clients. examples/ holds a deliberately naive client and a corrected one that run the Integration Lab scenarios against the gateway. The naive client opens the matching finding for each scenario and the corrected one is written to open none.
• Doctor screens. Doctor (list and detail with evidence, fix, verification and actions), Releases, Users, a health tile on Overview, insight badges on traces, an error-class filter on Traces and an Insight notifications setting in Settings, Project.
• Operations. GET /api/v1/projects/{id}/detector-runs shows what each detector did on its last passes. New Prometheus metrics (spanlight_detector_runs_total, spanlight_detector_duration_seconds, spanlight_insights_open, spanlight_detector_projects_skipped_total) feed a Doctor row on the Grafana dashboard. A new runbook covers detector runs, a project that times out, muting and the explanation budget.

Changed

• insight.opened joins the notification payloads. Webhook, Slack, email and PagerDuty channels render it next to alert events; PagerDuty uses a dedup key that is stable for the insight and never sends a resolve for one.
• The gateway reads x-spanlight-release (1 to 128 characters) into the trace, so release comparison covers traffic that has no SDK.
• Gateway Retry-After handling. The provider’s retry-after-ms header is now passed back to the client next to retry-after, and the gateway honours and records the millisecond value first.
• A Lab malformed_json call is recorded as failed. It still answers 200 with the cut body, but its span has status error and the message 200 FAULT_MALFORMED_JSON: …, so a client’s reaction to it is judged like any other failure.

SDK

• Request hash and finish reason on LLM spans. The OpenAI and Anthropic wrappers now send a request_hash (a stable hash of the model and request parameters, computed before payload capture so it exists even when inputs are not stored) and the provider’s raw finish_reason (the first choice’s finish_reason for OpenAI, stop_reason for Anthropic, streams included). Manual spans can call span.set_request_hash(...) and span.set_finish_reason(...).

Settings

Variable	Default	Effect
DETECTORS_ENABLED	true	Schedule the detector job. Off, no new insights appear.
EXPLAIN_MODEL	claude-sonnet-5-5	The model behind “Explain with Claude”. It must have a price.
EXPLAIN_MONTHLY_BUDGET_USD	1.00	Most an organization may spend on explanations per UTC month. 0 turns them off.
USER_STATS_ENABLED	true	Schedule the refresh of per-user daily statistics behind the Users page.

Migrations

Run spanlight migrate (or restart the stack) after upgrading. Migrations 0400 to 0405 add the error class, request hash and finish reason columns on spans, the insights, detector runs and insight explanations tables, the project’s insight channel list, the per-user daily statistics table and a release index on traces. Migrations 0400, 0401 and 0405 build their indexes CONCURRENTLY: writes continue, and on a large table the build takes a few minutes and extra disk.

[0.4.0] - Not tagged; included in 0.6.0

Alerts and budgets: get told when error rate, latency, volume or cost crosses a line, and stop runaway spend at the gateway.

Added

• Alert channels. Organization-wide channels for email, Slack incoming webhooks, signed webhooks and PagerDuty (Events API v2). Email goes only to verified members of the organization. A channel can be tested, and its delivery log shows every send with its status, attempts and last error; a failed delivery can be retried by hand. Slack URLs, PagerDuty routing keys and webhook signing secrets are encrypted with CREDENTIALS_KEYS, never returned, and a webhook’s secret is shown once, when it is created or rotated.
• Signed webhooks. Each request carries X-Spanlight-Event, X-Spanlight-Delivery, X-Spanlight-Timestamp and an HMAC-SHA256 X-Spanlight-Signature, with a documented verification procedure and test vector. Failures are retried with backoff for up to 8 attempts.
• Alert rules. Threshold rules and anomaly rules (a band of a chosen number of standard deviations around the average of the previous windows) on error rate, p95 latency, p95 time to first token, LLM calls, tokens and cost, filtered by environment, provider or model, with a 7-day preview before you save. Rules are evaluated every 60 seconds with a cooldown after each resolve, can be muted for up to 30 days, and their events can be acknowledged. A metric with no data never fires or resolves a rule.
• Budgets. Daily or monthly (UTC) spend limits per project, gateway key, end user or model, each set to notify or to block. A budget notifies its channels when its spend passes the amount and again when it recovers.
• Blocking budgets. The gateway answers 402 BUDGET_EXCEEDED, in the OpenAI or Anthropic error shape, to calls in the scope of a blocking budget that is exceeded. Ingestion is never blocked. Blocking lags spend by up to one evaluation, about a minute.
• Weekly digest. Every Monday at 08:00 UTC, members with a verified email get one summary per project of spend, LLM calls, error rate, p95 latency, top models by cost and alerts fired. It can be turned off per project in project settings and needs an email provider.
• Dashboard screens. Alerts (Rules and Channels) and Budgets, with the rule editor and preview, mute and acknowledge, the event timeline, and the delivery log.
• Grafana alert panels. Rules evaluated by outcome, transitions by kind, evaluation duration, outbox backlog, deliveries by kind and outcome, and budget blocks. /health/ready reports alert_evaluation_lag_s.
• spanlight alerts evaluate. Runs one evaluation pass by hand, with --dry-run to roll it back and --json for scripts, which is also how the evaluation load test is timed.
• Documentation. Guides for alerts and channels, budgets and the weekly digest, plus a runbook for stuck and failed deliveries, a rotated webhook secret and late evaluation.

Changed

• Outbox leases and fencing. A worker now claims an outbox row with a 60 second lease and a fencing token and sends outside any database transaction. A worker whose lease expired can no longer settle the row, and a crashed worker’s rows are claimed again by the next run. Delivery is at least once.
• Failed alert payloads are kept. Payload reduction after a delivery settles now keeps the payload of a failed alert delivery so it can be retried; sent rows and failed transactional mail (verification, reset, invite) are reduced as before.
• Egress protection is a shared module. The check that refuses private, loopback, link-local and reserved addresses on every connection moved from the gateway into app/core/egress.py and now also guards webhook, Slack and PagerDuty deliveries.
• Outbound URLs stay out of logs. httpx and httpcore log at WARNING and Sentry’s httpx integration is off, so a Slack or webhook URL can never reach a log line or an error report.
• Model filters match snapshot names. An alert or budget filter on gpt-4o also matches gpt-4o-2024-08-06 and gpt-4o-latest, but not gpt-4o-mini, the same rule the price table uses.
• SMTP on port 465. With SMTP_PORT=465 the email sender connects with implicit TLS (SMTPS) and checks the certificate, for networks that block port 587.
• The gateway budget guard is real. The hook added in 0.3.0 now reads the evaluated state of blocking budgets instead of allowing every call.

Security

• Alert email cannot be used to write to strangers: recipients must be verified members, checked when the channel is saved and again when the message is queued (ALERT_EMAIL_ANY_RECIPIENT lifts it for a closed deployment).
• Webhook URLs must be https:// and resolve only to public addresses, checked when the channel is saved and on every connection, so a DNS change after the check does not get around it. Redirects are never followed, and response bodies are capped before they reach last_error. WEBHOOK_ALLOW_PRIVATE_TARGETS lifts both rules for a receiver inside a private network.
• Channel secrets are never written to a log, the audit log or last_error.

Settings

Variable	Default	Effect
ALERTS_EVALUATION_ENABLED	true	Schedule the once-a-minute alert and budget evaluation.
WEEKLY_DIGEST_ENABLED	true	Schedule the Monday digest email.
WEBHOOK_ALLOW_PRIVATE_TARGETS	false	Allow http:// and private addresses for webhook channels.
ALERT_EMAIL_ANY_RECIPIENT	false	Allow email channels to send to addresses that are not verified members.
PAGERDUTY_EVENTS_URL	https://events.pagerduty.com/v2/enqueue	Where PagerDuty channels send events.

Migrations

Run spanlight migrate (or restart the stack) after upgrading. Migrations 0300 to 0307 add the outbox lease columns, the alert channel, rule, state, event and budget tables, the weekly digest switch on projects, and three indexes. Migrations 0302 (end-user traces), 0306 (a channel’s deliveries) and 0307 (weekly digest lookups) build their indexes CONCURRENTLY: writes continue, and on a large table the build takes a few minutes and extra disk.

[0.3.0] - Not tagged; included in 0.6.0

The LLM gateway and the Integration Lab: point the official OpenAI or Anthropic client at Spanlight with one base URL and every call is traced, priced, limited and, if you configure it, routed, retried, cached and fault-injected.

Added

• LLM gateway. POST /gw/v1/chat/completions, POST /gw/v1/responses, POST /gw/v1/messages and GET /gw/v1/models accept OpenAI and Anthropic requests, streaming and tool calls included, and forward the provider’s answer unchanged. Errors come back in the provider’s own shape with a stable spanlight_code. Each call is recorded as one llm span with the gateway’s route, target, attempts and overhead, written after the answer is sent (a span that cannot be written is counted in spanlight_gateway_record_failures_total), and can join a trace you already started with the x-spanlight-* headers.
• Provider credentials. Organization owners store OpenAI, Anthropic and OpenAI-compatible API keys. Keys are encrypted with CREDENTIALS_KEYS, never returned, and can be checked, rotated and re-sealed under a new master key with spanlight reseal-credentials. The gateway refuses to connect to private, loopback, link-local and other reserved addresses, on every connection, unless GATEWAY_ALLOW_INSECURE_BASE_URLS is set for a local model server.
• Routes. Weighted targets, per-target model aliases, retries with backoff and Retry-After, fallbacks to the next target, and a time budget for the whole call. Every save is a numbered version with revert, and a stale edit is refused with 409 ROUTE_VERSION_CONFLICT.
• Gateway keys and limits. Project-scoped spl_gw_… keys, accepted as a bearer token or x-api-key, with a route, an environment, requests and tokens per minute, an allowed-model list and default tags. GATEWAY_ORG_RPM_CEILING caps an organization across its keys.
• Response cache. Opt-in per key with a time to live. Exact match on the surface, route, route version and request body, for non-streaming 200 answers up to 1 MB. X-Spanlight-Cache reports hit, miss, off or bypass, and a project’s cache can be purged.
• Budget guard hook. The gateway asks a budget guard before every provider call and answers 402 BUDGET_EXCEEDED when it says no. The guard that ships allows every call; budget rules arrive in a later release.
• Price overrides. An organization can set its own per-model prices (/api/v1/orgs/{org_id}/price-overrides). They win over the built-in table for spans ingested afterwards, and such spans record pricing_version override:<id>.
• Integration Lab. Fault profiles inject nine provider failures into non-production keys: expired or denied keys, rate limits, an unsupported parameter, server errors, a cut-off JSON body, a stream that ends early, a slow first byte and a timeout. Faulted calls are marked by X-Spanlight-Fault, a FAULT_<SCENARIO> code and a lab:<scenario> trace tag.
• Standalone gateway. GATEWAY_MODE=standalone runs the gateway as its own process (python -m app.gateway, same image) behind the standalone-gateway Compose profile, so LLM traffic can scale and fail apart from the dashboard API. disabled turns it off. There is a new runbook for it.
• Demo through the gateway. The demo project’s LLM calls now go through the gateway with the demo project’s own key, so the demo exercises the same path as a real application.
• Gateway load test. The k6 load tests gain a scenario that measures the time a call spends in Spanlight against a fake provider, with a target of under 20 ms at p95, and docs/performance.md explains the method.
• Dashboard screens. A Gateway section with Overview (traffic, errors, cache hit rate, fallbacks and retries per target), Keys, Routes with a versioned editor, Credentials and Lab. New Prometheus metrics cover gateway requests, overhead, upstream time and attempts; Grafana panels show requests, overhead, attempts, cache hits and spans not recorded.
• Documentation. Gateway guides for the quickstart, routing, the cache, the Lab, credentials and error responses.

Changed

• CREDENTIALS_KEYS now also protects provider credentials. Rotate it in two phases and run spanlight reseal-credentials before dropping an old key (see the rotate-secrets runbook).
• Audit log. Gateway configuration changes (credentials, routes, keys, fault profiles, cache purges and price overrides) are recorded as audit events.

Security

• Provider API keys are write-only: they are never returned by the API, written to a log or kept in an Idempotency-Key record, and responses that describe a credential carry Cache-Control: no-store.
• Gateway keys are stored as hashes, compared in constant time, and redacted from logs and spans.
• The gateway never forwards the client’s Authorization, x-api-key, cookies or x-spanlight-* headers to a provider, does not follow redirects, and sends no CORS headers.
• Fault injection cannot run on a key whose environment is production, checked when a profile is attached and again on every call.

[0.2.0] - Not tagged; included in 0.6.0

Production hardening: accounts and access control, a versioned API, durable background work, backups and exports, observability of Spanlight itself, and documentation.

Added

• Accounts. Email verification and password reset by email (links carry the token in the URL fragment; verification links expire after 24 hours, reset links after 1 hour). Sign in with GitHub and Google. Two-factor authentication with an authenticator app and recovery codes, which an organization owner can require for all members.
• Access tokens. Personal access tokens (spl_pat_…) for scripts and CI, with a read or write scope and an optional expiry, revocable at any time.
• API keys. Project API keys carry scopes (ingest:write, traces:read) and an optional expiry. Existing keys keep ingest:write and never expire.
• Idempotency keys. State-changing API requests accept an Idempotency-Key header, so a retried request runs once.
• Dashboard at scale. Hourly rollups with bucketed latency histograms back every time window longer than 24 hours; those responses are marked "approximate": true. Shorter windows still read the spans directly.
• Shared rate limits. Limits for ingestion, the demo and bearer reads are enforced in Postgres, so they hold across replicas and across restarts. A limited request gets 429 with Retry-After.
• Object storage and backups. Any S3-compatible store (AWS S3, Cloudflare R2, MinIO) can hold nightly pg_dump backups, kept for 14 daily and 8 weekly copies, and spanlight restore loads one back.
• Exports. Export traces as CSV or JSONL, and the audit log as CSV. Download links last one hour and files are deleted after seven days.
• Price endpoints. GET /api/v1/prices lists the price table, and GET /api/v1/projects/{project_id}/unpriced-models shows which models in your traffic have no known price.
• Organization and project administration. Rename or delete an organization or a project through the API, with a typed confirmation for deletes.
• Observability of Spanlight itself. Optional self-tracing with OpenTelemetry, a readiness check (/health/ready) that reports the database, the migrations and the worker, and Prometheus metrics for the worker (jobs, outbox, notifications, rollups) on WORKER_METRICS_PORT.
• Operations. Runbooks for deploying, rolling back, restoring, rotating secrets, revoking keys, scaling, ingestion spikes and service level objectives, plus a threat model and security documentation.
• Documentation site. Quickstart, Python SDK, OTLP, self-hosting, configuration, API reference, security, runbooks and this changelog, served at /docs/ by the web image.
• Load tests. k6 scripts and a seeder in backend/load/ measure ingestion (500 spans a second), the dashboard overview and the trace list (10 million spans) against the Compose stack, and a manually started workflow runs them on a GitHub-hosted runner. Method and targets are in docs/performance.md.

Changed

• Versioned API. Dashboard routes moved from /api/<resource> to /api/v1/<resource>. The old paths return 404. Ingestion (/v1/traces, /v1/otlp/traces) keeps its own version and is unchanged. A future breaking change will ship as /api/v2, with /api/v1 kept for at least 12 months.
• Postgres shared memory. The Compose postgres service gets 256 MB of /dev/shm instead of Docker’s 64 MB default, which parallel queries on large tables could exhaust (“could not resize shared memory segment”).
• Dashboard queries at 10 million spans. Migration 0016 swaps the spans time index for a covering one, so a 24 hour overview is answered from the index and not from the table. It is built CONCURRENTLY: writes continue, and on a large table it takes minutes and extra disk.
• Row-level security policies read their settings once per statement. Migration 0017 rewrites the tenant-isolation and worker-bypass policies of traces, spans, the rollup tables and exports so current_setting() is evaluated once per query instead of once per row; which rows are visible does not change. It takes a brief exclusive lock on each table and gives up after 5 seconds if it cannot get it, so run it again at a quieter moment.
• A busy database answers 503, not 500. A request that waits more than 5 seconds for a connection (API_POOL_TIMEOUT_SECONDS) or has a statement that runs longer than 10 seconds (API_STATEMENT_TIMEOUT_SECONDS) now gets 503 SERVICE_UNAVAILABLE with Retry-After: 5.
• Postgres JIT off in Compose. The postgres service starts with jit=off, which costs more than it saves on the dashboard and rollup queries.
• More than one replica. Rate limits and idempotency state live in Postgres, so the API can run as several replicas.
• Optional features report themselves. A request that needs an unconfigured integration (email, object storage, OAuth, CREDENTIALS_KEYS) answers 409 NOT_CONFIGURED and names the setting; a background job that needs one ends as skipped_not_configured.

Security

• Two-factor seeds are encrypted at rest with the keys in CREDENTIALS_KEYS.
• Sign-up is throttled per client IP.
• Password hashing and checking (argon2) run off the event loop, on a small dedicated executor that caps how many hashes run at once, so a burst of sign-ins cannot starve other requests or exhaust memory.
• Requests to /api/v1 are limited to 1 MiB (413 above that). Deeply nested JSON sent to ingestion answers 400 INVALID_JSON instead of 500.
• BACKUPS_ENABLED and BACKUP_DATABASE_URL are set only on the worker, the only service that runs backups. The object storage keys stay on both services, because the API signs export download links.
• Responses that reveal a secret (a new API key or access token, recovery codes, a two-factor setup secret) carry Cache-Control: no-store.
• The /metrics token is compared in constant time.
• The web image runs Caddy as a non-root user.
• Export files are written under a separate key for each attempt, so a retry or a lease takeover can never delete a file another attempt finished. Download links force the browser to save the file instead of showing it.
• A database that is too busy to answer answers 503 with Retry-After, not 500.
• Row-level security policies read their settings once per query. Which rows are visible does not change.

[0.1.0]

Initial version.

Added

• Tracing. Python SDK (spanlight) with @observe, spans and wrappers for the OpenAI and Anthropic clients, native JSON ingestion (POST /v1/traces) and OTLP ingestion (POST /v1/otlp/traces, JSON and protobuf, OpenTelemetry GenAI attributes).
• Dashboard. Overview with latency, token, cost and error metrics, a trace explorer with a span waterfall, and session views.
• Cost. Versioned per-model prices; a model without a known price shows as unknown, never as $0.
• Teams. Organizations, projects, roles, invites, API keys and an audit log.
• Privacy. Postgres row-level security, secret redaction before storage, and per-project switch for payload capture.
• Self-hosting. One Docker image with api, worker and web roles, a Compose file and a Railway deployment guide.

---

This page is generated from CHANGELOG.md in the repository.