Skip to content
SpanlightSpanlightSpanlight

Data, limits and operations

How Spanlight treats your data at scale and how to operate it. Settings are listed in Configuration, and step-by-step procedures are in the runbooks.

Before you start

  • Shell access to the Spanlight host, to run backups and restores
  • For exports and backups: object storage (S3_BUCKET, S3_ACCESS_KEY and S3_SECRET_KEY)
  • For nightly backups: BACKUP_DATABASE_URL naming a role that can bypass row-level security
  • The worker running

Metrics over long windows are approximate

Section titled “Metrics over long windows are approximate”

The overview, time series and per-model views aggregate spans. Whole hours that are already rolled up come from hourly rollups: one row per project, hour, environment, provider, model and span kind, with counts, tokens, cost and a bucketed histogram of latencies. The rest of the window, the partial hours at its edges and the most recent hour or so, is read from the spans themselves. This keeps a 24-hour or 30-day chart fast with millions of spans.

Counts, errors, tokens and cost are exact either way, because the two parts never overlap. Latency percentiles are exact only when the window reads no rollup hour, which in practice means the last hour. Otherwise they are estimated from the histograms: each bucket is about 1.5 times wider than the one before, so an estimate can be off by roughly 25 %, and a latency above 300 seconds is reported as 300 seconds. The API marks those responses "approximate": true and the dashboard says so. Cost stays unknown, never zero, when no call in the group has a known price; GET /api/v1/projects/{project_id}/unpriced-models shows which models in your traffic are missing from the price table.

The worker rewrites the last three hours of rollups every five minutes and the rest of the last 48 hours about once an hour. A span that arrives more than ten minutes after it started can therefore take up to five minutes to show in a chart, or up to an hour if it arrives more than about two hours after its hour ended; and one that arrives more than 48 hours after it started is not reflected until the range is backfilled by hand with spanlight rollups backfill --project <project id> --from <date> --to <date> (at most 90 days at a time). The rollups are derived data and can always be rebuilt from the spans.

Rate limits

Section titled “Rate limits”

The Limits reference lists every rate limit and size limit. The main buckets are below.

The main ones, each a token bucket per caller kept in Postgres:

What Limit
Ingestion, per API key 50 requests a second, bursts of 100
Reads with an API key or a personal access token, per credential 20 requests a second, bursts of 40
Starting a demo session, per client address 10 an hour

A limited request gets 429 RATE_LIMITED and a Retry-After header in whole seconds. The Python SDK retries 429 and honours Retry-After. If you send traces yourself, prefer fewer, larger batches over many small ones.

Because the buckets are in Postgres, the limits hold across any number of API replicas. If the limiter cannot reach the database, requests are let through and the failure is logged and counted, so a hiccup in the limiter never becomes an ingestion outage.

Sign-in, signup, password reset, verification email and invitation limits count attempts per time window instead. The main ones are listed in Accounts and sign-in, and all of them in the Limits reference. Set TRUSTED_PROXIES so that limits and session records see the real client address; see Self-hosting.

Exports

Section titled “Exports”

Exporting traces as CSV or JSONL, the limits, and requesting exports through the API are covered in Exports.

Backups and restore

Section titled “Backups and restore”

With object storage, BACKUP_DATABASE_URL and BACKUPS_ENABLED (on by default), the worker runs pg_dump once a day, from 03:00 UTC, and uploads the dump to your bucket. It keeps the newest backup of each of the last 14 days and of the last 8 Sundays. Without them the nightly job ends as skipped_not_configured and takes no backup, so confirm the prerequisites in the runbook before relying on it.

BACKUP_DATABASE_URL must name a role that can bypass row-level security. The application role cannot dump the tenant tables, so using it makes the dump fail instead of producing an incomplete one.

A restore always goes into a new, empty database, with spanlight restore, and refuses a database that already has tables. Restoring a database backup has every command. Rehearse a restore at least once: a backup you have never restored is a hope, not a backup.

Watching Spanlight itself

Section titled “Watching Spanlight itself”
  • Health. /health/live says the process is up. /health/ready also checks the database and the migrations, and the worker heartbeat while WORKER_REQUIRED is true (the default). It answers 503 when one of them is wrong, and also when more than 1000 notifications are overdue by over ten minutes.
  • Metrics. With METRICS_TOKEN set, the API serves Prometheus metrics at /metrics, and the worker does the same on WORKER_METRICS_PORT. They cover requests and latency, ingested and rejected spans, rate-limit rejections, idempotency outcomes, finished jobs, pending notifications and rollup duration. The service level objectives page has alert rules built on them.
  • Self-tracing. Set OTEL_EXPORTER_OTLP_ENDPOINT to send OpenTelemetry spans about Spanlight’s own requests and queries to a collector. Request bodies, headers and query parameters are never recorded.
  • Errors and logs. SENTRY_DSN reports errors without request bodies, cookies or query strings. Logs are JSON lines by default (LOG_JSON, LOG_LEVEL), and each carries the request ID that the X-Request-ID response header returns.

Load tests and performance targets

Section titled “Load tests and performance targets”

The latency objectives are in the service level objectives: a p95 of at most 200 ms for ingestion and at most 300 ms for the dashboard. The load tests hold Spanlight to them under load: ingestion of 500 spans a second, and the dashboard overview and the trace list at 20 requests a second each. The recorded run stored 1 million spans; the 10-million-span stress profile exists but has not been recorded yet.

They are checked with k6 scripts against the shipped Docker Compose stack, so you can repeat the measurement on your own hardware. The scripts, the seeder that fills the database with a 28-day history, and the instructions are in backend/load in the repository. The Load test workflow runs the same scripts on a GitHub-hosted runner: smoke (100 000 spans) checks that the setup works, and full (10 million spans) takes hours. A script fails when it misses its latency target, when more than 1 % of requests fail, or when it cannot hold the request rate.

Measure before you tune. Self-hosting lists the four Postgres settings that matter most on a larger host.

Check it worked

Section titled “Check it worked”
  • Long windows. On Overview, choose a time range longer than the last hour, such as the default 24 hours. The header shows the chip Estimated · hourly rollups, and its tooltip reads “Percentiles are estimated from hourly latency histograms, accurate to about ±25 %. Counts and cost are exact; a span that arrives late can take up to an hour to appear.” The API response has "approximate": true.
  • Health. GET /health/ready answers 200 with "status":"ok".
  • Backups. After the first run at 03:00 UTC, spanlight backup list shows a dump from that night. Without the prerequisites above the job ends as skipped_not_configured and no dump appears.

Troubleshooting

Section titled “Troubleshooting”
  • A percentile differs between the last hour and a longer range. By design: the last hour reads the spans, longer ranges estimate percentiles from hourly rollups. See Metrics over long windows are approximate.
  • Recent spans are missing from a long-window chart. Rollups are rewritten every five minutes for the last three hours and about once an hour for the rest of the last 48 hours. A span that arrives more than 48 hours late needs spanlight rollups backfill.
  • Cost shows “—”. No call in the group has a known price. GET /api/v1/projects/{project_id}/unpriced-models names the models; see How cost is calculated.
  • 429 RATE_LIMITED. Wait Retry-After seconds and send fewer, larger batches. See Rate limits.
  • /health/ready answers 503. The database, the migrations, the worker or the notification backlog is failing; see the troubleshooting page.
  • No backup appears. The job needs object storage, BACKUP_DATABASE_URL and BACKUPS_ENABLED; a dump with the application role fails on purpose. See Backups and restore.

Next steps

Section titled “Next steps”
ExportsExport traces as CSV or JSONL.
Self-hostingWhat runs, how to expose it and how to keep it healthy.
ConfigurationEvery environment variable the API and worker read.
Restoring a database backupRestore a backup into an empty database.