Self-hosting

Spanlight is one Docker image with three roles and Postgres. Postgres is the only datastore, and it also holds the job queue, so there is no Redis or message broker to run.

Before you start

Docker with the Compose plugin
Three secrets for deploy/.env: POSTGRES_PASSWORD, APP_DB_PASSWORD and SECRET_KEY
A reverse proxy or load balancer that terminates TLS, if people reach Spanlight from outside

What runs

Service	Role	Listens
`postgres`	PostgreSQL 17. Traces, spans, users, sessions, rollups and the job queue.	5432, internal
`migrate`	Applies the database migrations as the database owner, creates the unprivileged `spanlight_app` role, then exits.	none
`api`	The FastAPI application: authentication, ingestion and queries.	8000, internal
`worker`	Background jobs: retention, cleanup, hourly rollups, the daily attribution rollups, notifications, backups and exports, demo traffic.	9100 for `/metrics`, internal
`web`	Caddy. Serves the dashboard and this documentation, and proxies `/api`, `/v1` and `/health` to the API.	`WEB_PORT`, 8080 by default

The worker’s attribution refresh rebuilds the daily rollups behind attribution windows longer than 24 hours every 15 minutes. Set ATTRIBUTION_ENABLED=false to stop it: those windows then stop updating, and windows of 24 hours or less stay live. See Configuration.

The API and worker connect as spanlight_app, a role that is neither a superuser nor able to bypass row-level security. Row-level security on the trace tables is what keeps one project’s data from another’s even if a query forgets its filter, so never point DATABASE_URL at the database owner.

Docker Compose

Requirements: Docker with the Compose plugin.

git clone https://github.com/dheerajkarwasra/spanlight.git
cd spanlight
cp deploy/.env.example deploy/.env       # fill in POSTGRES_PASSWORD, APP_DB_PASSWORD and SECRET_KEY
docker compose -f deploy/compose.yaml --env-file deploy/.env up -d

The stack runs the released images named in deploy/.env: SPANLIGHT_IMAGE (api, worker and the optional gateway) and SPANLIGHT_WEB_IMAGE (web), published on GHCR for every release. To build from the checkout instead, comment out both lines and start with --build; the images are then tagged spanlight:local and spanlight-web:local.

The dashboard is at http://localhost:8080 (change the port with WEB_PORT). Confirm that the stack is healthy as described in Check it worked.

Spanlight does not terminate TLS. Put your own reverse proxy or load balancer in front of the web service, set APP_BASE_URL to the public https:// URL, and set TRUSTED_PROXIES to the proxy’s address ranges so rate limits and session records see real client addresses. With an https base URL cookies become Secure and the application refuses to start with the default SECRET_KEY.

Optional add-ons

Each is switched on by its own variables; see Configuration for all of them.

Email (address verification and password reset): set EMAIL_PROVIDER to resend or smtp and the values it needs.
Sign in with GitHub and Google: set the client ID and secret of a provider, and register <APP_BASE_URL>/api/v1/auth/oauth/<provider>/callback as its callback URL.
Two-factor authentication: set CREDENTIALS_KEYS, and back its value up. Without it, authenticator enrolment answers 409 NOT_CONFIGURED.
Object storage, backups and exports: set S3_BUCKET, S3_ACCESS_KEY and S3_SECRET_KEY (and S3_ENDPOINT for a store other than AWS). Nightly backups additionally need BACKUP_DATABASE_URL. To try this locally without a cloud account, add the MinIO file: docker compose -f deploy/compose.yaml -f deploy/compose.minio.yaml --env-file deploy/.env up -d. Export downloads are links straight to the store, so the public S3 endpoint (S3_PUBLIC_ENDPOINT, or S3_ENDPOINT when that is unset) must use https when the dashboard is served over https; otherwise browsers block the download as mixed content.
More than one API replica: rate limits and idempotency state live in Postgres, so replicas share them. deploy/compose.replicas.yaml is an example that runs two.
Metrics: set METRICS_TOKEN and scrape api:8000/metrics and worker:9100/metrics on the internal network with Authorization: Bearer <token>. deploy/prometheus.yml is an example scrape configuration. /metrics is not served through the web port.
Error reporting and self-tracing: SENTRY_DSN, and OTEL_EXPORTER_OTLP_ENDPOINT for OpenTelemetry spans about Spanlight’s own requests and queries.

Outbound connections

Spanlight connects out to the providers your gateway credentials, judges and demo traffic use, and to the services you configure (email, sign-in providers, object storage, Sentry). One more connection is on by default: every five minutes the worker polls the public status pages of three providers, so the Provider health page and the Doctor can show their incidents. If your network restricts egress, allow these hosts on port 443:

status.claude.com
status.openai.com
status.cloud.google.com

The feed addresses are fixed in the code; no setting changes them. The requests use https only, follow no redirects, give up after 10 seconds, read at most 2 MB, refuse any host that resolves to a private or local address, and connect directly: HTTP_PROXY and HTTPS_PROXY are not read.

In an air-gapped network, set PROVIDER_STATUS_POLLING_ENABLED=false on both the api and the worker. The worker then fetches nothing, and the API says so: the incident list answers "polling": "disabled" and the dashboard reads “Provider status feeds are off on this server” instead of showing an empty or stale list. Incidents stored earlier stay readable.

Tuning Postgres

The Compose file runs Postgres 17 with settings sized for a host with 2 vCPUs and 4 GB of memory that runs the api, the worker and Postgres together. Postgres needs about 1.5 GB free for them (shared_buffers alone is 1 GB); on a smaller host it can fail to start or be killed for lack of memory, which deploy/smoke.sh and docker compose logs postgres show. Lower shared_buffers and effective_cache_size there; on a bigger node raise them.

Setting	Shipped	Postgres default	Why it matters here	On a bigger node
`shared_buffers`	`1GB`	`128MB`	Postgres’ own page cache. The dashboard reads windows of recent spans through an index that outgrows 128 MB quickly.	About 25 % of the memory you give Postgres. The operating system caches the rest.
`effective_cache_size`	`3GB`	`4GB`	Allocates nothing. It tells the planner how much data is probably cached (this plus the operating system’s cache), and the planner chooses index scans over table scans more readily when the data fits.	50 to 75 % of the memory you give Postgres.
`work_mem`	`16MB`	`4MB`	Memory for one sort or hash. The latency percentiles sort every model call in the window, and larger windows spill to disk at the default.	16 to 32 MB. It is used per sort or hash, per query, so many connections multiply it: keep connections times a few times `work_mem` well inside your memory.
`maintenance_work_mem`	`256MB`	`64MB`	Memory for vacuum and index builds.	Up to about 1 GB.
`max_connections`	`200`	`100`	Each api replica can open 30 connections; 200 fits six replicas and a worker (see the scaling runbook).	Only as many as the replicas need. Each connection costs memory.
`wal_compression`	`on`	`off`	Compresses the full-page images that ingestion writes into the WAL, for a little CPU.	Keep.
`random_page_cost`	`1.1`	`4`	The planner’s cost of a random page read, against `1` for a sequential one. The default describes a spinning disk.	Keep on SSD or NVMe; `4` on spinning disks.
`autovacuum_vacuum_cost_limit`	`1000`	`-1` (`vacuum_cost_limit`, 200)	How much work autovacuum does before it pauses. Traces are rewritten as their spans arrive, and the default lets dead rows pile up.	Keep, or raise on fast disks.
`jit`	`off`	`on`	JIT compilation takes longer than the dashboard and rollup queries it is meant to speed up.	Keep off.

To change them, override the command of the postgres service in an override file. An override replaces the whole command, so repeat every value, including jit=off. For a host with about 8 GB for Postgres:

services:
  postgres:
    command:
      - postgres
      - -c
      - jit=off
      - -c
      - shared_buffers=2GB
      - -c
      - effective_cache_size=6GB
      - -c
      - work_mem=16MB
      - -c
      - maintenance_work_mem=512MB
      - -c
      - max_connections=200
      - -c
      - wal_compression=on
      - -c
      - random_page_cost=1.1
      - -c
      - autovacuum_vacuum_cost_limit=1000

docker compose -f deploy/compose.yaml -f deploy/compose.tuning.yaml --env-file deploy/.env up -d postgres
docker compose -f deploy/compose.yaml -f deploy/compose.tuning.yaml --env-file deploy/.env exec postgres \
  psql -U postgres -tAc "SHOW shared_buffers"

Changing shared_buffers restarts Postgres, so expect a short outage. On a managed Postgres, set the same parameters in the provider’s console.

Railway

The public demo runs on Railway: Railway Postgres plus three services built from the same Dockerfile. The Railway guide walks through it step by step, including the secrets and the infrastructure-as-code file in deploy/railway/.

Upgrading and operating

Back up before every upgrade. These pages cover the routine work:

Upgrade Spanlight: pin a version, back up, rebuild and check the migrations.
Back up and restore: nightly backups to object storage, and a restore drill.
Troubleshooting: no trace arrived, spans rejected, a failing readiness check, a stopped worker or email that does not send.

The runbooks cover each procedure with the commands and the output to expect:

Admin commands

The spanlight command inside the api container is the operator’s tool:

docker compose -f deploy/compose.yaml --env-file deploy/.env exec api spanlight --help

It creates users (create-user), resets passwords (reset-password) and two-factor authentication (reset-2fa), refreshes the price table (sync-prices), runs migrations (migrate), rebuilds older days of the attribution rollups (attribution backfill), and restores a backup (restore; see the restore runbook).

Check it worked

Check the stack with:

curl http://localhost:8080/health/ready

A healthy instance answers 200 and a body like {"status":"ok","database":"ok","migrations":"ok","worker_heartbeat_age_s":7.4,"outbox_backlog":0}: the seconds since a worker last reported, and the number of notifications waiting to be sent. /health/ready answers 503 when the database is unreachable, when the schema is not the one this build expects, when no worker has reported in the last two minutes (unless WORKER_REQUIRED=false), or when more than 1000 notifications are overdue by over ten minutes. Then open http://localhost:8080/login in a browser: it shows the sign-in page. (The root address shows the landing page to a signed-out visitor.)

Troubleshooting

Compose stops with required variable POSTGRES_PASSWORD is missing a value: set POSTGRES_PASSWORD, or the same for APP_DB_PASSWORD or SECRET_KEY. A required secret is not in deploy/.env. Fill in the three secrets and run the command again.
The api refuses to start with “SECRET_KEY must be set when APP_BASE_URL uses https”. The base URL is https:// but SECRET_KEY is still the development default. Set a long random SECRET_KEY.
/health/ready answers 503, or the connection fails. Read the field that is not ok, or check the services with docker compose -f deploy/compose.yaml --env-file deploy/.env ps; see Read the readiness check first.
Sign-in works over http://localhost but not behind your proxy. With an https APP_BASE_URL cookies are Secure. Reach the dashboard over https, and check APP_BASE_URL and TRUSTED_PROXIES.
An export download is blocked in the browser. The dashboard is on https and the public S3 endpoint is not. Set S3_PUBLIC_ENDPOINT to an https address.
Two-factor enrolment answers 409 NOT_CONFIGURED. CREDENTIALS_KEYS is not set; see Optional add-ons.
A trace never arrives from your application. The SDK defaults to port 8000, which Compose does not publish; see The SDK points at the wrong host.

Next steps

ConfigurationEvery environment variable the API and worker read.

Upgrade SpanlightMove a running deployment to a new version.

Back up and restoreNightly backups and a restore drill.

Deploy to RailwayRun Spanlight on Railway.