Skip to content
SpanlightSpanlightSpanlight

Alert webhooks

A webhook channel makes Spanlight send a signed POST with a JSON body to a URL you choose whenever an alert rule fires or resolves, a budget is exceeded, or the Doctor opens a critical insight. Use it to open tickets, page a rota, or feed your own tooling. This page lists the events, the request, how to check that a request really came from your Spanlight, and what happens when your endpoint is down.

Webhook channels belong to an organization. Create one under Alerts → Channels, choose Webhook and enter the URL. Spanlight shows the signing secret once, when the channel is created and again when you rotate it. Copy it then; it cannot be read back.

Before you start

  • The admin or owner role in the organization, to create a webhook channel
  • CREDENTIALS_KEYS set on the server, because the signing secret is stored encrypted
  • An https endpoint on a public address, unless the operator sets WEBHOOK_ALLOW_PRIVATE_TARGETS
  • The worker running, because it sends the deliveries

Events

Section titled “Events”
event Sent when
alert.fired An alert rule’s condition became true.
alert.resolved The condition stopped being true. A resolved budget rule also sends this event, with budget set.
budget.exceeded A budget reached its amount.
insight.opened The Doctor opened a critical insight in a project that lists this channel under Insight notifications. See insight.opened.
test You pressed Test on the channel’s row. The body is smaller: version, event, occurred_at, org and url.

The request

Section titled “The request”
POST /your/endpoint HTTP/1.1
Content-Type: application/json
User-Agent: spanlight/1.0.0
X-Spanlight-Event: alert.fired
X-Spanlight-Delivery: 0192f5a0-1111-7000-8000-00000000d1e1
X-Spanlight-Timestamp: 1760087700
X-Spanlight-Signature: sha256=<64 hex characters>
Header Meaning
X-Spanlight-Event The event name from the table above.
X-Spanlight-Delivery The id of this delivery. It stays the same on every retry, so it is the key to deduplicate on.
X-Spanlight-Timestamp When this attempt was signed, in Unix seconds. It changes on every attempt.
X-Spanlight-Signature sha256= and the hex HMAC-SHA256 of <timestamp>.<body> under your signing secret.

The body is compact JSON in UTF-8 (no spaces between tokens). Decimal numbers are strings, and times are UTC in YYYY-MM-DDTHH:MM:SSZ form.

{
  "version": "2026-10-01",
  "event": "alert.fired",
  "event_id": "0192f5a0-0000-7000-8000-0000000000e1",
  "occurred_at": "2026-10-10T09:15:00Z",
  "org": { "id": "0192f5a0-0000-7000-8000-00000000a001", "name": "Acme" },
  "project": { "id": "0192f5a0-0000-7000-8000-00000000b001", "name": "Support Copilot" },
  "rule": {
    "id": "0192f5a0-0000-7000-8000-00000000c001",
    "name": "Error rate in prod",
    "kind": "threshold",
    "metric": "error_rate",
    "comparator": "gt",
    "threshold": "0.05",
    "window_minutes": 15,
    "filters": { "environment": "production" }
  },
  "value": "0.083",
  "threshold": "0.05",
  "started_at": "2026-10-10T09:15:00Z",
  "resolved_at": null,
  "budget": null,
  "url": "https://app.example.com/0192f5a0-0000-7000-8000-00000000a001/0192f5a0-0000-7000-8000-00000000b001/alerts/0192f5a0-0000-7000-8000-00000000c001"
}
  • threshold at the top level is the threshold that applied. For an anomaly rule it is the anomaly threshold at that moment, while rule.threshold is null.
  • budget is null for rule alerts. For budget events it is { id, name, scope, scope_id, period, amount_usd, spent_usd, action, resets_at }, rule.kind is budget and rule.window_minutes is null.
  • resolved_at is set on alert.resolved.
  • New fields can appear in a later version. Ignore fields you do not know.

insight.opened

Section titled “insight.opened”

The Doctor sends this event once each time a critical insight becomes open: when it is first found, when a resolved one comes back, and when a mute ends and the problem is still there. A warning or info insight, and a critical one that is merely seen again, send nothing; a warning that later grows to critical stays quiet. A channel receives it only when an admin has added it to the project’s insight channels under Settings, Project, Insight notifications. The request is signed, retried and deduplicated exactly like the alert events above; only the body differs.

{
  "version": "2026-10-01",
  "event": "insight.opened",
  "event_id": "0192f5a0-0000-7000-8000-0000000000e2",
  "occurred_at": "2026-10-10T09:30:00Z",
  "org": { "id": "0192f5a0-0000-7000-8000-00000000a001", "name": "Acme" },
  "project": { "id": "0192f5a0-0000-7000-8000-00000000b001", "name": "Support Copilot" },
  "insight": {
    "id": "0192f5a0-0000-7000-8000-00000000f001",
    "kind": "retry_storm",
    "label": "Retry storm",
    "severity": "critical",
    "fingerprint": "5c1d0a9be3f24d7a8a61c0b2e4f93d17",
    "title": "Identical failing requests repeated up to 24 times",
    "summary": "3 bursts in the last 15 minutes sent the same request 5 or more times within 60 seconds after it failed (2 distinct requests).",
    "failure_layer": "client",
    "suggested_fix": "Retry with exponential backoff and jitter, at most 3 attempts, and only on retryable errors (429, 5xx, timeouts). Never retry 400, 401 or 403. The official SDKs do this with `max_retries=2` (three attempts in all).",
    "verification": "Run the `provider_5xx` lab scenario on a test key: one round shows at most 3 attempts with growing gaps.",
    "occurrences": 1,
    "first_seen_at": "2026-10-10T09:30:00Z",
    "last_seen_at": "2026-10-10T09:30:00Z",
    "evidence": {
      "trace_ids": ["4bf92f3577b34da6a3ce929d0e0e4736", "0af7651916cd43dd8448eb211c80319c"],
      "metrics": { "storms": 3, "largest_storm": 24, "hashes": 2 },
      "window": { "start": "2026-10-10T09:15:00Z", "end": "2026-10-10T09:30:00Z" }
    }
  },
  "url": "https://app.example.com/0192f5a0-0000-7000-8000-00000000a001/0192f5a0-0000-7000-8000-00000000b001/doctor/0192f5a0-0000-7000-8000-00000000f001"
}
  • fingerprint names the problem within the project and is stable across reopenings, so it is a good key for your own ticket or incident. insight.id is also stable.
  • evidence.trace_ids holds up to 20 traces, most recent first. evidence.metrics depends on the kind: decimals are strings, counts are integers and an unknown value is null. evidence.window is the span of data the detector looked at.
  • occurred_at is the detection that opened the insight. failure_layer is one of client, request, agent, provider, platform or traffic.
  • suggested_fix and verification are the catalogue text for the kind, as shown on the insight page; they contain Markdown-style backticks and no HTML.
  • There is no insight.resolved event: an insight resolves after a quiet day, which is not a signal worth waking anyone for.
  • A PagerDuty channel gets a trigger with severity critical whatever severity you set on the channel, and a dedup key that is stable for the insight. The dedup key stops a second incident only while the first is still open. It never gets a resolve, so close the incident in PagerDuty when you have fixed the cause.

Verify a request

Section titled “Verify a request”

Check every request before you act on it:

  1. Read the raw body bytes. Do not parse and re-serialize them; any change to the bytes changes the signature.
  2. Read X-Spanlight-Timestamp and reject the request if it is more than 300 seconds away from your clock, in either direction. This stops someone replaying a captured request later.
  3. Compute HMAC-SHA256(secret, "<timestamp>." + body) with the signing secret as the key, write it as lowercase hex and prefix it with sha256=.
  4. Compare it with X-Spanlight-Signature in constant time.
  5. Deduplicate on X-Spanlight-Delivery. Delivery is at-least-once, so a retry can reach you after you already handled the first attempt.
import hashlib
import hmac
import time


def verify(secret: bytes, timestamp: str, body: bytes, header: str, tolerance: int = 300) -> bool:
    try:
        sent_at = int(timestamp)
    except ValueError:
        return False
    if abs(time.time() - sent_at) > tolerance:
        return False
    mac = hmac.new(secret, timestamp.encode() + b"." + body, hashlib.sha256)
    expected = ("sha256=" + mac.hexdigest()).encode()
    return hmac.compare_digest(expected, header.encode())
import { createHmac, timingSafeEqual } from 'node:crypto';


export function verify(secret, timestamp, body, header, toleranceSeconds = 300) {
  if (Math.abs(Date.now() / 1000 - Number(timestamp)) > toleranceSeconds) return false;
  const expected = 'sha256=' + createHmac('sha256', secret).update(`${timestamp}.`).update(body).digest('hex');
  const a = Buffer.from(expected);
  const b = Buffer.from(header);
  return a.length === b.length && timingSafeEqual(a, b);
}

Reference vector

Section titled “Reference vector”

Use this to test your implementation. With the secret whsec_testsecret (the ASCII text), the timestamp 1760000000 and the body

{"version":"2026-10-01","event":"alert.fired"}

the signature is

sha256=4cc8b9a14fd1521d3e288d2cc68077b30c14aeeaa10627c561859d0418d069a4

Responses and retries

Section titled “Responses and retries”

Answer with any 2xx status as soon as you have stored the event; Spanlight does not read the body of a 2xx answer and waits at most 10 seconds. Do slow work after you answer.

Your answer What Spanlight does
Any 2xx The delivery is done.
408, 429, any 5xx, a timeout or a connection error Retried.
Any other 4xx Failed for good. A retry would get the same answer.
Any 3xx Failed for good. Redirects are never followed; give the channel the final URL.

A retryable failure is tried again after 30 seconds, then after 1 minute, 2 minutes and so on, doubling each time up to a cap of 1 hour between attempts, for 8 attempts in all. After the last one the delivery is marked failed and stays visible, with its error, in the channel’s delivery list, where you can retry it by hand. The error shows the status and the start of your response body, never the URL or the secret.

PagerDuty test events

Section titled “PagerDuty test events”

A PagerDuty channel sends Events API v2 events instead of webhooks, and does not use the signature described above. Test on a PagerDuty channel triggers an incident with severity info and resolves it straight away, with the dedup key spanlight-test-<channel id>. It does not page anyone and leaves no open incident, although the two events appear briefly in the service’s activity. If the resolve cannot be sent, the incident stays open until you resolve it in PagerDuty.

URL and secret handling

Section titled “URL and secret handling”
  • The URL must start with https:// and must resolve to a public address. Addresses that are private, loopback, link-local or reserved are refused when you save the channel and again when Spanlight connects, so a DNS change after the check does not get around it. An operator running Spanlight inside a private network can lift both rules with WEBHOOK_ALLOW_PRIVATE_TARGETS (see Configuration).
  • Anyone in your organization who can read channels can see the webhook URL. Put tokens in the signing secret and verify the signature; do not put them in the query string.
  • Rotating the secret takes effect at once. Requests already queued are signed with the new secret when they are sent, so switch your receiver to accept the new secret before you rotate, or accept both for a while.

Check it worked

Section titled “Check it worked”

Press Test on the webhook channel’s row under Alerts, Channels. A tile under the row reads Test sent, and the row’s Not tested yet changes to a Verified badge.

Your endpoint received one request with X-Spanlight-Event: test and a body of version, event, occurred_at, org and url. Run the verification steps on it: the computed signature must equal X-Spanlight-Signature. Deliveries on the row lists the test with the status Sent.

Troubleshooting

Section titled “Troubleshooting”
  • The tile reads “Test failed: 401 Unauthorized”, or another 4xx status. Your endpoint refused the request. The tile shows the status and the first part of your response body. Spanlight sends no token, so allow the request by its signature, not by a header you expect. A 4xx other than 408 and 429 is final; fix the endpoint, then press Test again.
  • The tile reads “Queued for retry: Timed out waiting for the receiver” or “Queued for retry: Could not connect: …”. The endpoint did not answer within 10 seconds, or the connection failed. The worker retries; see Responses and retries. Check the URL and that the endpoint is reachable from the server.
  • “Blocked: the target resolves to a private or reserved address”, or a refusal when you save the channel. The URL must be https:// and resolve to a public address. Use a public URL, or ask the operator to set WEBHOOK_ALLOW_PRIVATE_TARGETS. See URL and secret handling.
  • The signature never matches. Sign the raw bytes, not re-serialized JSON, with the secret shown when you created or last rotated the channel, and compute <timestamp>.<body>. Test your code against the reference vector. A rotated secret applies to requests already queued.
  • Saving the channel answers 409 NOT_CONFIGURED. CREDENTIALS_KEYS is not set on the server. See Alert rules and channels.
  • A test or alert never arrives and the delivery stays Queued. The worker is not running; see the alerts runbook.

Next steps

Section titled “Next steps”
Alert rules and channelsGet told when error rate, latency, volume or cost crosses a line.
Budgets and the weekly digestSpend limits that notify or block gateway calls.
The DoctorFindings about how your application calls LLMs.