Alert rules and channels

An alert rule watches one metric of a project and notifies channels when the metric crosses a line, and again when it comes back. Rules belong to a project and are managed under Alerts, Rules; channels belong to the organization and are managed under Alerts, Channels. Budgets use the same channels and the same delivery path.

Anyone who can see a project can read its rules and events. Only organization admins and owners can create, edit, mute or delete rules and channels, acknowledge events, and retry deliveries. A member sees the controls disabled.

Before you start

• The admin or owner role in the organization. Every role can read rules and events
• The worker running, with ALERTS_EVALUATION_ENABLED on (the default)
• To save a Slack, webhook or PagerDuty channel: CREDENTIALS_KEYS set on the server
• To send email: an email provider (EMAIL_PROVIDER) set on the server

Channels

A channel is a place alerts go. One channel can serve any number of rules and budgets in the organization, and an organization can have up to 20.

Kind	Sends	You provide
Email	One message per recipient	1 to 10 addresses
Slack	A message with the metric, value, threshold and a button that opens the alert	A Slack incoming-webhook URL (https://hooks.slack.com/…)
Webhook	A signed POST with a JSON body, see Alert webhooks	An https:// URL
PagerDuty	An Events API v2 trigger when a rule fires and a resolve when it recovers	A routing key and a severity (critical, error, warning or info)

Who can receive email. Every recipient must be a member of the organization with a verified email address; anything else is refused when you save the channel (422 RECIPIENT_NOT_MEMBER). The check is made again when the message is queued, so an address removed from the organization after the channel was saved stops receiving alerts. This keeps alert email from being used to send messages to strangers. An operator can lift the rule with ALERT_EMAIL_ANY_RECIPIENT (see Configuration). Email channels need an email provider; without one, testing the channel answers 409 NOT_CONFIGURED.

Secrets are shown once. The Slack URL, the PagerDuty routing key and a webhook’s signing secret are stored encrypted with CREDENTIALS_KEYS and never returned by the API, written to a log or kept in the audit log. A webhook’s signing secret is generated by Spanlight and shown only when the channel is created and when you rotate it. Saving a Slack, webhook or PagerDuty channel without CREDENTIALS_KEYS answers 409 NOT_CONFIGURED. A webhook URL, unlike its secret, is visible to every member, so authenticate with the signature and not with a token in the URL.

Test a channel. Test on the channel’s row delivers a test message straight away and reports sent, failed or pending (a failure that a retry might fix; the worker keeps trying). A successful test marks the channel verified; editing its target or secret clears that mark. Each channel allows 10 tests an hour. A PagerDuty test triggers and immediately resolves an info incident (see Alert webhooks).

Delivery log. Deliveries on the channel’s row opens a list of its deliveries, newest first, with status (Queued, Sent, Retrying or Failed), attempts, the event it carried and the last error. The error shows a status and the start of the response, never a URL or a secret. For an email channel there is one row per recipient. A failed delivery can be retried by hand with Retry; it keeps its id, so a webhook receiver sees the same X-Spanlight-Delivery.

Deleting a channel leaves the rules that named it untouched; they skip the missing channel.

Rules

A rule has a name, a metric, a window, a condition, optional filters, a cooldown and up to 10 channels. A project can have up to 100 rules.

Metrics

Metric	What it measures over the window	With no data in the window
llm_calls	Number of LLM calls	0
error_rate	LLM calls that ended in error divided by LLM calls	Unknown
error_calls	Number of LLM calls that ended in error, shown as Failed LLM calls	0
p95_ms	95th percentile duration of LLM calls	Unknown
ttft_p95_ms	95th percentile time to first token, over calls that have one	Unknown
tokens	Input plus output tokens of all spans	0
cost_usd	Cost of all priced spans	0 with no spans; unknown when spans exist and none has a price

Unknown is never treated as zero. A rule whose metric is unknown in a window does not change state: it neither fires nor resolves. The rule’s pill reads “No data yet” until the first evaluation pass and then shows its state (OK for a rule that has not fired); its last value stays “—” until a value exists. That is deliberate: a quiet hour should not look like a recovery from a latency problem. If you want an alert for silence, use llm_calls with the condition “below 1”, which reads 0 for an empty window.

cost_usd counts only priced spans, so it is a lower bound when some models have no known price, like the overview.

Filters narrow the rule to an environment, a provider or a model. The environment matches exactly. A model filter matches the model exactly or with a snapshot suffix (-YYYYMMDD, -YYYY-MM-DD or -latest), like prices do: gpt-4o also covers gpt-4o-2024-08-06 but not gpt-4o-mini.

Two more filters apply to error_rate and error_calls only, and the rule editor shows them as Error class and Rate-limit kind (“Any” by default) only for Error rate and Failed LLM calls: error_class (one error class, such as provider_5xx) and rate_limit_kind (one rate-limit kind, such as spend_cap). They narrow the failed calls a rule counts, never the calls a rate is divided by: error_rate with error_class rate_limit is the rate-limited calls divided by all LLM calls. “Any spend_cap refusal in 15 minutes” is error_calls above 0 with rate_limit_kind spend_cap. On any other metric these filters are refused with 422, for example filters.error_class: applies to error_rate and error_calls. With either filter, calls the Integration Lab faulted are not counted, so a Lab drill only fires rules without them.

Threshold rules

A threshold rule fires when the metric over the last window_minutes is above (gt), at or above (gte), below (lt) or at or below (lte) a fixed threshold. The window is 5 to 1440 minutes. “Error rate above 5 % over 15 minutes in production” is a threshold rule.

Anomaly rules

An anomaly rule compares the metric with its own recent past instead of a fixed line. Spanlight takes the baseline_windows (3 to 30) windows of the same length that came directly before the current one, back to back, and computes:

mean   = average of the baseline values that are known
stdev  = sample standard deviation of those values (divisor n - 1)
spread = max(stdev, 0.1 × mean)
upper  = mean + sensitivity × spread        (conditions gt, gte)
lower  = max(mean − sensitivity × spread, 0)   (conditions lt, lte)

The rule fires when the current value is past that line. sensitivity is 0.5 to 5 and defaults to 3; a higher value means fewer alerts. The 0.1 × mean floor stops a perfectly steady baseline from making every small wobble an anomaly. Baseline windows with an unknown value are left out, and with fewer than 3 known values there is no baseline: the rule records “no data” and does not change state until enough history exists. In a notification, the threshold is the line at that moment.

Preview

The rule editor draws the last 7 days of the metric, with the anomaly band for an anomaly rule, before you save. Points are spaced at least an hour apart (up to 168 of them). Older hours are read from hourly summaries, so a percentile in the preview is approximate; the editor says so.

Cooldown, mute and acknowledge

• Cooldown (0 to 1440 minutes, default 15). After a rule resolves it stays quiet for this long, even if the metric breaches again, so a metric that flaps around the line does not send a message every minute.
• Mute silences notifications for up to 30 days. A muted rule keeps evaluating and recording events; it only sends nothing. Unmute at any time.
• Acknowledge marks an event as seen by you, for the team’s own bookkeeping. It does not stop the alert, change its state or send anything. An event can be acknowledged once, open or resolved.
• Editing a firing rule resolves it. Changing a rule’s kind, metric, condition, threshold, window, filters, baseline or sensitivity, disabling it, or deleting it ends its open event at that moment and sends alert.resolved to its channels (unless it is muted). The next evaluation starts from a clean state, and the cooldown applies. Renaming, changing channels or cooldown, and enabling keep the state.

How evaluation works

The worker evaluates every enabled rule once a minute. Each rule is evaluated in its own transaction. A state change, its event, and every notification it causes are written together, so a crash can never leave an event without its notifications, or notifications without an event.

• State. A rule is ok or firing. It moves to firing on the first breach (unless it is inside its cooldown) and back to ok on the first pass that does not breach.
• Where the numbers come from. Whole hours that ended more than an hour ago come from the hourly rollups; the recent part of the window and any partial hour at its edges come from the raw spans. Counts, tokens and cost are exact either way. A percentile merged from rollup histograms is approximate (a histogram bucket is about 1.5 times wide, so an estimate can be off by roughly 25 %), while a percentile over raw spans only is exact. A rule on a short window therefore reads mostly exact numbers and one on a long window mostly approximate ones; see Metrics over long windows.
• Delivery is at least once. Notifications go through a transactional outbox. If a worker stops after a provider accepted a message but before Spanlight recorded it, the message is sent again after a 60 second lease expires. Webhook receivers deduplicate on X-Spanlight-Delivery and PagerDuty on its dedup key; Slack and email have no such key, so a rare duplicate is possible there.
• Retries. A failure that might pass, such as a timeout, 429 or 5xx, is retried 30 seconds later and then at doubling intervals up to an hour, 8 attempts in all. A failure that cannot pass, such as a 4xx answer or a blocked address, fails at once. A failed delivery stays in the delivery log until you retry it.
• Lag. Evaluation runs one pass a minute, so an alert fires up to about a minute after the metric crosses its line, plus the time the data takes to arrive. GET /health/ready reports alert_evaluation_lag_s, the age of the last finished pass; a healthy worker keeps it under about 90.
• Outage. If evaluation stops, rules keep their last state and nothing fires or resolves until it runs again. Operators can switch it off with ALERTS_EVALUATION_ENABLED, or run one pass by hand with spanlight alerts evaluate (add --dry-run to roll the pass back).

Metrics for operators

The worker exports Prometheus metrics for the evaluation (spanlight_alert_rules_evaluated_total, spanlight_alert_transitions_total, spanlight_alert_evaluation_duration_seconds) and for deliveries; the bundled Grafana dashboard has alert panels for them.

Settings

Variable	Default	Effect
ALERTS_EVALUATION_ENABLED	true	Schedule the once-a-minute evaluation. Off, no alert fires.
WEEKLY_DIGEST_ENABLED	true	Schedule the weekly digest.
WEBHOOK_ALLOW_PRIVATE_TARGETS	false	Allow http:// and private addresses for webhook channels.
ALERT_EMAIL_ANY_RECIPIENT	false	Allow email channels to send to addresses that are not verified members.
PAGERDUTY_EVENTS_URL	https://events.pagerduty.com/v2/enqueue	Where PagerDuty channels send events.

Full descriptions are in Configuration.

Check it worked

Save a channel, then press Test on its row. A tile under the row reads Test sent, and the row’s Not tested yet changes to a Verified badge. Then save a rule and open Alerts, Rules. Right after saving, the rule’s pill reads No data yet. After the worker’s first pass, within about a minute, it reads OK or Firing. OK can also mean the metric had no data in the window: open the rule and look at its last value, which reads “—” when it is unknown. See the first row below.

Troubleshooting

You see	Check
A rule never fires	Its state: “No data yet” means the rule has not been evaluated yet. After that, an OK rule whose last value reads “—” had no data in the window, or too little history for an anomaly baseline. Check the filters, in particular the environment and model spelling, and that ALERTS_EVALUATION_ENABLED is on.
A rule fired but nobody was told	The rule may be muted, or have no channel. Open the channel’s delivery log: a Failed row shows why.
A channel says Failed	A 4xx is permanent: fix the URL or key, then Retry. A timeout or 5xx keeps retrying by itself.
“Blocked: the target resolves to a private or reserved address”	Webhook, Slack and PagerDuty targets must be public. For a receiver inside your own network, an operator can set WEBHOOK_ALLOW_PRIVATE_TARGETS.
RECIPIENT_NOT_MEMBER	The address is not a member of the organization, or its email is not verified. Invite the person first.
NOT_CONFIGURED when saving a channel	CREDENTIALS_KEYS is not set on the api and the worker.
Delivery rows pile up, or evaluation is late	See the alerts runbook, in particular a stuck outbox and evaluation lag.

Next steps

Budgets and the weekly digestSpend limits that notify or block gateway calls.

Alert webhooksSigned requests for alerts, budgets and critical insights.

The DoctorFindings about how your application calls LLMs.

Run alertsStuck deliveries, failed deliveries and late evaluation.