Skip to content
SpanlightSpanlightSpanlight

OTLP ingestion

Spanlight accepts OTLP/HTTP traces, so an OpenTelemetry exporter can report to it without the Python SDK. The native JSON endpoint (POST /v1/traces) is described at the end of this page.

Before you start

  • A running Spanlight instance that your exporter can reach
  • A project API key with the ingest:write scope, which is the default scope
  • An OpenTelemetry exporter that sends model calls over OTLP/HTTP, as JSON or protobuf, with OpenTelemetry GenAI, OpenInference, Vercel AI SDK or Traceloop attributes

Endpoint

Section titled “Endpoint”
URL POST <host>/v1/otlp/traces, for example http://localhost:8080/v1/otlp/traces
Authentication Authorization: Bearer spl_live_…, a project API key with the ingest:write scope (the default scope of a new key)
Content types application/x-protobuf or application/json. Anything else is 415.
Compression Content-Encoding: gzip or deflate
Limits 5 MiB per request (after decompression) and 1000 spans per request, otherwise 413
Rate limit 50 requests per second per key, with a burst of 100. Over the limit is 429 with Retry-After.

The response is 200 with {} (JSON) or an empty ExportTraceServiceResponse (protobuf). If some spans were rejected, the body carries partialSuccess with rejectedSpans and an errorMessage naming the first few. A body that is not valid OTLP is 400 with code INVALID_OTLP. An API key is not a personal access token: a token on this endpoint is 401.

Configure an exporter

Section titled “Configure an exporter”

Point the exporter at the full traces URL and pass the key as a header. With the standard OpenTelemetry environment variables:

Terminal window
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://localhost:8080/v1/otlp/traces
export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer%20spl_live_..."

Use the traces-specific endpoint variable. The general OTEL_EXPORTER_OTLP_ENDPOINT makes exporters append /v1/traces to it, which is Spanlight’s native endpoint, not the OTLP one. The %20 is the URL-encoded space that the OpenTelemetry specification requires in header values.

In Python with the OpenTelemetry SDK:

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor


provider = TracerProvider(
    resource=Resource.create(
        {"service.name": "support-bot", "deployment.environment.name": "production"}
    )
)
provider.add_span_processor(
    BatchSpanProcessor(
        OTLPSpanExporter(
            endpoint="http://localhost:8080/v1/otlp/traces",
            headers={"Authorization": "Bearer spl_live_..."},
        )
    )
)
trace.set_tracer_provider(provider)

A request by hand, to check the key and the endpoint:

Terminal window
now=$(date +%s)
curl -sS http://localhost:8080/v1/otlp/traces \
  -H "Authorization: Bearer $SPANLIGHT_API_KEY" \
  -H "Content-Type: application/json" \
  --data-binary @- <<EOF
{
  "resourceSpans": [{
    "resource": {"attributes": [{"key": "service.name", "value": {"stringValue": "curl-test"}}]},
    "scopeSpans": [{"spans": [{
      "traceId": "5b8efff798038103d269b633813fc60c",
      "spanId": "eee19b7ec3c1b174",
      "name": "chat",
      "kind": 3,
      "startTimeUnixNano": "${now}000000000",
      "endTimeUnixNano": "$((now + 1))000000000",
      "attributes": [
        {"key": "gen_ai.request.model", "value": {"stringValue": "claude-haiku-4-5"}},
        {"key": "gen_ai.usage.input_tokens", "value": {"intValue": "120"}},
        {"key": "gen_ai.usage.output_tokens", "value": {"intValue": "40"}}
      ]
    }]}]
  }]
}
EOF

A successful call prints {}. Spans that started more than 90 days ago are rejected, so use a current timestamp.

How attributes map

Section titled “How attributes map”

Spanlight reads four conventions: the OpenTelemetry GenAI attributes (gen_ai.*, also written by LiteLLM and by Traceloop’s provider spans), OpenInference (llm.*), the Vercel AI SDK (ai.*) and Traceloop’s workflow spans (traceloop.*). It works out for each span on its own which one the span speaks, so one trace may mix them, and fills the gaps of the other three from any gen_ai.* attributes on the same span. The convention it used is stored on the span as spanlight.convention, and the attributes it did not read are kept on the span. Every attribute, fallback and error rule is listed in OTLP attribute mapping. OTLP compatibility shows, for each recorded exporter library and version, which fields it fills and which it does not send, and the integration guides for the Vercel AI SDK, LiteLLM, OpenLLMetry and OpenInference show the setups those recordings were made with.

Frameworks often report a call’s usage twice, on the step and on the provider call inside it. Spanlight counts it once, on the innermost span that reports it, and marks the outer span as a usage rollup; see Usage counted once.

Cost is computed on the server from the model and the token counts. A span with an unknown model, or without token counts, has an unknown cost, not zero. As with the SDK, secrets are redacted from payloads before they are stored, and a project can switch payload capture off. Include cached tokens in the input token count.

A failed span gets an error class from its status or message. When it is a 429 or an overloaded provider, it also gets a rate-limit kind, worked out from the status message only: the OpenTelemetry GenAI attributes carry no provider error type or code, so a 429 with no message has the kind unknown. To give Provider health more to work with, set the provider (gen_ai.provider.name) and keep the provider’s error message in the span status. A client may also set the provider.error.type, provider.error.code and provider.ratelimit attributes itself; they are filtered and read like those of a native span.

Trace metadata

Section titled “Trace metadata”

Give a trace metadata, such as a customer or a feature, with attributes named spanlight.metadata.<key>. Spanlight reads them on any span of the trace, not only the root, and on the resource, so a resource attribute such as spanlight.metadata.team tags every span of the service. A span’s value wins over its resource’s for the same key.

from opentelemetry import trace


span = trace.get_current_span()
span.set_attribute("spanlight.metadata.customer_id", "acme")
span.set_attribute("spanlight.metadata.feature", "summarize-thread")

The key is the rest of the name, lower-cased, and must then be 1 to 64 characters of a-z, 0-9, _, . and -. A value is 1 to 256 characters; a number or boolean is stored as its text. A trace keeps at most 32 keys. These attributes are not stored again on the span.

The metadata attributes of the other conventions are read the same way, and lose to spanlight.metadata.<key> when both name a key: ai.telemetry.metadata.<key> and ai.settings.context.<key> (Vercel AI SDK), traceloop.association.properties.<key> (Traceloop) and the OpenInference metadata attribute. See Trace fields.

An invalid key or value is dropped on its own, never the span. The response then carries a warning in partialSuccess, with rejectedSpans at 0 when no span was rejected:

{"partialSuccess": {"rejectedSpans": "0", "errorMessage": "trace metadata: 2 entries dropped (invalid spanlight.metadata.* key or value, or over 32 keys)"}}

The count can be higher than the entries you sent: an invalid resource attribute is counted once for each span of the resource.

Native JSON

Section titled “Native JSON”

POST /v1/traces takes the format the Python SDK sends, and is the simplest way to report from a language without an SDK:

{
  "spans": [
    {
      "trace_id": "5b8efff798038103d269b633813fc60c",
      "span_id": "eee19b7ec3c1b174",
      "name": "answer",
      "kind": "llm",
      "status": "ok",
      "start_time": "2026-10-09T10:00:00+00:00",
      "end_time": "2026-10-09T10:00:01.500+00:00",
      "provider": "anthropic",
      "model": "claude-haiku-4-5",
      "usage": {"input_tokens": 120, "output_tokens": 40, "cached_tokens": 0},
      "input": {"messages": [{"role": "user", "content": "Hi"}]},
      "output": "Hello!",
      "trace": {"name": "support-request", "user_id": "u_42", "session_id": "chat-123", "tags": ["beta"]}
    }
  ]
}

Timestamps are ISO-8601 with an offset, or integer Unix nanoseconds. The response lists what was stored:

{"accepted": 1, "rejected": []}

Each span is validated on its own, so one bad span appears in rejected (with its index, span id and reason) without failing the batch. A span that is sent again with the same ids updates the stored one, so retrying a batch does not double-count by design. The limits, the key scope and the rate limit are the same as for OTLP. Validation rules, limits and the merge rules for trace fields are listed under ingestion in the API conventions.

Check it worked

Section titled “Check it worked”

Run the request by hand from Configure an exporter. A successful call prints {}. Open Traces for the same project: the list has a trace named chat, the name of the root span. Open it: the span shows the model claude-haiku-4-5 and its token counts, 120 in and 40 out. A trace from your own exporter carries the name of its root span, or the service.name when that span has no name.

Troubleshooting

Section titled “Troubleshooting”
  • The exporter gets 422 VALIDATION_ERROR or 400 INVALID_JSON. It sent to /v1/traces, the native endpoint, because it appended the path to a base URL. Set OTEL_EXPORTER_OTLP_TRACES_ENDPOINT to the full URL. See The exporter appends /v1/traces.
  • 401 UNAUTHORIZED. The Authorization header did not arrive, the key is wrong or revoked, or you used a personal access token. Encode the space as %20 in OTEL_EXPORTER_OTLP_HEADERS and use a project API key. See The Authorization header does not reach Spanlight.
  • 415 UNSUPPORTED_MEDIA_TYPE, or a gRPC exporter cannot connect. Spanlight accepts OTLP over HTTP only, as protobuf or JSON. Set OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf. See The protocol is not OTLP over HTTP.
  • 403 KEY_SCOPE. The key was created without the ingest:write scope. Scopes are fixed at creation, so create a new key. See The key lacks the ingest:write scope.
  • My agent’s cost is doubled. Spanlight counts a call’s usage once when the outer span and the provider call are in the same trace and linked by their parent span ids: the outer span is then a usage rollup and is left out of totals. Check that your instrumentation passes the trace context from the step to the provider call. Traces stored before your server had usage rollups are corrected when one of their spans arrives again, or by an operator with spanlight recompute-usage-rollups --since 30d. See Usage counted once.
  • The answer is 200 but a trace is missing. The response body holds partialSuccess with rejectedSpans. Read the errorMessage, then see Spans are rejected.

Next steps

Section titled “Next steps”
Python SDKInstall the package and trace functions and provider calls.
Releases and usersCompare two versions of your app and see which users cost the most.
The DoctorFindings about how your application calls LLMs.
API keys, tokens and administrationScoped API keys and personal access tokens.