Contao crawler leaks auth credentials to external hosts

Summary

Contao's crawler tries to prevent confidential HTTP client options from being sent to external domains by creating a scoped client: full options for root page origins, cleaned options for everything else. The cleaner removes Cookie and Authorization headers, but it removes the non-Symfony option names basic_auth and bearer_auth instead of Symfony HttpClient's real auth_basic and auth_bearer options.

When contao.crawl.default_http_client_options contains Basic or Bearer authentication for a protected staging/production site, those credentials remain in the "clean" client used for external links or configured additional URIs. An attacker who can get an external URL crawled, for example through a link on a crawled page while the broken-link checker is enabled, can receive the crawler credentials.

Technical Detail

Root Cause

// core-bundle/src/Crawl/Escargot/Factory.php:175-209 @ e550b92a01ef625bd546e6c3956dd200af05ebf0
private function createHttpClient(array $options = []): HttpClientInterface
{
    $options = array_merge_recursive(
        [
            'headers' => [
                'accept' => 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
                'user-agent' => self::USER_AGENT,
            ],
            'max_duration' => 10,
        ],
        array_merge_recursive($this->getDefaultHttpClientOptions(), $options),
    );

    $cleanOptions = $this->cleanOptionsFromConfidentialData($options);

    if ($options === $cleanOptions) {
        return ($this->httpClientFactory)($options);
    }

    $scopedOptionsByRegex = [];

    foreach ($this->getRootPageUriCollection()->all() as $rootPageUri) {
        $scopedOptionsByRegex[preg_quote($this->getOriginFromUri($rootPageUri))] = $options;
    }

    return new ScopingHttpClient(($this->httpClientFactory)($cleanOptions), $scopedOptionsByRegex);
}

// core-bundle/src/Crawl/Escargot/Factory.php:226-247 @ e550b92a01ef625bd546e6c3956dd200af05ebf0
foreach ($options as $k => $v) {
    if ('headers' === $k) {
        foreach ($v as $header => $value) {
            if (\in_array(strtolower($header), ['authorization', 'cookie'], true)) {
                continue;
            }

            $cleanOptions['headers'][$header] = $value;
        }

        continue;
    }

    if ('basic_auth' === $k || 'bearer_auth' === $k) {
        continue;
    }

    $cleanOptions[$k] = $v;
}

Symfony HttpClient authentication options are auth_basic and auth_bearer; Contao's own manual documents auth_basic for crawler Basic Authentication. Because the cleaner only strips basic_auth and bearer_auth, the "clean" default client for non-root-page hosts still carries the real auth options. The existing factory test intends to assert that Authorization is not sent to www.foreign-domain.com, but its mock client factory ignores the $defaultOptions argument, so it does not catch auth options that survive into HttpClient::create($cleanOptions).

Suggested Mitigation

Strip the actual Symfony HttpClient authentication option keys from the clean client. Include NTLM as a defensive extension because Symfony documents it as another auth option.

-            if ('basic_auth' === $k || 'bearer_auth' === $k) {
+            if (\in_array($k, ['auth_basic', 'auth_bearer', 'auth_ntlm', 'basic_auth', 'bearer_auth'], true)) {
                 continue;
             }

Also update the factory test so the mock factory records or preserves $defaultOptions; otherwise the test does not verify what HttpClient::create($cleanOptions) receives in production.

Impact

• Direct primitive: disclosure of crawler Basic/Bearer credentials to an external host reached by the crawler.
• Chain potential: if those credentials protect a staging or pre-publication environment, an attacker can use them to access that environment. The impact depends on what the leaked credential unlocks.
• Realistic exploitation: a content editor adds a link to https://attacker.example/probe on a page that the crawler visits. When an administrator or scheduled maintenance run starts the broken-link checker with crawler Basic/Bearer authentication configured, the request to the attacker URL includes the generated Authorization header.