Scrapy allows redirect following in protocols other than HTTP
Impact
Scrapy was following redirects regardless of the URL protocol, so redirects were working for data://, file://, ftp://, s3://, and any other scheme defined in the DOWNLOAD_HANDLERS setting.
However, HTTP redirects should only work between URLs that use the http:// or https:// schemes.
A malicious actor, given write access to the start requests (e.g. ability to define start_urls) of a spider and read access to the spider output, could exploit this vulnerability to:
- Redirect to any local file using the
file://scheme to read its contents. - Redirect to an
ftp://URL of a malicious FTP server to obtain the FTP username and password configured in the spider or project. - Redirect to any
s3://URL to read its content using the S3 credentials configured in the spider or project.
For file:// and s3://, how the spider implements its parsing of input data into an output item determines what data would be vulnerable. A spider that always outputs the entire contents of a response would be completely vulnerable, while a spider that extracted only fragments from the response could significantly limit vulnerable data.
Patches
Upgrade to Scrapy 2.11.2.
Workarounds
Replace the built-in retry middlewares (RedirectMiddleware and MetaRefreshMiddleware) with custom ones that implement the fix from Scrapy 2.11.2, and verify that they work as intended.
References
This security issue was reported by @mvsantos at https://github.com/scrapy/scrapy/issues/457.
Affected software
Click a technology to see all vulnerabilities affecting it.
| Platform | Package | Vulnerable versions | Fixed in |
|---|---|---|---|
| Python (PyPI) | scrapy | < 2.11.2 | 2.11.2 |
References
- CVSS vector
CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:H/I:N/A:N- Weakness (CWE)
CWE-552- Platform
- Python (PyPI)
- Package
scrapy- Disclosed
- 14.05.2024
- Updated
- 11.10.2026 12:18
Sources and licenses
OSV / GitHub Advisory Database - GHSA-23j4-mw76-5v7h - CC BY 4.0
Vulnerability history: scrapy
All published vulnerabilities of this package (Python (PyPI)), newest first. Shows whether this is a one-off slip or a package that needs watching.
- Total entries
- 14
- First → latest
- 09.2017 09.2026
- Without a fix
- 1
- Exploited in attacks
- 0
Severity breakdown
- High 6
- Medium 8
Disclosures per year
- 2017: 1
- 2018: 0
- 2019: 0
- 2020: 0
- 2021: 1
- 2022: 3
- 2023: 0
- 2024: 6
- 2025: 1
- 2026: 2
-
Scrapy allows redirect following in protocols other than HTTP this entry
Know which vulnerabilities affect your application
With the DockRay integration and software inventory enabled, DockRay checks reported component versions against known vulnerabilities and notifies you about confirmed matches. Coverage depends on the supported technology and available advisory data.