Inefficient Regular Expression Complexity in nltk (word_tokenize, sent_tokenize)
Impact
The vulnerability is present in PunktSentenceTokenizer, sent_tokenize and word_tokenize. Any users of this class, or these two functions, are vulnerable to a Regular Expression Denial of Service (ReDoS) attack.
In short, a specifically crafted long input to any of these vulnerable functions will cause them to take a significant amount of execution time. The effect of this vulnerability is noticeable with the following example:
from nltk.tokenize import word_tokenize
n = 8
for length in [10**i for i in range(2, n)]:
# Prepare a malicious input
text = "a" * length
start_t = time.time()
# Call `word_tokenize` and naively measure the execution time
word_tokenize(text)
print(f"A length of {length:<{n}} takes {time.time() - start_t:.4f}s")Which gave the following output during testing:
A length of 100 takes 0.0060s
A length of 1000 takes 0.0060s
A length of 10000 takes 0.6320s
A length of 100000 takes 56.3322s
...I canceled the execution of the program after running it for several hours.
If your program relies on any of the vulnerable functions for tokenizing unpredictable user input, then we would strongly recommend upgrading to a version of NLTK without the vulnerability, or applying the workaround described below.
Patches
The problem has been patched in NLTK 3.6.6. After the fix, running the above program gives the following result:
A length of 100 takes 0.0070s
A length of 1000 takes 0.0010s
A length of 10000 takes 0.0060s
A length of 100000 takes 0.0400s
A length of 1000000 takes 0.3520s
A length of 10000000 takes 3.4641sThis output shows a linear relationship in execution time versus input length, which is desirable for regular expressions. We recommend updating to NLTK 3.6.6+ if possible.
Workarounds
The execution time of the vulnerable functions is exponential to the length of a malicious input. With other words, the execution time can be bounded by limiting the maximum length of an input to any of the vulnerable functions. Our recommendation is to implement such a limit.
References
- The issue showcasing the vulnerability: https://github.com/nltk/nltk/issues/2866
- The pull request containing considerably more information on the vulnerability, and the fix: https://github.com/nltk/nltk/pull/2869
- The commit containing the fix: 1405aad979c6b8080dbbc8e0858f89b2e3690341
- Information on CWE-1333: Inefficient Regular Expression Complexity: https://cwe.mitre.org/data/definitions/1333.html
For more information
If you have any questions or comments about this advisory:
- Open an issue in github.com/nltk/nltk
- Email us at nltk.team@gmail.com
Software afectado
Haz clic en una tecnología para ver todas sus vulnerabilidades.
| Plataforma | Paquete | Versiones vulnerables | Corregida en |
|---|---|---|---|
| Python (PyPI) | nltk | < 3.6.6 | 3.6.6 |
Referencias
- https://github.com/nltk/nltk/pull/2869
- https://github.com/nltk/nltk/commit/1405aad979c6b8080dbbc8e0858f89b2e3690341
- https://github.com/nltk/nltk/security/advisories/GHSA-f8m6-h2c7-8h9x
- https://github.com/nltk/nltk/issues/2866
- https://nvd.nist.gov/vuln/detail/CVE-2021-43854
- https://github.com/nltk/nltk
- https://github.com/pypa/advisory-database/tree/main/vulns/nltk/PYSEC-2021-859.yaml
- Vector CVSS
CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H- Debilidad (CWE)
CWE-400- Plataforma
- Python (PyPI)
- Paquete
nltk- Publicada
- 23.12.2021
- Actualizado
- 11.10.2026 13:19
Fuentes y licencias
OSV / GitHub Advisory Database - PYSEC-2021-859 - PyPA Advisory Database; CC BY 4.0
OSV / GitHub Advisory Database - GHSA-f8m6-h2c7-8h9x - CC BY 4.0
Historial de vulnerabilidades: nltk
Todas las vulnerabilidades publicadas de este paquete (Python (PyPI)), de la más reciente a la más antigua. Muestra si es un fallo aislado o un paquete que hay que vigilar.
- Entradas en total
- 52
- Primera → última
- 08.2019 09.2026
- Sin corrección
- 6
- Explotada en ataques
- 0
Distribución por gravedad
- Crítica 5
- Alta 32
- Media 13
- Baja 2
Divulgaciones por año
- 2019: 1
- 2020: 0
- 2021: 2
- 2022: 1
- 2023: 0
- 2024: 1
- 2025: 0
- 2026: 47
-
Inefficient Regular Expression Complexity in nltk (word_tokenize, sent_tokenize) esta entrada
Identifica las vulnerabilidades de tu aplicación
Con la integración de DockRay y el inventario de software activado, DockRay compara las versiones notificadas de los componentes con las vulnerabilidades conocidas y avisa de las coincidencias confirmadas. La cobertura depende de la tecnología compatible y de los avisos disponibles.