NLTK: Corpus readers follow symlinks outside trusted roots despite pathsec enforcement

Summary

Several corpus readers still step outside NLTK's symlink-aware trusted-root model. They derive in-root paths from trusted corpus state, convert those paths back into plain strings, and reopen them with built-in open() rather than nltk.pathsec.open().

Details

• Vulnerability type: Path traversal and symlink boundary bypass
• Affected component: nltk.corpus.reader.ipipan, nltk.corpus.reader.crubadan, nltk.corpus.reader.lin
• Affected versions: Published 3.9.4 and current source v3.10.0-rc2 both reproduced.
• Patched versions: Not yet patched
• Root cause: Root-derived paths are reopened with raw open() without preserving the trusted-root boundary.

IPIPANCorpusReader opens header.xml derived from morph.xml, CrubadanCorpusReader opens table.txt directly, and LinThesaurusCorpusReader opens simN.lsp paths returned from its own root helpers. Under pathsec.ENFORCE=True, a symlink placed inside the trusted corpus root can point outside the root and still be parsed successfully. It was confirmed parsed outside-root content is returned through public methods such as channels(), domains(), categories(), langs(), crubadan_to_iso(), synonyms(), and scored_synonyms().

PoC

Preconditions

• The application processes attacker-influenced corpora inside a trusted NLTK data root or trusted corpus directory.

Steps

1. Create a trusted corpus root and keep pathsec.ENFORCE=True with that root allowlisted.
2. Place symlinked reader inputs such as header.xml, table.txt, or simN.lsp inside the root and point them to external files.
3. Instantiate the corresponding corpus reader and call its normal public methods.
4. Observe that parsed outside-root values are returned even though pathsec.open() blocks the same symlink targets.

Minimal reproducible excerpt

{'ipipan': ['LEAK', 'TOPSECRET', 'CLASSIFIED'], 'crubadan': ['LEAK'], 'lin': [('LEAK', 9.5)]}

Impact

An attacker who can stage corpus files or symlinks under a trusted data root can disclose outside-root content through normal corpus-reader results, defeating the boundary NLTK documents for shared and untrusted-input environments.

Remediation

Preserve PathPointer and required_root semantics end to end. Replace direct open() calls with nltk.pathsec.open() or a reader helper that keeps the trusted-root boundary intact.

References

• https://github.com/nltk/nltk/blob/3.9.4/nltk/corpus/reader/ipipan.py#L162-L192
• https://github.com/nltk/nltk/blob/3.9.4/nltk/corpus/reader/crubadan.py#L74-L98
• https://github.com/nltk/nltk/blob/3.9.4/nltk/corpus/reader/lin.py#L40-L43
• https://github.com/nltk/nltk/blob/v3.10.0-rc2/nltk/pathsec.py#L521-L545

Fix + full-codebase audit (verified)

I swept every raw file open in the corpus readers, not just the three the umbrella named:

ipipan already validates via the earlier #3727 fix — unchanged.

Fix

Each site now calls nltk.pathsec.validate_path(path, required_root=…) before opening. Where the reader has a concrete corpus root, the check is scoped with required_root (rejects any escape outside that root). XMLCorpusView carries no root, so it falls back to the global data-root sandbox via getattr(self, "_root", None) — which also avoids an AttributeError on the bare-string path.

Reproduced (captured)

raw open(symlink) reads: 'TOPSECRET_OUTSIDE_ROOT'          <- the bypass
validate_path(symlink, required_root): ValueError -> BLOCKS the escape
validate_path(legit in-root): PASSED                       <- loads normally

Honest residual

The global-sandbox fallback (toolbox, named_entity, xmldocs-view) is only as tight as the allowed-roots list, which currently includes the system temp dir. Scoping every reader with required_root and removing the temp dir from the allowed roots would harden it further (separate advisory / task).

Tests

test_corpus_reader_pathsec.py — symlink escape rejected, in-root file allowed, XMLCorpusView string-fileid no AttributeError, MTEFileReader out-of-root rejected. 46 existing corpus/toolbox tests pass; all edited modules import (no circular import). pre-commit (black/isort/ruff) clean.

Scope caveat

validate_path blocks every symlink escape variant (verified) and equals pathsec.open()'s guarantee, but does NOT block hardlinks (no symlink to resolve; tracked separately as GHSA-f794-5jv7-7672) or the validate-then-open TOCTOU race (shared by pathsec.open; needs O_NOFOLLOW/openat).