RabbitMQ Java client has frame-level OOM: Math.min(maxInboundMessageBodySize, 0) defeats frame size enforcement

Vulnerability

In AMQConnection.java (line 435-436), after Connection.Tune negotiation, the frame-max limit is set via:

_frameHandler.setFrameMax(
    Math.min(this.maxInboundMessageBodySize, frameMax));

When frameMax = 0 (meaning "unlimited" per AMQP spec), Math.min(67108864, 0) = 0. This value is then passed to Utils.framePayloadLimit(0) which returns Integer.MAX_VALUE (line 77-79 of Utils.java):

static int framePayloadLimit(int frameMax) {
    if (frameMax <= 0) {
      return Integer.MAX_VALUE;
    }
    // ...
}

This completely defeats the maxInboundMessageBodySize protection (default 64MB) at the frame level.

Attack Scenario

A malicious AMQP server (or MITM) sends Connection.Tune with frameMax=0:

1. Client defaults: requestedFrameMax = 0 (ConnectionFactory.DEFAULT_FRAME_MAX, line 82)
2. negotiatedMaxValue(0, 0) = Math.max(0, 0) = 0 (line 673-676)
3. Math.min(maxInboundMessageBodySize, 0) = 0 — 64MB cap defeated
4. framePayloadLimit(0) = Integer.MAX_VALUE — no frame size enforcement
5. Attacker sends a single frame with frameSize = 0x1FFFFFFF (~500MB)
6. Frame.readFrom() (line 135) executes new byte[frameSize] — OOM crash

The frame does not need to be a body frame — method frames, header frames, or heartbeat frames with a crafted size field all trigger the allocation before any content-level check fires.

Root Cause

The AMQP spec uses frameMax=0 to mean "unlimited", but Math.min treats it as the integer value zero. The intent of line 435-436 was to take the smaller of the two limits, but when one limit uses 0-means-unlimited semantics, Math.min always selects the zero, disabling the other limit.

Impact

• Default configuration is vulnerable: Both requestedFrameMax (client) and legitimate servers' frameMax in Tune may be 0
• Single-frame OOM: One malicious frame triggers up to ~2GB allocation (Integer.MAX_VALUE bytes)
• Bypasses existing protection: maxInboundMessageBodySize (introduced to cap allocations at 64MB) is entirely defeated at the frame level
• Different from ValueReader OOM: This is a frame-layer allocation in Frame.readFrom(), not a value-layer allocation in ValueReader.readBytes()

Affected Code

• AMQConnection.java:435-436 — Math.min with 0-means-unlimited
• Utils.java:77-79 — framePayloadLimit(0) returns Integer.MAX_VALUE
• Frame.java:135 — new byte[frameSize] allocation site
• ConnectionFactory.java:82 — DEFAULT_FRAME_MAX = 0

Suggested Fix

int effectiveFrameMax = (frameMax == 0)
    ? this.maxInboundMessageBodySize
    : Math.min(this.maxInboundMessageBodySize, frameMax);
_frameHandler.setFrameMax(effectiveFrameMax);

This treats frameMax=0 as "use maxInboundMessageBodySize as the cap" instead of "zero".