Netty: Fragmented ClientHello records trigger quadratic pre-handshake reassembly in default SNI parsing

Summary

Netty's default SNI entrypoint reparses and recopies previously received ClientHello fragments on every additional TLS handshake record. A remote peer can send a small first record that advertises a large ClientHello length and then drip the body in many tiny records, causing superlinear (quadratic) CPU work before the handshake completes. With 4095 one-byte fragments, the handler recopies 8,386,560 bytes from only 24,579 bytes on the wire — a 341× amplification ratio.

Affected Entrypoints

• io.netty.handler.ssl.SniHandler — default constructors
• io.netty.handler.ssl.SslClientHelloHandler — pre-handshake ClientHello aggregation path

Vulnerable Code Locations

• handler/src/main/java/io/netty/handler/ssl/SniHandler.java:85
• handler/src/main/java/io/netty/handler/ssl/SslClientHelloHandler.java:75 (decode entry)
• handler/src/main/java/io/netty/handler/ssl/SslClientHelloHandler.java:165 (handshakeBuffer.clear)
• handler/src/main/java/io/netty/handler/ssl/SslClientHelloHandler.java:174 (writeBytes re-copy)
• codec-base/src/main/java/io/netty/handler/codec/ByteToMessageDecoder.java:294 (cumulation retention)

Exploit Path

1. TCP connection → SniHandler → SslClientHelloHandler.decode
2. First record: TLS handshake header declaring large ClientHello length (e.g., 4096 bytes)
3. Attacker sends thousands of tiny follow-on handshake records (1 byte each)
4. On each fragment: handshakeBuffer.clear() + writeBytes() re-copies ALL accumulated body bytes
5. Total bytes copied = n*(n+1)/2 where n = number of body bytes → quadratic
6. Event-loop CPU exhausted before SslHandler takes over

Impact

• Vulnerability Type: Inefficient Algorithmic Complexity
• An unauthenticated network attacker can drive disproportionate CPU consumption on the Netty event loop
• Affects all Netty deployments using SniHandler for TLS termination (the default SNI path)
• No privileges, user interaction, or special configuration required
• Can degrade or stall TLS connection handling for all clients on the affected event loop
• The attack requires only modest bandwidth (~25 KB) to trigger significant CPU work

Credits

Found by a security research team from the University of Sydney, focusing on detecting open source software vulnerabilities. Liyi Zhou: https://lzhou1110.github.io/ Ziyue Wang: https://zyy0530.github.io/ Strick: https://str1ckl4nd.github.io/ Maurice: https://maurice.busystar.org/ Chenchen Yu: https://7thparkk.github.io/