Netty: Memory Leak in DNS Record Decoder via Malformed Domain Names

Summary

A memory leak can be caused in Netty's DNS codec by sending malicious DNS packets containing invalid domain names. Because the leak occurs incrementally per packet, sustained malicious requests will cause a gradual Denial of Service.

Details

Inside io.netty.handler.codec.dns.AbstractDnsRecord, the parsed domain name string is passed to IDN.toASCII(name). If the domain name contains characters that violate IDNA rules, IDN.toASCII throws an IllegalArgumentException.

Because this exception occurs inside the constructor before the DnsRecord instance can assign the buffer to its content field for later release, the ByteBuf whose reference count was incremented (or newly allocated) is never released, resulting in a direct memory leak.

There are several places where variants of this leak happen:

• io.netty.handler.codec.dns.DefaultDnsRecordDecoder#decodeRecord(java.lang.String, io.netty.handler.codec.dns.DnsRecordType, int, long, io.netty.buffer.ByteBuf, int, int) invokes in.retainedDuplicate() or creates a new buffer out when constructing DefaultDnsRawRecord
• io.netty.handler.codec.dns.DnsCodecUtil#decompressDomainName allocates a new ByteBuf and passes it to encodeDomainName(). If the decompressed domain name contains a null byte (\0), encodeDomainName() throws an IllegalArgumentException, leaking the newly allocated buffer.

Impact

Resource Exhaustion. Any application utilizing Netty's DnsRecordDecoder (such as DnsNameResolver or custom DNS servers) is vulnerable.