OpenSSL patches HollowByte DoS flaw in silent updates

Okta’s red team discovered a denial-of-service bug in OpenSSL called HollowByte that allowed an 11-byte TLS handshake to trigger large preallocated receive buffers. OpenSSL maintainers applied the fix in version 4.0.1 and backported it to 3.6.3, 3.5.7, 3.4.6 and 3.0.21 without a public announcement.

The flaw existed because older OpenSSL versions used the four-byte length field in the TLS handshake header to reserve receive buffers before any application data arrived. A minimal 11-byte payload could claim a much larger message and cause OpenSSL to allocate up to about 131 KB per connection while the worker thread waited indefinitely for data that never came.

Glibc’s memory allocation behavior increased the impact. The GNU C Library keeps small and medium freed blocks for reuse instead of returning them immediately to the operating system. When an attacker opened many connections and randomized the claimed sizes, freed chunks could not be reused and the server process grew in memory even after connections closed. Recovering the memory required terminating the process.

In tests, Okta reported a 1 GB machine became unresponsive after 547 MB of memory were fragmented and frozen. On a 16 GB system the attack locked roughly 25% of total memory while staying below typical connection limits, so simple caps on connections did not stop it. Any server, runtime or database linked against vulnerable OpenSSL releases can be affected, including Apache, NGINX, Node.js, Python, Ruby, PHP, MySQL and PostgreSQL unless they are updated to a patched build.

The OpenSSL change alters when and how receive buffers grow: the library now increases buffer size only as actual bytes arrive instead of trusting the length declared in the handshake header. That behavior was added in 4.0.1 and backported to the listed 3.x releases.

Okta’s analysis noted: “The worker thread then blocks, waiting indefinitely for data that will never arrive.” The report also stated: “By launching waves of connections with randomized claimed sizes, an attacker prevents the allocator from reusing those freed chunks.”

Administrators are advised to verify OpenSSL versions on affected systems and apply the patched releases or vendor-supplied updates to mitigate the risk of HollowByte-style denial-of-service attacks.