OpenSSL 'HollowByte' Flaw Could Freeze Servers by Locking Memory

OpenSSL 'HollowByte' Flaw Could Freeze Servers by Locking Memory

A critical vulnerability in OpenSSL, dubbed 'HollowByte,' allows attackers to freeze servers by locking up memory indefinitely, even after the attacker disconnects. The flaw, discovered and named by Okta's Red Team, was patched in June but received no CVE identifier, advisory, or changelog entry, making it difficult to detect and mitigate.

How HollowByte Works

The vulnerability exploits the way OpenSSL handles TLS handshake messages. Every message carries a 4-byte header that declares the body's length. Older OpenSSL versions allocate memory based on this header before validating the message, allowing attackers to claim large buffer sizes and lock up memory.

For inbound ClientHello messages, the maximum allocation is 131 KB. The server's worker thread then waits indefinitely for a message body that never arrives, effectively freezing the connection. This memory remains locked even after the attacker disconnects, as the glibc allocator holds onto the memory for reuse rather than returning it to the system.

Memory Fragmentation and Denial-of-Service

Okta's testing revealed that varying the claimed buffer size on each connection prevents glibc from reusing the memory, leading to heap fragmentation and increased resident set size. In one test, a 1 GB server was overwhelmed with 547 MB of memory locked in fragments, while a 16 GB server had 25% of its memory frozen without exceeding the connection limit. This makes traditional connection-limiting defenses ineffective.

OpenSSL's Response and Patch Details

OpenSSL quietly patched the vulnerability in releases 4.0.1, 3.6.3, 3.5.7, 3.4.6, and 3.0.21 on June 9. However, the fix was not classified as a vulnerability, and no CVE or advisory was issued. The project's security team treated it as a 'bug or hardening' fix, despite similar issues receiving CVEs in the same release.

For example, a TLS 1.3 certificate-compression bug (CVE-2025-66199) and a QUIC memory-exhaustion flaw (CVE-2026-34183) both received identifiers, while HollowByte did not. This inconsistency has raised concerns about OpenSSL's triaging process.

Impact and Mitigation

The lack of a CVE or advisory makes it difficult for system administrators to identify and mitigate the vulnerability. Downstream distributions like Red Hat, which backport fixes without updating version numbers, further complicate detection. Users are advised to check package changelogs or consult maintainers to confirm whether the patch has been applied.

For self-built OpenSSL installations, upgrading to the patched releases and restarting services is recommended. However, the fix only applies to TLS; the DTLS handshake path remains unpatched, as fixing it would require more invasive changes.

Conclusion

The HollowByte vulnerability highlights the importance of transparent communication in security patching. While OpenSSL has addressed the issue, the lack of a CVE or advisory leaves users and administrators in the dark, underscoring the need for better disclosure practices in open-source security.