datahacker Blog

A newly discovered security vulnerability exists in most modern operating systems which allows a malicious actor to hijack an active TCP-based VPN connection, without the need to break its encryption.

If you use a VPN for any purpose, you are likely impacted.

Just when you thought it was safe to use your VPN at least until quantum cryptography becomes an existential threat, a new vulnerability has been announced affecting virtually ALL VPNs. This threat circumvents the VPN's encryption methods, and allows a malicious actor to deduce the following:

• Whether a connected user is using a VPN
• Make positive inferences about the websites the user is visiting
• Inject data into the TCP stream

CVE-2019-14899: Inferring and hijacking VPN-tunneled TCP connections

Significance

On affected systems, a Virtual Private Network (VPN) cannot be guaranteed to provide confidential data communication between two endpoints. Previously, the prevailing wisdom was a strong VPN provided protection from Man-in-the-Middle (MitM) attack vectors. In fact, this is one primary purpose of a VPN; preventing snooping or hijacking of a connection between two endpoints by a host device that lies between them. The classic example of this is a hijacked wireless access point (WAP).

Thanks to the researchers who announced their findings on December 4 (2019), we now know nearly all VPN software on almost all operating systems are vulnerable to a rather simple, malicious payload injection attack.

Contents

This article explains this critical vulnerability in simple terms: what it is, how it works, and how to protect yourself. It is heavily slanted toward Linux (as is everything on this website), and does not go into all the technical nuances. I have attempted to keep the technical details light while conveying enough information that even a novice user will be capable of exercising a bit of precautionary detective work on their own system if desired.

Quick Facts

This threat was isolated in early August 2019, but the discovery team withheld announcing it until December 4, because they needed an opportunity to conduct additional testing and notify operating system maintainers.

The attack may be carried out by a compromised router (Man-in-the-Middle or MitM vector) or an adjacent user in the same virtual IP space as the victim's outbound connection from the VPN server to the WAN (normally the Internet). Perhaps the most disturbing piece of news about this new vulnerability is how insidiously simply its attack method is, and the fact it completely bypasses any encryption method. This is possible because the attack method takes advantage of TCP's default handling by the network adapter and operating system kernel.

Known impacted VPN software includes IPsec, IKE/IKEv2, OpenVPN, and WireGuard. Known impacted operating systems are Linux (almost all distros), FreeBSD, OpenBSD, MacOS, iOS, and Android. Nearly every VPN service protocol is impacted.

To reiterate, there are two (2) possible attack entry points where encrypted VPN connections may be compromised:

1. Man-in-the-Middle (MitM) attacker when either endpoint (network peer) device is running a vulnerable operating system and vulnerable VPN service.
2. The attacker is an adjacent user in the same virtual IP space as the victim.

An important point is the attacker must have knowledge of the virtual IP space of the victim's client, though this is not an especially difficult task. For example, this is ordinarily easy to determine in a corporate environment, and even in deference to commercial VPN services, certain practices are common. Many VPN servers utilize the default virtual IP space of the underlying protocol (based on default configurations) and/or blocks of adjacent virtual IP ranges. Some publicize their virtual IP ranges. Alternatively, knowledge of the virtual IP space of the VPN server may be known to the attacker by some other means. Regardless, it should be presumed it would not be difficult for a malicious actor to acquire this information.

Threat Summary

If you're going to tell everyone the sky is falling, you'd better be able to back it up, right?

Currently just a reference point, CVE-2019-14899 seems innocuous. For if you were to scan Mitre corporation's CVE database (Common Vulnerabilities and Exposures) - the offical bug tracker of the Internet - as of this writing in early December 2019, you won't find anything interesting tied to that CVE number. If you're astute, you will notice whatever "it" happens to be was first reported on August 10, 2019. Since then, it's gone dark. Nothing. Nada.

So, why all the red flag waving? Well, apparently at that time the researchers were contemplating releasing their findings, until they began to realize the problem they had discovered might in fact be much, much larger than they originally thought. Fast forward to last week. What changed? Last Wednesday, on December 4th, three (3) researchers at the University of Mexico (Jedidiah Crandall, Beau Kujath, and William Tolley) dropped a veritable bombshell email via the Open Source Security Mailing List at OpenWall, a veteran of the Internet and clearinghouse of open source security software and security related discussions pertaining to open source software.

While the UNM researchers made it clear they were not revealing all the details of their discovery, they did disclose the aforementioned CVE placeholder, how simple of a concept the threat is, and how incredibly widespread the damage potential is. They also provided sufficient details about the process a hacker would need to take in order to leverage the vulnerability, lending credibility to their claims. Their email letter to the community reads in part:

"We have prepared a paper for publication concerning this vulnerability and the related implications, but intend to keep it embargoed until we have found a satisfactory workaround.... We are also reporting this vulnerability to the other services affected, which also includes: Systemd, Google, Apple, OpenVPN, and WireGuard, in addition to distros [almost all Linux distributions] ... for the operating systems affected." (emphasis added)

Translation: We're working on publishing a paper. It's a huge issue (almost every operating system and VPN service). We've reached out to the organizations that produce the impacted products. We decided not to publish full details here out of concern there is currently no known fix. We'll be back.

Even if you successfully mitigate the attack risk on your device, it is still possible the device on the other end of your VPN connection could be compromised, potentially resulting in a malicious payload reaching your system.

For example, if a commercial VPN server were compromised, you could be impacted in spite of your efforts. Until the full analysis of the problem is revealed, it is possible the mitigation steps discussed in this article may be ineffective.

Thankfully, the authors did explain the issue well enough to provide some guidance and proactive steps a responsible system administrator might take in the interim. Though I have to say, based on my interpretations, mitigation efforts at this point are limited. IPsec actually probably has the most robust work-around. Options are very limited for other VPN platforms, and this entire issue is quite frankly operating system agnostic. The ideal approach for any VPN other than IPsec - if one is worried about this new risk - is probably to avoid using TCP tunneling VPNs PERIOD. Until further notice.

Overall, I suspect the researchers are attempting to surreptitiously notify the public while simultaneously turning up the heat on the organizations they are purportedly imploring to take notice of a significant gaping hole in their network stack. If you've followed networking for any length of time, you've likely garnered these things don't change very often. This is especially true in the Linux world, where we are still using networking code in the core operating system that is over 20 years old.

Point is, as the old U.S. Navy saying goes, "A fleet only moves as fast as its slowest ship." Meaning, it takes time to get everyone onboard when a change in direction is warranted and consensus is required.

So, while this all unfolds, here are some details about this threat for you to consider, to help you ascertain whether or not you should be concerned:

• Only TCP based tunnel VPNs are affected
• SOCKS proxies and TOR are not vulnerable to this attack vector
• Targets outbound connections from the VPN server, between Internet-facing virtual IP address and the destination on the Internet (such as outbound connections via a commercial VPN service provider on behalf of a VPN user)
• Does not directly affect the connection between VPN client and server (but it may be able to insert a payload that gets routed back to the client device)
• Primarily a threat to Linux distributions (due to how the Linux kernel handles TCP)
• Due to how the hijack works, this is primarily a threat targetted at VPN servers (e.g. commercial VPN service providers)

Scope

The vulnerability is two-fold.

1. It allows a malicious attacker to determine the virtual IP address of the VPN user.
2. Once the user's virtual IP address is known, the attacker may inject arbitrary payloads into the ongoing encrypted connection.

This vulnerability is known to affect the following:

• VPN protocols: IPsec, IKE/IKEv2, OpenVPN, WireGuard
• Operating Systems: Biggest impact to Linux OS's, including but not limited to the following:
  • Ubuntu 19.10 (systemd)
  • Fedora (systemd)
  • Debian 10.2 (systemd)
  • Arch 2019.05 (systemd)
  • Manjaro 18.1.1 (systemd)
  • Devuan (sysV init)
  • MX Linux 19 (Mepis+antiX)
  • Void Linux (runit)
  • Slackware 14.2 (rc.d)
  • Deepin (rc.d)
  • FreeBSD (rc.d)
  • OpenBSD (rc.d)

Most Ubuntu Distros Not Affected

Ubuntu distributions prior to Ubuntu 19.10 are not affected unless a system administrator has modified a machine's systemd configuration and the default settings in sysctl.d/50-default.conf in the systemd repository were changed from “strict” to “loose” mode. Ubuntu 19.10 happens to use a version of systemd modified on November 28, 2018 and applied to the Ubuntu 19.10 distro. To my knowledge it is not possible to "roll back" that update in Ubuntu 19.10.

systemd (a False Flag)

It was initially widely reported the problem was related to systemd (a system and service manager that runs as a daemon on many Linux operating systems). However, that is not the case. Any connection between this vulnerability and systemd has to do with its configuration and not systemd per se. The vulnerability has been found on Linux operating systems that use alternatives to systemd, with similar results. Therefore, systemd itself is not the problem. It is true the problem was originally discovered on systems utilizing systemd, and I would imagine the researchers originally suspected therein lay the root of the problem. However, subsequent testing on their part disproved that theory, and we now know it is much wider spread. On the other hand, one of the potential mitigation strategies is specific to systemd-based operating systems.

Attack Process/Description

The simpler approach from the malicious actor/hacker's perspective is via the Man-in-the-Middle (MitM) scenario. Since the researchers have not yet gone into detail regarding how the adjacent virtual IP address host attack works, I'll stick with explaining how the more likely (and potentially more insidious) MitM attack may occur.

The hack works by sending bogus challenge ACK (acknowledgement) messages to the victim's client device. Based on whether or not a response is received, an attacker is able to discern characteristics of the secure connection. Once this information is known to the attacker, they may hijack the secure connection by inserting TCP packets disguised as legitimate traffic from either network peer. The size and number of packets are visible to the attacker due to its MitM position. This allows the attacker to hijack the TCP connection and bypass the encryption provided by the VPN protocol.

The attack is comprised of three (3) steps:

1. Determine the VPN client’s virtual IP address by sending SYN-ACK packets to every known virtual IP address within the known range. Any active host will respond with an RST, revealing to the attacker which virtual IPs are currently active.
2. Using a virtual IP address to make inferences about active connections.
3. Using encrypted replies to unsolicited packets to determine the sequence and acknowledgment numbers of the active connection in order to hijack the TCP session.

Risk Mitigation Strategies

The following solutions will theoretically solve the problem. The first has been tested and appears to work on Linux devices. However, it merits further testing as to-date follow-on tests have not been comprehensive. The other proposed solutions are unknown in terms of their effectiveness. All of these solutions merit further investigation. However, for the moment - considering the circumstances - these ideas are better than doing nothing.

1. Ensure rp_filter value is set to "strict" (Linux).
2. Obfuscation by making all TCP packets in the encrypted channel the same size (e.g. IPsec TFC Padding).
3. Adjust the challenge ACK process so that after a specified number of failed attempts, subsequent attempts are ignored.¹
4. Allowing a host to respond with equivalent-sized packets after exhausting the challenge ACK limit could prevent an attacker from making correct inferences about the underlying traffic.

Changes to rp_filter should be considered a temporary workaround until each operating system and VPN server software are evaluated to determine if a more robust alternative is feasible. I say this because fixing this problem with rp_filter breaks functionality required by some use-case scenarios (particularly routers). Longer term, this problem warrants re-evaluating some aspects of the Linux network stack.

Solutions

At this time there is no known permanent fix. Several mitigation strategies exist but their long-term effectiveness is unknown. A permanent fix will likely involve a patch to the affected operating systems and/or VPN services. The following are short-term suggestions to reduce your risk of becoming a victim.

1. IPsec: Enable Built-in Payload Padding

If you happen to be using IPsec or IPsec/IKEv2, you are in luck! A built-in capability of IPsec (disabled by default) will mitigate risk of this attack. A drawback to this method is it may slow your connection.

This only works when using ESP mode in IPsec.

IPsec supports a feature called TFC Padding (Traffic Flow Confidentiality Padding). Part of the ESP protocol (Encapsulated Security Payload), IPsec's TFC Padding function is capable of facilitating a defense by creating equal-sized packets to prevent a malicious actor from making accurate inferences about the observed connection. TFC pads TCP packets so they appear as uniform lengths. It is limited in its abilities, but is a step in the right direction.

How It Works (TFC)

ESP natively supports padding, but is limited to 255 bytes. TFC extends this functionality by padding the payload itself. Naturally, this requires unambiguous handling by both networking peers. They must both understand TFC is being used. The standard method of communicating this fact to the other peer is to insert protocol ID 59 into the Next Header field, indicating a dummy payload follows. The dummy load is then wrapped in a standard ESP header and trailer (tail hash), as seen in this diagram:

In IPsec tunnel mode, the original IP header, transport layer headers, and data payload are encrypted. Thus, because the dummy payload is contained within the encrypted TCP payload, the hacker will be unable to detect it even though the protocol ID 59 value is present. Recall the hacker is not breaking any encryption with this attack vector. They are attempting to hijack the connection without interrupting the encrypted payloads. Therefore, even with deep packet inspection the hacker will not be able to view the protocol 59 (which if they could, would give away the fact there's a dummy payload and what length it is). The key here is the TFC payload is treated like a normal TCP data payload, including wrapping it with ESP encryption and authentication.

IPsec/TFC Reference Information

For more information on IPsec TFC Padding, see RFC 4303: IP Encapsulating Security Payload (ESP) [2005] and/or Traffic Flow Confidentiality in IPsec: Protocol and Implementation by Kiraly, et al (2008).⁴

2. Use a Proxy Server

Use a proxy server as an alternative to a VPN. A proxy server effectively sanitizes the TCP packets en route between the source (client) and destination. There is no virtual IP process attack surface.

3. Use Tor

Tor - also known as The Onion Router - is basically a mesh network. Tor bounces internet traffic through relays run by thousands of volunteers around the world, making it very difficult for anyone to identify a connection's true location. From a practical standpoint, it works like a proxy server. From a network interface perspective, the packets coming into a Tor node get redirected to localhost (127.0.0.1), where they are handled by the Tor process and then sent out again. There is no virtual IP target at the network layer for the attacker to thwart.

4. Linux: Modify rp_filter Configuration

rp_filter is the Linux Reverse Path Filter; an extension to the built-in Linux firewalls (iptables and nftables). Reverse Path Filtering is a safety mechanism built into most routers and designed to prevent IP address spoofing. When a machine with reverse path filtering enabled receives a packet, it checks the rp_filter setting, which determines whether or not the source IP address in the IP header of the incoming packet should be verified or not.

Most routers maintain an rp_filter cache in memory, so this check isn't necessarily run every single time a new packet comes in. The router will have recent traffic cached, speeding up the process (otherwise it basically pings the remote host, and depending on network congestion this may be a bother if it is forced on every incoming packet if there is no rp_filter IP address cache).

In the past, rp_filter only functioned one way: what is now known as "strict" Reverse Path (RP) filtering. Now, most Linux distributions have another option called "loose." Here is how they differ:

• Disabled: RP filtering is turned off. Incoming packets are not filtered.
• Strict: Before accepting the packet, the source of the received packet is checked to see if it's reachable through the network interface the packet arrived on. If not, the packet is dropped.
• Loose: Before accepting the packet, the source of the received packet is checked to see if it's reachable via ANY network interface. If not, the packet is dropped.

The standard usage in Linux is to define the rp_filter setting with an integer configuration value. 0 is OFF. 1 is Strict. 2 is Loose.

The "loose" behavior was first implemented by Red Hat, but has since been adopted by other Linux distros. The default behavior for *most* distros is strict. However, some disable it by default and a few set it to loose. The latter scenario is becoming more common as it is more likely to just work for any given use-case configuration "out of the box." For example, Red Hat, CentOS, and Ubuntu as of 19.10 all default to the "loose" setting.

rpfilter is an extension present in iptables and nftables. It was added in Linux kernel 3.4.4.

IPv4 (Linux)

You may verify your Linux machine's current rp_filter setting for IPv4 like so: cat /proc/sys/net/ipv4/conf/default/rp_filter

rp_filter needs to be set to "strict" mode. This is crucial. If you are unable or unwilling to locate and modify the rpfilter configuration file, you can force a change during the device's boot-up process. On a machine using systemd, create a configuration file and place it the appropriate directory so that systemd will execute the instructions on boot.² For example, rpfilter may be forced to use its default mode by creating a 1-line configuration file, such as /etc/sysctl.d/51-rpfilter.conf

net.ipv4.conf.all.rp_filter=1

IPv6 (Linux)

rp_filter only applies to IPv4 traffic. To apply comparable strict filtering to IPv6, you will need to insert a new firewall rule.

Add a new rule to the Mangle table in the PREROUTING chain, like so:³

ip6tables -t mangle -I PREROUTING -m rpfilter --invert -j DROP

Hardening systemd

It has been widely reported this vulnerability is attributed to a post-11/28/2018 update of systemd, but this is not correct. The issue is not limited to systemd. Testing by the discovery team indicates other Linux service management daemons and various networking stacks are also affected (e.g. sysV and rc.d).

References