This article is Part 2 in a series of articles about how Virtual Private Networks (VPNs) work. Part 1 of the series, How VPNs Work outlined VPN functionality at a high level. Part 3 (Encryption and Authentication) in this series explores security and encryption pitfalls you should be aware of, including deprecated ciphers, how some VPN service providers conceal their use of lower level encryption methods, and explains Perfect Forward Secrecy and why it is important.
The purpose of this article (Part 2) is to clarify the difference between VPN protocols and VPN services. It explains the most prevalent VPN protocols to assist you in making an informed decision when choosing a sensible VPN platform that meets your needs.
Once a decision is made to use a Virtual Private Network (VPN), several important questions and decisions should follow:
- Do you understand the difference between VPN protocols offered by a third-party VPN service provider?
- Which one is best for you?
- Why are there different types of VPNs? Why use one versus another?
- Should I create my own VPN or use a 3rd party service provider?
- Which VPN protocol should I use?
- How do I set it up?
- How do I use it?
- Do you have a particular VPN service provider in mind?
- Do you have a particular VPN protocol in mind?
- Have you chosen a particular VPN service?
VPN Service Architecture
VPNs form peer-based virtual networks. There are two (2) types of them, which you can think of as VPN base structures: Client-Server VPNs and Mesh VPNs.
What's the difference?
Traditional Client-Server VPNs
Most VPNs establish a simple dual (2) peer client/server virtual network.
Mesh Networks
A small, but growing number of VPN protocols are actually mesh network VPN architectures. Let's clarify what a mesh network is. A mesh network is a peer-to-peer network composed of two or more devices, where each device is able to communicate with at least one peer, and the together the peers collectively form a non-hierarchical network. You may think of a mesh network as a one-to-many relationship, while a client/server peer network is a one-to-one relationship.
VPN Protocols
Regardless of the type of network, in order to communicate and exchange data, every peer device on a network must share at least one network protocol. Common examples are IP and Ethernet.
Being a virtual network, a VPN is no different. A peer on a virtual network must use the same protocol or the VPN won't work. There are many, many products on the market that create virtual private networks. The primary distinction between them is which VPN protocols they support. The most confounding question to the majority of people is which VPN service provider to use, followed closely by choosing a VPN protocol. You can think of VPN protocols as languages. Imagine two people who need to speak the same language in order to communicate clearly. Protocols establish the framework by which data (ideas) will be exchanged.
The chart below describes various characteristics of the most common VPN Protocols, listed alphabetically. It shows each protocol's possible Virtual Network Interface (VNI) types, tunnel and packet encryption and authentication capabilities, supported service layers, and whether or not the VPN protocol is open source.
The "IP v6" column indicates whether or not the Virtual Network Interface (VNI) is capable of supporting IPv6 traffic.
| VPN Protocols | ||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Virtual Network Interface (VNI) | IP v6 |
Service Layer |
Tunnel | Packet | L3 Net Protocol | Open Src |
||||||||||||||
| Protocol | TAP | TUN | PPTP | L2TP | SSTP | SSH | WG | 2 | 3 | PPP | Auth | Enc | Int | Con | Auth | Enc | TCP | UDP | ||
| Hamachi | Yes | -- | -- | -- | -- | -- | -- | Yes | Yes | No | No | Yes | Yes | Yes | No | No | No | Yes | Yes | No |
| L2TP | -- | -- | -- | Yes | -- | -- | -- | No | Yes1 | No | ONLY | No | Yes | No | No | No | No | No | Yes | No |
| L2TP/IPsec | -- | Yes | -- | Yes | -- | -- | -- | No | No | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No | Yes | No |
| OpenConnect | Yes | Yes | -- | -- | -- | -- | -- | Yes | Yes | Yes | No | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes |
| OpenVPN | Yes | Yes | -- | -- | -- | -- | -- | Yes2 | Yes | Yes | No | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes |
| PPTP | -- | -- | Yes | -- | -- | -- | -- | Yes | Yes | No | ONLY | No | Yes | No | No | No | No | Yes | No | No |
| SoftEther | Yes | Yes | -- | -- | -- | -- | -- | Yes | Yes | Yes | No | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No | Yes |
| SSH | Yes | Yes | -- | -- | -- | Yes | -- | Yes | No | Yes | Yes | Yes | Yes | Yes | No | No | No | Yes | No | Yes3 |
| SSTP | -- | -- | -- | -- | Yes | -- | -- | Yes | No | Yes | Yes | Yes | Yes | Yes | No | No | No | Yes | No | No |
| StrongSwan | Yes | Yes | -- | -- | -- | -- | -- | Yes | Yes | Yes | No | Yes | Yes | Yes | Yes | Yes | Yes | No | Yes | Yes |
| Tinc 1.x | Yes | Yes | -- | -- | -- | -- | -- | Yes | Yes | Yes | No | Yes | Yes | Yes | No | Yes | Yes | Yes | Yes | Yes |
| WireGuard | -- | Yes | -- | -- | -- | -- | Yes | Yes | No | Yes | No | Yes | Yes | Yes | Yes | Yes | Yes | No | Yes | Yes |
1 Behaves like a Layer 2 protocol, though technically it is not.
2 Versions 2.3.x and higher only.
3 Various open source versions exist, but not all versions are open source.
The following clarifies select column abbreviations referenced above:
Tunnel
- Auth : authentication (key negotiation)
- Enc : encapsulation
- Int : tunnel integrity
- Con : confidentiality (e.g. conceal underlying packet type and/or IP address)
Packet
- Auth : packet authentication
- Enc : packet encryption
VPN Protocol Authentication and Encryption Libraries
The chart below describes details of Layer 3 (IP tunnel) authentication and encryption supported by each protocol.
The "IP v6" column in the chart below indicates whether or not the protocol can be implemented over an existing IPv6 connection.
| VPN Protocol Layer 3 Auth/Encryption Details | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| VPN Protocol | IP v6 |
Ports | Fixed | Layer 3 | NAT Trav4 |
Tunnel | Anti Replay |
Hash Ciphers5 | Authentication | Encryption Ciphers8 | Encryption Library | |||
| TCP | UDP | X.509 | PSK6 | PFS7 | ||||||||||
| IKEv2 | Yes | 500 45009 |
Yes | No | Yes | Yes10 | IPsec | Yes | SHA-1 SHA-2 |
Yes | Yes | Yes | -- | Oakley,ECDSA ISAKMP,SKEME |
| IPsec11 | Yes | -- | -- | -- | -- | No | TLS | Yes | SHA|MD5 | Yes | Yes12 | Yes | AES|DES|3DES | ISAKMP (IKE2) |
| L2TP | No | 1701 UDP | Yes | -- | -- | No | PPP | Yes | -- | No | No | No | CHAP|EAP|PAP|SPAP | -- |
| L2TP/IPsec | No | 500,1701 4500 |
Yes | No | Yes | No | TLS | Yes | See IPsec |
Yes | Yes | Yes | See IPsec | IKEv2 (ISAKMP) |
| OpenVPN | Yes | 1194 | No | Yes | Yes | No | TLS | Yes | SHA-1 SHA-2 |
Yes | ? | Yes | RSA|ECDHA/E AES-CBC|AES-GCM Camellia|Blowfish |
OpenSSL MbedTLS13 LibreSSL14 |
| PPTP | No | 1723 UDP | Yes | -- | -- | No | PPP | No | -- | No | No | No | CHAP|EAP|PAP|SPAP | -- |
| SoftEther | Yes | 443,992 1194,5555 |
Yes | Yes | No | Yes | SSLv3 or TLS 1.0 |
Yes | SHA-1 MD5 |
Yes | Yes | Yes | AES|RC4|DES|3DES | OpenSSL |
| SSTP | Yes | 443 | Yes | Yes | No | No | SSL/TLS | Yes | -- | No | No | No | -- | -- |
| Tinc 1.x | Yes | 655 | No | No | Yes | Yes | SPTPS TLS 1.2 |
Yes | HMAC SHA-256 |
No | Yes | Yes | ECDHA/E AES-CTR |
OpenSSL LibreSSL |
| WireGuard | Yes | 51820 | No | No | Yes | No | ? | Yes | SipHash BLAKE2s |
No | Yes | Yes | Curve25519 ChaCha20,Poly1305 |
-- |
4 Network Address Translation (NAT) Traversal
5 Hash ciphers provide Integrity regarding the payload; they ensure the message has not been tampered with in transit.
6 Pre-Shared Keys
7 Perfect Forward Secrecy
8 Based on most recent protocol version.
9 Used only if necessary for NAT traversal (NAT-T).
10 Optional. Must be enabled explicitly.
11 IPsec (Internet Protocol SECurity) is a network protocol framework widely used in VPNs, including VPN protocols (e.g. L2TP/IPsec) and services. Learn more What Is IPsec?
12 Do not use in Aggressive Mode as the hash is exposed in plain text (source: https://en.wikipedia.org/wiki/IPsec#Alleged_NSA_interference)
13 Previously known as PolarSSL.
14 Support is version specific (e.g. OpenBSD 6.x + OVPN 2.4.8). LibreSSL support is dicey and subject to breaking if OpenBSD's network API changes.
VPN Server Protocol Support
Having reviewed the most widespread VPN protocols above, it's time to explore which VPN server types support them. By server type, I mean the VPN server program itself. Most VPN software may be run in either client mode or server mode (and some can do both simultaneously). The focus of the chart below is regarding VPN software operating as a server, and the VPN protocols it understands. Most VPN servers only understand one protocol. For instance, OpenVPN only understands OpenVPN; you cannot establish an L2TP connection to it, for example. Streisand is on the other end of the spectrum. A jack-of-all-trades, Streisand allows connections from at least seven (7) different VPN protocols, and even supports sslh.
All of this matters when you are considering either creating a VPN server or you wish to setup a VPN client. You need to know which VPN protocols any given solution is capable of conversing with. If you're shopping for a third-party VPN provider, are you dead-set on a specific VPN provider? Then you need to know which VPN protocols they support and choose a VPN client application that is compatible. Or if you are in the reverse situation - you have chosen a specific VPN client to use - then you need to know which VPN server software is capable of communicating with your VPN client software.
You can think of VPN server and client applications like an electrical plug and wall receptacle. The client (electrical appliance) plugs into the server (wall receptacle). A VPN server and client must both support the same protocol. Who determines which protocol will be used when more than one is supported? It is a negotiation process (but in a tie, the client wins). First, both peers must support the same VPN protocol. Once that is established, either peer may have stronger security requirements than the other. If both support multiple protocols to begin with, the client will determine which is used. As the client is the peer initiating the connection, it calls the shots with regards to which protocol(s) it understands. Thankfully, many VPN servers support multiple protocols. This makes VPN connections more flexible. Most public VPN service providers have written custom server software to detect incoming VPN clients and route them to the best option out of a suite of supported VPN protocols, however that process is not necessary if the server operator wishes to stand-up a VPN server with "out of the box" functionality. Depending on which VPN server software is chosen, it is possible to accommodate a variety of incoming protocols.
The chart below correlates VPN server types to their supported VPN protocols, and corresponding VPN implementation methods, including Layer 2 and/or Layer 3 support, and whether TCP, UDP, or both network protocols are supported. The "IP v6" column indicates whether or not the VPN client/server connection may be initiated over an existing IPv6 connection or not. And finally, if the VPN server supports mesh networking, that is also indicated in the chart.
| VPN Server Protocols | |||||||
|---|---|---|---|---|---|---|---|
| Server Type | Supported VPN Protocols | Network Protocols | |||||
| Layer 2 | Layer 3 | Mesh | |||||
| IP4 | IP6 | TCP | UDP | IP6 | |||
| Hamachi | Hamachi | Yes | Yes | No | No | No | Yes |
| L2TP | L2TP | Yes | No | No | No | No | No |
| L2TP/IPsec | L2TP/IPsec | No | No | No | Yes | No | No |
| OpenSwan | IPsec,L2TP/IPsec OpenSwan |
Yes | Yes | No | Yes | Yes | No |
| OpenVPN | OpenVPN | Yes | Yes | Yes | Yes | Yes | No |
| PPTP | PPTP | Yes | No | Yes | No | No | No |
| SSTP | SSTP | No | No | Yes | No | No | No |
| Streisand | OpenConnect,OpenSSH OpenVPN,Shadowsocks,sslh15 Stunnel,Tor bridge,WireGuard |
Yes | Yes | Yes | Yes | Yes | No |
| StrongSwan | IPsec | Yes | Yes | No | Yes | Yes | No |
| Tinc 1.x | Tinc|TAP|TUN | Yes | Yes | Yes | Yes | Yes | Yes |
| SoftEther | L2TP/IPsec,L2TPv3/IPsec,OpenVPN SoftEther,SSTP,EtherIP/IPsec |
Yes | Yes | Yes | Yes | Yes | No |
| WireGuard | WireGuard | No | No | No | Yes | Yes | No |
15 sslh is a protocol multi-plexer application. It combines multiple protocols on the same port, with built-in support for HTTP, SSL, OpenVPN, TLS, SSH, Tinc, XMPP, SOCKS5. Any protocol can be supported by differentiating it with a regular expression.