Virtual Private Networks

How VPNs Work - Part 2: Protocols

This article is Part 2 in a series of articles about how Virtual Private Networks (VPNs) work. Part 1 of the series, How VPNs Work outlined VPN functionality at a high level. Part 3 (Encryption and Authentication) in this series explores security and encryption pitfalls you should be aware of, including deprecated ciphers, how some VPN service providers conceal their use of lower level encryption methods, and explains Perfect Forward Secrecy and why it is important.

The purpose of this article (Part 2) is to clarify the difference between VPN protocols and VPN services. It explains the most prevalent VPN protocols to assist you in making an informed decision when choosing a sensible VPN platform that meets your needs.

Once a decision is made to use a Virtual Private Network (VPN), several important questions and decisions should follow:

  • Do you understand the difference between VPN protocols offered by a third-party VPN service provider?
  • Which one is best for you?
  • Why are there different types of VPNs? Why use one versus another?
  • Should I create my own VPN or use a 3rd party service provider?
  • Which VPN protocol should I use?
  • How do I set it up?
  • How do I use it?
  • Do you have a particular VPN service provider in mind?
  • Do you have a particular VPN protocol in mind?
  • Have you chosen a particular VPN service?

VPN Service Architecture

VPNs form peer-based virtual networks. There are two (2) types of them, which you can think of as VPN base structures: Client-Server VPNs and Mesh VPNs.

What's the difference?

Traditional Client-Server VPNs

Most VPNs establish a simple dual (2) peer client/server virtual network.

Mesh Networks

A small, but growing number of VPN protocols are actually mesh network VPN architectures. Let's clarify what a mesh network is. A mesh network is a peer-to-peer network composed of two or more devices, where each device is able to communicate with at least one peer, and the together the peers collectively form a non-hierarchical network. You may think of a mesh network as a one-to-many relationship, while a client/server peer network is a one-to-one relationship.

VPN Protocols

Regardless of the type of network, in order to communicate and exchange data, every peer device on a network must share at least one network protocol. Common examples are IP and Ethernet.

Being a virtual network, a VPN is no different. A peer on a virtual network must use the same protocol or the VPN won't work. There are many, many products on the market that create virtual private networks. The primary distinction between them is which VPN protocols they support. The most confounding question to the majority of people is which VPN service provider to use, followed closely by choosing a VPN protocol. You can think of VPN protocols as languages. Imagine two people who need to speak the same language in order to communicate clearly. Protocols establish the framework by which data (ideas) will be exchanged.

The chart below describes various characteristics of the most common VPN Protocols, listed alphabetically. It shows each protocol's possible Virtual Network Interface (VNI) types, tunnel and packet encryption and authentication capabilities, supported service layers, and whether or not the VPN protocol is open source.

The "IP v6" column indicates whether or not the Virtual Network Interface (VNI) is capable of supporting IPv6 traffic.

VPN Protocols
Virtual Network Interface (VNI) IP
v6
Service
Layer
Tunnel Packet L3 Net Protocol Open
Src
Protocol TAP TUN PPTP L2TP SSTP SSH WG 2 3 PPP Auth Enc Int Con Auth Enc TCP UDP
Hamachi Yes -- -- -- -- -- -- Yes Yes No No Yes Yes Yes No No No Yes Yes No
L2TP -- -- -- Yes -- -- -- No Yes1 No ONLY No Yes No No No No No Yes No
L2TP/IPsec -- Yes -- Yes -- -- -- No No Yes Yes Yes Yes Yes Yes Yes Yes No Yes No
OpenConnect Yes Yes -- -- -- -- -- Yes Yes Yes No Yes Yes Yes Yes Yes Yes Yes Yes Yes
OpenVPN Yes Yes -- -- -- -- -- Yes2 Yes Yes No Yes Yes Yes Yes Yes Yes Yes Yes Yes
PPTP -- -- Yes -- -- -- -- Yes Yes No ONLY No Yes No No No No Yes No No
SoftEther Yes Yes -- -- -- -- -- Yes Yes Yes No Yes Yes Yes Yes Yes Yes Yes No Yes
SSH Yes Yes -- -- -- Yes -- Yes No Yes Yes Yes Yes Yes No No No Yes No Yes3
SSTP -- -- -- -- Yes -- -- Yes No Yes Yes Yes Yes Yes No No No Yes No No
StrongSwan Yes Yes -- -- -- -- -- Yes Yes Yes No Yes Yes Yes Yes Yes Yes No Yes Yes
Tinc 1.x Yes Yes -- -- -- -- -- Yes Yes Yes No Yes Yes Yes No Yes Yes Yes Yes Yes
WireGuard -- Yes -- -- -- -- Yes Yes No Yes No Yes Yes Yes Yes Yes Yes No Yes Yes

1 Behaves like a Layer 2 protocol, though technically it is not.

2 Versions 2.3.x and higher only.

3 Various open source versions exist, but not all versions are open source.


The following clarifies select column abbreviations referenced above:

Tunnel

  • Auth : authentication (key negotiation)
  • Enc : encapsulation
  • Int : tunnel integrity
  • Con : confidentiality (e.g. conceal underlying packet type and/or IP address)

Packet

  • Auth : packet authentication
  • Enc : packet encryption

VPN Protocol Authentication and Encryption Libraries

The chart below describes details of Layer 3 (IP tunnel) authentication and encryption supported by each protocol.

The "IP v6" column in the chart below indicates whether or not the protocol can be implemented over an existing IPv6 connection.

VPN Protocol Layer 3 Auth/Encryption Details
VPN Protocol IP
v6
Ports Fixed Layer 3 NAT
Trav4
Tunnel Anti
Replay
Hash Ciphers5 Authentication Encryption Ciphers8 Encryption Library
TCP UDP X.509 PSK6 PFS7
IKEv2 Yes 500
45009
Yes No Yes Yes10 IPsec Yes SHA-1
SHA-2
Yes Yes Yes -- Oakley,ECDSA
ISAKMP,SKEME
IPsec11 Yes -- -- -- -- No TLS Yes SHA|MD5 Yes Yes12 Yes AES|DES|3DES ISAKMP
(IKE2)
L2TP No 1701 UDP Yes -- -- No PPP Yes -- No No No CHAP|EAP|PAP|SPAP --
L2TP/IPsec No 500,1701
4500
Yes No Yes No TLS Yes See
IPsec
Yes Yes Yes See IPsec IKEv2
(ISAKMP)
OpenVPN Yes 1194 No Yes Yes No TLS Yes SHA-1
SHA-2
Yes ? Yes RSA|ECDHA/E
AES-CBC|AES-GCM
Camellia|Blowfish
OpenSSL
MbedTLS13
LibreSSL14
PPTP No 1723 UDP Yes -- -- No PPP No -- No No No CHAP|EAP|PAP|SPAP --
SoftEther Yes 443,992
1194,5555
Yes Yes No Yes SSLv3 or
TLS 1.0
Yes SHA-1
MD5
Yes Yes Yes AES|RC4|DES|3DES OpenSSL
SSTP Yes 443 Yes Yes No No SSL/TLS Yes -- No No No -- --
Tinc 1.x Yes 655 No No Yes Yes SPTPS
TLS 1.2
Yes HMAC
SHA-256
No Yes Yes ECDHA/E
AES-CTR
OpenSSL
LibreSSL
WireGuard Yes 51820 No No Yes No ? Yes SipHash
BLAKE2s
No Yes Yes Curve25519
ChaCha20,Poly1305
--

4 Network Address Translation (NAT) Traversal

5 Hash ciphers provide Integrity regarding the payload; they ensure the message has not been tampered with in transit.

6 Pre-Shared Keys

7 Perfect Forward Secrecy

8 Based on most recent protocol version.

9 Used only if necessary for NAT traversal (NAT-T).

10 Optional. Must be enabled explicitly.

11 IPsec (Internet Protocol SECurity) is a network protocol framework widely used in VPNs, including VPN protocols (e.g. L2TP/IPsec) and services. Learn more What Is IPsec?

12 Do not use in Aggressive Mode as the hash is exposed in plain text (source: https://en.wikipedia.org/wiki/IPsec#Alleged_NSA_interference)

13 Previously known as PolarSSL.

14 Support is version specific (e.g. OpenBSD 6.x + OVPN 2.4.8). LibreSSL support is dicey and subject to breaking if OpenBSD's network API changes.

VPN Server Protocol Support

Having reviewed the most widespread VPN protocols above, it's time to explore which VPN server types support them. By server type, I mean the VPN server program itself. Most VPN software may be run in either client mode or server mode (and some can do both simultaneously). The focus of the chart below is regarding VPN software operating as a server, and the VPN protocols it understands. Most VPN servers only understand one protocol. For instance, OpenVPN only understands OpenVPN; you cannot establish an L2TP connection to it, for example. Streisand is on the other end of the spectrum. A jack-of-all-trades, Streisand allows connections from at least seven (7) different VPN protocols, and even supports sslh.

All of this matters when you are considering either creating a VPN server or you wish to setup a VPN client. You need to know which VPN protocols any given solution is capable of conversing with. If you're shopping for a third-party VPN provider, are you dead-set on a specific VPN provider? Then you need to know which VPN protocols they support and choose a VPN client application that is compatible. Or if you are in the reverse situation - you have chosen a specific VPN client to use - then you need to know which VPN server software is capable of communicating with your VPN client software.

You can think of VPN server and client applications like an electrical plug and wall receptacle. The client (electrical appliance) plugs into the server (wall receptacle). A VPN server and client must both support the same protocol. Who determines which protocol will be used when more than one is supported? It is a negotiation process (but in a tie, the client wins). First, both peers must support the same VPN protocol. Once that is established, either peer may have stronger security requirements than the other. If both support multiple protocols to begin with, the client will determine which is used. As the client is the peer initiating the connection, it calls the shots with regards to which protocol(s) it understands. Thankfully, many VPN servers support multiple protocols. This makes VPN connections more flexible. Most public VPN service providers have written custom server software to detect incoming VPN clients and route them to the best option out of a suite of supported VPN protocols, however that process is not necessary if the server operator wishes to stand-up a VPN server with "out of the box" functionality. Depending on which VPN server software is chosen, it is possible to accommodate a variety of incoming protocols.

The chart below correlates VPN server types to their supported VPN protocols, and corresponding VPN implementation methods, including Layer 2 and/or Layer 3 support, and whether TCP, UDP, or both network protocols are supported. The "IP v6" column indicates whether or not the VPN client/server connection may be initiated over an existing IPv6 connection or not. And finally, if the VPN server supports mesh networking, that is also indicated in the chart.

VPN Server Protocols
Server Type Supported VPN Protocols Network Protocols
Layer 2 Layer 3 Mesh
IP4 IP6 TCP UDP IP6
Hamachi Hamachi Yes Yes No No No Yes
L2TP L2TP Yes No No No No No
L2TP/IPsec L2TP/IPsec No No No Yes No No
OpenSwan IPsec,L2TP/IPsec
OpenSwan
Yes Yes No Yes Yes No
OpenVPN OpenVPN Yes Yes Yes Yes Yes No
PPTP PPTP Yes No Yes No No No
SSTP SSTP No No Yes No No No
Streisand OpenConnect,OpenSSH
OpenVPN,Shadowsocks,sslh15
Stunnel,Tor bridge,WireGuard
Yes Yes Yes Yes Yes No
StrongSwan IPsec Yes Yes No Yes Yes No
Tinc 1.x Tinc|TAP|TUN Yes Yes Yes Yes Yes Yes
SoftEther L2TP/IPsec,L2TPv3/IPsec,OpenVPN
SoftEther,SSTP,EtherIP/IPsec
Yes Yes Yes Yes Yes No
WireGuard WireGuard No No No Yes Yes No

15 sslh is a protocol multi-plexer application. It combines multiple protocols on the same port, with built-in support for HTTP, SSL, OpenVPN, TLS, SSH, Tinc, XMPP, SOCKS5. Any protocol can be supported by differentiating it with a regular expression.