Having explained the process behind gateway routes in the previous article, what is a split gateway?
This article covers split gateways and split routes. It describes reverse engineering existing split routes. An introduction on how to create split routes and split gateways is described in the next article in this series, Split Gateway Tutorial.
Before you get serious about creating a split gateway, and especially a split default gateway, I recommend reviewing A Brief History of Linux Networking and Linux Networking Best Practices. You may also find the Linux Networking F.A.Q. to be informative.
The following topics are explained in this article:
- Analysis of existing routing tables using route and ip route commands
- Gateways vs. Routes
- Split Routes
- Single Routing Table
- Multiple Routing Tables
- Split Gateways
- Examples
This article does NOT cover how to create split gateways or custom routing tables. Those subjects are covered in the next article, Split Gateway Tutorial.
Gateway or Route?
What is the difference between a gateway and a route?
A route is a path to a device local to the current host (i.e. physically connected).
A gateway is a router.
A gateway connects the current host to remote hosts (devices).
Viewing a routing table with route or netstat, most routes are one of the following types:
- Route: A path (connection) to a local device
- Gateway: Path to a router capable of forwarded the current packet to an addressable device at a remote destination
- Host: A specific, addressable, single, local device
Why Do We Need Gateways?
Many tutorials describe the default gateway as the route to the Internet, but that is not always true and is an oversimplification. While often true figuratively, the reality is it's more complicated. Gateways are entrusted with handling packets traveling beyond the local network. The default gateway is special because - as its name implies - by default, it is the route ALL traffic will be routed to that hasn't found a more specific route in the routing table.
You might be thinking this is odd because you thought gateway routers' purpose was only to connect you to the Internet. This is a common misconception. Here's the deal: If the host device is not directly and physically connected to other hosts on the route, then that route better point to a gateway, or your traffic is going nowhere. A gateway route tells iproute2 and netfilter, "this is a remote destination." If it's not local, it's remote. If a destination is a remote IP address, point it to a gateway.
The "Work" in "Net-work"
IP networks are compartmentalized. From a hardware perspective, the networking capabilities of a machine on a network (a host) are generic. They are capable of forming datagrams based on a virtually infinite possibility of variations. At a physical level, they transmit bits of data - ones and zeros - represented as electronic bursts over a wire. The operating system of a host quantifies those signals into an organized stream of information. And as long as that block of data is standardized, and the same standard is understood by other devices the host may communicate with, we have a sytem of communicating information. However, there is a limit to how well this process functions, becaause there is a limit to where that host can physically reach. It is limited by the physical wiring of the network. So, how do you get information from a local host to a remote host, regardless of distance?
The answer of course, is via a gateway. A gateway is both a host (it is addressable and has an IP address of its own - but only on the local, physical network) and a route (it forwards data packets elsewhere). What is special about gateways is they form the next layer of the networking system. They are at the next layer of compartmentization. Gateways are routers that connect to other gateways (routers), specifically for the purpose of forwarding a local host's message (datagram) to a remote host that the originating host is incapable of reaching, but it has an address. The gateway simply passes that message along, and gets it closer to its destination. Ultimately, a gateway will be reached where the destination host address is local, and that gateway will deliver the message. The process is then complete.
Gateways act as proxies to remote networks. If the destination is not local, the packet must be handed to a gateway. We typically think of routing gateways as internet gateways, but that is not always the case. Gateways also connect private networks, such as in large organizations with multiple networks and multiple sub-nets.
Split Routes
Route splitting means "spliting" or dividing what could be a single network traffic route into multiple routes. A "split" is a routing decision technique. One way to think of split routes is as conditional routing.
How can you "split" a route? Splitting routes is accomplished through a combination of route and rule manipulation, normally using the tools ip route and ip rule. In some cases you may be able to manage it with the old route tool, but I strongly recommend against that practice. route and its cousin, netstat are fine when used as read-only resources, but they are outdated when it comes to writing data and should be avoided for that purpose.
You may split your network traffic by:
- Destination IP address range
- Hardware interface
- Different gateways
When route splitting is used, it's often for the purpose of splitting a default gateway. For example, you might normally direct your internet traffic out say, interface eth0. But, let's say you have two interfaces on your machine: eth0 and eth1, and you want to be sure your outbound traffic is using the same interface that its associated inbound traffic came from. A typical default gateway will always push your outgoing packets through a single interface. Let's say that's eth0. In order to make sure all the packets go out the proper interface, you will need to find a way to force the packets that came in via eth1 to go back out of eth1. You can do this via a combination of rules and routes. By marking the inbound eth1 packets, you may then look for the mark during the outbound packet sending process, and when those marks are seen, re-route the outgoing packet through the eth1 interface, and all will be well. While that's normally referred to as "splitting a gateway," it is an example of route splitting as it is the routes (or paths) that you are actually modifying.
The technique is not common and is not widely understood, which is a shame as it can be a powerful and useful methodology. Another useful example is splitting a VPN. A topic relatively unknown to most people who are not network administrators by trade, it's a subject I would think ought to be of strong interest to the majority of hobbyist and prosumer VPN users. The gist of it is splitting your outbound traffic so only traffic that needs to go through a VPN, passes through it.
One of the reasons split routes are not used frequently is you are forcing matching conditions to make a hard fork. Think carefully about where you're aiming the fire hose before you commit route changes! Minor mistakes in implementation can quickly wreak havoc on your network traffic with unintended consequences. Split routes should always be thoroughly vetted on a non-production test system prior to implementation. I can't stress this enough, especially in conjunction with a VPN.
ip... rules...
Routing rules (ip rule) are often, but not always necessary for effective route splits. ip rule is better equipped to handle preparatory tasks such as packet mangling and address substitutions (e.g. NAT - Network Address Translation) compared with ip route. If you have a complex scenario, plan on utilizing both ip rule and ip route commands to make it work. Either way, be certain you are choosing the most efficient tool for the job, with the least potential for collateral damage to packets you don't intend to divert.
Here are a few noteworthy scenarios when managing network traffic using split routes may be a prudent approach:
- Selective branching of outbound internet traffic between multiple gateways
- Splitting sub-network traffic by network interface
- Routing select traffic through a VPN gateway
- Load balancing your gateway traffic
- Redundancy; if a route (or a gateway) goes offline, another picks up the slack
- Failsafe circuit breakers
The technique applies not only to routing tables, but also routing rules (ip rules). The combination of the two can be used to create powerful network traffic filters that improve performance, protect privacy, or form part of a disaster recovery plan, just to name a few uses.
Routing Tables Review
A brief refresher on Linux routing tables is in order. It's important to understand the caveats regarding single vs. multiple routing tables, especially with regards to default gateways.
Here are some core concepts you should be familiar with before embarking on creating split routes or split gateways:
- Three (3) default routing tables that come pre-installed in Linux: main, local, and default
- You cannot delete the main or local tables
- The default table is not used
- You can delete the default table; however you are strongly advised not to do this as it may have negative, adverse consequences; leaving it harms nothing
- Do not enter any routes into the default table
- If you are not using custom tables, limit your custom routes to the main table only
- Split routes may exist in the main or custom defined tables
- There may be only one default route per table
The information above (and more) is explained in greater detail in Linux Routing Table Architecture
Split Gateways
Split gateways are simply a form of route splitting. A gateway that could be represented as a single route is split into more than one route. It's rare to find this concept in use within a routing table unless it concerns the default gateway.
Are you certain a split gateway is what you need? Ask yourself what you are trying to accomplish and whether or not this is the proper method. Splitting gateways - especially default gateways - is somewhat complex and can easily result in unintended consequences if you make a mistake. If a problem can be solved by another method - such as an alternate routing table - may be a better choice in the long run. Food for thought.
Splitting The Default Gateway
The default gateway is special. If an outbound packet fails to match any route in the current routing table, and there is a default gateway present, that is where the packet will be sent. It is a catch-all for network traffic.
As a result, by design there can be only one (1) default gateway (if you're a fan of the movie Highlander (1986), just picture its infamous one-liner). What happens if you use more than one? Nothing, except any after the first will be ignored. Only one is permitted per routing table, per network interface. This makes sense if you think about it. The default route defines the "default" or automatic selection of a route if no matching route is found for a packet. When you define a default route, you are establishing the floor. If no better match is found for a packet, it is sent there.
So, what if there is more than one network interface, you have an equal number of default routes in the same routing table, and you want those routes to be chosen under particular circumstances? In order to function properly, you must define rules that instruct iproute2 how you want it to funnel the packets.
Note: This method doesn't work for every circumstance. Best practice if you need multiple default routes is to use custom tables with rules.
Also, it is possible to trick the routing software and kernel into allowing more than one entry in the routing table that points to a default gateway, using the same interface. The key to the technique involves ensuring no IP address is duplicated in the default routes. This is called a split default gateway or default gateway split. Sometimes when people mention a split gateway, this is what they are referring to.
Several methods are available to manage network traffic using routes and rules. This discussion pertains to using only routes. However, it should be noted that a combination of routes and rules is a superior approach and lends much finer control over outcomes. The route-only approach is very limited in its abilities. It is detailed here as a learning exercise.
It Takes Two (or more)
The premise of a split gateway is routing traffic between two or more routes based on specified criteria. The default gateway is a bit different from other gateways. For starters, iproute2 will not allow more than one default gateway. Or will it? Actually, it is possible to trick it into allowing multiple default route entries provided there is no overlap in the assigned IPv4 ranges for each route declaration.
Another "trick" per se is to create default gateways in more than one route table. With this strategy, you may create as many default gateways as you like, so long as each routing table only has one, or as described above no IPv4 address has overlap for the default gateway within each table.
The net result is the validation of the default routes can be spoofed a bit, though there are limits to how far you can push it. There is some validation going on at this stage. Ensure there is no overlap in your default gateway rules. If there is, they will fail to function correctly and may result in unpredictable behavior (e.g. dropped packets).
Benefits of Split Default Gateways
One of the coolest uses of split default gateways is dynamically splitting your internet traffic between your normal outbound gateway and a Virtual Private Network (VPN).
If you are willing to learn a little bit about network routing and put some effort into understanding what your server or other host device is actually trying to accomplish on-line, and then customizing how your network traffic is delivered remotely. Now, in this day and age of what seems like ubiquitous bandwidth, you might think that's a non-issue, but I disagree. However, the benefits of what I'm about to explain are truly dependent on your use-case scenario. If privacy or speed are your top concerns, you should listen up.
Single ("Main") Routing Table Solutions
These examples demonstrate solutions using iproute2's main routing table. The main table is one of three (3) system default routing tables (main, local, and default are their names). The main table is the ONLY system routing table you should ever modify.
First, the single-table solution. This technique is tricky because of the fact every route will be evaluated at the same time (this is not true in multi-routing table configurations). While this can be advantageous in some scenarios, single-table designs work best with small network configurations where there aren't a lot of routes. In a larger system, it is difficult to troubleshoot a single table with hundreds or thousands of routes. Either way, it's a double-edged sword.
The main benefit is all your routes are in front of you in one spot, making routing management simpler. On-the-other-hand, if you make a mistake and hose up your routing table, it may be more difficult to recover versus a multi-table arrangement where you may be able to disable or redirect a particular table with suspected faulty routes while troubleshooting.
Multiple Routing Tables
There are multiple routing tables. In fact, you can theoretically have up to 255 routing tables in total. The process of how these tables are managed is discussed in iproute and Routing Tables.
Multiple tables allow the possibility of multiple default gateways. This can be really useful for certain routing strategies, such as failover network designs. However, there is a catch. If you have a default gateway route (a.k.a. default route) in more than one table, only the default route in the FIRST table that is processed will be applied. Take this into consideration if you're attempting to establish multiple default routes for multi-faceted gateway routing. Depending on your implementation plans, you may need to place all your routes in a single table.
Examples
Below you will find some split routing examples, broken down into two categories: Single Table methods using the default routing table for user created routes (main), and Multiple Table methods that utilize custom routing tables as part of their solution.
This article is intended to introduce you to the logic process behind split gateways. Therefore, the examples herein are limited strictly to route-based solutions. More complex scenarios are explained in the Split Gateway Tutorial.
Example #1: Single Routing Table, Dual NICs, Dual Gateways
This is an example of what not to do. There is a single routing table (main). There are two interfaces connected to the local device: eth1 and eth2. As you can see in the table below (per route -n command), each network interface (NIC) is assigned its own gateway. Technically, this is OK because the interfaces are different, even though they are in the same table and use the same IPv4 range. However, think about the problem this creates and you'll see why this practice is discouraged. I'll explain what is wrong after you have a moment to examine the routing table.
Kernel IP routing table | |||||||
Destination | Gateway | Genmask | Flags | Metric | Ref | Use | Iface |
---|---|---|---|---|---|---|---|
10.10.0.0 | 0.0.0.0 | 255.255.255.0 | U | 0 | 0 | 0 | eth1 |
192.168.15.0 | 0.0.0.0 | 255.255.255.0 | U | 0 | 0 | 0 | eth2 |
0.0.0.0 | 10.10.0.1 | 0.0.0.0 | UG | 0 | 0 | 0 | eth1 |
0.0.0.0 | 192.168.15.1 | 0.0.0.0 | UG | 0 | 0 | 0 | eth2 |
So, what's wrong with this table? I'll give you a hint. There's no packet mangling occurring before the packets hit this outbound route (i.e. no rules via ip rule have been created to help segment traffic).
There are two (2) problems:
- Any packet not matching one of the two LAN routes will be sent out via eth1, regardless of which interface it entered on.
- Any packet routed to a gateway (does not match the criteria of either LAN) will always be routed out eth1 via the gateway at 10.10.0.1. Do you see why? The first gateway entry in the routing table (3rd line) will always win precedence, because the two gateways have equal weight in terms of how the routes are evaluated. This means the first one will ultimately be chosen, every time. The second gateway will never be used. Lines 3 and 4 have equal weight in precedence because they both cover the entire range of IPv4 addresses and are equal bit lengths. Therefore, the first entry will win the tie.
Following best practices, the proper way to handle this situation is create separate tables for each interface to house its rules, and use ip rules to tag incoming packets from at least one of the interfaces. Routes can then be created that deliver the packets properly. This subject is covered in the how-to discussion of split gateways in the Split Gateway Tutorial. I have mentioned it here to give you an idea of what you don't want to see in your routing tables!
By definition, there can be only one default route. Contrary to that fact, it is possible to trick the routing code into allowing multiple default gateway paths through a variation on the split gateway method. The technique is explained below in this article.
Example #2: Single Routing Table, Split Gateway, Default to VPN
When you have a client-side VPN, a virtual network adapter is created. This example has a type of split gateway also known as a split VPN. There are two (2) entries in the routing table pointing to the VPN network interface (tun0). The routing table performs selective routing of packets based on the destination address, splitting the VPN route into two possible destinations. Note the VPN is the default route, so in this traffic gets routed to one VPN address or the other by default.
Kernel IP routing table | |||||||
Destination | Gateway | Genmask | Flags | Metric | Ref | Use | Iface |
---|---|---|---|---|---|---|---|
10.10.0.0 | 0.0.0.0 | 255.255.255.0 | U | 0 | 0 | 0 | eth0 |
0.0.0.0 | 138.17.92.14 | 0.0.0.0 | UG | 0 | 0 | 0 | tun0 |
0.0.0.0 | 138.17.92.15 | 128.0.0.0 | UG | 0 | 0 | 0 | tun0 |
From this table we can derive the following about the current host's network connections:
- LAN on 10.10.0.0 - 10.10.0.255
- Half of the traffic bound for the Internet 0.0.0.0 - 128.0.0.0 is routed to a VPN gateway at 138.17.92.14
- All other traffic 128.0.0.0 - 255.255.255.255 is routed to a VPN gateway at 138.17.92.15
As you can see, the example above still has only one default route, but it is split between two (2) VPN gateways.
By definition, there can be only one default gateway per routing table. However, through careful routing definitions it's possible to trick the kernel and divide this traffic as well.
Server side VPNs manage VPN connections. The client-side is what most people are accustomed to. If you are using a VPN, you are on the client side. If you are a VPN service provider, you are managing the server portion. In the case of the latter, I'm going to presume you know what you are doing (hopefully).
Example #3: Single Routing Table, Split Gateway, Multiple LANs
Same as #2 above, with multiple Local Area Networks (LANs) and printed with route instead of route -n (i.e. hostnames shown).
Kernel IP routing table | |||||||
Destination | Gateway | Genmask | Flags | Metric | Ref | Use | Iface |
---|---|---|---|---|---|---|---|
10.10.10.0 | 0.0.0.0 | 255.255.255.0 | U | 0 | 0 | 0 | enp2s0 |
192.168.15.0 | 0.0.0.0 | 255.255.255.0 | U | 0 | 0 | 0 | enp2s0 |
default | hostname_vpn1 | 0.0.0.0 | UG | 0 | 0 | 0 | tun0 |
default | hostname_vpn2 | 128.0.0.0 | UG | 0 | 0 | 0 | tun0 |
Breaking down the table above, we see there are two (2) LANs and two (2) VPN hosts:
- A LAN exists on 10.10.0.0 - 10.10.0.255 and uses interface enp2s0
- Another LAN exists on 192.168.15.0 - 192.168.15.255 and uses interface enp2s0
- The default route points to a split VPN gateway. The split is 1/2 of the possible addresses split between two different VPN hosts called hostname_vpn1 and hostname_vpn2.
Example #4: Single Routing Table, Split Default Gateway (Partial VPN)
Here is a tighter focus on splitting default routes. This is a very basic example where the default route is split between a VPN interface (tun0) and a normal network interface (eth0). I know it's odd. This is for demonstration purposes only and not representative of a real routing table.
Kernel IP routing table | |||||||
Destination | Gateway | Genmask | Flags | Metric | Ref | Use | Iface |
---|---|---|---|---|---|---|---|
0.0.0.0 | 10.10.0.3 | 128.0.0.0 | UG | 0 | 0 | 0 | tun0 |
0.0.0.0 | 10.10.0.4 | 0.0.0.0 | UG | 0 | 0 | 0 | eth0 |
What's going on here? The default route has been carved up into two (2) sections, by using the genmask (netmask). Notice the destination field remains the same on both lines. The combination of destination and genmask on both lines results in 1/2 of possible IPv4 addresses sent to a gateway on the eth0 interface and 1/2 sent to a gateway on the tun0 interface. The important point here is no IPv4 address is duplicated in the range within the table, as assigned to the default route.