Thinking of creating split routes or split gateways? This article is a basic introduction, but it is a hands-on tutorial and will walk you through the building blocks of constructing split gateways, and particularly split default gateways.
Do not make changes to default gateway connections in routing tables unless you are either physically connected to the machine (local) or on the same LAN if remotely connected. Failure to follow this advice is likely to result in a disconnect and unrecoverable error (reboot required).
What this article is:
- How to split a gateway
- Guidance on how to load balance network traffic
What this article isn't:
- A deep dive on anything
- An overview of Split Gateways
Simple Split Gateway
This example explains a very simple split gateway concept. This is just to demonstrate what a "split gateway" means. "Splitting a gateway" - in its most basic form - means simply dividing (splitting) network traffic that could be routed to a single gateway between two or more gateways in a routing table.
You can use any IPv4 address you like under the destination address, however the genmask (netmask) MUST be a valid CIDR notation value (i.e. 0-32 bits selected), of which there are only 33 possible values. Furthermore, the genmask/netmask must correlate to the destination IPv4 address with regards to its class (i.e. A, B, or C).
The easiest way to think of this is each number in the netmask must be greater than or equal to the corresponding numeral in the destination address (i.e. where destination = x.x.x.x and netmask = y.y.y.y, "y" must be greater than or equal to "x" for each corresponding number).
If you're not clear what your options are here, you may wish to consult a Netmask/Genmask conversion table.
Imagine you have just one interface (eth0) and a routing table with a default gateway at 192.168.1.1, and you wish to split that gateway into two gateways at addresses 192.168.1.2 and 192.168.1.3, both of which are connected to your eth0 network interface. So, what you are starting with in your routing table - relevant just to the original gateway - would be something like this:
root@test-server:~# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.1.1 0.0.0.0 UG 0 0 0 eth0
Focusing solely on your routing table, you decide to split your new gateways such that the one at 192.168.1.2 routes only traffic to 111.0.0.0 - 111.255.255.255 and the new gateway at 192.168.1.3 will become your default route.
Since you already have an existing default gateway, you must remove it first. Otherwise, when you attempt to add another one, you'll get this message:
RTNETLINK answers: File exists
When you attempt to add a new route, the iproute2 process checks the syntax of your command and cross-references it against existing routes in the target routing table. It will not allow you to create a duplicate route to the same IPv4 address over the same interface. If you try, you will receive an error message.
To avoid this problem, first remove your old default gateway at 192.168.1.1:
ip route del default via 192.168.1.1
ip route del (route delete) command must match the route name EXACTLY.
Then setup your new split gateways like this:
ip route add 111.0.0.0/255.0.0.0 via 192.168.1.2
ip route add default via 192.168.1.3
This concludes an over-simplified example of establishing a split gateway. In reality, if you want to do this you need to understand more depth about the problem you're attempting to solve. The splitting of a gateway route is just part of a solution. For example, using different gateways to route traffic in this manner could result in mis-directed packets between the gateway routers, depending on a variety of factors. The gist of it is if you begin a network connection through one gateway, you should make sure your network configuration will maintain the entire connection along the same local path, as long as that connection is established. Otherwise, you could end up with dropped packets or the whole connection could get dropped. But again, this example was meant as a very basic tutorial and example to familiarize you only with the concept of split gateways. Now, let's move on to a real example that is also for demonstration purposes only, but this one will actually work.
Load Balancing with Two Network Interfaces
This example demonstrates establishing a load balancer between two outbound internet gateways using routing tables and routes. There are a variety of ways to spin this, so take this for what it is - a very basic example of the routing and route table concepts behind network load balancing.
Tools and knowledge required:
- ip route command
- ip rule command
- Network routes
Scenario: A LAN (Local Area Network) with Two (2) Outbound gateways, and two (2) outbound network interfaces to the Internet. The goal is to split outgoing traffic half-and-half between the two interfaces. In the interest of simplicity, outbound traffic will be split between the two (2) network interfaces based on destination IPv4 address. This isn't true load balancing, but it keeps this example as simple as possible.
Prerequisites
To make this work, you need at least a basic understanding of ip route and ip rules. If you want to create sophisticated VPN or gateway branching behavior, you will also need a good grasp of iptables. For now, let's focus on some simple examples to nail down the basic concepts.
The following are presumed to be in place before you begin:
- Linux operating system (any distro)
- SSH or direct physical access to your Linux terminal
- root user access (sudo)
- Any other appropriate or necessary user credentials or definitions
Step 1: Add new routing tables to the Master Routing Table
In order to prevent interfering with any other routes or routing rules, we will create a new routing table for each of the outbound interfaces. Open the Master Routing Table for editing and add two new tables to it (use any available numbers you like between 100 and 200).
nano /etc/iproute2/rt_tables
Add these lines:
200 test1
199 test2
Step 2: Create new routing tables
Now, let's make use of the new tables. We will presume the two (2) tables we are interested in are exclusive to internet traffic, and the LAN is accessed via a separate, 3rd interface (e.g. eth0). We will ignore the LAN interface for the purpose of this exercise.
Setup the new routing tables:
ip route add 10.10.10.0 dev eth1 src 10.10.10.10 table test1
ip route add default via 10.10.10.10 table test1
ip route add 192.168.1.0 dev eth2 src 192.168.1.1 table test2
ip route add default via 192.168.1.1 table test2
Now, there is the meat of the whole setup. Notice what we're doing here. First of all, these are outbound gateways, so we need to be able to capture any traffic that is not local. That means casting a wide net of all possible IPv4 addresses. At the same time, these gateways have to be local to our current host device. So, each gateway router must have an IPv4 address on the LAN.
For this example, we will place one gateway router at 10.10.10.10 (the eth1 interface) and the other at 192.168.1.1 (the eth2 interface). To make this happen, we need to first establish a route from the local device to each router (it's IPv4 address). That's what you see above in the 1st and 3rd lines; they establish routes from the current (local) device to the gateway routers. You can see from examining the gateway router declarations that they are on the same local network as the local device. Beneath those lines we establish the fact each of those routers are gateway routers, and assign them default routes. Notice this is all going on in their respective tables, so each router is able to have its own "default" route unto itself.
Let's pause and review the results one more time.
There is a gateway router at 10.10.10.10.
ip route add default via 10.10.10.10 table test1
Accessible to the local device via its eth1 network interface, which is attached to a network at 10.10.10.0.
ip route add 10.10.10.0 dev eth1 src 10.10.10.10 table test1
There is a gateway router at 192.168.1.1.
ip route add default via 192.168.1.1 table test2
Accessible to the local device via its eth2 network interface, which is attached to a network at 192.168.1.0.
ip route add 192.168.1.0 dev eth2 src 192.168.1.1 table test2
Step 3: Leverage the Main Routing Table
Now, here we are going to use a little trickery to accomplish our main goal. As you may recall from previous articles in this series, the main table is the table name built-in to iproute which by design contains all user-defined routes. Since iproute is a legacy tool, when its successor (iproute2) was created, it kept the main table concept, even though it expanded Linux' number of user-defined tables from one (1) to 255. Unless you've mucked around with the Master Routing Table, the main table will always be called, and it will be called prior to our custom tables, but that's OK. We can use that to our advantage in this case by leveraging the main table to gain control and force the load balancing we want.
Next, we will make the main table aware of our two gateway routers and the interfaces they are connected to. There's no reason we can't make other tables aware of them. Remember, tables are processed independently of one another, so what is the point in doing this? The point is that in a moment we are going to setup some rules instructing the local device to use our two (2) new tables (test1 and test2) with their dedicated network interfaces (eth1 and eth2.). However, it's good to setup an insurance policy via the main table, since it is going to be called by default for any traffic that doesn't get captured by the new rules we will create. This helps ensure any outbound traffic to a gateway will be processed as we wish.
So, here goes. Inform the main table that these two (2) new routes exist, how to reach them (device), and the last, but important part (src) says that if a packet is sent out the corresponding interface and via the given route, and if the local device is the source of that packet, then use the src IPv4 address as the packet's source address. In a moment, you'll see why.
ip route add 10.10.10.0 dev eth1 src 10.10.10.10
ip route add 192.168.1.0 dev eth2 src 192.168.1.1
Did you notice in the commands above, there is no table name specified? This is because the default or presumed table name - when none is provided to ip route is the main table.
Step 4: The Rules
Now, we finally add our routing rules. This is the logic that will force which route to use, based on the interface. It's the magic that ensures alignment between each interface and its corresponding gateway, by associating each interface with the IPv4 address of its corresponding gateway router. At this point, all this work might seem convoluted, but you must realize that iproute2 is very literal. It presumes nothing. You must teach it.
For the first time, we'll now use the ip rule tool.
ip rule add from 10.10.10.10 table test1
ip rule add from 192.168.1.1 table test2
As you can see, we have low linked a relationship between each gateway host (it's IPv4 address) and the routing tables.
Step 5: Who's Up First?
At last we have one more important step, and that is to define what will be the default route for outgoing packets. For this, we must return back to the main table and clarify how new outgoing packets are going to be sent out. At this point, we've already setup handling packets that are part of an existing connection on either interface (eth1 or eth2), but we have not clarified what should happen when our local device is initiating an outbound connection that is a new connection. For that, we must declare a default gateway in the main table.
If you've been paying attention you realize this is now a conundrum. How can we do that when our goal is load balancing? We can't have just one gateway (interface out), can we? No. At least if we do that we won't meet our goal. Thankfully, there is a solution. However, before I get into that I want to point out this demonstrates the fact you could have an alternative goal in mind that dovetails nicely into this example. Namely, what if you wanted to have a primary outbound interface that you prefer, and a secondary outbound interface serving as a backup? Well, we are going to get into that in the next example, because that is a fale-safe or fail-to-backup solution. My point is this solution (load balancing) and failsafe or backup solutions are very similar, as you'll soon see.
Back to our regularly scheduled program.... The final step is to setup the default gateway in the main table, to control outbound connection paths when the connection originates from the local device. To fix this missing link, ordinarily, this would be a single line and it would identify a single interface and router IPv4 address. For example, if we wanted all outbound connections originating from our local device to be pushed out on device eth1, we would setup a single default route, like this:
ip route add default via 10.10.10.10
Or if we wanted all the traffic originating from the local device to go out eth2 instead, we'd do this:
ip route add default via 192.168.1.1
But since we are splitting or load balancing, its a little different. We want to split the routes such they are assigned equal ranges of outbound IPv4 traffic based on the destination IPv4 address. This is relatively straightforward. We'll just segment the top half and bottom half of all possible IPv4 addresses, like so:
ip route add 0.0.0.0/128.0.0.0 via 10.10.10.10
ip route add 128.0.0.0/128.0.0.0 via 192.168.1.1
These commands first push all possible IPv4 addresses to the gateway at 10.10.10.10 (which will route them through eth1 per the rules and other routes we created). Then, the 2nd line usurps half of the IPv4 addresses (128.0.0.0 and above) and redirects them to the gateway at 192.168.1.1 on eth2. The ranges become:
- eth1: 0.0.0.0 - 127.255.255.255
- eth2: 128.0.0.0 - 255.255.255.255
Note these commands also designate these routes are pointing to gateways (use of "via" in the command line indicates the command line target IPv4 is a gateway).
Step 6: Making Routing Changes Persistent
The final step is to make our routing table changes persistent across reboots. While the Master Routing Table retains its contents (since it's a file), each of the actual routing tables are loaded into memory on system startup. From that point it can be modified, but any changes need to be stored properly or they will disappear when the routing table is flushed. This happens automatically on a reboot, as the routing table is empty when the system starts up and it must be loaded anew. These steps below will ensure your changes are applied everytime the local device is restarted.
Logged in as the root user, edit your local device's interface file for the network interface associated with the changes you've made above.
nano /etc/network/interfaces
What you see will vary, depending on how your network is currently configured (and particularly your LAN or LANs). Find each interface you need to edit. In this example, that would be eth1 and eth2. What you see is going to look different depending largely on whether your LAN interfaces are already assigned a static IP address or use DHCP to get their local IPv4 address.
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
source /etc/network/interfaces.d/*
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
auto eth1
iface eth1 inet dhcp
# Other network interface
auto eth1
iface eth2 inet dhcp
Beneath the eth1 section, add the following lines:
## static ip config START ##
up /sbin/ip route add 0.0.0.0/1 via 10.10.10.10 dev eth1
down /sbin/ip route delete 0.0.0.0/1 via 10.10.10.10 dev eth1
## static ip config END ##
Your file should then look like this:
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
source /etc/network/interfaces.d/*
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
auto eth1
iface eth1 inet dhcp
## static ip config START ##
up /sbin/ip route add 0.0.0.0/1 via 10.10.10.10 dev eth1
down /sbin/ip route delete 0.0.0.0/1 via 10.10.10.10 dev eth1
## static ip config END ##
# Other network interface
auto eth1
iface eth2 inet dhcp
0.0.0.0/1 = 0.0.0.0/128.0.0.0
Beneath the eth2 section, add the following lines:
## static ip config START ##
up /sbin/ip route add 128.0.0.0/1 via 192.168.1.1 dev eth1
down /sbin/ip route delete 128.0.0.0/1 via 192.168.1.1 dev eth1
## static ip config END ##
Your file should then look like this:
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
source /etc/network/interfaces.d/*
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
auto eth1
iface eth1 inet dhcp
## static ip config START ##
up /sbin/ip route add 0.0.0.0/1 via 10.10.10.10 dev eth1
down /sbin/ip route delete 0.0.0.0/1 via 10.10.10.10 dev eth1
## static ip config END ##
# Other network interface
auto eth2
iface eth2 inet dhcp
## static ip config START ##
up /sbin/ip route add 128.0.0.0/1 via 192.168.1.1 dev eth2
down /sbin/ip route delete 128.0.0.0/1 via 192.168.1.1 dev eth2
## static ip config END ##
128.0.0.0/1 = 128.0.0.0/128.0.0.0
Now, either reboot or restart your network service, like so:
systemctl restart networking
Verify your new routes are intact:
route -n
They should be there (if not, start over as you have missed a step somewhere above).