Routes and Rules

Routers manage routes. Some routers act as gateways. A gateway is a forwarding router that forms a bridge between local and remote networks. A gateway is required when sending a packet between networks that are remote to one another. It's called a "gateway" because it is an opening, or path, to another network the local device is not capable of connecting to directly (physically). The purpose of this article is dive a little deeper into gateways.

Analyzing the output of network route inspection tools such as ip route, route, and netstat will allow you to identify network gateways. The details of how to use those tools are discussed in several other articles on this website, including:

• Route and Netstat
• Routers, Routes, Subnets, and Netmasks

Identifying a Gateway

Let's briefly review the headers in the route command's output:

There's a column here for Destination, a column for Gateway, a column for Genmask, and a column for Flags. The other columns are unimportant in terms of evaluating a route's purpose. Destination and Genmask go together and are used to calculate the route's routing prefix and wildcard mask. Together, they clarify the scope and size of the route. Meanwhile, the Gateway and Flag values identify what each route points to.

You will encounter three (3) common flags over and over in routing table displays from route or netstat. They are:

• G : destination is a gateway; the route is indirect
• H : destination is a host; the route is direct
• U : destination is a route; the route is direct

Direct vs. Indirect Routes

You may have noticed the "direct" and "indirect" comments above. What do those terms mean? Direct means the route is directly accessible (to the local device). This means it's physically accessible ("on the wire"). An indirect route is the opposite ("can't get there from here"). To reach a destination on an indirect route, the packet must utilize a forwarding router as a proxy to reach the destination.

Gateway Column vs. Gateway Flag

The layout of routing table displays can cause confusion. Depending on which tool you are using to view router tables, the gateway column will display as one of the following four (4) values for each route:

What is the significance of the Gateway column versus the presence or absence of a Gateway flag?

You need both. The gateway column value in the routing table display identifies the IPv4 address of the gateway for the associated network route. And the gateway flag (G) must be present for the route.

Here's a list of requirements to help clarify:

• A valid IPv4 address must be specified for the gateway address
• An empty gateway value (0.0.0.0 or *) is valid, but the route is not a gateway
• If the Gateway flag (G) is not present, the route is not a gateway
• If the Gateway flag (G) is present, the gateway column must contain a valid IPv4 address, and cannot be 0.0.0.0
• The Up flag (U) must be present; if not, it means the Gateway is not reachable (connection is down)
• The device pointed to by a valid gateway route must be a forwarding router

0.0.0.0

The IPv4 address 0.0.0.0 (or *) has a special connotation, though its meaning varies depending on where it is located in the route instruction. When found under the Destination or Genmask columns, it influences the destination address range and size of the route by manipulating the routing prefix. When under the Destination column, it is sometimes represented by the word, Default. This is because 0.0.0.0 establishes its valuea is the lower boundary IPv4 address in the IPv4 address range covered by its particular line in the routing table. The Genmask is then used to create the upper boundary of the range. The net result is any packets not filtered out by other routes will be trapped by this "default" route.

Under the Gateway column it means, "A gateway is not specified." This is normally indicative of a route pointing to a Local Area Network (LAN).

Local Area Network (LAN) Connections

Like a host, a gateway device must be addressable (have an IP address). If you see 0.0.0.0 (or *) in the Gateway column output, it means, "unspecified" or "empty." In other words, there is no gateway. A Gateway column value of 0.0.0.0 (or *) combined with the presence of an UP flag (U) and absence of a Gateway flag (G) tells you the route is a Local Area Network (LAN), and the local device is physically connected to all the devices on the LAN.

The Loopback Adapter

We've now discussed direct and indirect routes, what gateways are, what 0.0.0.0 and * mean, and how to identify a route pointing to a LAN. There is one more important odd-ball scenario worth mentioning. I've included it here in this section on Gateways - even though it's not a gateway - because functionally, it acts like a gateway.

Some local applications encapsulate data using IP protocols such as TCP or UDP packets and send it to other internal programs on the same device. When this occurs, the packets need to be routed internally. However, there is a problem. These packets are in the outbound packet queue, yet they really belong in the incoming packet queue. How is this conundrum resolved? By redirecting the current packet from the outgoing packet queue into the incoming queue instead. This is accomplished via the loopback adapter.

Also known as the local loopback adapter, local loop, local adapter, or lo interface (pronounced "el" "oh" or "LO"), the loopback adapter is a unique type of virtual network interface. It takes outgoing packets and redirects them back to the local device's inbound IP packet queue. Of course, as with many things, solving the aforementioned problem is not quite that simple. An interface by itself is just a conduit. Something needs to direct those packets to the loopback interface: routes in a routing table.

Loopback is NOT localhost

Quick point of clarification. The loopback adapater is NOT the same thing as the localhost.
The loopback adapter is a virtual network interface. Localhost is a virtual IP address.

The Local Routing Table

Depending on which network route management tool interface you are using, you may never notice this routing table, but it's there. Take a look through your Master Routing Table, and you'll find a table named local with ID number 255. In case you need a refresher and don't feel like reading about it, I'll re-cap what this means for you. The Master Routing Table refers to all the routing tables on the local device. It keeps track of these tables by name and an associated index ID. The ID range is 0-255. Lower index number tables receive precedence over higher table IDs. Table index 0 (zero) gets first dibs. It gets processed before every other table. Conversely, index number 255 is always handled last. And by default that is... wait for it... the local table. As each routing table is processed, if a route in the current table matches the current packet, that route will be used and subsequent routing tables will be ignored.

If you've been paying attention, you should have realized what I'm going to tell you next! What happens to those packets addressed to the local device itself? They will not find a match in any of the routing tables until they hit routing table ID # 255 - the last one - called local. This means those packets have to go through being screened by every other route in your route database. It also means if anything goes wrong in table 255 (i.e. can't find a match to the current packet), then the packet falls on the floor and generates an error (or sometimes nothing happens and it just disappears into oblivion).

Modifying the local table is not for the faint-of-heart. Do NOT do that unless you absolutely know what you are doing.

Friends Don't Let Friends Modify Their Local Routing Table

The local routing table is dedicated to handling routing packets destined for the loopback adapter, which ultimately delivers them to the localhost. You can add your own routes referencing the loopback adapter interface in any table. If you wish to do so, I recommend adding them in a table other than the local table. The local routing table is part of the built-in infrastructure that keeps your local device running properly, and it's simply a best practice to incorporate any routes referencing the loopback adapter in another table, such as a custom table (preferred), the default table, or the main table.

You may find it helpful to review the route types incorporated into the ip route tool. "Local" is one of them. It redirects to the loopback adapter, providing a simple method of creating new routes where you want that functionality, without modifying the local route table.

Now You See It, Now You Don't

One of the drawbacks to the older route and netstat commands is they're only capable of displaying the main routing table. Why? The fact is they are just old tools. So old they pre-date the concept of multiple route tables. They were developed when there was only one routing table. To view the local routing table, you will have to use ip route.

~# ip route show table local

I cannot stress enough that you should be very cautious when manipulating the local routing table, if ever.

For those of you who can't help yourself, remember the system reset button is your friend!

It's For Me! (localhost)

There are a few special (read: reserved) IPv4 addresses in the Linux networking world. One of them is 127.0.0.1 - localhost - the pseudonym for the local device. In spite of the fact there is a dedicated IPv4 address to represent itself, addressing a packet to 127.0.0.1 (the localhost IPv4 address) does not actually cause the packet to be sent to the local device as one might expect. Rather, it works the same way as described above regarding the loopback adapter. In other words, any packet sent to the "localhost address" (127.0.0.1) must be intercepted by a route in a routing table processing the outgoing IP packet queue. The route in question then needs to feed it back into the incoming IP packet queue. Only then can it be processed by the local device.

Examples

Here are a few different examples to help reinforce the concepts described in this article.

Default Route Re-cap

Reminder: Every routing table needs a default route. The default route must be a gateway, and it must route all network traffic. When the routing code attempts to match a packet by destination address in the routing table, if there is no more specific route for the packet it will be sent out via the default route. On a server that is not a router or firewall, you'll normally have a default gateway that is represented by a single line in the main routing table.

A Basic Gateway Example

Suppose you have a simple network with two entries in your main routing table. One is the route for the local network, which tells the kernel which IP addresses are directly reachable. And the second is the “default gateway,” which tells the kernel in order to reach the rest of the Internet, traffic should be sent to the gateway. The gateway must be a router or firewall device.

Here you see the top line in the routing table (output above derived via route -n command) is a network path with the U (Up) flag set. We can surmise this must be the LAN. The Destination of 0.0.0.0 is equivalent to default. If you viewed this routing table using the route command only (no switches), you would see the following output:

One or the other can be easier to read, depending on your preferences. In the output above, notice on the first liine, the gateway IPv4 address (0.0.0.0) is replaced with an asterisk (*). This signifies there is no gateway associated with that route. This is just a different way of representing the same information. Likewise, the second line now shows default as the destination and has a gateway flag (G flag).

Regardless, there’s only one line comprising the entire default route path. The 2nd line in both examples tells us:

1. This route represents the default network path
2. The default path points to a gateway (router) named hostname at IPv4 address 192.168.1.1
3. Any packet not already captured by a more specific addressable route will be routed to the gateway (default route)

There are two crucial pieces of information to note:

1. If a route represents a gateway, it will have an IPv4 address under the Gateway column AND the G flag must be set under the Flags column
2. Both the Destination and Genmask columns for the gateway are set to 0.0.0.0

Balancing a Wire: Route Splitting by Range

A common technique in server load balancing is splitting a range of similar IP addresses between two or more routers. The idea is to divide the performance load based on destination address. There are a variety of reasons why you might want to do this, such as geo-location, hardware limitations, or redundancy. For example, you might have a farm of web servers that accepts incoming connections via a single, powerful firewall server which acts as a gatekeeper. You decide it's prudent to split those incoming connections between two routers in order to better manage the load and imrpove your customers' experience. Your first step is setting up outgoing routes from the firwall server forwarding each connection to one router or the other.

Regardless of the reason, I'll show you a very basic model of this concept.

An extension to this same technique may be applied to a Gateway.
Split Gateways are covered in the next document in this series: Split Gateways

The order of the routes in the routing table is not important. Remember, the kernel will choose the longest or most specific matching route for a packet. However, it is a best practice to organize routes in a routing table from least to most specific filtering, as it makes it easier for humans to troubleshoot the table.

Route splitting creates routes with duplicate destination addresses and leverages the netmask as a filter. When the genmask (netmask) is 0.0.0.0 for example, everything passes through. When it is 255.255.255.255, only a single address will be able to pass.

Here is a basic example of route splitting (branching logic):

Notice the destination IP addresses are the same, and they are not specific to a single IPv4 address, but rather represent a range (in this case x.x.x.0 or /24 or a 255 address sub-net). However, the Genmasks are different on each line. This is the key. While initially it appears as if there are duplicate entries in the routing table (not allowed), the fact is they are not duplicates because the Genmask is different. The Genmask becomes the filter; in this case splitting the /24 address range into two halves (x.x.x.0 through x.x.x.127 and x.x.x.128 through x.x.x.255).

Now, why would you want to do this? At first this seems redundant. Notice both routes connect to a LAN (Local Area Network). They both have only an "UP" flag (U). What is the point in splitting a single route into two routes when one would suffice? Well, actually they are not the same. Take a look at the interface (Iface) for both and notice they are different. Aha! There are two (2) LANs here. Now, we can see the load balancing is between two sub-nets.

If the packet destination is between 192.168.10.0-192.168.10.127, the destination is accessed via the eth0 interface. If the packet destination is between 192.168.10.128-192.168.10.255, the packet will be routed via the eth0 interface.

Loopback and Localhost

To illustrate the point above regarding packets sent to localhost, here is an excerpt from a typical local routing table of a Linux server:

local 127.0.0.0 dev lo  proto kernel  scope host  src 127.0.0.1
local 127.0.0.0/8 dev lo  proto kernel  scope host  src 127.0.0.1
local 127.0.0.1 dev lo  proto kernel  scope host  src 127.0.0.1
broadcast 127.255.255.255 dev lo  proto kernel  scope link  src 127.0.0.1

If we place those route instructions in a table, we can more easily digest what is going on.

Let's pick apart the first line. The route type is being delivered to the loopback device (route type = local). The destination is IPv4 address 127.0.0.0. We can see the device that will be utilized to send the packet (remember, we are currently in the outbound IP packet queue) is the loopback adapter (dev lo). The destination target is a host (scope host). And the source IPv4 address needs to be 127.0.0.1 (src). This last instruction in the route tells the kernel to alter the outgoing packet right before it is sent, such that no matter what it's source IPv4 address was before, it is changed to 127.0.0.1. Since the interface is the loopback adapter, this packet will be handed to the incoming IP packet queue, where it will be treated like any other packet. The incoming process will see the packet originated from 127.0.0.1 (the source address, i.e. localhost). It will see the packet is addressed to 127.0.0.1 and will hand it to the localhost process. Voila! Our previously outbound IP packet has been properly redirected back to the local device, and the internal process will now receive it.

Now, you might be wondering, "Why are there several nearly identical routes here?" These are slight variations on the same theme. For instance, the example we just picked apart is designed to capture packets addressed to the local device that are - for whatever reason - identified as bound for the LAN. Take a look a the 2nd line, immediately below it in the table above. Notice their similarities? In fact, those first two (2) lines are nearly identical. Their only difference is the scope. The first says, "link" and the second says, "host." The difference? "Link" means it is associated with the local network (LAN), while "host" means it's associated with a destination address on the local host. It's a very subtle distinction. The bottom line is together they will capture any packets targeted at 127.0.0.0 regardless of what their scope is. There is another type of host called "global" or "universe" (depending on implementation), but those are only applicable to routers. Thus, there is no need for a global scope route and that's why you don't see one in the table above.