Have you ever wondered why Linux network management is so confusing? You need only make a cursory review of the history of Linux networking tools to understand how it got this way.
It's not surprising so many people become confused attempting to comprehend Linux network routing functions, given the manner in which they have evolved. The names of several important commands and processes are nearly identical to others belonging to completely different tool generations.
Legacy code has a way of sticking around in Linux for a very long time. Networking code is no different.
From the mid-1990's to 2001, the Linux networking code experienced a massive expansion of its architecture. The code remained more or less untouched until 2014, and even then the milestone was an overhauled interface to the kernel. The core remained virtually unchanged. Fast forward to 2019, and we are still using 20+ year old networking code in our Linux servers! Amazing when one takes into consideration the evolution in anecdotal technologies since then, such as network security, identity, and block chains to name a few.
The historical timeline of Linux user-based networking tools:
The First Configurable Linux Firewall
Linux kernel 1.2.1 - released in 1996 - unveiled a command line driven firewall administration tool for the first time, named ipfwadm; an abbreviation for IP FireWall ADMinistration. The first packet-level screening interface for Linux where a sysadmin could tweak how the kernel handled network packets, ipfwadm was replaced a few years later (1999) with ipchains (itself subsequently replaced by iptables the following year).
Fast forward to 2014. Nftables is introduced as a replacement and consolidation of the venerable iptables and several other little-known networking tools such as arptables and ebtables. Nftables (technically, nf_tables) combined the functionality of those tools into a single interface, which then hooked into netfilter (the kernel-based packet filtering framework present in Linux kernels 2.4 and beyond). The main benefit to nft was the consolidation of aforementioned tools plus extensions of these tools (e.g. CONNTRACK) into a single application.
Ironically, on arrival nftables received - and continues to enjoy - a lukewarm welcome within the Linux networking community.
Allow me to pause here for a moment and reflect on the terms at play so far:
- chains are network packet routing paths and processes that segment network traffic based on its destination
- ipchains is a version of the Linux firewall; while it makes use of chains, it is not the same thing as chains
- netfilter is the internal firewall found inside modern (version 2.4+) Linux kernels
- nf_tables stands for NetFilter Tables and is the name of the netfilter sub-system within the Linux kernel
- nftables is the name of a command-line interface (CLI) driven administrative tool called nftables that may be used to add, remove, and edit the firewall rules in nf_tables
- nft is the command line statement executed to launch nftables
With me so far? And so it begins to become clear to the reader why Linux networking nomenclature is so confusing.
Chains and ipchains
The chains concept in Linux networking is very primitive. Adding insult to injury for neophytes, Linux incorporates a networking process-based concept called chains (also known as ip chains) and there is a version of the Linux firewall called ipchains (identical pronunciation and nearly identical spelling). It is not uncommon for Linux networking neophytes to become confused as to which one someone is referring to in conversation, or that there is a distinct difference between them.
The networking chains concept was introduced in Linux kernel 1.3.66 (1996) and marked a milestone branch in networking logic. Incorporated as ipfwadm version 2 (ipfwadm2). The old ipfwadm process was a linear routing system. All network packets traversed the same block of code. The new chains concept partitioned networking traffic into three (3) distinct paths, depending on the destination of any given network packet:
- IN: IP packets inbound with a destination address of the current host's IP address or the localhost adapter
- OUT: outbound IP packets originating from the local host adapter
- FORWARD: outbound IP packets not originating from the localhost adapter and the destination IP address is not the current network adapter's IP address
Most of the time, when network administrators are discussing chains or ipchains, they are NOT talking about this old Linux firewall. They are usually referring to the other definition of chains.
Easily confused with ipchains (a major overhaul of the Linux networking core which debuted in 1999), the "chains" concept itself dating back to Linux kernel 1.3.66 in 1996 when there were only three: IN, OUT, and FORWARD. The concept of "chains" in ipchains entails each of the three main processes in routing network packets through a server. Those packets will either be incoming to a local destination, outgoing to another device and originating from a local service, or outgoing to another device and not originating from a local service.
You may think of the chains as highways inside your server. They are able to direct traffic flow along a particular direction, but cannot modify the packets. Although primitive compared to more recent routing processes, ipchains remains engrained in the Linux' networking DNA.
With the advent of ipfwadm2's chains concept, the processing of IP packets became more organized and paved the way for the introduction of iproute.
The RPDB Revolution
Released in 1997, Linux kernel 2.0 was a watershed event for the Linux community. It included a complete rewrite of the networking backend. A new network management tool debuted. Called iproute, the Routing Policy DataBase (RPDB) was born.
The RPDB created the concept of a table of routing tables; a "master" routing table, if you will. Prior to the 2.0 kernel, Linux used the traditional model of a single routing table. Shortly afterward the advent of ipchains arrived with the 2.2 kernel, and in September 1999, version 2 of iproute was released. A set of tools originally written by Alexey Kuznetsov that expanded the capability of the RPDB, iproute2 added an additional layer of rules which today we think of as filtering rules. This controversial change allowed some functions of ipchains to be duplicated by the RPDB. This in turn led to a split in philosophies of where and how network packet mangling and Network Address Translation (NAT) should be handled, and which tools ought to be interacting with netfilter (Linux' network management kernel layer). Still relevant today, when using ip route and ip rule, you're using functions of iproute2. It superseded several original iproute tools including route, ifconfig, and netstat.
Approximately six months later, another initiative called iptables went live with the 2.3 kernel. It replaced ipchains with a foundation for a more robust solution and more or less wrestled the reins of various functions away from iproute again. When the Linux 2.4 and 2.6 kernels were released a short time later, a number of functions of iproute2 were deprecated, solidifying iptables lead in packet mangling and NAT functions. While ipchains' legacy of chains remained, iptables expanded the chains concept to five (5) built-in chains and created the ability of users to add new, custom chains to the network management process.
Another point-of-confusion is netfilter. Perhaps because though it is spoken of regularly, yet there is no netfilter command. Netfilter is a process responsible for network packet management and NAT (Network Address Translation). It is effectively the network traffic cop inside your server.
netfilter is comprised of two parts: 1) a set of kernel hooks; and 2) a tool that translates commands to kernel actions. The various xx-tables tools interact with the translation portion of netfilter.
The next significant evolution in network packet management within Linux was iptables. An evolution of ipchains and still widely used today, iptables cannot direct traffic flow, but can modify network packets.
Tables are called by chains and provide a finer level of detailed packet filtering. The benefit of this approach is all filtering rules are located in one place. Packet directional flow control is handled by the chains and they in turn call the tables as needed. A table contains a set of packet filtering instructions, but cannot alter the direction the packet is flowing.
It is important to note although the command ipchains is deprecated in Linux, chains themselves still exist and are a critical component of iptables.
nftables is available in kernels 3.13 and later, but is not required. iptables still works.
Nftables functions differently from iptables and its extensions. It completely bypasses netfilter's translation tool and passes data directly to netfilter's kernel interface. Placing this in context, if you were to remove netfilter, nftables would continue to function properly but iptables and associated xx-tables commands would break.
nftables bypasses the traditional iptables hooks and the netfilter API, tapping directly into the nf_tables process.
While nftables may eventually replace it, for now, iptables remains the de-facto Linux firewall administration tool. Regardless of which tool you choose, there is no sign of iptables disappearing anytime soon, given its widespread usage and popularity.
The advantages of nftables over iptables are:
- More concise command structure; consolidates filtering to fewer command lines
- Chains are fully configurable; unlike iptables, base chains do not exist
- Concatenations are supported
- New protocols are supported without kernel updates
- Expressions replace and combine the concepts of matches and targets
- Dual stack IPv4/v6 implementation
Moving from iptables to nftables is a paradigm shift both intellectually and practically. Prior to nftables, Linux' approach to networking has been to embed all controls into the operating system kernel. Nftables creates a virtual machine that manages all mangling acts independently upon the networking packets, before presenting the final packet version to the kernel.
There are only two possible paths whereby a future Linux iteration could abandon iptables code: a hard fork (remove all iptables code), or a "soft" fork where the new system maintains hooks to the old system for compatibility. To date, we currently have the latter. Without a concerted effort to push developers off of iptables and onto nftables, I hesitate to even call it a 'soft' push. More like an alternative option. Rather, iptables and its variants continue to represent the gold standard in Linux networking tools, simply by virtue of momentum and familiarity (and in spite of their often clunky interfaces or command structures). Netfilter's translation code is deeply buried in the Linux kernel. So much so that it will require a major effort to untangle it if the ultimate goal of nftables is to be realized (deprecating iptables). As I write this content – over five years after the release of nftables - the current Linux kernel is 4.19 and there are no signs of the elimination of iptables on the horizon.
The iptables code is well documented and - in spite of its inefficiencies - it works. Unless someone forces a change, it's easier to continue using 20-year old code than address the task of a monumental re-write, and likely break lots of things until the fact they are broken is discovered and corrected. As the world has become more dependent on networking-based applications such as "cloud computing," the effort required to make an eloquent conversion is greater than ever. To wit, my preceding comment in the introduction: reliability trumps innovation. The Linux networking roadmap is prima facie evidence of this design philosophy.
As if the underpinnings of network packet management in Linux were not confusing enough, Ubuntu introduced another layer of complexity and confusion with the introduction of the ufw tool (Uncomplicated FireWall). Originally released with Ubuntu 8.04 LTS in April 2008, ufw purported to be, "The default firewall configuration tool for Ubuntu." (UFW, 2017). Claimed to be the "default," yet by default it is disabled. The reality is UFW simply adds another layer; a front-end to iptables.
IPv4 and IPv6
Any discussion of networking would be remiss without a brief discussion of IPv6. Netfilter handles both IPv4 and IPv6 address translation and routing. Meanwhile, it’s split in iptables. IPv6 is handled by ip6tables, while iptables handles IPv4 only. The default routing configuration in Ubuntu has a small amount of IPv6 routing built-in (just a few very broad capabilities that ensure your IPv6 packets don't get blocked by the IPv4 routing process). However, due to the substantial difference in address complexity of IPv6 vs. IPv4, I'm omitting an in-depth discussion of IPv6 routing from this guide, though the basic concepts are the same.
1 ip6tables is the IPv6 equivalent of iptables. This document is limited to discussing iptables, which only covers IPv4.
2 The Address Resolution Protocol (ARP) packet filter rules in the Linux kernel firewall modules. Not discussed in this document.
3 ebtables is a filtering tool that enables transparent filtering of network traffic passing through a Linux bridge. It is beyond the scope of this document.