Here are a few helpful tips related to rule and route testing.
Don't Forget to Flush!
If you make changes to ip rules, ip routes, or iptables and wish to utilize them prior to the next system reboot, don't forget to flush the cache before you change iptables and after you change ip routes. Flushing the cache forces the kernel to reload the iptables and routing databases.
Do NOT Flush ip rules
It's important to flush the cache for routes and tables, but do NOT do that for ip rules. It will break your server's connectivity.
mark vs. fwmark: What's the Difference?
Are you banging your head into a wall trying to understand the relationship between "mark," "fwmark," and "firewall mark?" Well, the good (and simple) news is they are all the same thing. The difference is fwmark is the reference name used by ip rules, and mark is the reference name used by iptables. All three (3) of these terms refer to "marks" applied by the CONNMARK and CONNTRACK netfilter modules.
Persisting Routes, Rules, and Tables Across Reboots
Routing Policy DataBase (RPDB) rules and iptables commands need to be loaded into the kernel’s memory when the server starts up. The master routing table is the only portion of the RPDB that will automatically repopulate after a server restart. The remainder of the RPDB and iptables commands must be reinstated after every restart. There are several methods of doing this, but it can be a little tricky as of Ubuntu 16.04 due to significant infrastructure changes from 14.04.