Multiple NIC:s server behind NAT-router - part II

This part is actually harder to understand technically speaking, so for now I'm just going to leave you with a script that does the job. Invoke the script somewhere from init.rc, the order compared to the dyn-DNS script doesn't matter and it's perfectly alright to invoke this before the dyn-DNS script.

Note however that the script can fail if the NIC isn't ready when the script is run. It will also stop working if a NIC is removable (USB WLAN for example), in which case it has to be rerun as routing-tables will be flushed and internal IP-numbers probably different anyway thanks to DHCP. This script is robust however and you could add it to crontab as well, with a quite slow update rate say once an hour. Or better yet, have a daemon detect when a link is broken and reestablished and run the script then.

Also note, that even though one NIC will have a proper back-route in the default table, it doesn't hurt to add one more table/route/rule-set to cover the issue of not knowing which NIC:s will be up first and which ones will be secondary.

Here's the script. Invoke it with one argument, the NIC-name (you can get the NIC-names from the command ifconfig):

	#! /bin/bash
	IFNAME=${1}
	NIC_IPNR=$(ifconfig | \
	grep $IFNAME -A3 | \
	grep "inet addr" | \
	cut -f2 -d":" | \
	cut -f1 -d " ")
	LNET=$(echo $NIC_IPNR | cut -f1-3 -d".").0
	GW=$(echo $NIC_IPNR | cut -f1-3 -d".").1
	source logger.sh
	if [ "X$IFNAME" == "X" ]; then
	logger "First argument (IFNAME) must be present. Exiting..."
	exit 1
	fi
	if [ "X$NIC_IPNR" == "X" ]; then
	logger "IF [$IFNAME] not up or doesn't exist. Exiting..."
	exit 1
	fi
	logger "$(basename $0): Setting up return-route for IF: [$IFNAME]"
	logger " NIC: [$NIC_IPNR]"
	logger " NET: [$LNET]"
	logger " GW: [$GW]"
	#Check if there exist a specific table for the interface
	EXIST_TABLE=$(cat /etc/iproute2/rt_tables | \
	grep -Ev '^#' | \
	grep $IFNAME)
	OTHER_TABLE_IDS=$(cat /etc/iproute2/rt_tables | \
	grep -Ev '^#' | \
	grep -vE 'local|main|default|unspec' | \
	cut -f1 -d" " | \
	sort -n)
	NEXT_ID=$(( $((sed -e 's/[[:space:]]\+/\n/g' | tail -n1) <<< $OTHER_TABLE_IDS) + 1))
	if [ "X${EXIST_TABLE}" == "X" ]; then
	logger "Table missing for interface [$IFNAME]. Creating one as ID [$NEXT_ID]"
	echo "$NEXT_ID $IFNAME" >> /etc/iproute2/rt_tables
	fi
	logger ">>> Table:"
	cat /etc/iproute2/rt_tables | \
	grep -Ev '^#' | \
	grep $IFNAME | \
	logger
	if [ "X$(ip route show table ${IFNAME})" == "X" ]; then
	logger "Table is not set, setting"
	ip route add $LNET/24 dev $IFNAME src $NIC_IPNR table $IFNAME | logger
	ip route add default via $GW dev $IFNAME table $IFNAME | logger
	fi
	logger ">>> Routes in table [${IFNAME}]:"
	ip route show table ${IFNAME} | logger
	if [ "X$(ip rule | grep $IFNAME)" == "X" ]; then
	logger "Rule is not set, settig"
	ip rule add from $NIC_IPNR/32 table $IFNAME | logger
	ip rule add to $NIC_IPNR/32 table $IFNAME | logger
	fi
	logger ">>> Rules fo table [${IFNAME}]:"
	ip rule | grep $IFNAME | logger

view raw set_routes.sh hosted with ❤ by GitHub

Basically, what the script is doing is creating a new table for each new interface it's ever seen (which shouldn't be too many), and to this table create a specific routing table with it's own default route (which will be the "router" that's on the same sub-net as the NIC).

To that table there's also rules saying "what-ever comes in, must go back the same way".

Multiple NIC:s server behind NAT-router

Say that you'd like to create an El-Cheapo server accessible from the Internet, or just want to access your home PC from work AND that you have some reason to have two "routers" (i.e. NAT-firewall gizmos, commonly but erroneously named "routers").

Me for example, I don't have any physical line as the wireless broad-band (3G/4G) in the neighborhood is very good, though not as reliable as a physical line would be. I'm also very much into the idea of personal freedom and I'm into coastal sailing and to like the idea of being able to bring my stuff with me.

(I also don't want to pay the network-provider extra for housing servers, which I'm lucky not to have to from a legal standpoint because my provider is very liberal. But some others Fascist bastards in this country do have the stomach to charge for inbound traffic and they have even set up limitations for it already. Officially the reason is that they want to get payed for their services, especially for voice traffic. WTF - who cares about voice nowadays!? There's no doubt in my mind the real reason is to limit file-sharing using torrents, like this is going to be such a big hindrance. Bah, twisted trolls. May they rot in Hell... Anyway, back to the story...)

The plans for my accounts are very different. One is very fast and quite cheap downstream and has also unlimited quota - to my kids enormous joy and pleasure). The other costs virtually nothing, but is very throttled downstream, but not upstream. This is probably a mistake by the operator, but who am I to correct their mistakes. They're not correcting mine :-P. Furthermore, I'd like my "server" to be accessible by name(s), not by number and the IP-numbers change from time-to-time each time there's been a glitch an a "router" restarts.

Add to this the fact that radio-based networks of various flavors are by nature not as stable as their wired cousins, what do you do? Hmm... Interesting. Challenge accepted :-)

What you could do is use the old spare router and account you have left over and attach it to an extra NIC, and then use that extra router for inbound traffic (i.e. traffic initiated from the outside in) and the other one for outbound traffic (i.e. the other way around).

But here's the problem. Since these "routers" are not real routers, they change the packets IP headers address fields and piggy-back onto another field (proc ID) to translate back and forth. This is called NAT and is the trick behind why IPV4 has survived for so many years longer was anticipated. In practice, each household has become a network of it's own where all inside nodes share the same public IP-number.

This works extremely well, so to the point that what was once believed to be a IP-number shortage and the end of IPV4, and the big motivation behind the gigantic infrastructure undertaking of IPV6 a decade ago, now mostly seems to have become just a fart up-wind.
 
It was originally thought that the Net-traffic would be distributed fairly equal, which in a sense is somewhat true. But it seems no-one in the late 90' or early 20' though about that there would be that much more consumers (clients) than data-providers (servers), and that initiation of traffic would be mostly unidirectional. Big data-providers hosting your e.mail and so forth has certainly helped in that direction, even if their motivation IMO is quite questionable. NAT solved this problem over-night, and whats even better, the infrastructure cost was completely shifted away from back-bones and sub-nets, down to the end users (IPv6 would had meant a global paradigm shift, affecting each router and each computer in the world. Some people still believe IPv6 is coming, personally I consider it stillborn). Not realizing this in time and before shouting "wolf!", scaring up the whole world, Now that's what I'd call a "once in a life-time blunder"... (only superseded bu the Y2K ditto). The only outcome of such mistakes is a total loss of trustworthiness and even if there's a real need behind, any solution will be greatly delayed.

However there's a few real technical down-sides about keeping IPv4 via NAT:s in favor for IPv6, and this writing is about one of them:

For multi-homed hosts and concerning in-bound traffic: What comes in on one NIC, must go back via the same NIC (!).

This won't happen auto-magically unless you do some wizardry. The reason is that the default route in your server will always choose one NIC over the other, and if it's not the same as where it came from, the second "router" will do a NAT, and the originating outside client-socket will not be able to pair the returning package with the one it sent. Normally, it would be perfectly alright for packets to take different routes one way than the other. But if there's a NAT in the way, this just won't work and all returning packets will be lost in void.

Here's how you solve it:

Create (a) specific route(s) and rules.

This needs to be done eithe for the inferior router, or can be replicated for all your routers. The "inferior" router in this context is the one which don'y not have your servers default route set up first in it's routing table. The order of the list may differ, it's up to the OS to decide, but for computers the primary NIC is often choosen by time it got ready. Note however that this may vary a lot. Smart-phones for example tend to prefer WLAN over WAN in the (sometimes incorrect assumption) that the user would prefer WLAN or that your WLAN is always faster (which in my case for example is almost never true).

Before we continue, lets assume the following for your multi-homed host. Each NIC belongs to a separate test-networ. Networks in 192.168/16 are used for this for this purpost, so:

NIC1: 192.168.0.X  / IF1: eth0  / GW1: 192.168.0.1
NIC2: 192.168.1.Y  / IF2: wlan0 / GW2: 192.168.1.1

Lets furthermore assume that each "router" runs a DHCP-serice and that X and Y are any number in the allowed subnet's DHCP range ( 192.168.0. and 192.168.1. respectively). I've found it best to either allocate a DHCP entry in each "router" or to let it be fixed close to either range-end of the sub-net. For one because normally your server would be headless and besides it saves you the trouble of physically change work-plae just to look up the IP when something isn't working as it should. Except for choosing different network numbers for each subnet and make sure they don't get in contact with each other (your DHCP:s will fail if they do), the actual numbers are not that important as you will soon see. The above is just some friendly advice prto help debugging should you be needing it doing so. Personally I prefer reserve a number in the DHCP, but not some really old routers won't allow you to do that and your only choice will be the second option of narrowing the subnet down and to allocate a fixed address outside of the DHCP range, but still within the subnet range.



There's yet besides routing still the problem about the dynamic DNS thingy and the wish to reach the server's NIC:s by name, not number. Most "routers" nowadays have a dynamic-DNS updater built in. But what if your favorite dyn-DNS provider isn't supported by your router? Then you will have to run the updating from a host in the inside. We're going to do that in a cron script at the host that's supposed to be always up: teh Seeer-Veeer. Note that many of the once free-as-in-free-beer dyn-DNS service providers (like one of the true originals "DynDNS" for example to many peoples dismay) has started to charge for their service, way too much considering the actual "service" if you ask me. Remember that our "server" belongs to the fine category of El-Cheapos...





Set up a free dyn-DNS account at any of the free providers - for example: http://freedns.afraid.org/ and create two host-entries (under the same account if you wish). This service-provider has the elegant solution considering updating DNS-entries. The only thing needed is to access a randomly generated and lightly encrypted URL, known only by you, once in a while or when changes of your IP:s occur. But you want to do that with the right NIC.

So we start by creating a specific route via the right IF connected to the subnet where the intended router is at. Here's a couple of scripts doing just that for you. Note that only the first one contain the logic, the two following are just some helper scripts that can be reused whence we get as far as to routing. Put the following somewhere root can use them (/root/bin perhaps?).

	#! /bin/bash
	# This script updates a dynamic-DNS record for a specific NIC of your host.
	# Script is made for freedns.afraid.org but should in principle work for
	# other providers as well that allow DNS updates by simply loading a
	# specific URL.
	#
	# Usage: dyndns_update.sh IF SHA1
	#
	# As this script is meant to be run from cron, it will be fairly silent,
	# except when something goes wrong in an unexpected way or when a change of
	# IP address actually occurred. (note, some of the fails are normal and not
	# unexpected). To see the output of the log, export the environment variable
	# LOGFILE to "--" (any other content of that variable will write to this
	# file-name).
	#
	# Example: (export LOGFILE="--"; dyndns_update.sh eth1 $MYHOST)
	IFNAME=${1}
	SHA1=${2}
	SHOST=${3-"freedns.afraid.org"}
	SHELL=$(basename $0)
	#Set export of this to "--" if you want output visible
	LOGFILE=${LOGFILE-"/tmp/update_dyndns.log"}
	source logger.sh
	source pipe_logic.sh
	if [ "X$IFNAME" == "X" ]; then
	logger "First argument (IFNAME) must be present. Exiting..."
	exit 1
	fi
	if [ "X$SHA1" == "X" ]; then
	logger "Second argument (SHA1) must be present. Exiting..."
	exit 1
	fi
	NIC_IPNR=$(ifconfig | \
	grep $IFNAME -A3 | \
	grep "inet addr" | \
	cut -f2 -d":" | \
	cut -f1 -d " ")
	LNET=$(echo $NIC_IPNR | cut -f1-3 -d".").0
	GW=$(echo $NIC_IPNR | cut -f1-3 -d".").1
	URL="http://${SHOST}/dynamic/update.php?${SHA1}"
	if [ "X$NIC_IPNR" == "X" ]; then
	logger "IF [$IFNAME] not up or doesn't exist. Exiting..."
	exit 1
	fi
	logger "$(basename $0): Updating dynamic DNS to interface: [$IFNAME]"
	logger " NIC: [$NIC_IPNR]"
	logger " NET: [$LNET]"
	logger " GW: [$GW]"
	logger " URL: [$URL]"
	logger
	# Try removing temporary route first, in case last invocation didn't complete.
	route -v del -host $SHOST >/dev/null 2>&1
	#set -e #Trigger e-mail if something goes wrong from here on & if run in cron
	logger "Setting temporary route for [$SHOST]"
	route -v -A inet add -host $SHOST gw $GW | logger
	#route -v -A inet add -host $SHOST gw $GW
	netstat -anr | logger
	#netstat -anr
	logger "Updating dyn-DNS [$URL]"
	RESPONSE=$(wget -O - $URL 2>/dev/null)
	logger "Removing temporary route [$SHOST]"
	route -v del -host $SHOST | logger
	logger
	logger "Response from service provider [$SHOST] for [$IFNAME]:"
	logger "$RESPONSE"
	set +e #Stop trigger error mails (grep returns fail is not found)
	if [ \
	"X$(echo $RESPONSE | grep 'has not changed')" == "X" -a \
	"X${LOGFILE}" != "X--" \
	]; then
	# Something else is responded. Make it visable in syslog (if used from
	# cron), might be an error (or at least interesting).
	echo "Response from service provider [$SHOST] for [$IFNAME]:"
	echo "$RESPONSE"
	fi

view raw dyndns_update.sh hosted with ❤ by GitHub

	#! /bin/bash
	# Helper functions to make nice log-output to the right destination
	#
	# Format is a ":" separated list with each column meaning the following:
	# 1) TS: Time since epoc in nano-seconds. Numerical format makes it ideal
	# or sorting. Recover to human readable date by executing:
	# date -d "${TS}"
	# 2) PPID: Parent ID. Cront, init or your shell.
	# 3) PID: PID of the executing shell
	#
	# This file is meant to be sourced
	if [ -z $LOGGER_SH ]; then
	LOGGER_SH=logger.sh
	source pipe_logic.sh
	LOGFILE=${LOGFILE-"--"}
	#Log input from pipe
	function logitp() {
	TS=$(date +%s.%N)
	read LINE
	if [ "X${LOGFILE}" == "X--" ]; then
	cat <<< "${TS}:${PPID}:${$}:${SHELL}:${LINE}"
	else
	cat <<< "${TS}:${PPID}:${$}:${SHELL}:${LINE}" >> ${LOGFILE}
	fi
	}
	#Log input from arguments
	function logita() {
	echo "${@}" | logitp
	}
	#Figure out from where to where to log
	function logger() {
	if input_is term; then
	logita "$@"
	else
	logitp
	fi
	}
	if [ "$LOGGER_SH" == $( basename $0 ) ]; then
	#Not sourced, do something with this.
	echo "Supposed to be sourced, but might just as well do some tests..."
	# The following tests can be done to cover all cases:
	# logger.sh "This is a test"
	# => STDOUT: "$X:This is a test"
	#
	# echo "This is a test" | logger.sh
	# => STDOUT: "$X:This is a test"
	#
	# (export LOGFILE="logfile"; logger.sh "This is a test")
	# => logfile: "$X:This is a test"
	#
	# (export LOGFILE="logfile"; echo "This is a test" | logger.sh)
	# => logfile: "$X:This is a test"
	#
	# Where $X above means: ${TS}:${PPID}:${$}:${SHELL}
	if [ "X${LOGFILE}" == "X--" ]; then
	echo
	fi
	logger "$@"
	if [ "X${LOGFILE}" != "X--" ]; then
	echo "Check log-file [${LOGFILE}]"
	fi
	fi
	fi

view raw logger.sh hosted with ❤ by GitHub

	#! /bin/bash
	# Helper functions to determine if in/out is a attached to TERM or PIPE
	#
	# TERM in this context means attached to a console (screen or keyboard
	# depending on context), i.e. potentially interactive.
	#
	# PIPE means anything else: Pipe, redirect to file, redirect to variable
	# e.t.a.
	#
	# This file is meant to be sourced
	#
	# Note: Using return-values are needed as testing for console / terminal
	# becomes impossible when capturing output, since capturing itself goes via a
	# pipe (i.e. resulting answer would always be a pipe regardless of what's
	# being tested)
	#
	# This however also might become cumbersome if/when script sourcing uses
	# 'set -e'. The test are designed with this in mind (if on fail should not
	# trigger exit), but it might require better error handling to avoid trigger
	# false positives.
	if [ -z $PIPE_LOGIC_SH ]; then
	PIPE_LOGIC_SH=pipe_logic.sh
	function whatis_output() {
	if [ -t 1 ]; then
	echo "TERM"
	else
	echo "PIPE"
	fi
	}
	function whatis_input() {
	if tty -s; then
	echo "TERM"
	else
	echo "PIPE"
	fi
	}
	function output_is() {
	if [ $# -ne 1 ]; then
	echo "output_is() needs exactly one argument" 2>&1
	exit 1
	fi
	ARG=$(tr '[:lower:]' '[:upper:]' <<< $1)
	case $ARG in
	'TERM')
	if [ -t 1 ]; then
	return 0
	fi
	;;
	'PIPE')
	if ! [ -t 1 ]; then
	return 0
	fi
	;;
	*)
	echo "output_is(): Unknown argument [$1]" 2>&1
	exit 1
	;;
	esac
	return 1
	}
	function input_is() {
	if [ $# -ne 1 ]; then
	echo "input_is() needs exactly one argument" 2>&1
	exit 1
	fi
	ARG=$(tr '[:lower:]' '[:upper:]' <<< $1)
	case $ARG in
	'TERM')
	tty -s && return 0
	;;
	'PIPE')
	tty -s || return 0
	;;
	*)
	echo "input_is(): Unknown argument [$1]" 2>&1
	exit 1
	;;
	esac
	return 1
	}
	if [ "$PIPE_LOGIC_SH" == $( basename $0 ) ]; then
	#Not sourced, do something with this.
	echo "Supposed to be sourced, but might just as well do some tests..."
	# The following tests can be done to cover all cases:
	# echo | pipe_logic.sh | cat --
	# => PIPE / PIPE
	#
	# pipe_logic.sh | cat --
	# => TERM / PIPE
	#
	# echo | pipe_logic.sh
	# => PIPE / TERM
	#
	# pipe_logic.sh
	# => TERM / TERM
	#
	echo
	whatis_output "$@"
	if output_is term; then
	echo "output is terminal"
	fi
	if output_is pipe; then
	echo "output is pipe"
	fi
	echo
	whatis_input "$@"
	if input_is term; then
	echo "input is terminal"
	fi
	if input_is pipe; then
	echo "input is pipe"
	fi
	fi
	fi

view raw pipe_logic.sh hosted with ❤ by GitHub

Export your SHA1:s that you've got from freedns to make life easier. For the sake of it, I'm using my own case as example. SHA1:s obfuscated naturally:

 export PI="aSecretString1="
 export KATO="aSecretString2="

Now you can test it it works:

 dyndns_update.sh eth0 $KATO
 dyndns_update.sh wlan1 $PI

The scripts should be fairly scilent, unless something goes wrong or when a IP address is changed. You can try repeate the above but swap interfaces. Try pinging the names using an externally connected computer (use a shell in your smart-phone for example), ICMP will probably not be replied until the next part with routing, but the IP-addresses should be updated.

Add the corresponding information in a crontab running each 5 minutes. Remember that cron lines must be unbroken and that crontab has only a very limited set of variables. I.e. you can't use shell-expansion and hence you need to put the complete SH1-strings in there:

 3,8,13,18,23,28,33,38,43,48,53,58 * * * * dyndns_update.sh eth0 aSecretString1=
 4,9,14,19,24,29,34,39,44,49,54,59 * * * * dyndns_update.sh wlan1 aSecretString2=
 
Note that freedns has more then your hearts desire when it comes to registered suitable domain names. There's is no actual need to consider any other, even if you already have one. It's hard to compete with free. Besides freedns is also very good: Any needed updates usually propagate world-wide within seconds.

Next update will be about the actual routing...