Deploying 464XLAT for IPv6-only clients on a small WISP network with Mikrotik routers

Motivation

The world is out of IPv4 addresses. While some can be bought and traded, ARIN is no longer issuing new assignments to organizations in the USA. My WISP, Black Mesa Wireless, came up against this problem last year when we asked ARIN for another block of IPv4 addresses and were told we were out of luck. While some ISPs NAT many customers through a few public IP addresses, there are many reasons not to do that and I refuse to. We assign each customer a public IP address. Now we will have to assign most of them IPv6 addresses only, and not IPv4.

It’s easy to get an allocation of IPv6 addresses, and there are so many that we should not run out for many times my lifetime, if ever. ARIN allocated us a /36 with very little difficulty or fanfare. The first thing we had to do was figure out how to split this up. After doing a bunch of reading about best practices, we came up with a sparse allocation scheme that should be OK should we grow to many times our current size. I will leave that for the reader to research, or maybe one day write a separate article.

All of that settled, we had to figure out how to provide good service to our customers even if they were running with only an IPv6 public address (or rather, subnet) on their WAN. T-Mobile has led the way on this, deploying 464XLAT on their network over the past few years. I have made use of this as a T-Mobile and then Project Fi customer, and it works just fine. While NAT64 is an option, it still breaks some applications and fails if IP literals (i.e. ip addresses written out rather than host names) are used in web sites or services. 464XLAT uses a scheme to NAT IPv6
traffic out IPv4 addresses and retains those mappings in a stateful way for the duration of a connection. This transparently works for pretty much everything.

Implementation

Deploying 464XLAT requires two special functions– client-side translation (CLAT) to translate client-side IPv4 traffic to encapsulated IPv6, and provider-side translation (PLAT), to translate from encapsulated IPv6 back to IPv4. At this time (December 2017) I am still not aware of any consumer routers that support CLAT out of the box. We would love it if our wireless CPE manufacturers (Ubiquiti and Mimosa) would support CLAT on their devices, or our preferred router manufacturer (Mikrotik) would support it on their devices, but none of them do yet. However, the OpenWRT and LEDE projects (which forked a while ago but are now merging back together) do support CLAT. Therefore, after some research, we found a few models of consumer routers that we could flash with LEDE, and deploy to customers. I will cover this in more detail shortly.

On the provider side, some routers such as Juniper routers do support PLAT, but we are already invested in Mikrotik, and they still do not. However, using the Jool project along with BIND9, it is possible to run PLAT on a Linux server with very minimal setup. We have a Proxmox VE HA cluster on which we run two PLAT nodes, which are set up to fail over using different OSPF costs, and additionally they will fail over to other machines in the cluster if one goes down. As these nodes are required for IPv6 customers to access the IPv4 internet, it is very important that they stay up. In the future we may consider moving to Juniper routers if Mikrotik doesn’t eventually address this need.

PLAT Implementation

Installing and configuring Jool and BIND9 is pretty easy. We are standardized on Ubuntu 16.04 LTS at the moment, so I will use that in my examples. Note that if you are doing this in a virtual machine (as we do), it needs to have full hardware virtualization, and not run in a container, as Jool is a kernel module.

Jool

Here are the instructions for installing the Jool kernel module. In this example we will use DKMS.

First, install dependencies:

root@plat-demo:~# apt -y install build-essential linux-headers-$(uname -r) dkms
... (output) ...
root@plat-demo:~# git clone https://github.com/NICMx/Jool.git
... (output) ...
root@plat-demo:~# dkms install Jool
... (output) ...
DKMS: install completed.

Now we must tell the machine to insert the jool module at boot with the correct settings. We are using the well-known prefix of

64:ff9b::/96

which is the default used by many CLAT devices on the IPv6 side. You will need to supply your own IPv4 subnet for translation. Here we use a documentation-reserved prefix. Open

/etc/rc.local

in your favorite editor and add the line:

/sbin/modprobe jool pool6=64:ff9b::/96 pool4=192.0.2.0/28

You will need to assign the pool4 addresses on the appropriate interface. We are doing this via OSPF so I will omit it for now. You could script it or add all the addresses on alias intefaces, we have it in our QUAGGA configuration. You will also need to make sure you are routing properly to the IPv6 prefix, which is beyond the scope of this writeup. Once all that is done, manually run the command you added to

/etc/rc.local

You should then be able to ping an IPv4 address through the PLAT device by pinging, for example,

64:ff9b::8.8.8.8

.

BIND9

This is even easier than Jool.

root@plat-demo:~# apt -y install bind9

Once that is done, use your favorite editor to open

/etc/bind/named.conf.options

and make sure it looks like this:

acl my_net {
        // Use your public prefixes here, these are documentation prefixes.
        2001:DB8::/32
        198.51.100.0/24;
        203.0.113.0/24;
        // You may want to include any RFC1918 subnets you use on your network
        10.0.0.0/8;
        192.168.0.0/16;
};

options {
        directory "/var/cache/bind";

        dnssec-validation auto;

        auth-nxdomain no;    # conform to RFC1035
        listen-on-v6 { any; };

        # This is the key. Note that you can write multiple of these if you need
        # more IPv6 prefixes.
        # "64:ff9b::/96" has to be the same as Jool's `pool6`.

        // use the well-known prefix
        dns64 64:ff9b::/96 {
                # Options per prefix (if you need them) here.
                # More info here: https://kb.isc.org/article/AA-01031
        };
        recursion yes;

        allow-recursion { my_net; };
};

Save it and restart with:

systemctl restart bind9

Assuming you have proper firewall and routing settings in place, you should now be able to get NAT64 by doing dns lookups to this server. For example, Slashdot still doesn’t have IPv6 deployed (shame on them). However, BIND9 will synthesize an IPv6 address for you:

root@plat-demo:~# dig AAAA @(dns-server-ip) slashdot.org

; <<>> DiG 9.10.3-P4-Ubuntu <<>> AAAA @(dns-server-ip) slashdot.org
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49918
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 4, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;slashdot.org.                  IN      AAAA

;; ANSWER SECTION:
slashdot.org.           292     IN      AAAA    64:ff9b::d822:b52d

;; AUTHORITY SECTION:
slashdot.org.           25241   IN      NS      ns2.dnsmadeeasy.com.
slashdot.org.           25241   IN      NS      ns1.dnsmadeeasy.com.
slashdot.org.           25241   IN      NS      ns3.dnsmadeeasy.com.
slashdot.org.           25241   IN      NS      ns4.dnsmadeeasy.com.

;; Query time: 33 msec
;; SERVER: #53()
;; WHEN: Sat Dec 23 09:52:52 MST 2017
;; MSG SIZE  rcvd: 156

As you can see, it has appended the well-known prefix. Now, we can ping this address from a host on the network with a route to the PLAT machine:

[brock@demo-client]-(~)-> ping6 64:ff9b::d822:b52d
PING 64:ff9b::d822:b52d(64:ff9b::d822:b52d) 56 data bytes
64 bytes from 64:ff9b::d822:b52d: icmp_seq=1 ttl=235 time=73.1 ms
64 bytes from 64:ff9b::d822:b52d: icmp_seq=2 ttl=235 time=72.9 ms
64 bytes from 64:ff9b::d822:b52d: icmp_seq=3 ttl=235 time=73.5 ms
64 bytes from 64:ff9b::d822:b52d: icmp_seq=4 ttl=235 time=73.4 ms

--- 64:ff9b::d822:b52d ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3004ms
rtt min/avg/max/mdev = 72.912/73.277/73.587/0.421 ms

That completes PLAT setup!

CLAT Implementation

This is a bit more tricky– we have to provide customers with a router that will do CLAT, so that their internal IPv4 devices will send out traffic via IPv6. OpenWRT/LEDE firmware will do this, but you have to find one of the many devices that will support these firmwares, make a firmware image, and then deploy it.

We are using some older linksys routers that are quite adequate for the plans most of our customers use, the EA3500, primarily. We use the LEDE image builder to make a custom image that has what we need in it. You can download that here:

LEDE Imagebuilder for Kirkwood.

Note that on this hardware you currently need to install the openwrt factory image first, then the LEDE sysupgrade image.

You can download that here:

OpenWRT Factory Image

So, flash the router from factory with the OpenWRT image. Then, on Linux extract the imagebuilder and change into that directory. Run the following command to build with 464xlat:

umask 022
make image PROFILE=linksys-audi PACKAGES="464xlat luci-ssl"

When it finishes you should have a file:

bin/targets/kirkwood/generic/lede-17.01.4-kirkwood-linksys-audi-squashfs-sysupgrade.tar

You need to upgrade the router with that:

scp bin/targets/kirkwood/generic/lede-17.01.4-kirkwood-linksys-audi-squashfs-sysupgrade.tar root@192.168.1.1:/tmp/sysupgrade.tar
ssh root@192.168.1.1 sysupgrade -n /tmp/sysupgrade.tar

When that completes, you should be able to access the router running LEDE at https://192.168.1.1.

You need to log in (default is root, no password), go to network settings, and change the protocol for the IPv4 WAN to 464XLAT. In the IPv6 wan settings, change your DNS server to point to the IPv6 address of your PLAT device.

You should then be able to reboot the router, and 464XLAT should be working. Once you have a template router config just how you want it, you can extract the config files and give them as input to the image builder. This will give you a ready-to-go router firmware. We have done this plus additional scripting to set a wifi SSID and password, and print labels automatically.

Conclusion

I have seen very good throughput on this setup, basically limited by the client (CLAT) router’s wireless connection or CPU, the PLAT is pretty performant on VMs on some old Xeon servers. However, you need to make sure that the servers have ethernet offloading enabled as per this documentation. For me it made the difference between 0.5Mbps and 150Mbps throughput.

I highly recommend you test your setup with this IPv6 tester site.

I am happy to answer questions about this and also to take suggestions. Happy routing!

Booting from a Samba share in old Supermicro IPMI

I picked up some older Supermicro machines from eBay (GREAT way to get some real server hardware, btw), but they have a VERY old IPMI firmware and no way to upgrade to a more modern version.

They can’t do standard virtual media — you have to specify a SMB share with an ISO on it to mount. Once you do that, you have to make it bootable in the BIOS and move it up the boot order.

I wasted about an hour trying to figure out why I couldn’t enable the device “PepperC Virtual Disc” in the boot order — the answer is, you have to use the ‘x’ key to remove unused devices from the 8 available boot slots, then you can select the PepperC device and use the ‘x’ key to enable it, and then you can move it up/down the boot order with the +/- keys.

Hopefully this saves someone else some grief, I was losing my mind trying to figure out the problem.

Resizing Cryptswap in Ubuntu 16.04

I recently migrated my system to NVMe drives (great decision, by the way), and part of my reason for doing so was much faster swap, for some outrageously memory-hungry finite element mesh generation stuff.

I also wanted to enlarge the swap. I use LVM on top of software RAID, and encrypted home directories on top of that, which means the Ubuntu automatically set up encrypted swap for me. I had a non-encrypted swap volume at:

/dev/vg0/swap

I didn’t have much luck finding information about how the cryptswap is set up by the installer, which is why I’m writing this. It turns out that the crypt swap is configured in /etc/crypttab, where I have a line like this:

cryptswap1 /dev/vg0/swap /dev/urandom swap,offset=1024,cipher=aes-xts-plain64

What this means is that the system will create a crypt device called /dev/mapper/cryptswap1 at boot using a random seed, on top of /dev/vg0/swap. It will then run mkswap and swapon on the encrypted device.

This latter part is specified in /etc/fstab like:

/dev/mapper/cryptswap1 none swap sw 0 0

So, if your base unencrypted swap partition is an LVM logical volume, all you have to do is use lvextend to make it larger and (the easiest way) reboot. On reboot the larger device will automatically be used in its entirety.

Exim4 system-wide conditional email forwarding

This is a guest post written by Steve Goertz after he solved this particular issue.

Use:

  • Filtering and forwarding all emails received by exim4
  • Filtering by specific email components (sender information, subject, etc.)
  • Forwarding all filtered emails to a specific email address or email addresses

Assuming a working instance of exim4:

Create a filter file for exim4 using the appropriate filters and syntax as found in the exim4 filter documentation here: http://www.exim.org/exim-html-current/doc/html/spec_html/filter_ch-exim_filter_files.html

You may want to create a directory for exim4 filters and place the file in that directory, like:

/etc/exim4/conf.d/filters/filter_name

For our particular use, a conditional section and  deliver command were the only necessary components. A filter file will look something like this:

#Exim filter <<== do not edit or remove this line!

##Filter description, so you remember what you were trying to do.

if
  $sender_address is “sender@address.example” and 
  $header_subject does not contain “foo” and 
  $message_body contains “bar”
then
  deliver “recipient@other.example”
endif

Filters will then be placed as the first router in the exim4 router config ( found at /etc/exim4/conf.d/router/router_name ). Depending on your configuration the router name may vary, or you may need to add one. The filter should be formatted as follows:

filter_name:
  driver = redirect
  allow_filter
  file = /path/to/filter (in this case /etc/exim4/conf.d/filters/filter_name)
  user = exim4_user

It is essential that the  user variable match the user that owns exim4.  If not, the filter will not function and email traffic will not pass through the first filter to the remaining filters and all regular email processing will stop. You can probably figure this out by checking the init script for exim4 or using the command: 

ps aux | grep exim4

to see whom exim4 is running as.

After the filter file has been generated and correctly referenced in the router config, rebuild the exim4 config using: 

sudo update-exim4.conf.template -r

And restart the service

sudo service exim4 restart

Test the email server to ensure that it is working as intended.

This configuration will prevent emails from arriving at their intended destination, which was our need. Also, if the incoming email does not meet all of the requirements above, it will pass through to the next router in the router configuration file.

Dealing With Dogs While Cycling

I’ve been cycling a lot this season, but off-leash dogs are the rule here rather than the exception. They like to chase me on the bike, and even the ones that don’t seem likely to directly harm me are liable to cause an accident.

To combat this, I use pepper spray. To keep the pepper spray handy, I use this simple device I found on Amazon.com. I was skeptical when I first saw it that it would be stable enough, but actually it has worked quite well. I use Fox OC spray with it, and it has never failed to stop an aggressive dog.

The spray is a bit pricey, but if you ride the same route or routes regularly the dogs eventually start to learn and you don’t need it that much.

I have a dog and I don’t like pepper spraying dogs, but irresponsible owners have created dangerous conditions for me and other cyclists and pedestrians, so it has come to this.

Ride safely.