Category Archives: Networking

Deploying 464XLAT for IPv6-only clients on a small WISP network with Mikrotik routers

Motivation

The world is out of IPv4 addresses. While some can be bought and traded, ARIN is no longer issuing new assignments to organizations in the USA. My WISP, Black Mesa Wireless, came up against this problem last year when we asked ARIN for another block of IPv4 addresses and were told we were out of luck. While some ISPs NAT many customers through a few public IP addresses, there are many reasons not to do that and I refuse to. We assign each customer a public IP address. Now we will have to assign most of them IPv6 addresses only, and not IPv4.

It’s easy to get an allocation of IPv6 addresses, and there are so many that we should not run out for many times my lifetime, if ever. ARIN allocated us a /36 with very little difficulty or fanfare. The first thing we had to do was figure out how to split this up. After doing a bunch of reading about best practices, we came up with a sparse allocation scheme that should be OK should we grow to many times our current size. I will leave that for the reader to research, or maybe one day write a separate article.

All of that settled, we had to figure out how to provide good service to our customers even if they were running with only an IPv6 public address (or rather, subnet) on their WAN. T-Mobile has led the way on this, deploying 464XLAT on their network over the past few years. I have made use of this as a T-Mobile and then Project Fi customer, and it works just fine. While NAT64 is an option, it still breaks some applications and fails if IP literals (i.e. ip addresses written out rather than host names) are used in web sites or services. 464XLAT uses a scheme to NAT IPv6
traffic out IPv4 addresses and retains those mappings in a stateful way for the duration of a connection. This transparently works for pretty much everything.

Implementation

Deploying 464XLAT requires two special functions– client-side translation (CLAT) to translate client-side IPv4 traffic to encapsulated IPv6, and provider-side translation (PLAT), to translate from encapsulated IPv6 back to IPv4. At this time (December 2017) I am still not aware of any consumer routers that support CLAT out of the box. We would love it if our wireless CPE manufacturers (Ubiquiti and Mimosa) would support CLAT on their devices, or our preferred router manufacturer (Mikrotik) would support it on their devices, but none of them do yet. However, the OpenWRT and LEDE projects (which forked a while ago but are now merging back together) do support CLAT. Therefore, after some research, we found a few models of consumer routers that we could flash with LEDE, and deploy to customers. I will cover this in more detail shortly.

On the provider side, some routers such as Juniper routers do support PLAT, but we are already invested in Mikrotik, and they still do not. However, using the Jool project along with BIND9, it is possible to run PLAT on a Linux server with very minimal setup. We have a Proxmox VE HA cluster on which we run two PLAT nodes, which are set up to fail over using different OSPF costs, and additionally they will fail over to other machines in the cluster if one goes down. As these nodes are required for IPv6 customers to access the IPv4 internet, it is very important that they stay up. In the future we may consider moving to Juniper routers if Mikrotik doesn’t eventually address this need.

PLAT Implementation

Installing and configuring Jool and BIND9 is pretty easy. We are standardized on Ubuntu 16.04 LTS at the moment, so I will use that in my examples. Note that if you are doing this in a virtual machine (as we do), it needs to have full hardware virtualization, and not run in a container, as Jool is a kernel module.

Jool

Here are the instructions for installing the Jool kernel module. In this example we will use DKMS.

First, install dependencies:

root@plat-demo:~# apt -y install build-essential linux-headers-$(uname -r) dkms
... (output) ...
root@plat-demo:~# git clone https://github.com/NICMx/Jool.git
... (output) ...
root@plat-demo:~# dkms install Jool
... (output) ...
DKMS: install completed.

Now we must tell the machine to insert the jool module at boot with the correct settings. We are using the well-known prefix of

64:ff9b::/96

which is the default used by many CLAT devices on the IPv6 side. You will need to supply your own IPv4 subnet for translation. Here we use a documentation-reserved prefix. Open

/etc/rc.local

in your favorite editor and add the line:

/sbin/modprobe jool pool6=64:ff9b::/96 pool4=192.0.2.0/28

You will need to assign the pool4 addresses on the appropriate interface. We are doing this via OSPF so I will omit it for now. You could script it or add all the addresses on alias intefaces, we have it in our QUAGGA configuration. You will also need to make sure you are routing properly to the IPv6 prefix, which is beyond the scope of this writeup. Once all that is done, manually run the command you added to

/etc/rc.local

You should then be able to ping an IPv4 address through the PLAT device by pinging, for example,

64:ff9b::8.8.8.8

.

BIND9

This is even easier than Jool.

root@plat-demo:~# apt -y install bind9

Once that is done, use your favorite editor to open

/etc/bind/named.conf.options

and make sure it looks like this:

acl my_net {
        // Use your public prefixes here, these are documentation prefixes.
        2001:DB8::/32
        198.51.100.0/24;
        203.0.113.0/24;
        // You may want to include any RFC1918 subnets you use on your network
        10.0.0.0/8;
        192.168.0.0/16;
};

options {
        directory "/var/cache/bind";

        dnssec-validation auto;

        auth-nxdomain no;    # conform to RFC1035
        listen-on-v6 { any; };

        # This is the key. Note that you can write multiple of these if you need
        # more IPv6 prefixes.
        # "64:ff9b::/96" has to be the same as Jool's `pool6`.

        // use the well-known prefix
        dns64 64:ff9b::/96 {
                # Options per prefix (if you need them) here.
                # More info here: https://kb.isc.org/article/AA-01031
        };
        recursion yes;

        allow-recursion { my_net; };
};

Save it and restart with:

systemctl restart bind9

Assuming you have proper firewall and routing settings in place, you should now be able to get NAT64 by doing dns lookups to this server. For example, Slashdot still doesn’t have IPv6 deployed (shame on them). However, BIND9 will synthesize an IPv6 address for you:

root@plat-demo:~# dig AAAA @(dns-server-ip) slashdot.org

; <<>> DiG 9.10.3-P4-Ubuntu <<>> AAAA @(dns-server-ip) slashdot.org
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49918
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 4, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;slashdot.org.                  IN      AAAA

;; ANSWER SECTION:
slashdot.org.           292     IN      AAAA    64:ff9b::d822:b52d

;; AUTHORITY SECTION:
slashdot.org.           25241   IN      NS      ns2.dnsmadeeasy.com.
slashdot.org.           25241   IN      NS      ns1.dnsmadeeasy.com.
slashdot.org.           25241   IN      NS      ns3.dnsmadeeasy.com.
slashdot.org.           25241   IN      NS      ns4.dnsmadeeasy.com.

;; Query time: 33 msec
;; SERVER: #53()
;; WHEN: Sat Dec 23 09:52:52 MST 2017
;; MSG SIZE  rcvd: 156

As you can see, it has appended the well-known prefix. Now, we can ping this address from a host on the network with a route to the PLAT machine:

[brock@demo-client]-(~)-> ping6 64:ff9b::d822:b52d
PING 64:ff9b::d822:b52d(64:ff9b::d822:b52d) 56 data bytes
64 bytes from 64:ff9b::d822:b52d: icmp_seq=1 ttl=235 time=73.1 ms
64 bytes from 64:ff9b::d822:b52d: icmp_seq=2 ttl=235 time=72.9 ms
64 bytes from 64:ff9b::d822:b52d: icmp_seq=3 ttl=235 time=73.5 ms
64 bytes from 64:ff9b::d822:b52d: icmp_seq=4 ttl=235 time=73.4 ms

--- 64:ff9b::d822:b52d ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3004ms
rtt min/avg/max/mdev = 72.912/73.277/73.587/0.421 ms

That completes PLAT setup!

CLAT Implementation

This is a bit more tricky– we have to provide customers with a router that will do CLAT, so that their internal IPv4 devices will send out traffic via IPv6. OpenWRT/LEDE firmware will do this, but you have to find one of the many devices that will support these firmwares, make a firmware image, and then deploy it.

We are using some older linksys routers that are quite adequate for the plans most of our customers use, the EA3500, primarily. We use the LEDE image builder to make a custom image that has what we need in it. You can download that here:

LEDE Imagebuilder for Kirkwood.

Note that on this hardware you currently need to install the openwrt factory image first, then the LEDE sysupgrade image.

You can download that here:

OpenWRT Factory Image

So, flash the router from factory with the OpenWRT image. Then, on Linux extract the imagebuilder and change into that directory. Run the following command to build with 464xlat:

umask 022
make image PROFILE=linksys-audi PACKAGES="464xlat luci-ssl"

When it finishes you should have a file:

bin/targets/kirkwood/generic/lede-17.01.4-kirkwood-linksys-audi-squashfs-sysupgrade.tar

You need to upgrade the router with that:

scp bin/targets/kirkwood/generic/lede-17.01.4-kirkwood-linksys-audi-squashfs-sysupgrade.tar root@192.168.1.1:/tmp/sysupgrade.tar
ssh root@192.168.1.1 sysupgrade -n /tmp/sysupgrade.tar

When that completes, you should be able to access the router running LEDE at https://192.168.1.1.

You need to log in (default is root, no password), go to network settings, and change the protocol for the IPv4 WAN to 464XLAT. In the IPv6 wan settings, change your DNS server to point to the IPv6 address of your PLAT device.

You should then be able to reboot the router, and 464XLAT should be working. Once you have a template router config just how you want it, you can extract the config files and give them as input to the image builder. This will give you a ready-to-go router firmware. We have done this plus additional scripting to set a wifi SSID and password, and print labels automatically.

Conclusion

I have seen very good throughput on this setup, basically limited by the client (CLAT) router’s wireless connection or CPU, the PLAT is pretty performant on VMs on some old Xeon servers. However, you need to make sure that the servers have ethernet offloading enabled as per this documentation. For me it made the difference between 0.5Mbps and 150Mbps throughput.

I highly recommend you test your setup with this IPv6 tester site.

I am happy to answer questions about this and also to take suggestions. Happy routing!

Disabling Mikrotik Hotspot DNS Proxying for Authenticated Users

My wireless ISP (WISP) uses the Mikrotik hotspot feature with RADIUS on the back end to authenticate our users. This implements a captive portal that redirects all DNS requests so that the user is taken to a login page if they’re not logged in. Once they log in once, the system associates their radio with their account, and they don’t have to log in anymore under normal circumstances.

However, once logged in, users still have all their DNS requests proxied through the routers. A lot of users want to use their own DNS (like OpenDNS or Google Public DNS), and that’s fine with me, but a user ran the namebench utility and found that their DNS was being forcibly proxied.

It took some hunting, but I finally found this post on the Mikrotik forums which details how to get around this. Basically:

  • The hotspot adds dynamic DNS redirect rules. If you go to /ip firewall nat and just print, these rules don’t show up. If you do print dynmic they do. The relevant lines are:

    2 D chain=hotspot action=redirect to-ports=64872 protocol=udp dst-port=53 log=no log-prefix=""
    3 D chain=hotspot action=redirect to-ports=64872 protocol=tcp dst-port=53 log=no log-prefix=""
  • We still want non-logged-in-users to have their DNS redirected, so we need to add something here that will enable authenticated hotspot users through. The magic incantation here (because it’s entries 2 and 3) is set 2,3 hotspot=!auth, which results in the following:

    2 D chain=hotspot action=redirect to-ports=64872 protocol=udp hotspot=!auth dst-port=53 log=no log-prefix=""
    3 D chain=hotspot action=redirect to-ports=64872 protocol=tcp hotspot=!auth dst-port=53 log=no log-prefix=""

And now namebench works as expected.

Preventing BGP Advertised Route Flapping in Mikrotik RouterOS

I am not an expert on this, I just wanted to document a problem I had and a solution I found today, in a concise way. Comments correcting me or suggesting better ways are very welcome.

I have a network running OSPF internally, and advertising routes to the upstream ISP over BGP at two separate edge routers (multi-homed, single ISP). We discovered last night that internally bringing down any of the subnets we advertise results in the dropping of those routes from the tables of the edge routers (as expected). This drops the advertisements. What we did NOT expect was that flap damping from upstream of us then null-routes that subnet for up to a few hours.

So, how do we retain our adaptive internal routing (OSPF) while avoiding route flap? I was a bit stumped about this, but I found a more complex article that describes a multi-homed BGP setup. A key part of that setup was a little trick to avoid this problem. Nameley, set up a static, black hole route for the subnet on the edge router, with maximum distance. This way, even if the OSPF route disappears, the router still “knows” a route to the subnet and won’t drop the advertisement.

For example, if you want to advertise the subnet 1.1.1.0/24, you should add a static route like


/ip route add dst-address=1.1.1.0/24 type=blackhole distance=254 comment="prevent flapping of the route over BGP"

I’ve tested it and it seems to work as expected. The route is not active as long as the OSPF route is in the routing table. If it disappears, the black hole route becomes active.

Comments? Suggestions?