- Software engineer by day
- I care too much about RFCs
- In my spare time, I work on my hackerspace’s network.
I’m an unabashed text console enthusiast. I like to manage my equipment via command lines and SSH; it means I have a uniform interface when dealing with machines all over the place, and I can use it from pretty much anywhere-even my phone.
But command line interfaces just aren’t common on most consumer equipment sold today, in fact, they basically don’t exist. That’s a huge bummer for me, and caused me to start looking for other options.
Be it ARP table overflows, memory leaks, overheating, or poor antenna performance, I’ve just not had good luck with consumer routers.
Weak configuration options-often limited to port forwarding or sticking a machine in a DMZ-and minimal firewall options beyond simply providing NAT.
Clearly, there are lots of opportunities for the software we run on our routers to face security vulnerabilities. However, when we’re picking out our stack ourselves, we have the option to limit our risk by choosing a minimal set of tools-and updating quickly, or as quickly as our vendor can (^^,)
There are insecure default settings on so many routers, and they often support management protocols which are also inherently insecure.
We don’t use SHA-1 any more, but I can guarantee there’s a lot of equipment out there that still uses it, or even requires it
Netgear’s routerlogin.net private cert was found to have been leaked while I was writing this talk and that’s far from the only security issue we’ve seen.
Turns out, Cisco’s NX-OS’ management protocol has a stack overflow vulnerability, which, combined with the OS’ weak ASLR, lack of stack canaries, and the privileged context of the CDP parser means that a successful attack is devastating.
Above a certain complexity, systems encounter emergent behaviors-unexpected and undesigned-for modes of operation that can manifest as failures and errors in ways that you may not understand. Actually building some familiarity with how networks work at a low level can help you comprehend what’s happening when your systems fail.
When building out a network, you’re taking a bunch of complex systems, and by integrating all of them together, creating an even more complex meta-system. Debugging this system without context is difficult, to say the least, and you’ll have more of a fighting chance with a deeper understanding of how a router actually operates-and what goes into network design.
While inexpensive, consumer routers are often extremely limited; if you look at the OpenWRT wiki, there’s an entire warning about 4/32 devices, which have only 4MB of RAM and 32MB of flash storage; these devices can barely fit their firmware when they’re shipped out, let alone run extra tools! By using commodity desktop hardware, we can swap out components, such as adding higher-bandwidth NICs (10GbE hardware is pretty inexpensive now) or other specialized additions.
Instead of trying to figure out what strange configuration is present, or divine what IP address has gotten assigned, all of your familiar command line tools, like src_sh{ip addr} or src_sh{ip route} are there, close at hand.
A full operating system on your router offers you lots of opportunities to try out different tools or technologies
I host my Unifi Controller on my router as a Docker container, which avoids using a separate physical device
I’m starting with hardware selection, but really both hardware and software have to be considered in tandem, since one will inform the other. This may evolve as you build your system, too! Maybe you decide that one approach isn’t working for you after you’ve done some basic setup. That’s okay. Just remember to take notes and log your progress.
Obviously, what capabilities you want to focus on will define how you select hardware to some extent; the short form is that as you want to have more capabilities captured in your router, you’ll need more resources in general; there are some areas where you can optimize for a specific function.
4GB of RAM is more than enough for a basic firewall; if you want to add more services, however, you’ll want more. Once you start considering virtualization, 16 or 32 might be needed or useful.
Any recent CPU will be powerful enough–here your concerns are cost and heat, mainly. Low-power embedded CPUs are great for low- or no-noise operation. I do recommend multi-core, and if you’ll be doing more crypto on-board (TLS termination, VPN endpoint) your CPU should have SIMD instructions.
120GB SSDs are $20 or less on Newegg; a lot of boards designed for a usecase like this are also built to be booted off of flash drives or SD cards.
If you’re interested in doing more edge computing on your router-logging, traffic shaping, running an HTTP server for a personal website (why not?) you might want additional storage. This doesn’t need to be an SSD, spinning platters will do fine.
This is the heart of choosing what you’ll use as a router; at a minimum, you need two interfaces: one external, one internal. If you’ve got a separate switch you’re using for your internal network, that may be all you need; if you want other machines directly connected to your router, however, you’ll need more.
Assuming your motherboard has at least two NICs, you can forego all of the complexity and just use those devices.
If you want to do a little more than the motherboard can support directly, either in capacity or speed (or both!) an add-in card might be a good choice. With a PCIe card, you can get SFP+ ports and full 10GbE speeds.
One issue I’ve encountered is, if you install an expansion card NIC, some motherboard firmwares will disable the onboard NIC by default. You may want that! In my case, I really didn’t. So, if your NIC suddenly doesn’t work when you install an additional one, you should look for this in the BIOS/EFI.
A key point with NIC selection is driver support; depending on your OS choice, you may be more or less limited here! I recommend picking your NIC in conjunction with your OS or after-so you can confirm what support you’ll have.
You may encounter a situation where you want the core components of your network to be on one subnet, and to have client devices on another; if you’ve got a switch that can allocate ports to VLANs, then you can use one larger, more capable switch to handle multiple subnets in parallel-but that does require that your drivers support multiple VLANs on the same connection, otherwise you’ll need a separate connection from the router for each subnet.
When choosing a NIC for your build, you should keep your throughput requirements in mind; if you host a media server inside your network, you may want to have higher throughput on your LAN connection than your WAN. Just try to keep your upstream and downstream in excess of what you’ll need for your connection.
Often slightly more expensive, but featureful systems. When looking at prebuilt equipment that runs pfSense or other similar firewall-oriented operating systems, you’ll see purpose-built, but closer to commodity hardware than if buying from a vendor who builds a custom router OS like Ubiquiti or MikroTik.
Mostly a NAS vendor, they have some switches and network equipment, including what they call a smart edge switch, which looks a lot like what we’re describing here.
Ship with a pfSense license, this is great if you want an integrated solution.
Any relatively recent SFF desktop will do, as long as you can install the parts you want to use in it!
Useful boards for custom installations; if you’re very space constrained or have limited need for expandability, this might be a good choice.
Order parts from Supermicro, Tyan, other vendors, and build yourself a box! This is a more expensive route, but can be very rewarding. If you’re building with Supermicro, do be aware that working in their cases can be a little like kitbashing, and might involve some initially questionable approaches. <Photos of my router build.>
There’s plenty of 1U servers with fantastic loadouts available for relatively low prices. A manufacturer I was recently introduced to is HYVE, who have some really cute boxes; they’ve got a focus on warm-temp datacenters, and they’re blue! <Photos of Duwamish>
10 Gigabit hardware is getting really inexpensive! If you’ve got 8 PCIe gen 2 lanes available, you can have a dual SFP+ card. If you’ve got more, you can do oh so much more. I’ve found multi-10-Gig/multi-gigabit cards, which can be handy for virtualization down the line.
If you get a wireless card that can run in AP mode, you can use your router as your access point-and even trade out parts to upgrade to new standards as they become available.
You can use Tensorflow in hardware now, relatively cheaply. They’re $35 at Mouser. What would I do with these? I have no idea yet! But if you wanted to use some tool like greylog to capture logs, you could run an ML model analysis on it, and do so more efficiently than just on your CPU.
While TPM support is limited generally, you can use it as a source of HWRNG, and it can sometimes be used to offload cryptographic computations.
This is an area where there’s lots of hot-headed opinions, and lots of options without significant distinction. Decide what matters to you from the axes I’ll present, and remember that this choice is not permanent. You can try something, decide it doesn’t work, and change it out! That’s okay!
Some options are more configurable through webpages and graphical environments, but are less configurable through text interfaces; careful configuration of interfaces and potentially a VPN may be needed to remotely manage some of these stacks.
For a more complete list to explore, check out the list on Wikipedia.
- firewall-cmd (on any distro that supports it)
- shorewall
- VyOS - both free and paid, this one’s complicated. Derivative of Brocade Networks’ OS.
- raw nftables
The mon0wall derivatives (pfSense, OPNsense) are all FreeBSD derivatives; in other cases, you may prefer running a Linux kernel-for familiarity’s sake, or because of hardware support.
Paid support options exist for many firewall-oriented distros and derivatives; generally speaking, there’s also community support available, but you may or may not find what you need in forums, especially when dealing with unusual hardware or network configurations.
For our demo here, I’m going to use Fedora, and I’m going to configure it with very low-level tools, to highlight fine details that graphical environments might gloss over.
This part is probably the most ordinary aspect of this build. I’m going to use Fedora Server, and I’m going to leave any graphical components out, while making sure to set up SSH from the beginning.
Before we start configuring components, it’s a good idea to have a few tools quick at hand.
I’m not normally a nano user, but it’s got a very straightforward interface, and all I really care about here is being able to edit files on the machine. I don’t care about having the most ergonomic environment for my daily driver needs.
Again, you may prefer other tools, but this is one I’m generally comfortable with, and can usually get around in easily. If you’re going to remote into your machine a lot, you may want to configure it to be comfortable for you–but that is not the goal of this exercise.
NetworkManager, systemd-networkd are both viable options; here I’m using systemd-networkd since it’s what I’ve been using at home.
Configuration files will be applied in alphanumerical order, and later configurations will override earlier ones; it’s reasonable to have baselines defined in low-numbered scripts, and more custom configurations in high-numbered scripts. All network setups do this.
Useful for your gateway address on internal networks, or if you have a static IP allocation from your ISP.
Handy when your upstream address is provided by DHCP; less handy if you’re trying to have routes declared statically.
One useful configuration is to deliberately block IP forwarding on an interface, to restrict the potential for devices (IoT in particular) to leak information you might not want visible on the broader network. In this case, you’d still run DHCP on the interface, but you would not present a route to that network, and you would deliberately block forwarding for that NIC.
Normally, to collect information from IoT devices that aren’t routing to the broader network, you need a dual-homed machine to collect logs or video streams, for instance; when you’re running a full Linux install on your router, it is a dual-homed machine, and can provide the access you need. If you want to do more to limit your firewall’s exposure, you might consider a virtual machine; we’ll talk about that in <a href=”* Virtualized firewall”>* Virtualized firewall.
From ip-sysctl.txt in the Linux kernel documentation,
Possible values are: 0 Do not accept Router Advertisements. 1 Accept Router Advertisements if forwarding is disabled. 2 Overrule forwarding behaviour. Accept Router Advertisements even if forwarding is enabled.
Note that this means we’ll want to set accept_ra to 2 specifically on our
WAN interface for IPv6 support.
Not really a “versus” here, but configuring nftables directly instead of using a
frontend is a viable path, and if you have custom logic for null-routing
specific IPs, you might want to have your own custom tooling writing your .nft
files and applying them. For this talk, we’ll use firewalld. It’s close to the
same syntax, but has some nice-to-have details like port numbers having service
names.
It’s hard to stress this enough. Blocking ICMPv6 is a great way to cause arbitrary, difficult-to-diagnose slowdowns if you have IPv6 support enabled. This isn’t going to improve your security posture, SLAAC with security extensions will handle that.
Part of the fun of running your own firewall is being able to host services on it that would be limited or impossible on consumer hardware, and might be difficult to configure on commercial hardware. Some of these can be quality of life improvements (and may even be relatively straightforward to configure, as some of them would need to exist inside your network as well.)
Being able to remotely connect to your firewall and check on the state of your local network is useful–especially if you’re away from home, and want to make sure that a service is working. However, you’ll want to take some precautions.
Add at least one public key to your user’s authorized_keys, validate that you can connect with that key, then disable password authentication. Best practices for SSH keys include using a different one on each device that needs to connect, and potentially using a different key for each service that you connect to with them. I recommend using Ed25519 as the cipher; it’s short and powerful.
Most linux installs do this by default now, but it’s always good to check.
This is optional, but reduces the number of random drive-by attempts that you’ll see. I’m a fan of port 24. This can be done just with a firewall rule, even!
In cases where you look at your journalctl logs and see lots of
attempted connections, you might consider adding fail2ban, and setting
some sufficiently high threshold–10 attempts, maybe. Remember when doing
this that it can bite you, too.
Here we get into nice-to-have functionality, beyond the absolutely necessary. If you want direct access to hosts inside your network from a remote location, you’ll want to use a VPN.
There’s several approaches to this, largely depending on how willing you are to DIY parts of the solution. Helpfully, Torvalds merged Wireguard into his branch for 5.6, so we should be able to rely on that being in production kernels soon, and you won’t need to install separate modules to get it working. There’s a few frontends for Wireguard under development now, but if you don’t want to use those there’s OpenVPN, which is significantly more mature.
Instead of memorizing your IP address, why not use a domain name? Services like no-ip can offer you a way to get back to your own machines relatively easily, but if you’re willing to pay a little for a domain, there are plenty that are very cheap–and then you just need to set up a script to auto-update your DNS record regularly. Personally, I like Hurricane Electric for my DNS hosting needs, but there’s lots of options out there. Pick one you like, and have fun!
Once you’ve got a domain name, clearly you need a website, too. Consider getting a Let’s Encrypt cert, while you’re at it! Note that this is a fantastic service to run in a container, or on another machine with port forwarding. If you take the port forwarding route, just make sure to enable forwarding for as many ports as you’ll end up using–often both 80 and 443 are sufficient.
All the tools you’d normally use to diagnose network connection issues
apply-layer 3 and above tools to determine inter-network connectivity like
traceroute and ping, but now we’re also looking to understand firewall
issues like if DNS requests are being blocked, or even some layer 2 issues, like
whether or not our router’s seeing ARP requests. This is where we start caring
about the difference between received packets and processed packets, and where
tools that interact directly with the NIC come in.
Some tools outside of the normal desktop network toolkit, then, that we’ll want:
- tcpdump
- Collect logs directly from the firewall, on a given interface
- Wireshark
- Collect and examine pcaps, packet captures. Lets you investigate failures at your leisure, and sift through captures
If these tools don’t work on your router, you have a NIC that doesn’t support promiscuous mode (can pass through all packets, not just those for its MAC address)-most should, but it’s possible some won’t. In that situation, you’ll need to find a different NIC to use if you want to be able to debug with these tools.
Sometimes cables go bad. Sometimes NICs are bad. When a NIC’s bad, one often has few options for recourse, but generally only a cable is bad. This might be as trivial as one of the conductors failing, or it may be as strange as a cable that’s fine unless a specific frame goes through it. Bin the cable. Life’s too short for bad cables.
There’s a great early-internet meme of the 500 mile email. Configuration errors can happen to the best of us, and they will manifest in myriad ways. Strange traffic patterns, machines that can’t communicate with each other, things which work outside the network not working inside… you name it!
This is a fun one. Sometimes your logical error is not in software-it’s in how you think about the situation. An assumption fault can come up all sorts of ways-routes that don’t work, network segments not being shared-but when they’re encountered, diagramming them can help.
Failures happen, unfortunately. Power outages strike, hard drives crash, stray voltage kills your SSD’s controller…
… and then you need to get into your router.
Backing up key information matters a lot; this can be in the form of a full-system backup, or as a build and design log. I’m a big fan of logging my builds in markdown, or Org mode, or whatever journalling mechanism is most convenient-then storing it in Git and keeping it off-site, on a service like Gitlab, Bitbucket, or Gitlab.
Maybe your only laptop with an SSH key for your router on it had a hard drive failure, or you had a device stolen. In this case, if you can’t get into your router because you’ve properly locked it down, you’re in a pickle.
Assuming you have physical access to the server, you can connect a serial console, and, assuming you’ve enabled it ahead of time for the machine, you can sign in directly through that interface. If you have IPMI that supports integrated KVM, such as a Supermicro machine or a suitably-licensed Dell or HP board, you can do the same direct login.
Whether the private key is stashed in a password manager, on a USB stick, or encoded in a QR code, having an extra public key provisioned on your router, with the private key on a separate physical device allows you to maintain access to the router, from any device with an SSH client, without pre-provisioning it.
With this mechanism, especially if the key is stored in some sort of off-site backup, you’ll want to regenerate your break-glass key after resolving your lack of access, and stash the new one in its place; otherwise, your extra credential is now just another credential that you can use from that or any machine.
Being the gateway device that has the most internet-facing surface area, your firewall is a prime target for attack. By running the firewall as a virtual machine instead of as the host, you can apply more restrictions to the firewall than you could with it running as the main OS. Now, it can only access storage or any other device that is assigned to it.
A common approach to securing multiple virtual services is to run a firewall VM and have it act as the gateway for all of your virtual machines, instead of having the host also operate as the gateway; this approach allows you to have hidden services inside the network you’ve created, and treat your virtual network as you would a physical network.
Updates to a virtual machine, depending on the approach, can be applied extremely quickly and with little downtime.
Basically all the questions we asked above apply twice, now; we need to determine how much physical RAM our VM host needs, and of that amount, how much the VM needs. A multi-core processor, and preferably with a lot of cores, is essential.
One common approach is to connect the VM to one of the host machine’s NICs through a sort of bridge.
One option is to make your host also route packets, although this might be said to defeat the purpose of the firewall here.
In this mode the NIC passes all packets it receives to the kernel, which means it can respond to multiple MAC addresses if the host(s) so choose; this is a common approach to allow one or multiple VMs to share a network connection with a host.
For my virtual firewall setup, i’ve opted to dedicate an entire physical card
with multiple ports to the firewall, and thereby made my VM host indirectly
connected to the main network. To achieve this I ended up adding an instruction
to load pcistub as the driver for a specific device to the kernel command
line.
I’ve been setting up an OPNsense VM to operate as the firewall for my hackerspace; the server I purchased to function as our new firewall is far more powerful than is necessary for a firewall alone, so I figured I’d host some extra machines alongside the firewall, and get more use out of the server.
Well, the Chelsio NIC I added to the server required a fair amount of
massaging to let me actually do the passthrough; I ended up needing to
use the pcistub kernel command line option for the whole card to stop the
kernel from loading the drivers for it.
If you’re using virsh to establish your networks, for a virtual network where
the firewall VM is the gateway, you’ll need to specify how all your VMs attach
to it by configuring the network in their XML config to not have a forwarding
entry.
Seen as a “defense in depth” strategy, this takes the medieval walled city approach to network design. Here, we have potentially two levels of network; we might want to keep “soft hosts”–personal computers, other relatively unsecured systems–behind a more restricted firewall, while still allowing machines operating as servers to have a more porous environment to work in–potentially with other untrusted devices there as well (such as IoT devices).
The standard MTU is 1500B; this is fine in a reasonably fast network, but does have a not-insignificant amount of overhead. That MTU includes the IP header, after all, and especially on an IPv6 network, that can be quite large.
There’s unfortunately issues with using DHCP to announce the MTU of a network; many DHCP clients will simply ignore your stated value and use 1500, so until jumbo frames in consumer OSes start showing up more often, we’re going to have to set this aside.
An interesting technology but not widely supported; the primary vendor who’s pushing for this tech is Chelsio; they’ve attempted in the past to get offload support built into the Linux kernel, but were rebuked on the grounds that this moves kernel decisions into a black box; we may yet see some changes in this attitude, but it is generally outside the scope of this talk.
Let’s build a router real quick!
Set up a virtual firewall for other virtual hosts (mayyybe?)
For the purposes of this section, we’ll focus on a minimal firewall setup that covers the basic needs of a firewall, with clear extension points. The base image is Alpine Linux, an ultra-minimalistic distro based on musl-libc and busybox, which has an absolutely miniscule footprint and makes it great for demonstrating the low-level approach to this setup.
This will need the --internal flag.
Set up alpine in Docker?
#+NAME demo-Dockerfile
FROM alpine:latest
# nftables is the firewall we'll be using
# tinyssh is a server, that doesn't support password auth-so we need keys
# openrc is the package that will handle adding init services.
RUN apk add --no-cache nftables tinyssh openrc
CMD ["/bin/bash"]
# Create the image!
sudo docker build -t alfire .
# Now let's run it.
sudo docker run -i -t alfireThat will get us connected interactively to the alfire docker container; we
can now start running some tasks.
mkdir ~/.ssh
chmod 700 ~/.ssh
touch ~/.ssh/authorized_keys
# Have your public key copied and ready for this next step!
# We're going to append our public key to our authorized_keys so we can detach
# from the container and SSH into it.
echo "$PUBLIC_KEY" >> ~/.ssh/authorized_keysWe’ll create an extra key for the purpose of a break-glass; this is mostly for the exercise, I don’t expect anyone to need this. But! For the sake of the exercise, here’s how we’ll do it:
#+NAME demo-break-glass
cd ~/demo
ssh-keygen -t ed25519 -C break-glass -N "" -f id_ed25519.break-glass
# -t ed25519 : Use the current preferred elliptic curve system for key generation
# -C break-glass : set a comment of "break-glass", to identify the public key on
# sight
# -N "" : set an empty passphrase. For this key, we don't want to need to enter
# additional information to use it.
# -f id_ed25519.break-glass : filename. Easily recognized.Copy the public key’s entire contents, then repeat the echo command above to
set the break-glass key for your user.
This filesystem will tell you a lot about the configuration of your network,
and the files in /etc/sysctl.d will set values at boot which can also be
dynamically configured; files in this folder are read in sort order, which
is why files are usually prefixed with a number denoting importance, low to
high. An example of a line in one of these config files is as follows:
#+NAME ipv4-conf-martians-example
net.ipv4.conf.all.log_martians = TRUE
This line sets /proc/sys/net/ipv4/conf/all/log_martians to TRUE; that will
log any and all impossible addresses received by the machine to the kernel
log. This could be useful if you’re seeing a lot of unrouteable traffic on your
network, for instance.
To set up forwarding with networkd, you’ll want to include the following block in a configuration file matching each interface you want to forward traffic.
#+NAME networkd-forwarding
[Network]
IPForward=yes
For every interface that’s going to be routing traffic, when you check
/proc/sys/net/ipv4/conf/$IFACE/forwarding it should show 1; for
non-forwarding interfaces, it’ll show a 0. (If it shows any non-zero
value, that’s correct.
Declare your address and subnet with the following network block (you only need one block with each ehading, but this is useful to differentiate.)
#+NAME networkd-static-ip
[Network]
Address=172.24.3.1/24
Turning on DHCP is similarly straightforward.
#+NAME netword-dhcp
[Network]
DHCP=ipv4
For this example we’re going to be writing a few (just a few!) raw NFTables
commands; in my home setup I use firewalld because it’s just less complex.
As an extension of the concept of the * Non-routing subnet you can also have the firewall deny connections from an IoT device subnet into your main subnet or subnets.
This is a form of single-port, one-to-one NAT; with this forwarding, you pick a single external port to surface on your public IP address and forward it to a specific internal port on either the firewall or another internal system. An example of this might be an internal web server:
#+NAME firewalld-https-forward
# This also needs to be run in permanent mode, but the short demo works.
firewall-cmd --add-forward-port=port=80:proto=tcp:toport=80:toaddr=172.24.3.2
firewall-cmd --add-forward-port=port=443:proto=tcp:toport=443:toaddr=172.24.3.2With this port forwarded, our machine at 172.24.3.2 can now serve HTTP and
HTTPS content, and we can proceed to use (for example) Let’s Encrypt to
secure it. Note that if you want to use a server on a different port to handle
getting that certificate, you’ll need to add another exemption for it.