dpzmick.com

Homelab Act 1: VPNs

For the last 170 days (according to uptime on one my routers), I've been setting up a small homelab. Homelabs are kind of cool, and, the setup has been interesting. I'll be writing a few posts explaining the steps I took.

VPNs

Unfortunately, I don't have a public IP in my building, so setting up remote access had to be a little more involved than just opening a port for ssh. I found a tiny, passively cooled, quad-port intel NIC, celeron box on amazon, and figured I'd try setting up my own linux router and run my own VPN server in AWS or something.

This took a bit of work.

Networking

Before diving in, I needed to decide how to layout my network and what VPN tools to use. I read a ton about best practices, but this wasn't super helpful, so instead, I just started spinning up VMs and messing with VPN software and playing with my network settings.

Eventually I settled on wireguard for a VPN, mostly because I never actually got OpenVPN working anywhere. I also spent bunch of time trying to get IPSec and layer 2 tunneling to work, but decided that I didn't really want that anyway. Wireguard is easy, fast, and probably secure, so I'm using that for now.

The network is organized as:

  • One subnet for each "region"
    • My apartment is on the 172.16.1.xxx subnet (for snobs: 172.16.1.0/24)
    • A cloud region on the 172.16.2.xxx subnet
      • I wanted to be able to use cloud VMs for other services, so I found a provider that suported private networking
    • Wireguard speaks IP, so it needs a subnet too. It got 172.16.255.xxx
      • Each device connected to the VPN gets an IP on this subnet
  • Use one private, made-up subdomain for each "region." For example, you could use apartment.me.com, cloud.me.com, and vpn.me.com.
    • I'm using a private subdomains of a domain that I control to ensure that I never clash with anything that actually exists.
  • Have a DHCP server and DNS server available in each region (the DHCP host will always be xxx.xxx.xxx.1 and have hostname/dns name gateway)
    • DNS is authoritative for its subdomain
    • VPN region just routes DNS to the cloud region
    • The VPN gateway is just the VPN server
    • There is no DHCP for VPN subnet since all VPN clients wil have a static IP
  • The VPN server/cloud gateway has a public IP
    • Probably also want a firewall that drops everything other than VPN traffic (I'm using cloud provider firewall and a firewall on the server)

I haven't figured out ipv6 yet because I'm a bad person.

The 172.16.xxx.xxx prefix was selected to try and avoid conflicting with commercial subnets (10.xxx.xxx.xxx) and common private subnets (192.168.xxx.xxx). The entire 172.16.0.0/12 subnet is private, so we can do whatever we want in this range. Unfortunately, a local coffee shop I frequent uses an IP range that clashes with mine, so I have to get clever with routing rules when I am working there.

Install alpine linux on local router

The official install guides are good and more up to date than anything I'd have to say about this.

Setup Basic Networking

In /etc/network/interfaces: 

auto eth0
iface eth0 inet dhcp
        hostname gateway

auto eth1
iface eth1 inet static
        address 172.16.1.1
        netmask 255.255.255.0

eth0 is hooked to my ISP (so I get a DHCP ip), and eth1 was hooked to a tiny TP-Link switch I had laying around.

Firewall and NAT

After spending time banging my head against the iptable, I gave up and tried using a this thing built into alpine called awall. There's a pretty good Zero to Awall guide available which can get you started.

I don't want to explain a full example (the docs are again better than what I can do), but here's some highlights.

  • Enable packet forwarding

    The linux kernel must be told that it is allowed to forward packets. Put net.ipv4.ip_forward = 1 in a sysctl.conf file on alpine, see https://wiki.alpinelinux.org/wiki/Sysctl.conf This is probably needed for ipv6 as well, if you aren't a bad person who is ignoring ipv6, like me

  • Most (all?) for awalls config files can be written in yaml

    The Zero To Awall guide has this example:

    /etc/awall/private/custom-services.json: 

    {
    "service": {
        "openvpn": [
            { "proto": "udp", "port": 1194 },
            { "proto": "tcp", "port": 1194 }
        ]
    }
}

    But, you could also create an equivelent /etc/awall/private/custom-services.yaml if you want: 

    service:
  openvpn:
    - { proto: udp, port: 1194 }
    - { proto: tcp, port: 1194 }

Tricks

In case the internet every goes down, I sometimes need to refresh my ISP DHCP lease to get it to come back up. I stuck a checkinit.sh script into my $PATH somewhere, then added it to cron to run once a minute: 

gateway:~# crontab -l
# min   hour    day     month   weekday command
*       *       *       *       *       checkinet.sh| logger -t checkinet

gateway:~# cat $(which checkinet.sh)
#!/bin/sh

echo "Checking if internet still up"

# does not use our dns server, uses isp
if ! ping -c5 google.com; then
        echo "bouncing network interface"
        ifdown eth0
        ifup eth0
        #unbound needed to be restarted, dnsmasq appears to be fine with this
        #sleep 30
        #/etc/init.d/unbound restart # idk why this needs to happen
else
        echo "Internet still up!"
fi

This is really only testing if I can resolve google.com, since ping will probably work if I can reach DNS to resolve google, but whatever. The script gets me back up and going if I unplug stuff or if my ISP flakes out for some reason (which has only happened twice ever, this fixed it the second time), and it's never killed my internet spuriously, so I guess it works?

I also:

  • Cranked up the syslog file size and max files to keep around by editing an init file (probably the wrong way to do it)
  • Installed the S.M.A.R.T. tools (since there's an SSD in the thing)
  • Created a cron job to run smart tests sometimes and log it somewhere (which I've never looked at)

Setup dnsmasq as a DHCP server and DNS server

The arch wiki has wonderful docs for this. Just go read those.

All I really had to do in the end was:

  • Turn on DHCP and DNS servers
    • Enable dhcp-authoritative
    • Provide useful defaults to connected clients: dhcp-option=option:router,172.16.1.1
  • Tell dnsmasq what interfaces to listen on and from where to allow DNS queries
  • Tell dnsmasq which domain it is going to be authoritative for
    • domain=<whatever>.me.com and local=/<whatever>.me.com/
  • Configure dnsmasq to resolve gateway.<whatever>.me.com to the 172.16.1.1 host
    • Create a file called /etc/hosts.dnsmasq with the only the line 172.16.1.1 gateway
    • Tell dnsmasq not to read the /etc/hosts file with the no-hosts configuration option
    • Then, give dnsmasq the configuration addn-hosts=/etc/hosts.dnsmasq
    • This way, the local networking does not have to be tainted by anything I might want a fixed IP for.
  • Log a lot
    • dhcp-script=/bin/echo, log-queries, and log-dhcp

Download pi-hole's ad domain blacklist

From https://github.com/notracking/hosts-blocklists. Put the tracking domain lists somewhere then just set: 

conf-file=/path/to/domains.txt
addn-hosts=/path/to/hostnames.txt

In the dnsmasq config file. See the dnsmasq docs for an explanation of the difference.

Pay for and plug in some sort of Wireless Access Point

I bought a Unifi AP and followed the instructions to set it up. It works.

Setup alpine and DNS on a cloud server somewhere

Same as above mostly, just with a different made-up star trek themed subdomain.

Wireguard

Each device that can connect to the server needs a private/public key pair. The server contains a list of recognized public keys; only the devices in the server config can connect.

There's a wireguard-tooling package available that you can use to generate keys. Generate keys for each device (including the server): 

$ umask 077 # make sure no one can read your files
$ wg genkey | tee private_key | wg pubkey > public_key
$ ls
private_key public_key

Once you are done copying the contents of these files into the wireguard configs, delete them.

On the VPN server (cloud instance)

Create a wireguard server config at /etc/wireguard/wg0.conf. Note that I am not using the wg-quick interface for this or the apartment router. 

gateway:~# cat /etc/wireguard/wg0.conf
[Interface]
PrivateKey = ..... # put the contents of the private key file here
ListenPort = .... # 51820 seems to be standard port

# For each device that can connect to the VPN, create a [Peer] block

# gateway router in apartment
[Peer]
PublicKey = ..... # put the contents of the public key file here
# The AllowedIPs list is sort of like a routing table
# In this section, we specify which IPs may be reached by directing traffic to this peer.
# For the apartment router:
# - assign the VPN IP: 172.16.255.2 and
# - allow wireguard to route traffic from the VPN subnet to the 172.16.1.0/24 using this peer
AllowedIPs = 172.16.255.2/32, 172.16.1.0/24

# laptop
[Peer]
PublicKey = ..... # put the contents of the public key file here
# laptop is assigned a static ip.
# this static ip is the only thing I'm allowing the VPN network to access
AllowedIps = 172.16.255.3/32

# .... more peers here

Next, configure kernel's networking stack:

  1. create a new interface named wg0
  2. use the wg tool to set the interface config file
  3. set a static ip/netmask for this interface/subnet
  4. Add a routing table entry to route traffic from the cloud subnet to the apartment subnet over the wg0 interface

This is done on alpine by adding more stuff to /etc/network/interfaces: 

auto wg0
iface wg0 inet static
        address 172.16.255.1
        netmask 255.255.255.0
        pre-up ip link add dev wg0 type wireguard
        pre-up wg setconf wg0 /etc/wireguard/wg0.conf
        post-up ip route add 172.16.1.0/24 dev wg0
        post-down ip link delete wg0

On the apartment gateway

The router in my apartment is a VPN client, maintaining a persistent connection to the VPN server.

In /etc/wireguard/wg0.conf put something like: 

[Interface]
PrivateKey = .... # private key associated with this peer

[Peer]
Endpoint = <public ip of VPN server>:<port of VPN server>
PublicKey = ...... # public key goes here
PersistentKeepalive = 25  # keep the connection alive at all times
# Allow the apartment router to route traffic into:
# - VPN subnet
# - cloud subnet
AllowedIPs = 172.16.255.0/24, 172.16.2.0/24

Create the new interface in /etc/network/interfaces: 

auto wg0
iface wg0 inet static
        address 172.16.255.2
        netmask 255.255.255.0
        pre-up ip link add dev wg0 type wireguard
        pre-up wg setconf wg0 /etc/wireguard/wg0.conf
        post-up ip route add 172.16.2.0/24 dev wg0
        post-down ip link delete wg0

On a "dynamic" VPN client

On machines like my laptop, I want to easily bring the VPN up and down. This is easy to do with the wg-quick tool. wg-quick allows you to add a few more entries to the config file. When you run wg-quick up wg0, it will bring up the interface, configure routing, and PostUp/PostDown scripts.

Here's the config from my (arch linux/systemd) laptop: 

[Interface]
Address = 172.16.255.3/32
PrivateKey = .... # private key for this device
# After coming up, reconfigure my domain resolution.
# I'm on the vpn subdomain now. I resolve DNS queries with the cloud region's DNS server
PostUp = printf 'domain vpn.me.com\nnameserver 172.16.2.1' | resolvconf -a %i -m 0 -x
# dnsmasq caches queries, so restart it to make sure the cache is clean
PostUp = systemctl restart dnsmasq
# on teardown, undo the DNS resolver tweaks
PostDown = resolvconf -d %i

[Peer]
Endpoint = <server public ip>:<server public port>
PublicKey = ...... # public key for the server
PersistentKeepalive = 25
# Route *all traffic* through the VPN
AllowedIPs = 0.0.0.0/0, ::/0
# Alternatively, we could use a list like:
# AllowedIPs = 172.16.255.0/24, 172.16.2.0/24, 172.16.1.0/24
# to route only internal traffic through the VPN.
# This list can be as precise as you need it to be.
  • Laptop Lid

    When my laptop lid closes, I kill the wireguard connection with a systemd unit file. This seems to minimize confusion when I close my laptop and take it somewhere.

    In /etc/systemd/system/wg-down.service: 

    [Unit]
Description=Kill wg when machine goes to sleep
After=suspend.target

[Service]
Type=oneshot
ExecStart=sh -c '(ip link show wg0 && wg-quick down wg0) || true'

[Install]
WantedBy=suspend.target

Tweak dnsmasq config again

Make sure that the DNS servers know how to send queries to each other:

In the apt.me.com dnsmasq config: 

# Add other name servers here, with domain specs if they are for
# non-public domains.
server=/cloud.me.com/172.16.2.1
server=/2.16.172.in-addr.arpa/172.16.2.1

In the cloud.me.com dnsmasq config: 

# Add other name servers here, with domain specs if they are for
# non-public domains.
server=/apt.me.com/172.16.1.1
server=/1.16.172.in-addr.arpa/172.16.1.1

# Allow VPN to use the cloud-region's DNS server
server=172.16.2.1@wg0

Plug it all in

I plugged the new router box into the wall (on port 0), and plugged a small 4-port TP-link switch into port 1. Everything else is plugged into the TP-link switch.

Finally: use the system

Good

  • Wireguard is rock solid, even on my phone and from an airplane.
  • My local network performance is incredible
  • The tracker block lists noticeably effect load times for some sites
  • The latency/bandwidth I get back to my apartment is low/high, from everywhere I've been in Chicago

Bad

  • DHCP lease refreshes are slow for me right now
    • When my lease expires, sometimes I'll see connectivity blips
  • The latency/bandwidth I get from when connecting to my apartment or cloud instance in Chicago from somewhere like Florida seems poor
    • This is probably an issue with my choice of cloud vendor
  • The tracker block lists break lots of things, which is sometimes annoying
    • Many tracker links are broken (emailed, google sponsored, etc)
    • Facebook behaves strangely
  • I haven't setup ipv6

Overall, I'm extremely happy with how this turned out.

home switch-color-mode