2026-04-06

I’ve had a Git repository storing my dotfiles for well over a decade at this point, and I’ve had the same installation process for just about all of that time: A script/bootstrap that I originally copied from Zach Holman’s dotfiles, that works like this:

• It crawls the repository for files with a name *.symlink.
• It processes each found filename thusly:
  1. Remove .symlink
  2. Remove the first directoy name (this allows config files to be organised per topic)
  3. It adds a . to the start
• It creates a symlink from ~/$processedName to the original found file or directory.

This was perfect for traditional “dotfiles”, which are so named because they are configuration files in the root of a user’s home directory that begin with a dot.

XDG Took Over The World

The freedesktop organisation produce a series of specifications under the XDG (Cross-Desktop Group) moniker. One such specification is the XDG Base Directory Specification, which was first published in 2003. It contains within it advice for where applications should store configuration and runtime files such that they do not pollute a user’s home directory with an inconsistent spattering of hidden files. I haven’t been able to find much on the history of this specification, but my memory is that this was it was an obscurity until the spec was updated in 2010. This update coincides with the development of Systemd, which went pretty hard in supporting this, and I think is a large part in generating broad support for this spec.

Slowly, the tools I was using began to adopt the XDG base directory specification, or at least began to support it. For example, Git added support alongside their existing configuration files in 2012. My first exposure to this was Fish, which expected its configuration to live in ~/.config/fish/config.fish, and the next was Neovim that expected its configuration to live in ~/.config/nvim/init.vim. To start off, I just manually made symlinks from these directories to the relevant part of my dotfiles repository. It was surprising to me that I tolerated this for well over half a decade, but I think that speaks more to how often I set up a brand new user account than anything else.

But two years ago, I finally decided to adopt this specification myself, and give it first-class status. I updated script/bootstrap.sh such that files with .xdg.symlink would actually get mapped to ~/.config/$baseName. This has been in place since then and I don’t think I ever revisited it. Over time, however, I noticed something surprising: everything I was maintaining was an .xdg.symlink type installation, rather than a traditional “dotfile”. The workaround had become the standard path.

The other XDG Variable

I regret to tell you that I have hit the limits of my little workaround script. You see I only link things from ~/.config/*, which is the default value for XDG_CONFIG_HOME. However, if you followed any of the links to the XDG Base Directory Specification above (or if you’re just clever), you will know that there are other locations specified for different user-specific state. The pertinent one in this story is XDG_DATA_HOME. Here is what the spec says about it:

$XDG_DATA_HOME defines the base directory relative to which user-specific data files should be stored. If $XDG_DATA_HOME is either not set or empty, a default equal to $HOME/.local/share should be used.

Gee, thanks for clearing that one up. This is a little underspecified for my tastes, but it seems to be used mostly for things like history files. Lazy.nvim also stores downloaded plugins there. In short, it’s full of stuff that doesn’t need to be committed and is not important. Except for one thing.

On my desktop, I use the Sway window manager, and an assortment of ancillary tools that together form something like a desktop environment. All of these support theming, but none of them support consistent or automatic theme switching that I might get in something like KDE. I like to use a light theme during the day, and a dark theme during the evening or in other low-light situations. The solution to this exact problem is another little ancillary tool called Darkman. To get all of these different things working together nicely requires some custom scripting. Darkman provides hooks for these:

darkman can run custom executables (which can be simple shell scripts).

Scripts are searched in a directory named darkman inside paths defined in the XDG_DATA_DIRS as well as XDG_DATA_HOME. Each script receives the current mode (“dark” or “light”) as its first argument ($1).

These scripts absolutely belong in my personal dotfiles, although I can see why these are treated as state rather than configuration by Darkman. The solution, therefore, is to be able to symlink from ~/.local/share/darkman to somewhere in my repo.

Perhaps darkman.xdg-data.symlink?

Enter GNU Stow

Absolutely not. Enough is enough. I was considering options to evolve this script into something a little more sophisticated, was turning over ideas in my head, playing with different constraints and compromises. Before progressing any further with such a project, I wanted to look at some existing art, to see if maybe there was an existing tool that did what I wanted, or perhaps gave me an alternative way of looking at it. After all, I’ve been using the same thing for a good portion of my life at this point.

I know that some people have used Stow for this. Stow was initially designed to symlink components of individually packaged software into system level directories so that it can more easily be used. I’m pretty sure it’s mostly a set of Perl scripts, and it was initially released before I was born, so it’s all green flags so far. Installing packages like this is a very similar job to installing dotfiles into a user’s home directory, and Stow makes this even more similar by the assumptions it makes at the outset: crucially, Stow will symlink things from the source directory into the same relative locations in its immediate parent. This means that if my dotfiles are cloned to ~/dotfiles, then by default, Stow will put things in ~. Hell yeah.

Stow has added more affordances for dotfiles in particular through its life. The most obvious of these is a --dotfiles flag which — while linking — rewrites directory names from dot-* to .*, meaning that ones dotfiles repo need not be a mess of hidden files.

After a careful read of the manpages, I was able to replace the venerable and now erstwhile script/bootstrap.sh with a new, sleek, modern, fresh, tasty, install.sh that does little other than call stow:

#! /usr/bin/env bash

function main {
    stow \
        --verbose \
        --stow . \
        --target ~ \
        --restow \
        --dotfiles \
        --ignore "\.DS_Store" \
        --ignore "install.sh" \
        --ignore "bin" \
        --ignore "accessories"
}

main

I chose to abandon the top-level topic directories, and keep my dotfiles repo as a partial mirror of my home directory. The reasoning here is that since I pretty much exclusively use XDG-compliant software, that topic distinction already exists within the dot-config and dot-local/share directories. While I no longer use Zsh, I wasn’t quite ready to eradicate its config, so it lives in dot-zsh and dot-zshrc, but that is [a] rare enough that I don’t really feel the need to solve this at a systems level, and [b] obviously named enough that nobody should be confused by this.

I’m delighted that in 2026, I’m able to pick up an ancient piece of tooling, and over the course of a morning, totally rearrange my dotfiles to accommodate a new need. In a career that is increasingly uncertain in a world that is increasingly bleak, I have found that silly and nerdy things like this bring me a load-bearing amount of joy.

2025-10-08

In this post, I’m going to explain my architecture as it pertains to networking and service discovery. In broad strokes I’ll outline some of my philosophy, and lay the groundwork for actually having real services running that I might want to use. So far it’s all been Linux and hardware and super low-level stuff, but I promise that we are getting somewhere. This will be the last entry in “Project Sunshine” for a little while. I am by no means finished, but after this I plan to write some articles that are more sharply focused on specific technical aspects as opposed to these rambling overtures. There are a few things I’ve done here that I think are particularly novel (or that I’ve fought particularly hard for), and I think they are worth sharing in a way that makes them easier to find. I will be returning to this series when I have something significant to write about with my server configuration: some new hardware, or my backup strategy loom as the likely next entries.

The Fleet

In addition to my home server, I am also running a small VPS to act as an edge node. This will be responsible for hosting everything that is exposed to the wider internet, such as this very blog, and my source code. This frees up my home server to be safely tucked away behind Tailscale on my residential connection. The VPS itself is hosted in Toronto with Vultr. I’m a new customer there, but so far I have only bombastic praise: I was able to supply my own cloud-init user data, they support configurable rDNS, their customer support is prompt. They have great region availability, and have been the lowest-friction host I think I’ve ever worked with.

My work here would not be complete without naming them. Both Tailscale and all laws of good taste require it. Please meet Orca and Plankton. Orca is the home server. Big and full of wisdom. Plankton is the smallest VPS I can get, and provably replaceable. (Plankton is in fact already on its third incarnation.) These are in keeping with a new marine theme I seem to have struck up: my home PC is called Terrapin.

Network Topology

As alluded to above, Tailscale is a really fundamental element of my infrastructure. All of my local networking is routed through Tailscale IP addresses. They are static for all time, they route directly, and it was super easy to set up. I’ve got all of my devices attached to it and I’m not yet paying them. Every feature I’ve needed has been no more than three clicks away. It works great, and I can’t imagine considering anything else for what I’m doing here. Having said that, this is still me we’re talking about here, so I was never going to stick with the defaults.

While it’s undeniably convenient, I found the Tailscale default TLD for nodes to be unsightly and restrictive. I plan to run multiple services and I want each of them to have separate domain names. While it would be nice if Tailscale supported subdomains, I’m kind of glad they don’t. It has allowed me to purchase perhaps my favourite domain name ever.

All of my internal services, all of my infrastructure nodes, all of everything that is not exposed to the internet is attached to the domain name max.plumbing. I bought this with my real money and it might be the best $80 or so that I’ve ever spent.

As well as pretty URLs, it was also very important to me to have TLS. A lot of apps and services warn if they are not running over TLS, even if (unbeknownst to them) they are running in an encrypted wireguard channel, and are locked inside a private network. I was also unwilling to deal with self-signed certificates. I don’t want to add a certificate to multiple isolated trust stores myself, let alone inflict it on somebody else I claim to care about. I use an iPhone, for crying out loud.

How are we to get all of these things done? Split DNS! Domain names are translated into routable IP addresses via a protocol called [the] Domain Name System. This protocol traces its origins back to a painstakingly maintained HOSTS.TXT file hosted at Stanford University, but is now a complex set of canonical servers and relays all over the world. When your computer makes a domain name lookup with DNS, it consults a specific DNS server — that is selected by either you or more likely your ISP — to find its answer. If that server doesn’t know, it passes it back up the chain, ultimately ending up at a server that says it is the authority on that domain name. The server that gets to be the authority is actually centrally maintained in a public database of domain names, indexed by the TLD, or top level domain. That’s your .com, .net, or even .plumbing.

Since we can select our DNS server, it is possible to override the DNS lookup process within a specific private network. This includes a Tailnet. Tailscale by default does exactly this to resolve its machine names, but they also provide an option to add other overlay servers to the chain. This technology allows me to provide public DNS records for max.plumbing, and an altogether different set of private records for anyone within my Tailnet. We can use the dig utility to demonstrate this:

mpwb@orca:~$ dig A orca.max.plumbing +short
100.x.y.z
mpwb@orca:~$ dig A orca.max.plumbing +short @9.9.9.9
155.138.139.214

With this one weird trick, I can have a completely different service graph exposed to my private network, while still having a public resolution for anything I wish. Right now I use that for TLS certificate verification. My private server is able to solve ACME challenges from Let’s Encrypt by having the HTTP challenge requests be proxied down to it by Plankton.

The actual DNS server I am running is called CoreDNS. It was really simple to set up, and configuring it in Tailscale was an absolute doddle. I am using Ansible to configure it all, the meat of which is visible here.

To handle TLS connections — and to proxy all of my HTTP/S requests to their final destinations — I am using Caddy. It builds TLS support in, and is alarmingly simple to configure. I remember when Nginx was the new kid on the block, and people praised its human-friendly configuration format compared to Apache. Caddy feels like still another leap forward. It automatically rewrites all insecure requests to HTTPS, and it handles the generation and renewal of TLS certificates.

Interlude: ACME

Automatic Certificate Management Environment, or ACME is a protocol that came up about a decade ago that has since revolutionised the process for facilitating encrypted web traffic. It used to be that to serve encrypted traffic over TLS, one had to procure a certificate from a certificate authority for some significant fee. These would have validities generally measured in a single digit number of years. That certificate would then have to be configured in your server and kept safe. Then, in 2016, The Internet Security Research Group launched Let’s Encrypt: a service that would give you certificates for free! The catch was that they were only valid for three months or so. This would be inconvenient to the point of being onerous under the traditional model, but the strategy here was to force people to automate their certificate management.

Having fewer people and shorter life spans for secrets are obvious boons to information security. Since then, numerous ACME clients to automate this process of creation and renewal have cropped up. While all the major web servers now integrate an ACME client, Caddy was the first to do it.

To obtain a certificate from an ACME authority, you need to perform some kind of verification to prove that you do indeed own the domain you are trying to certify. Under ACME, there are two primary methods: HTTP-01 and DNS-01. The former requires you to host some specific file at a provided URL, whereas the latter prescribes specific DNS records that must be publicly readable.

Given the inaccessible nature of Orca, I originally set up DNS-01, using a custom build of Caddy that integrated support for my DNS provider of choice, deSEC. Caddy would receive the challenge and use deSEC’s API to set records. However, this all got screwed up with my split horizon DNS, and I then resorted to HTTP-01 verification. At that point, I had to set up Caddy on Plankton to proxy these requests through. That worked perfectly!

I’d like to shout out a particular feature of Caddy here. I went through a lot of iterations while developing the elegant and straightforward process I described above. One early version involved using HAProxy as the front end and having a wildcard TLS certificate provisioned by Lego (a standalone ACME client). In setting this up, I ran afoul of Let’s Encrypt’s rate limits for setting up certificates, and found myself blocked for a week or so. This was bad and I don’t recommend it. I didn’t end up here through mistakes in configuration, it was because I was testing this setup in VMs that I would then blow away. So, a different kind of mistake. Let’s Encrypt has pretty harsh rate limits to ensure that misbehaving ACME clients are found out early, before failure to obtain a certificate can do any real damage. To protect fools like me from themselves, they also provide a staging certificate authority that can be used for preflight. If only I’d known about this beforehand. The feature of Caddy I’d like to shout out is that if it is unable to provision a certificate, for any reason, it automatically retries against Let’s Encrypt’s staging CA until it succeeds. I didn’t again hit their rate limits, even when I had tied myself in a hopeless knot with my DNS.

We now have our servers, and we have a general idea of how they are connecting. The missing piece is how we are going to actually deploy services.

Containerise Some of The Things

I adore containers. Docker came out at the start of my career, and it was one of those things that felt like magic for me once it clicked. It caused me to revisit Linux (for the first time) just to get near-native speeds with VM-like isolation. Some of the most enlightening moments of my early career were reading things like Jessie Frazelle’s blog, or learning about the jailbreaks that people were constantly finding and fixing.

Still, Docker is not without its issues: it requires a daemon to be running as root. It’s been a while since I heard about a serious jailbreak in Docker, but that’s still a fair bit of surface area risk! I’m also still a little bit wary of running things like databases in containers. I don’t know if that’s still the best advice or whether it’s vestigial FUD, but it gives me the willies.

Fortunately for me, in the decade or so since Docker launched the Open Container Initiative, there are other options. The option among those that is particularly interesting to me is Podman. It provides a mostly Docker-compatible API, but runs without a daemon, and can run without root access.

Interlude: What’s a Container?

To understand containers, I feel like one has to first understand VMs.

Inner Interlude: What’s a VM?

Computers are typically made up of software that runs on hardware. The device on which you’re reading this (unless you printed out, in which case baller move) probably matches this description. As I wrote in my last post, hardware is backed by software called drivers that know how to physically interact with it. The operating system boots up and uses common, shared, and pre-agreed-upon interfaces to interact with these drivers: block devices, character devices, and suchlike.

But we can implement those interfaces in software! We don’t have to have real hardware backing those things. A hypervisor is a piece of software that implements in software (or proxies access to) every piece of hardware an operating system needs to boot. This allows you to run a “computer” on virtualised hardware, ideally completely isolated from the actual hardware itself. My edge server Plankton is a VM, running on some giant server in some data centre.

Virtual machines always carry some kind of overhead, and depending on what you need to do, it can be significant. It is possible to mitigate many of these by giving direct access to some things, like storage devices or sections of RAM. However, this carries tradeoffs in resource consumption.

What if we could run something like a VM, but without pre-empting resource allocation or paying a hefty performance penalty?

Well. I’m so glad you asked. And I promise we will get to that, but we have to talk about the history of multitasking first. I swear it’s relevant.

Inner Interlude 2: A Brief History on Multitasking Kernels

As I write this, I have a Lofi Girl stream playing in the background; I have Wikipedia open in one window; I have the textbook Operating Systems: Three Easy Pieces in another. A quick scan of /System/Library/LaunchDaemons lists 419 services that are provided by Apple and considered essential to even run my computer. Our computers are doing hundreds of things literally all the time. That was not always the case.

In times of yore (like the 1950s), we had computers. Albeit slow and expensive, but definitely things we would recognise as computers by modern standards. One large deviation from modernity was in their approach to multitasking. If you needed to use a computer, you had to rent time on it! And then your thing would run, and then you would collect your results and let someone else have a go. There was a clear need to streamline this, to empower computers to do more than one thing at a time.

An early approach to multitasking was known as co-operative multi-tasking. This required that programmes cede control back to the operating system when they were able to, so that something else could be scheduled, and control later given back to the original process. This saw widespread adoption, including in early versions of both Windows and Mac OS. Despite that adoption, it is pretty clear in hindsight that it was only ever barely sufficient for the needs of the era, let alone the demands we make of our computers today. We didn’t have to wait very long, though: something better was just around the corner.

The late-1960s saw the development and release of a new operating system: Multics. Multics is the first time we saw an implementation of pre-emptive multitasking, which takes control of when to start and stop programmes from the programmes themselves and reserves that right for itself. This requires the ability to “interrupt” a running process, and sure enough Multics is the first time that we see the modern notion of software-driven interrupts (as opposed to the extant hardware-level interrupts of the time which allowed a programmer to halt execution of a programme, or to restart it once they had supplied input or read output). We still use this term to describe the same thing today. This powerful idea was a stepping stone to even more sophisticated notions that arose in successors to Multics, as we will see.

Through the 60s and early 70s, the folks at Bell Labs were developing the early versions of Unix. Unix is notable for a great many things, but today I want to focus on the concept of the process. A process is an abstraction of a running programme. A process presents a consistent view of low-level machine state: registers and memory to a programme that is running. This wrapper combined with interrupts gives us the building blocks for still more powerful features to delegate process management to processes, and giving us the idea of the process tree. In Unix, these primitives (still today) are called fork, exec, and wait.

These features themselves are a form of virtualisation. And the trend thereafter has been to virtualise basically everything. Memory access is virtualised and addresses are translated. Storage and other syscalls access runs through mandatory access control layers. This is done not only for security, but also to preserve the lie that we have access to an entire machine. Modern computers are incredibly complex. In fact, they contain more complexity than one could reasonably expect any programmer to hold when they want to build whatever new front-end to ChatGPT they’re expecting to become the next unicorn. The modern operating system’s built-in virtualisation is essential to how we understand and imagine computers to work today.

Interlude: Okay, what’s a Container?

With all of that backstory out of the way, we find ourselves without too much work to do. The modern kernel is a virtualisation device by design. The only thing we need to do now is convince it to lie about what else is going on on the computer. That’s what kernel namespaces do, and that’s how containers work. A container is basically just a collection of namespaces around a kernel’s existing virtualisation features.

For example, while running bash in my shell, I can use the $$ variable and the $PPID variable to see my process ID, and my parent’s id:

mpwb@orca:~$ echo $$
429982
mpwb@orca:~$ echo $PPID
429978
mpwb@orca:~$

However, using a Linux kernel feature known as namespacing, I can run a process that maps a process ID on the host to a different one in the container. To see this in action, we are going to start a container using Podman:

podman run -it --rm quay.io/fedora/fedora:43 bash

We now have a contained Fedora 43 installation, which is distinct from the Fedora 42 installation on the host. One difference is that btop is not installed. Let’s install it so that we have a long-running process that can be inspected:

bash-5.3# dnf install -yq btop
bash-5.3# btop --force-utf

![][ztop.png]

Right off the bat, there are some things that stick out. bash is running with PID 1, and btop is running with PID 22, and nothing else is here. That’s clearly not the case on my host, where it is very clear that the root process is Systemd:

mpwb@orca:~$ ps -p 1
    PID TTY          TIME CMD
      1 ?        00:01:39 systemd

We can inspect this a little bit further. Let’s look at all the processes I am currently running:

mpwb@orca:~$ ps -u "$(whoami)"
    PID TTY          TIME CMD
 429961 ?        00:00:00 systemd
 429963 ?        00:00:00 (sd-pam)
 430047 ?        00:00:00 catatonit
 430058 ?        00:00:00 dbus-broker-lau
 430059 ?        00:00:00 dbus-broker
 431356 ?        00:00:00 sshd-session
 431357 pts/1    00:00:00 bash
 431611 ?        00:00:00 sshd-session
 431612 pts/2    00:00:00 bash
 431820 ?        00:00:00 sshd-session
 431821 pts/3    00:00:00 bash
 431875 pts/3    00:00:02 podman
 431897 ?        00:00:00 pasta.avx2
 431900 ?        00:00:00 conmon
 431902 ?        00:00:00 bash
 432494 ?        00:00:00 btop
 432604 pts/2    00:00:00 ps

As you can see, it’s [a] not that many processes, and [b] you can see btop right there! Going further, we can inspect the tree of processes using pstree

mpwb@orca:~$ pstree -p -u "$(whoami)"
catatonit(430047)

conmon(431900)───bash(431902)───btop(432494)───{btop}(432495)

pasta.avx2(431897)

sshd-session(431356)───bash(431357)───pstree(432704)

sshd-session(431820)───bash(431821)───podman(431875)─┬─{podman}(431876)
                                                     ├─{podman}(431877)
                                                     ├─{podman}(431879)
                                                     ├─{podman}(431880)
                                                     ├─{podman}(431882)
                                                     ├─{podman}(431883)
                                                     ├─{podman}(431884)
                                                     ├─{podman}(431885)
                                                     ├─{podman}(431886)
                                                     ├─{podman}(431887)
                                                     ├─{podman}(431888)
                                                     └─{podman}(431899)

systemd(429961)─┬─(sd-pam)(429963)
                └─dbus-broker-lau(430058)───dbus-broker(430059)

This shows us a few interesting things:

1. We can see the current tree, where I logged in via SSH and ran pstree.
2. We also see the other session where I ran the podman command.
3. But outside of that tree, there is a process running something called conmon, and then bash, and then we see our btop command that’s running inside the container.
4. The container’s bash process is listed as PID 431902 here, whereas inside our container it says it has PID 1.
5. It also lists that process as being run by root, while we can clearly see in the pstree command above that it is owned by me: mpwb.

What is happening here is that Podman is using features of the Linux kernel to isolate the process we want to run without resorting to a hypervisor to run a complete virtual machine. This allows us to interact with the kernel, and with other devices in a lighter-weight fashion.

It’s not just PIDs and users that get namespaced in this way. Using another old Unix feature, chroot, it is possible to namespace filesystems and directories. This means that the processes in the container are running at /. In a sense, they are! But it’s a different root directory to my root directory. Different container implementations do different things, but Podman actually generates virtual file systems that aren’t accessible on the host without doing a little extra legwork. That legwork is out of scope for this, but I suggest reading podman-mount(1) if you’re curious.

This method of isolation is why I’m so jazzed about Podman’s ability to run containers as unprivileged users. Even if one exploits some issue in these mechanisms to break out of a container, they would still be stuck in an unprivileged account, subject to the standard Linux ACLs, Systemd’s isolation (it uses many of the same features to restrict access to resources in a granular fashion), and SELinux. Above the security benefits, rootless Podman containers allow me to simply organise my services without requiring all of my containers to be in one great big shared namespace.

Considering all this, here’s where I landed with respect to what goes in containers vs. what runs on my host:

• All of my “services” will run in containers.
• Anything that runs software used by multiple services can go on my host.

What are the things that multiple services might use? I’m glad you asked! Right now, it’s a pretty short list. On the default installation of Fedora 42, I have added:

• Podman (duh)
• Caddy
• PostgreSQL 17
• Postfix and OpenDKIM

I’ve also clarified that SELinux, Firewalld, and Socat are all available, and I am heavily dependent on Systemd (as I’ll explain later). There are a few primary motivators for pulling things onto my host:

1. Is it convenient and self-contained? I don’t want to have to install a dozen other daemons for something to work. If it’s a single binary with some straightforward configuration, the overhead of installing it outside a container is minimal.
2. Is it a database?
3. Is it something likely to be needed by multiple services? Caddy and Postfix are perfect examples here. In Caddy’s case, only one thing can bind to :443 on any given interface. For Postfix, networking from containers to other places is really complex. By centralising this on the host, I make one really hard problem into two much more manageable ones.
4. Can it even be containerised? I am running Samba for Time Machine backups of the household Macs, and placing that in a container is simply not an option.

If you can’t answer yes to at least one of these, then you’re going to go into a container. Currently, between both Orca and Plankton, the following services are running in containers:

• Forgejo
• CoreDNS
• Nextcloud

Plenty are going to be added to that list, but this is enough for me to write up my strategy as something that has been borne out and is working, rather than something purely theoretical.

The last piece of this puzzle for now is Systemd, which I am using to orchestrate my containers. Through Quadlet, Podman integrates with Systemd almost seamlessly. I will be discussing the particulars of this in its own post, but the gist of it is that my contained services all run as user level Systemd units in their own unprivileged accounts. Ansible makes it very easy to configure these, and I’ve gotten pretty handy at writing my own unit files. For services that need cron, I’ve set up Systemd timers within the same account that prod the container with the right command. I mount the ports from the containers at special high numbered ports on the host and listen only on the local interface. These are then forwarded to by Caddy, or socat when I need something non-HTTP.

Bibliography

This really closes the lid on what has been a long and involved project. I may only have an MVP, but it’s terribly exciting to have a working thing! I plan to document a lot of the implementation details. However, to reflect this milestone, and I had to do an awful lot for it, I’ve collected some of the resources I’ve leaned on while doing this whole thing.

• Jeff Geerling’s book Ansible for DevOps taught me just enough Ansible to be dangerous. I read all the book chapters and then skimmed the examples. But I kept coming back to this when I had a question like “What’s the best way to do this?”. Excellent orientation.
• The official Ansible documentation is comprehensive! In particular, I spent a huge amount of time poring over the specifics of the modules I was using to see what I had to set, what I could set, and so on. I have shied away from using any third-party modules so far, so this has basically been all I’ve needed.
• The Podman docs are extremely well-written and fun to read. Very instructive on how volumes and networks mount, as well as introducing me to Quadlet.
• The Podman docs were missing a few of the particulars on rootless containers. Luckily, this blog post by Christopher Smart filled in a lot of those gaps.
• This one page in Forgejo’s docs talks about deploying Forgejo as a rootless image. It wasn’t directly related to Podman, but I’m mentioning it here because it was the key to finally understanding why my volumes weren’t working.
• I wouldn’t have gotten anywhere on this without really getting to know Caddy, especially the proxy and TLS configuration. Their docs, especially the index of directives are a must.
• Here’s a nice page from Let’s Encrypt about TLS challenge types, which gives an exhaustive and authoritative list.
• I don’t think I’ve ever read anything by Dennis Ritchie that I didn’t adore, and this essay on The Evolution of the Unix Time-sharing System is no exception. An extremely readable early history of Unix that I referred to in order to shore up my historical notes.
• The iconic 2018 article C Is Not a Low-Level Language was my introduction to the idea that computers were lying to us all the time, and if you haven’t read it, I’d still recommend it today.

All in all, this has been so much fun. I’ve learned a crap-ton of new stuff, in addition to formalising and strengthening a couple decades’ worth of intuition and experience, and modernising a load of old server knowledge. I’ve already moved all of my stuff out of iCloud Drive, and I can’t wait to continue picking the low hanging fruit as I move towards freeing myself from the rent-seeking ghouls in Silicon Valley.

In a more general sense, I’d like to praise the idea of having meaningful projects, too! I’ve read three books to get this finished. Three books that I wouldn’t have otherwise read, which isn’t even to mention the reams of documentation I’ve pored over. It’s nice to be invigorated about technology again, especially with everything the way it is these days.

If you’ve joined me for all or part of this journey, either contemporaneously or after the fact, thank you! I hope it’s been as fun for you as it has been for me.

2025-09-28

Well I built the bloody thing. After my new hard drives arrived, I installed them and realised that they cut off the air intake on my upside-down PSU. So I ordered a different power extension cable that bends the other way, and put it in the correct way. To do this I had to take apart basically the entire thing. I swapped the orientation of my CPU cooler fan, and rearranged some of the cables to be a bit neater, too. I’m super happy with it.

Now I can really think about the software I want to run on this thing. Before going into the specifics, I wanted to start out with my use case. What am I really here for? There are lots of things that I ultimately would like to host myself, but to start out with, I have identified some low-hanging fruit that will allow me to lay a foundation:

• Nextcloud: A cloud storage application that is free for self-hosting. There are apps for every platform I care about, and this should allow me to replace Google Drive and iCloud Drive with my own infrastructure.
• Jellyfin: The whole reason I bought an Intel CPU. I want to be able to store and serve my movie collection to my Apple TV without having to deal with streaming services. I also have a local music collection that is rather large, and Jellyfin will happily serve that, too.
• macOS backups: It seems the correct way to support remote Time Machine backups these days is SMB, so very well: an SMB share for the three macOS devices in the household.
• CI: I have been running my own Forgejo instance on my VPS for some time, and I’d like to have a few things build on it. That server is too wimpy to be useful for that, but this one? I think we could be onto a winner.

Summary

This post is long and dense, so I’m back up at the top now to outline the decisions I made, and where I am currently with this setup. Just in case you want to skip to the good parts:

• I am running Fedora Server 42
• I am booting from my NVMe SSD, which is formatted (after UEFI et al) with an LVM VG that manages a few logical volumes for my root, home, var)
• I am running two 12TB HDDs as a mirrored VDev in a ZFS pool. No other VDevs for optimisation.
• I am planning to use the ZFS pool as data storage, and I am running my services and database servers from the NVMe drive.
• I’m very glad I’ve got 64GB of RAM for ARC.
• This post does not discuss backup plans. I need to be at pains to point out that redundancy is not a backup, though it narrow the spectrum of scenarios in which you need to restore from one. I am actively developing an off-site backup plan, but I’m not quite ready with it yet. Hopefully nothing breaks in the next two months or so.

Bibliography

This post is not a primary source. I read and watched a lot of things to plan this, and it would be extremely remiss of me not to point you to the countless genuine experts who helped me make my decisions. I truly cannot thank them enough for taking the time to share wisdom on forums, Reddit, YouTube, blogs, and more. Below I’ve attached some of the articles and other materials that I found particularly useful while researching all of this. All of it affected this post, some in more diffuse ways than others. In every case, learning about this stuff has been an absolute joy, in huge part thanks to the efforts of the community.

Outside of any of the below sections, I have consulted the book Operating Systems: Three Easy Pieces. I originally bought this to teach me some stuff about locking, but it has shown its stripes here too, particularly chapter 42.

(Official) Docs

• The entire System Administration page on OpenZFS’ official wiki is worth reading. It covers common workloads, a design overview, the works. Great reference, and not too long.
• ArchWiki is a fantastic resource in general, and this is no exception. The articles on Partitions, File Systems, and RAID were particularly useful for me.
• I spent a lot of time reading a lot of man pages, especially at first. In particular, filesystems(5) and lsblk(8) pointed me in the right direction.
• The RHEL documentation is often directly relevant for Fedora, especially in cases like this. The entire chapter on LVM was super helpful.

YouTube Videos

• Level1Tech has a lot of really great stuff, especially for novices. This intro to ZFS gave me some historical context (which I always find helpful), and really oriented me. Later on, I watched a video he made about Optane that I found super helpful.
• From there, this (90 minute) conference talk by George Wilson and Matt Ahrens from Delphix gave me some firm background. It’s from 2018, but ZFS hasn’t changed that much in the intervening seven years.
• 45Drives have a huge number of excellent videos about ZFS. Their Tuesday Tech Tips series is full of surprisingly in-depth videos. Here are a few I particularly enjoyed: ZFS Read & Write Caching, Accelerating ZFS Workloads with NVMe Storage. 45 Drives also spent a good 40 minutes designing a ZFS pool for a mixed workload. The scale is not relevant to me, but the aims definitely were.
• There’s a lot of old knowledge that has been cargo-culted around ZFS for a long time. This video from SpaceRex really helped clear up the value of L2ARC for me.
• Perhaps the best thing that SpaceRex video gave me was a link to a truly excellent talk at SDC 2019: Best Practices for OpenZFS L2ARC in the Era of NVMe.
• Lawrence Systems have also produced a number of good videos on ZFS. The introductory ZFS is a COW is very good, as is the much more recent conversation with Alan Jude: Is ZFS Ready for NVMe Speed?

Reddit and Forum Threads

The one thread I want to call out in particular is What are the bad things about zfs?. I think anyone considering ZFS should give this a thorough read.

Beyond that, the Level1Techs forums, the Lawrence Systems forums have been very valuable resources for specific questions. I haven’t yet had one that hasn’t already been asked.

Operating System

I decided pretty early on Linux. While I’m certainly BSD-curious, I just migrated to Linux on my desktop, and I like the idea of consistency. Furthermore, a familiar stack while I’m navigating a huge number of utterly new things is a welcome respite. Of course deciding on Linux is like deciding on “Asian Food” for dinner. There is no such thing, and the categorisation is so broad as to be almost useless. So let’s narrow it down a little. I come from the Red Hat tradition of Linux, in that I have typically run Fedora on clients and CentOS on servers. I know the tooling on these very well. I even quite enjoy SELinux. Having said that, I was not necessarily married to that world, either. I had a few

• Fedora: familiar, and matches my desktop. Frequent updates (modern tools and annoyance of upgrading, well-maintained and well-supported.
• Rocky Linux: Equally familiar. Longer term support, so less annoying and less modern, which is annoying in a different way.
• Debian: Major new version just came out, so everything is pretty up to date. Community-driven, which is a huge tick.
• Arch: A genuine consideration, since I plan to move to Arch on my desktop, and I enjoy consistency. I am also embracing control, which Arch grants to some extent.

I ruled out Arch fairly early on. I don’t want to be fiddling with the OS that much. I’m prepared to tolerate the at-least-annual updates of Fedora, because I know that they are relatively straightforward. But the level of tinkering required for Arch, and the ongoing management is difficult for me to justify for a server. I’d also like to unify the platform between my VPS and my home server. While Digital Ocean does support custom VM images, that seems like whole other kind of palaver that I don’t really want.

The next to go were Debian and Rocky. While Arch is a bit much, I already found frustration setting up a modern Rocky Linux server a few weeks ago. I really value having modern versions of things in my package manager. I’m not running a massive fleet here. It’s fine for me to be a little less conservative about what I install.

That left Fedora. It’s not exactly a ringing endorsement to simply be the last one standing, but I do genuinely enjoy this distribution. I like that everything in it is promptly updated, and I like the combo of firewalld and SELinux. I will also be leaning heavily on both Podman and Ansible, both of which are heavily supported by Red Hat, and are hence excellently supported by Fedora.

While the currently live version of Fedora is 42, the betas for 43 released only a few days before as I write this. I gave pretty serious consideration to starting there and saving myself that first update. I eventually decided not to. I felt that it was better to start out here and have the experience of an upgrade early in this system’s life, mostly to decide if I want to keep doing it.

Storage Configuration

I’m going to talk in a lot of detail about how I’ve chosen to arrange my storage. This is because there is a lot that goes into these decisions, and I find it really fascinating. I’ve learned a stunning amount about all this just in the last week or so, and I’m very excited to share it. I’m going to start out discussing the theory of how drives are arranged and organised, before running through how these are managed on a Linux system through the command line. Then I’ll talk about the file systems available to me, and why I made the choices I did.

Block Devices, Partitions, and Volumes

When I installed Linux for the first time ever, I had to partition my MacBook’s hard drive. That meant shrinking my HFS+ partition, and pre-allocating space for an Ext3 partition, as well as a swap partition. I was walked through this process with a very patient friend, and I did not understand what I was doing. Cut me some slack, I was but fourteen summers old. Things have changed a little since then, and for the better. This old 160GB HDD — and every stable storage device with which I’ve interacted with since — is exposed to Linux as a block device.

Interlude: Device Files

Unix loves the idea of presenting interfaces to things as files. Processes? Files. Streams? Files. Files? Believe it or not, also files. Devices are no exception to this rule. A device file is a special file that programmes can interact with through stdin/stdout syscalls mediated using a device driver. A device can be any number of things, like a printer, or a serial device, or a storage device. In Unix systems, device files are mounted under /dev.

Devices are varied, and the things they do are equally varied, however, they typically fall into a collection of categories. The operating system would like to be able to control as many classes of device as possible, and likewise devices would like to remain as OS-neutral as possible. To achieve this, device files were developed as a means of abstraction. There is a piece of software in the operating system that knows the details of how a device works. We call such a piece of software a driver. The other end of that driver — at least in Unix systems — is a device file. This exposes a standardised file-like interface to the operating system. This interface comes in one or both of two flavours: block, or character. The exact distinction has something to do with how they are read from and written to, but this has been enough of a diversion to me, someone who someday wants to set up their server.

In Linux specifically, storage devices, be they USB, SATA, NVMe, SAS, or anything else, are exposed as block devices, which is a raw way for the OS to interact with the device itself. To make this usable to a human, that device must be mounted. Mounting opens the device, interacts with it over a few standardised methods to work out the file system on the device, and then exposes that file system at a mount point, which is a directory. Different device types have different names under the /dev namespace. For example, my SATA drives will be /dev/sda and /dev/sdb, while my boot drive is /dev/nvme0n1.

Partitions

Getting back on topic, block devices can be further abstracted into partitions. A partition is a way for a single block device to expose multiple sub-block devices. On a SATA drive, these are numbered like /dev/sda1, or /dev/sda2, and so on. A partitioned drive allows you to run multiple file systems on that drive. This is essential for boot drives, because the boot process involves mounting increasingly sophisticated file systems until the operating system can be fully initialised. On secondary storage drives, partitioning is optional. As far as I can tell, you can just go ahead and format the entire storage device.

Partitions are pretty low level. To partition a block device, one must write a partition table at the start of it, which specifies regions that (oddly enough) partition the device. They are fixed in place. Their low level is actually a feature. They can be loaded by EFI firmware and BIOS. In fact, they are so low-level that you must choose the type of partition map based on whether your motherboard uses UEFI or BIOS. This very accessibility can cause ergonomic issues when trying to actually use them for storage management, though. If you’re anything like me, you can imagine higher level abstractions over these basic entities. If you’re anything like me, you’ve just motivated volumes!

Volumes

A volume manager is a tool that presents an abstraction over physical storage, and then re-presents block interfaces that can be formatted and mounted like drives or partitions. Typically, one feeds a volume manager physical volumes — which are either the storage devices themselves or a partition thereon — and exports volumes: a block interface that is mediated through the volume manager itself. Over partitions, volumes have several benefits. Chief among them is perhaps the ability to easily grow and shrink volumes. But it is also possible to have volumes share all of the available space on the physical media and use it as required, as opposed to having to segment that space ahead of time. Beyond that, volumes may also support using their physical devices in different ways to facilitate either higher throughput or redundancy.

The volume manager that I have on my server is called LVM2, which stands for Logical Volume Manager (2). It came by default with Fedora Server, and is a very common default in servers and on desktop. Out of the box, I was given a single volume group across (almost) my whole NVMe drive, and one logical volume was given to the OS for its root mount.

RAID

“Redundant Array of Independent Disks” (or RAID) is the name given to basically any technology that combines a collection of storage volumes into a single block device. This is done to improve performance, or to decrease the likelihood that one needs to restore from a backup. I am not going to go into detail on the different RAID levels and what they mean here, but as far as I can tell, all RAID (and RAID-like) implementations do a combination of three things:

1. Striping: When you write to a RAID device, it may split that write into multiple smaller ones and write them simultaneously to multiple physical devices, thus increasing your throughput. Likewise, when reading data, it must be read from multiple stripes as well.
2. Mirroring: When a device is mirrored, it means a write to that device must be replayed to multiple devices. As opposed to striping, this offers redundancy, but does not increase write throughputs, although it does allow data to be read faster.
3. Parity: When writing to a RAID device with parity, it will also write parity information somewhere. This allows restoration of data in the event of a physical failure without requiring the 1-1 storage loss of mirroring, and is typically mixed into larger arrays. The checksums combine data within the array in various ways, such that the data itself can be rebuilt from the checksums along with only sum of the data. In general, the more storage space you give to parity, the more sophisticated the checksums, and the more individual device failures you can survive.

In ye olde times, RAID was implemented on custom cards that slotted into servers. This is because parity computations were expensive enough — and the CPUs of the day slow enough — that it made sense to offload that calculation to a dedicated device. Today, those penalties are a lot less severe, and there are many, many software RAID implementations.

The classic implementation on Linux is mdadm, which constructs RAID arrays and exposes devices at /dev/md*. LVM2 also offers a RAID implementation, although this uses the standard tools provided by mdadm. In addition to the operating system and volume managers, more modern file systems also offer RAID. The file systems that do this also typically offer volume management, and that’s where this whole thing starts to get a lot muddier, from a conceptual point of view.

File Systems

A file system sits on top of a block storage device and exposes standardised APIs to the operating system to allow interacting with it in the way the OS expects. In Linux, the mount command is used to make a block storage device accessible at a certain location (or mount point). The file system will write a little bit of data in a standard place on disk to allow the device to be identified as belonging to that file system. After that, all bets are off because the file system has direct access to read and write whatever it likes from the device.

There are many file systems, and Linux supports more of them than most! For this story, I will be really focusing on three:

• XFS: Been around forever, often a default choice on server distributions, including RHEL and Fedora. Old school and relatively basic, but that means it’s outlived basically all of its issues. RHEL say it’s stable for multi-petabyte workloads, which is more than I can say for any software I’ve ever built.
• Btrfs: A newer, more modern file system that acts as a volume manager and RAID implementation. It’s the default on Fedora Workstation, but not on Server. Widely regarded as stable enough now, that has not always been true, and I’d avoid it for real production workloads. I’d have no issues with my use case. Fun fact, Btrfs stands for the B-TRee File System. This would stand it in good stead because I love b-trees, but it turns out basically all file systems use b-trees.
• ZFS: The last word in file systems. If there’s a modern file system out there, it’s likely taken inspiration from this one. This was developed by Sun Microsystems, who open-sourced it before they were scavenged by Oracle. Today, there is OpenZFS, and a private Oracle product. There is also Linux drama around ZFS, which I understand is part of why Btrfs exists.

After a lot of reading, these were my three options. This is a lot of options considering that I came into this basically certain I was going to use Btrfs for everything. The true lesson to take away from my entire experience here is that I had my mind turned around on this issue several times over the week or so that it’s taken me to pull this post together. I cannot recommend highly enough the act of learning: reading articles, papers, docs; watching videos, tutorials, lectures, conference talks. It is impossible to know going into it what will and will not be relevant. Study widely, it will pay dividends.

A full treatment of the internals of every file system I mentioned above should probably be out of scope of this post, so I’ll just say that I am using XFS on my NVMe drive for the OS, and have configured my HDDs as a ZFS pool.

Why XFS?

The primary reason I selected XFS is because ZFS is not a true part of the Linux project. Its implementation lags a little in kernel support, and mounting a root drive in ZFS is just a bit more hassle than I’m ready for. Defaults are powerful, and I trust XFS to do what I need it to do. Having said that, I would prefer a more sophisticated file system with some more advanced features. Btrfs and ZFS both offer that, and I’ll come back to talk about why I’m using XFS for anything later on, in the “Why not ZFS?” section.

Interlude: Journaling vs Copy-on-Write

Writing to disk has a serious problem: it touches the real world, and the real world sucks. It’s all nice to sit in our theoretical world where computers work, but in the real world they only barely work. And it is important for a file system to not commit nonsense to disk in the event of a sudden loss of power. How should a file system do this?

It turns out that file systems — like databases — often use a write-ahead log to track the integrity of data in the case of a crash. When a write to disk is executed, the data and its metadata must be written to the journal beforehand, and only once safely recorded there can they be written to disk proper. If power is lost sometime, then the log can be replayed and data is not lost. The journal has defined start and end blocks, and is written strictly in order so that partial writes to the journal can be detected and ignored. In that case, the write simply failed.

This raises a fairly significant issue: do we not have to write data twice? Yes, actually. And unlike databases, this is not a log structured system where the log is in fact the record. The log is only a contingency, so that write must be flushed immediately. This problem is so severe that most journaling file systems I know of don’t write the data twice. Instead, they just write the metadata to the journal. This is what XFS does by default, and all file systems I know that do write data to the journal have options to turn it off.

Copy-on-Write is an alternative way to protect data from sudden and catastrophic failures. Under such a scheme, data is never overwritten and so cannot be corrupted. That is to say that every write that takes place is a write of new data. Once that write has been committed, a pointer can be atomically updated and the “update” is complete, and the old address can be cleaned up if nothing else is pointing to it.

One issue that faces our naïve copy-on-write implementation here is performance, especially over time. By forcing all writes to take place on new data, we are inviting terrible fragmentation issues, especially on HDDs where the cost of random reads and writes is so much steeper. There are a few solutions to this:

1. By far the easiest option is to not support HDDs! This is a choice Apple made with APFS, which is a CoW file system. They later added support for HDDs by including a defragmentation tool.
2. Another option is to batch writes in a faster medium, like main memory. This allows our file system to buffer writes, then batch them to make them more sequential.

As an aside-on-the-aside here, the block-based nature of copy-on-write file systems also makes it easier to support transparent compression, and convenient creation of snapshots. In that case, the old state of the tree needs to be preserved, and any objects it points to must not be evicted.

Introducing ZFS

ZFS is a copy-on-write file system, designed to safely store large amounts of data in ways that do not really require one to trust the hardware on which it is running. ZFS is awesome. Here are a few highlights:

• ZFS uses integrated checksums to validate the integrity of each block.
• Where possible, ZFS will use those checksums to restore data when it detects errors.
• ZFS builds in a volume manager and RAID implementations.
• ZFS transparently compresses data.
• ZFS can encrypt data.
• ZFS has built-in support for replication.

ZFS Architectural Concepts

ZFS exposes its data as a storage pool called a ZPool. This is what you mount, read from, write to, et cetera. Inside a ZPool are one or more virtual devices — notional storage devices — called VDevs. Each VDev is made up of a number of actual physical devices. Redundancy is configured at the VDev level. So we might have 6 HDDs in a RAID pattern exposed as a VDev. ZFS will stripe reads and writes across all of the ZPools data VDevs, with no redundancy in place. So if you want redundancy, it is very important to configure it at the VDev level.

I snuck a new term in to that paragraph without defining it. I specified “data VDevs”. When you add a new VDev to your ZPool, you can configure it to be used for a number of special purposes. Beyond using it to store data, all use cases support increasing performance. This is often necessary because ZFS is slow by design, owing to its thorough nature. To compensate for this, ZFS has a number of optimisations, some of which are optional and exposed through VDevs.

Beyond your storage architecture and multiplexing data access through stripes, the primary optimisation layer is an in-memory cache called the ARC, which stands for Adaptive Replacement Cache. This can use a lot of memory, but should only use memory that would otherwise go unused on the system, and even then only up to half of it. It is a fairly straightforward block cache of data that is read from disk. Operating Systems have this built-in, it’s called a page cache. ZFS has one because the file system knows its own implementation better, and so can provide a more specialised solution. In particular, ARC maintains a cache of most-recently-used blocks (MRU). This is very standard for caches of all kinds, but this implementation can be vulnerable to scans of blocks larger than the entire cache. In a database context, think about updating a value on all rows in a table. To combat this, ARC also maintains a most-frequently-used list, and to be evicted from the ARC, you have to fall off both lists. The memory within ARC is also used to batch writes to disk to increase throughput and response times.

However, ARC is not always enough, and that’s where non-data VDevs come in. In particular, a VDev may be configured as:

• L2ARC: This is Level 2 ARC, and it’s basically just more ARC. In normal running, if there is an L2ARC VDev present, ARC is constantly feeding the entries at the back of its queue into the L2ARC. That way, if it gets evicted, it might be able to catch it in the L2. It seems that L2ARC is a little simpler than the ARC, in that it is a straightforward ring buffer, meaning that new entries bump the oldest from the cache.
• SLOG: The write architecture I described above is what happens, but the write is only confirmed to the client when that write is tagged as asynchronous. For so-called synchronous writes, the data must be persisted to a stable storage medium before it can be acknowledged as saved. Writes are still batched, and making the client application wait for five seconds for the write buffer to be flashed is not really an option. So ZFS will write this data to a persistent journal called the ZIL (ZFS Intent Log). By default, this lives in the data array. It can be slow, and it gunks up other activity on disk, especially with HDDs because they can only do one thing at a time. A SLOG is a separate log device for the ZIL, allowing that stream to be written to more quickly, and hence dramatically speeding up synchronous writes.
• Special VDev: When a Special VDev is present, metadata (and optionally very small files) are written there instead of to the main data array. This can have a huge impact on high metadata workloads, like list-heavy workflows.

These are all very cool, and each of them scratches a very particular nerd itch for me. However, they all also carry their own issues or complexities. These trade-offs have led me to integrate none of them into my home server right now. I am going to keep an eye on performance, and see what I can implement in the future to help alleviate issues that I do end up hitting.

There are a couple of issues that affect all three of these optimisations. First of all, each of these depend on having a storage device (or devices) that are considerably faster than HDDs. In 2025, I believed that this was either SSDs: either SATA or NVMe. However, NAND flash has a literal fatal flaw: it burns out when you write to it a lot. A lot of these workloads are write-heavy, and I am wary of burning out any drives I might use for these purposes. In the enterprise, there are actually specialised devices that don’t burn out, but I don’t feasibly have access to Optane or NVDimms at home. Of all of them, the SLOG is the one that gives me the most trouble here.

There are also some specific things to consider. For instance, L2ARC is useless if you don’t have data that is routinely falling out of ARC. I am anticipating very low workloads, and I have a lot of RAM considering my needs. ZFS includes excellent telemetry about ARC hits and misses that should allow me to diagnose issues that L2ARC might solve.

Special VDevs can be a huge accelerant for a number of workloads, especially for my SMB share and Nextcloud deployment. However, the metadata is the key to all the rest of the data in the storage pool! If that is lost or goes bad, the data on my HDDs is meaningless. Therefore, it becomes a potential point of failure for the entire ZPool. As a reminder, my server has two HDDs and one NVMe SSD. I’m not comfortable offloading a crucial component of my storage to only a single device.

While I am waiting, my current idea is to buy a couple of smaller SATA SSDs for my SLOG (these are very inexpensive), and a couple of larger ones for my Special VDev, arranged in some redundant way. This comes with a few snags, though. My motherboard only has four SATA ports, which requires that I buy a host bus adapter to add more. Adding four SSDs would also take up two of my drive enclosures, limiting me to only four HDDs. I think this would be okay, but it is worth noting. If I do take this route, it’s worth noting that these SSDs also suffer from the “fatal flaw” I mentioned above. I’m less worried about damaging separate devices that are not used for booting, especially when they are as cheap as they are.

Why not ZFS?

At this moment, I am deploying a mirrored VDev for my HDDs, and nothing else. Because I am lacking any other optimisation layers, there are some workloads that I am reticent to put on ZFS right now.

Chief among these is my database. I am planning to deploy a single PostgreSQL instance to store data for all of my services. I like PostgreSQL, and I don’t particularly care for running databases in containers, so this suits me. However, I want it to be performant and not give me headaches. Like any good database, PostgreSQL demands synchronous writes and so within ZFS today, my only option would be to have it completely dependent on the speed of my spinning drives. Given that I’ve got an NVMe drive just sitting there, this feels like a horrible waste. As such, I am keeping XFS on that drive and running my database workloads there, with frequent backups to my ZPool.

And that’s about it. If you read this entire thing, I hope you both enjoyed yourself and learned something. If not, I’m sorry! Although this post was very long, you have nobody to blame except yourself for reading it all. Thank you very much, and I’ll see you next time, when I’ll probably be writing about databases.

2025-09-23

The wired ethernet I wrote about in part 1 of Project Sunshine is working wonderfully. Laura just told me she’s never before seen an OS update download so quickly. With that infrastructure in place, it’s time to move onto the next critical part of de-clouding my life: actually getting a server. At the project kick-off, I knew a few things already:

1. I want to build it myself. None of this off-the-shelf NAS nonsense, I’m a computer person and I shall be building my own for this. Partially because this is my digital sovereignty project and I shall stand on principle, but mostly because I can meet my needs better with something custom, and also a little bit because I like building computers.
2. I demand silence. This thing will be at low but steady load most of the time, and at that load, I really want it to not be a nuisance. I live in an apartment, so it’s not an option to leave something noisy in a basement.
3. I hate power bills and love the planet. I want it to sip power! There will be at least an entire post about making this server as efficient as I can, but out of the gates I want to be deliberate with my component selection to maximise efficiency.
4. I am planning to stream media, which means I need graphics. I ruled out discrete GPUs pretty early, and all the Jellyfin and Plex docs advise against AMD Ryzen processors. Apparently, Intel’s Quick Sync technology is a bit of a moat.
5. I really want something that will do better for me in this role than a Mac Mini. This is a tall order! That thing is incredible. I’ll talk about this more a bit later.

With all of these things in mind, I set about planning out a build. While there are some newer cases out there for storage focused PC builds, the venerable Fractal Design Node 304 won out. I had also considered its larger cousin the Node 804. That case is awesome, but it has a much larger footprint, and comes with features that I don’t need. It’s still worth it, even factoring in the ITX tax.

This case fits six 3.5” hard drives in it, which is very exciting considering I haven’t owned a hard drive since 2013. This much storage gives me a lot of options, broadly in two dimensions: drive selection, and drive configuration. The latter is going to be its own post a bit later on. This post is just about hardware.

Drive Selection

Hard drives aren’t just higher capacity than SSDs, they are also considerably cheaper in $/TB, and will even outlast SSDs on a lot of workloads. That’s about what I knew coming into this, but it turns out there was a lot more knowledge coming my way. My decision-making regarding hard drives was primarily along three axes: sound, power consumption, and reliability. Alongside that, I also care about the price and the speed. The capacity is almost an orthogonal choice here: most hard drive lines come in a range of capacities, so that is something I can choose according to my needs. I very quickly learned that ordinary consumer hard drives were not going to fit my bill. For the kinds of things I’m planning, I will need NAS grade drives. I also learned that I want to make sure that they use CMR recording technology rather than the cheaper and denser SMR. SMR (Shingled Magnetic Recording) was introduced only a decade or so ago, and allows more data to be stored on a platter, but their design entails lower write speeds that all but entirely precludes doing anything with ZFS or RAID. Luckily for me, drives marketed as NAS grade almost entirely use CMR. There was a scandal about SMR drives being poorly labelled and shoved into existing NAS lines a few years ago, a scandal from which I can now benefit, because this information was easy to find.

It was harder to find information about another innovation in spinning rust drives over the last decade: helium filling. By hermetically sealing drives and filling them with helium, you can get some pretty serious efficiency savings, as well as better acoustic performance. As best I could tell, this was first sold in 2013, making it the second cool new thing that’s happened in this space since I last owned a hard drive. Unlike the recording technology, this is a lot harder to find on a product page. I had to go trawling through vendor data sheets to find this out, which is exactly what I did. Looking at the Seagate Ironwolf line, one can see that their 10TB drives are helium-filled, while their 8TB drives use air. This results in a 2.6W drop in idle power consumption, as well as a 10dB drop in idle sound. I cross-checked this with a few other vendors while comparison shopping, and this was pretty consistently the difference.

After a lot of comparison and looking around, I decided that the WD Red Plus line was probably best for me. The drives are quiet, power efficient, perfect for my needs. The only one in that line that is helium-filled is 12TB. That is honestly a little bit silly, but it does mean I can get away with two of them, rather than four or even more. It is also only one of the WD Red Plus 12TB drives that is helium-filled, since they have an air-filled model too. I needed to be very careful in buying WD120EFBX and not WD120EFGX.

Was I very careful? I’m sorry to report that I was not. I accidentally bought drives with the serial number WD122KFBX. These are 12TB WD Pro drives. Relative to Plus, Pro is more enterprise-focused, meaning that the drives are more reliable, support higher write volume, are faster, and are of course more expensive. I could have lived with this, except that that model is also air-filled. After taking a look at the data sheet again, I decided to RMA them and replaced them with the equivalent helium-filled drives: WD121KFBX.

Keen-eyed readers will note that that product code is different from the one I mentioned above. In my excitement to have found helium-filled drives, I did not note that I was still looking at WD Red Pro, rather than the WD Red Plus that I was actually aiming for. I didn’t notice this until I wrote this section and found the disagreement between SKUs. I am extremely embarrassed about this whole debacle. In short, buying hard drives is an absolute minefield and you should be very careful. Consult vendor data sheets, and paste SKUs into Newegg or something. Don’t rely on websites having good search features, because they don’t surface all the information one might care about, and it’s very easy to get confused between similarly named brands. Having said that, I do feel like I can fairly confidently recommend the WD120EFBX if you have a use case similar to mine.

While I am excited to find all sorts of ways to use the staggering capacities of modern 3.5” hard drives, I don’t want to rely on them all the time. Even the most demure hard drives are relatively noisy and energy intensive, so I want to use them as little as possible, and I certainly don’t want to boot off them. I am pairing these hard drives with a 1TB NVMe SSD. It is a PCIe Gen 4 drive, which is blazing fast compared to my hard drives. However, I may pin it to Gen 3 if that gives a meaningful power saving. I’m not sure yet, and making this thing as efficient as possible is going to be a future task. I went with the Crucial T500 line, which are well-reviewed but otherwise unremarkable.

Other Components

My first choice processor was the Intel 12400. This is an older midrange CPU with integrated graphics. It’s got great performance for the things I want it to do, with a modern enough stack that I can transcode movies all day long. Unfortunately, my ITX choices came calling, because I just couldn’t find a mini-ITX motherboard for the LGA1700 socket. Not at a reasonable price, anyway.

This ended up pushing me to buy the brand new successor to this chip, skipping forward four whole generations to the Core Ultra 225. Intel rebranded their CPU lineup a couple of generations back, so the names are totally different, but as far as I can tell, this is the same tier of processor. I wasn’t able to find much in the way of reviews of this chip, but it’s a 10-core (6 performance and 4 efficiency) 65W CPU that should be no match for my cooling.

I’m reusing 64GB of DDR5 that I had lying around. This is outrageously overprovisioned for this server, but I had it lying around. I had it lying around as a consequence of my choice of CPU cooler. I decided to use the Noctua L12Sx77 that was previously in my desktop here. I needed to buy a new CPU cooler for this server, and I chose to reconfigure my gaming PC slightly by installing a smaller CPU cooler. This Noctua unit is great, but it requires that I configure my Terra in the maximum CPU side, which dramatically restricts my GPU options. That’s probably the first component I will upgrade in a few years time, and I figured that I may as well make room for that now. I landed on the ID Cooling IS-55 for this, which is great, but my existing RAM wouldn’t fit under it. I bought new RAM, and so now I have 64GB of 6400MT/s RAM I can use here.

Finally, I chose the Corsair RM650E power supply. This is laughably overpowered for what I need. I’ll be surprised if I ever reach 50% of its rated output. But it was reasonably priced, and posts pretty good efficiencies at very low power. Ideally, I would have been able to find a lower power unit, but high end low power PSUs are hard to come by these days.

With that, I’m ready to actually assemble this thing.

The Build

Overall, it was a delight to build in this case. I replaced the stock fans with their Noctua equivalents (2x92mm in the front and 1x140mm in the back). This will help with my ambitions for silence, but my main motivation here was aesthetics. I’m chucking some beige into this because I think it looks sick.

I removed the fan control daughterboard from the back. The stock case fans did not support PWM, but the Noctua replacements do, so running them off my motherboard gives me much better control here. My motherboard only has a single fan header, so I was left with a snarl of y-splitters, but it all worked great.

I did have to install the power supply upside-down. The way the extension cable came in just wasn’t going to work with how close the plug was to the front of the case. This was okay, though there is one less screw connecting the PSU to the bracket. It also means the power supply is exhausting air upwards rather than down and through the vent that is literally in the case for the PSU exhaust. I’m not too worried about this, because [a] as I said I doubt the fan will ever turn on, and [b] it’s blowing that air up into the airflow of the case from front to back.

Apart from those hiccoughs, this build was pretty easy.

After assembling the system, it posted first time, and I installed Fedora 42 Server. I’ll be discussing my choice of operating system in a future update.

Benchmarking

At idle, this thing is blissful. CPU core temperatures are a balmy 28ºC. But I wanted to see how this build scaled thermally and acoustically as load increases. I first ran stress-ng to see sustained CPU load. On a 100% all-cores workload, my temperatures maxed out at 58ºC or so, with which I was delighted. I’ve got thermal headroom here. I don’t know what — if anything — I might want to do with it yet, but it’s nice to have.

After verifying that my new server was at least well-built, I decided to try some actual benchmarks. The Phoronix test suite is a community-built dizzying array of benchmarks for exactly this purpose. After reading enough of the docs to be dangerous, I merrily went ahead and ran phoronix-test-suite benchmark pts/compilation. This is a suite that compiles a lot of popular open source projects. There are a couple of things I learned running these benchmarks:

• Use Tmux for things like this. I started it off without this while at a cafe. This was a huge mistake that I didn’t notice until an hour or so later when I wanted to leave. I had to start the process again.
• Compiling code is kind of broken. Using make -j$(nproc) is not technically thread safe, so my benchmark runs would occasionally fail, and then succeed later. Phoronix will give up if it fails three times, but my GCC test failed twice before it got three successful runs.

With enough attempts and a tmux session, I did eventually get the compilation time test suite to run. If you’re super into it, you can check out the results on OpenBenchmarking. There’s not that much to see, though, as the results land about where I expected: a little faster than an Intel 12400, and a little slower than a Ryzen 9600X. More importantly for me, it was fast enough, very consistent (GCC took about 800s each time), and whisper quiet the entire time.

Did I Beat The Mac Mini?

Short answer: Of course I did!

Longer answer: there are a lot of workloads that a new Mac Mini would probably beat this PC at. I certainly think it would have a real good go at that Phoronix test suite, and it would likely do it with lower power consumption, too. But there are a few key ways that I’m happier where I am now than I would have been with a Mac Mini, and they’re why I chose to build this at all:

• I got to build it. This was a fun Max enrichment activity.
• It has space for built-in storage. For my purposes, I’d need an outboard hard drive enclosure, since Apple charges exorbitant prices for their storage upgrades.
• I can run whatever operating system I like.

If we spec up a Mac Mini, we need to make a few concessions to make it a fair fight. Apple gates 64GB of RAM behind the M4 Pro, and then they charge CA$900 above the base 24GB you get as standard. A Mac Mini with 32GB RAM and 1TB storage sets me back CA$1999. In addition to having less RAM, it also only has 1GbE out the back, which is a further downgrade from the 2.5Gbps LAN that I have previously written (at length) about installing.

On top of that, an external USB 3.2 Gen2 4-bay 3.5” HDD enclosure is listed at CA$249 on Amazon right now. So before we get to any hard drives, CA$2248, plus local taxes.

The total bill of materials for my components was:

• SSD: CA$144.99
• PSU: CA$124.99
• Case: CA$139.99
• CPU: CA$318.65
• Motherboard: CA$279.99

I did bring my own RAM and CPU Cooler, and upgraded the stock fans. The fans cost me CA$71.29. A set of 32GB RAM (to match the Mac) goes for about CA$140, and the CPU came with a cooler that you wouldn’t use, so let’s add CA$48.90 for the cooler I would have used instead.

All up we are looking at CA$1268.80. Relative to the Mac Mini, I’ve saved myself about a thousand bucks. This is not to lampoon Apple (though their prices for storage and memory are outrageous), that little computer is an excellent generalist. But I have a particular set of skills and a particular set of needs, and I love my little enthusiast-oriented server.

2025-09-20

As I have previously written, I recently moved house. Over Christmas, I moved to Canada and began (finally) to live with my partner. Her apartment in Vancouver only had room for a single desk. While I would occasionally borrow it, most of my work this year has been done from a cafe, or a kitchen table. Sometimes, from bed.

With our recent move to Toronto, we made space for a home office (for two) a priority. Laura is beginning a masters programme, and I no longer have an office to travel to and work from. It’s taken a month or so, but I finally have my desk set up and fully appointed. I’d like to take this chance to walk through the gear I have, the choices I’ve made, and delight in going through a few of the sentimental knick-knacks that share this space with me. Without further ado, I present to you, The Setup.

Section 1: Furniture

I bought the Desky Zero. I had a really hard time finding a nice desk that wasn’t a sit-stand desk. I know from experience that I prefer to sit comfortably and take frequent breaks to walk around. Unfortunately, I appear to be in the minority, as it seems that motorised sit-stand desks have well-and-truly taken over the market, and my options were either something cheap and crappy — my last home desk bowed in the middle because it was basically made of cardboard — or ludicrously expensive. Desky sell a fixed height model of their desk, and I opted for the hardwood desktop: white oak, which I chose because it’s a sustainable and durable choice.

I spent a bit of time looking around at office chairs, and the overwhelming recommendation coming out of the hive mind of YouTube and Reddit was that for its price, the Haworth Soji is an excellent choice. I got it in green (this will be a theme).

Lighting

I’m interested in some bias lighting for behind my screens, but I haven’t made a choice on that yet. And while I looked at various monitor light bars, none of these seemed like good fits for me:

• I didn’t want a dingus that sat on my desk. I like all my lights to be controlled through the same home interface.
• I refuse to add AA batteries to my life anymore.
• I wanted configurable temperature.
• I wanted to use a webcam, and all but the priciest options preclude that chance.
• Most monitor light bars seem designed for one landscape monitor. I like a vertical monitor next to my primary.

I figured that a nice diffuse light source would both look nicer, and do a better job of illuminating my space. I ended up treating myself to a genuine Noguchi lamp: The Akari 1N. It wasn’t cheap, but it’s gorgeous. I fell in love with the asymmetry of it. It was handmade out of washi paper and bamboo, and its story brings me joy every time I look at it.

Desk Mat

I wasn’t going to leave a beautiful hardwood desk at the mercy of my own clumsiness and carelessness regarding sharp things and cups. I went through a stunning number of options, but I eventually chose the Terrarium desk mat by Dwarf Factory, in green.

Section 2: Hardware

There are three computers on my desk. Two MacBook Pros, and a Linux workstation that I built earlier this year:

A silver 14” M1 Pro MacBook Pro (2021). Work-issued.

This is perhaps the finest laptop I’ve ever owned. I love it so much that I bought its successor for personal use. This thing is light, performs well, and has a battery that lasts forever. It serves my needs perfectly, and has been a transformative change from the line of laptops it replaced.

Black 14” M3 Pro MacBook Pro (2023). Personal device.

This inherits all of the benefits of my work laptop, and some. It got a bump in RAM (36GB from 32GB, which is a strange amount of memory to me), and more importantly, it is adorned with stickers I’ve collected over the years. A flash of personality that my corporate overlords neither desire nor deserve.

Fractal Terra PC

The first computer I ever built. I’d been on the fence about this for a while. After all, I already had plenty of computer between my two laptops. And while I did want to game, and I did want something with which I could return to Linux, it felt like a lot of work for very little return, especially when I had a PS5.

My move over the holidays gave me the opportunity, and I took it. In my post about using Linux, I told this whole story, so I shan’t tell it again. However, I never shared any specs, so I’ll do that here:

• CPU: AMD Ryzen 7 9700X: I love this thing. It performs only modestly better than the 7700X before it, but it does so with huge efficiency gains, which really help with this case.
• GPU: NVIDIA 4060 Ti 8G (MSI Ventus 2X): This GPU does what I need it to do, but was probably a mistake in the overall build. In hindsight, I should have purchased the AMD 9070 XT, or at least the variant of this card with 16GB of VRAM. I was at the end of the 40-series and cards were hard to get hold of, and I didn’t have the understanding that this build gave me. Still, I can’t justify upgrading right now, so this knowledge will have to be happy serving me in the future.

The other less interesting features include 2 2TB WD Black SN850X M.2 drives, the ASUS ROG Strix B650E-I motherboard, an ID Cooling IS-55 CPU cooler, and 64GB of DDR5 RAM. I had to make sure I got RAM that was small enough to fit under the cooler. I also replaced the CPU cooler fan with a Noctua 15mm fan, and placed another one of those at the bottom of the case under the power supply.

This whole PC runs Fedora. For software development, writing, study, and everything else like that, I use Sway. It works great with my dual monitors, fractional scaling, and everything else (despite the fact I have to start it with the --unsupported-gpu option). For gaming, I use XFCE4, and have the second monitor disabled.

Connectivity

My Macs connect to a Caldigit TS3+ with a single cable. This connects to ethernet, my two monitors, and a USB KVM that manages peripherals. The PC connects to all of these things directly.

Monitors

Speaking of monitors, there are two of them. I’m unable to use multiple monitors unless they are exactly the same. These are Dell 4K 27” panels. Not the ones with USB-C. My dock goes to the HDMI inputs and runs at 60Hz, while my PC gets the DP and can drive them at 120Hz, which is amazing. I notice it for sure when browsing, but especially when gaming.

Both monitors are on individual arms, also from Desky. I am suspicious of dual arms, mostly because I have a number of thick cables, and the cable management systems in most monitor arms barely support one monitor, let alone two.

Peripherals

Mouse

My mouse for the last forever has been the MX Master. Whenever I need a mouse, I get one, and currently I have the MX Master 3S. Fortunately, its 3 bluetooth modes map onto each of my three computers.

Keyboards

At present, my desk holds two keyboards. This is admittedly probably one too many. In the past I lived alone and worked in an office. At home, I had the clicky keyboard of my dreams, while at the office I kept a much more demure silent tactile configuration.

I built both of them myself. The lighter one is a NuPhy Gem 80, fitted with Kailh Box Navy switches. I love them, but if I had my time again, I’d likely go with the slightly softer Jades. The black one — my “stealth” keyboard — is a Keychron Q1 Max, and uses Kailh Deep Sea Silent Pro Box Tactile Whales. Both keyboards use the MT3 2048 Extended key caps in light and dark. I adore the texture, profile, and colours in these key caps. Keeping these the same makes switching a breeze.

Section 3: Stationery

For years I have used Studio Neat’s paper products, and their Panopads fit perfectly on my desks. I’m left-handed and I write sideways (the right side of the page is the top for me), so this layout works perfectly for me.

My mother bought me a Lumos Pro for Christmas. I loved it, and very quickly supplemented it with a Lumos Duo for some extra colours. I’m currently rocking black in the primary pen, and then green and red in the Duo. I use the 3mm writing tips, and these are a delight to look at, to write with, and to interact with generally. They have a tendency to roll (hence the rollstop on one of them; alas the Duo doesn’t really have space for one). These pens are the real deal. My partner has one, as do two of my best friends.

Section 4: Other Stuff

Some might call my desk “cluttered”. I prefer to call it “alive”. Everything here has a story, and it makes me happy. Most of the items are gifts, or souvenirs, that I’m going to self-indulgently itemise here.

Django and Ditto

To the right of my desk sits Django the teddy. When I was born, a particularly crafty aunt resurrected her grandmother’s tradition of handmaking teddy bears for grandchildren. She has one (original, and the prototype of Django), and my dad has one. He’s over 30 years old now, and he’s lost his clothes. He long since retired from bed duty, and now watches over my desk.

He is holding an embroidery that my dear friend Monica made for me, for my 29th birthday. Between his legs sits the cork from a magnum of Ruinart that my friends and I shared for my 30th birthday, as well as a fake flower that was a gift from a bartender on a night out in Washington DC with another dear friend (and colleague!) Peter.

To his left are a pair of clogs that were a gift from a trip Monica took to the Netherlands. And to his right is a hand crocheted cactus. This is the creation of a younger cousin (and daughter of the aunt who made Django). Next to the cactus is a Ditto made out of polymer clay. This I owe to another close friend of mine, Sam. She has a matching blue one, an in-joke about how we first met.

The Quarry (and Garage)

As well as being talented artists, Sam and Monica are both avid collectors and enjoyers of rocks and sea glass, respectively. Sitting on the felt mat are an assorted collection of such items, gifted at various times.

Alongside the geological oddities sit a few of my toy car collection. In my last house, I treated myself to one of those rugs that have a map of a pretend town on them. When I travelled to places, I would bring back a toy car for it. Really indulging my inner child there.

The rubbish truck (named so for its role, not its quality) is from a child’s toy shop in Washington DC. The wooden car (it’s abstract) on the left is something I bought from a local artist’s shop while visiting Melbourne (rather aptly for the Grand Prix). And the tiny car was a gift from Monica and our mutual friend Ash.

Finally, the cube is a fidget toy that was a Christmas gift from my oldest sibling. They have something of a collection of them, and when I visited home last Christmas, they knew it was a slam dunk. Of all the items on my desk, this is probably the one I touch the most (excluding my keyboards and mouse).

The Sunbathers

Underneath my lamp you’ll find two little guys. They don’t have names, but again were gifts from Monica and Sam. The German character from Monica’s European adventures, and the panda a birthday gift from Sam last year. His name is Maran, and he was made by The Lonely Animal Friendship Society. The Tempest tag also belongs to him.

2025-09-03

You’d think I’d’ve learned my lesson by now that beginning series of blog posts is a mistake, because it puts the onus on me to finish something off. I’ve achieved plenty in my life, but I’ve never achieved more than I would have otherwise by putting pressure on myself to “finish something off”. Evidently I have not learned my lesson, but I promise it’s going to be different this time!

I am on a mission to de-cloud my life. I pay too much money on an ongoing basis to companies I despise to do despicable things with my data while degrading the quality of the original service that led me to become a customer in the first place. As I wrote in my piece about Linux, I’ve really hit a wall with this stuff recently, and my computing tastes are changing drastically, and are doing so quickly.

I have the good fortune and the privilege to be able to significantly distance myself from online services by building my own server to run in my own home, to be used by me and my family. Not everyone can do this, and I think few people should. In an ideal world, online services would be public goods rather than privately-owned walled gardens. This is not a particularly original take. You can read about it in many places online. For instance, I found myself nodding along with Drew Lyton’s recent blog post on the topic: “The Future is NOT Self-Hosted”. Like the author, I do not think this is a good thing for everybody to do. However, as Drew also writes:

But if you really value control, freedom and flexibility — or you’re a software engineer that suffers from mild mania and hyperfixation — it can be a pretty fun time.

Yeah. I’m going to write all about self-hosting. The trials and tribulations, the fun things and the less-fun things. But not in this blog post.

This is actually a blog post about network cables. I’m not building a server yet, I need another month or so to acquire all the things I need. But I did just move house (my partner is starting a masters programme in a new city). We moved in the other day, and I need to configure our local network. I want it to be fast, I want what can be wired to be wired, and I want it to be able to support a home server. I haven’t had wired internet since I was using dial-up, so this is all quite the novelty. We moved into a pretty new building, and this comes with a couple of benefits:

1. Fibre to the unit, and
2. Ethernet in the walls.

Part 1.1: Internet

I signed up for a fibre plan that is supported by our building. I plugged it in and it all just worked. Great.

Interlude: Ethernet Primer

Before I go through what I actually did here (which isn’t a lot so don’t get too excited), I’d like to run through the highlights of what Max-from-a-week-ago would have considered more than anyone outside of professional networking needed to know about ethernet cables.

To start with the very basics, ethernet cable is cable over which data is transferred. Typically this is for networking, but ethernet cable is used for all sorts of data transfer, including access control, telephone cables, video, or any other signal. Ethernet cable always seems to be a set of twisted pairs, and comes in shielded or unshielded varieties.

Shielding? Twisted pairs? Transmitting anything over a physical medium involves introducing some amount of interference, or noise. This is caused by resistance in the medium, as well as external electromagnetic waves that hit the cable. Shielded cables include a foil around them that attenuates incoming electromagnetic waves to increase signal fidelity. This protects from interference from things like electrical current, fluorescent lights, air conditioners, and almost anything else that does stuff with wires. Shielding has its downsides: it’s stiffer and thicker, making it more difficult to work with, and it costs more. The cable I have in my walls, and in my patch cables, is unshielded, which is apparently the standard. I have seen shielded cabling like this on site at work, though.

Twisted pairs are another technique to reduce interference along a cable run. This is an Alexander Graham Bell invention, and the Wikipedia page goes into a lot more detail than I can here and is genuinely fascinating reading (It also discusses standards for shielding in some detail, which I also recommend reading. Then you too can know the difference between F/UTP and S/FTP.) The gist of it is that twisting a pair of wires equalises the average distance of each component from a source of interference, and sending the same signal down each component allows the error to be isolated at the end. It’s something like this:

We aren’t even close to done with the complexity inherent in ethernet cabling. Next up is the actual materials used in the wires. Ethernet can be either solid core or stranded core. That is to say that each component wire can be made of a single solid piece of material, or a lot of strands of a material. In ethernet cables, the material in question is always copper. Solid core cables experience less signal loss, but are also harder to bend, harder to stuff into RJ45 plugs, and easier to break. Stranded cables are the opposite side of those compromises. Generally speaking, solid core cables are the ones that are in wires, and get punched (technical term) into keystone jacks, which is the thing you plug an ethernet cable into. And the ethernet cables that you plug in (I think they are called patch cables) are usually stranded.

Ethernet cables are categorised into numbers, each representing different standards. These standards specify things like the materials, the number of pairs, the frequency of the twists, and then the throughput capacities that are supported, and the distances over which those capacities are supported. For example, Category (hereafter Cat) 5e (e for enhanced) was standardised in 2001, and guarantees at least gigabit speeds, and over short enough runs, is good enough for 2.5Gbps. Cat 6 is backwards compatible with Cat 5e, but it has stricter standards, and cannot run for as long. In return, you get support for 10Gbps. Again, each category supports both solid and stranded core, and offers different options for shielding.

Crossover Cables

I had heard at some point in my life that the pins of ethernet cables were configured for transmission or reception, and that when connecting nodes, these pins need to be swapped over to ensure that the transmission pin at the source is connected to a receiving pin at the destination. This sounded nightmarish and dizzying. Fortunately for me, a feature called auto MDI-X (Medium-Dependent Interface-Crossover) makes this basically a non-issue in home networking.

After that I stopped caring. Hopefully that doesn’t come back to bite me. Anyway, back to the main thread.

Part 1.2: Network Cabling

When I was inspecting what is now my new apartment, I found two ethernet ports: one in the main bedroom, and one in the living room. I would have preferred one in the second bedroom, since that’s what we are going to use as a shared office, but such is life. An added bonus here is that our Apple TV and Nintendo Switch dock can plug into the internet as well. I probably wouldn’t have bothered otherwise.

Measuring it out, I can have a 10ft ethernet cable from the living room drop to where a TV should go. Then I’ll need a switch, and then it will continue on to the second bedroom, which is just behind the living room. Then I’ll need a second switch to fan out to four other devices.

But what kind of switch should I get?

Interlude: Network Switches

Oh boy. I thought cables were complex. So you can’t just splice ethernet cable. The transmission requirements and the razor thin tolerances in the signal strength prohibit that. A switch is a powered piece of hardware that receives something on one of its ports, and retransmits it to one or more of its other ports.

Switches come in two variants: managed, and unmanaged.

A managed switch is the easier one to understand. A managed switch is a computer that has rules about what is allowed to be retransmitted, and the patterns in which that retransmission can happen. They can prioritise traffic, or indeed do pretty much anything else you can imagine doing with signal processing and a computer.

Unmanaged switches do something simpler, yet also less obvious, at least to me. My understanding is that it maintains an index of MAC addresses mapped to the port on which they reside, or at least the port that points towards them. If the switch doesn’t know how to route a particular request, it broadcasts it, and records the port on which it receives a response.

MAC addresses are the address structures at the link layer, as opposed to IP addresses, which are the addresses at the internet layer. They are lower level, less structured, and less abstract. In practice, the routing that the switch needs to do to find out where a particular device is located happens early during the networking relationship. Suppose a device wants to connect to another device on the same network: typically, it will send a request to an IP address. But to get to its destination, that IP address needs to be resolved into a MAC address. That happens with an ARP (Address Resolution Protocol) request. This is what the switch intercepts.

I am only running a home network. Nothing is exposed to the public internet, and all devices on it are implicitly trusted. Unmanaged switches are fine by me.

Switches also come in a variety of speeds. I didn’t want to bottleneck anything, but I’m also on some kind of budget. I chose 2.5Gbps, because 1Gbps was insufficient, and 10Gbps was (a) overkill, and (b) costly. Our two PCs only have 2.5Gbps ethernet, my Caldigit TS3+ dock has 1Gbps ethernet, and there’s only Cat 5e in the walls. This way I can keep my prices reasonable, and not bottleneck my internet.

I went with the TP-Link TL-SG105S-M2. While it doesn’t have a snappy name, it makes up for that by being a reasonably-priced, reasonably-sized five-port 2.5Gbps capable unmanaged switch. To be on the safe side, all my patch cables are Cat 6, and I think I’m going to plug my future home server into my fibre modem.

Taking a look in the network cabinet, there was a mixture of good news and bad news. The good news is that I could see two Cat 5e cables coming down. The bad news was that they weren’t really connected to anything I’d ever seen before.

With some reverse image searching, I learned that this is apparently a 66 punch-down block. These are designed for telephone cables (as pictured, how quaint), but builders of the era sometimes parked ethernet cables in them when there was no immediate use for them, but wanted the wires in the walls. Works out well for me.

I also learned a lot of other things. In no particular order:

• I can’t reasonably use that panel.
• Ethernet cable that goes in walls is solid core and not very flexible, whereas the patch cables we use for devices are stranded. Stranded patch cables tend to be crimped into RJ45 connectors, which is what I have always called an ethernet port. Solid core internal cables do not. Solid core ethernet cables are instead more commonly terminated into the “port” version of that connector, which is called a keystone jack. (I refuse to call them “female”)
• These aren’t just normal push-through or screw-in terminal blocks. You need a special tool called a punch-down tool.

I was a little nervous about playing in this space because:

1. I rent, and this is not my infrastructure. However, I’m going to do an excellent job, and I actually unwrapped this 66 block, indicating I’m the first occupant of this apartment ever to use it. Furthermore, I’m providing a legitimately useful and obvious improvement here.
2. I’ve never done any of this before, and I’m worried I’ll do it wrong. I’ve solved this by reading a lot and watching a lot of YouTube videos.

So I went onto the internet and I purchased a well-reviewed punch-down tool, some 1ft Cat 6 patch cables, and a dual-port keystone jack (I actually purchased five of these, because they came in packs of five. This stuff is really not designed for hobbyists).

Look out world, here I come.

The bottom one pictured is actually the second one I did. I was trying to match the length of the second one for its placement in the panel, and I didn’t quite get it right. But once the cover is on, you can’t really tell. I considered re-terminating it, but I wanted to test it first.

Part 1.3: Testing

Testing proved to be somewhat challenging! Our gaming PCs both have 2.5Gbps networking, but they are temporarily out of action owing to the move. I had a Caldigit TS3+ which supports 1Gbps ethernet, but lugging that around my house attached to my laptop, and attached to its power brick is quite the palaver. I ended up buying this Belkin dongle and plugging in one of the 1ft patch cables I got into it, and bam! Mobile testing setup.

I had planned to attach this to a laptop and walk around the house, but as I looked at it, I had the devilish idea to try my phone instead. And it works great! As soon as you plug it in, a new “Ethernet” option appears in the Settings app.

Starting simple, I plugged this directly into my router. Nothing happened. I turned off Wi-Fi, and suddenly I was given an IP address. I ran a speed test, and things were pretty good! Not much better than Wi-Fi in this exact location, but less than 30cm away from your access point is pretty close to ideal conditions. Great start.

Moving on, I plugged it into the two wall-based ethernet ports. iOS is a bit weird when you move this thing around, I had stale connections until I removed the dongle from the phone side as well, but I worked into a pretty quick routine. I fiddled with the router a little bit — removing the patch cables from the back to identify the cable drops — so that I could label them (you’re welcome, future nerd resident). Once I’d had this all tidied up, I moved onto testing at the switches. And these worked perfectly! From the office, the speed at the switch was a considerable upgrade over Wi-Fi.

Now it’s all ready for everything to be really actually set up, with desks and lights and all of those things. I am giddy and excited in ways becoming of maybe a six year old. I can’t wait.

2025-08-02

This is the story of how I have ended up on Linux for a large amount of my computer use, and all of my gaming. It’s a personal story, but I do genuinely believe that the carrot of really great gaming compatibility on Linux in 2025, along with the ever larger sticks that large tech companies are wielding against their users is fomenting an environment that makes this kind of change attractive. This is not a “How To”, this is a story.

I’ve only ever owned MacBooks as personal computers. My first was the aluminium unibody MacBook in 2008, followed by a 13” retina MacBook Pro in 2013, and an M3 Pro 14” MacBook Pro that I bought in 2023. Interspersed between that, I’ve had a collection of work computers, all of which have been Macs. I know the operating system really well at this point. I like it. I like my collection of indie apps. However, a need for digital sovereignty has been growing within me in the last couple of years, and there is one real area that macOS doesn’t excel in as a general purpose computing platform: games.

I’ve been a lifelong PlayStation kid. I’ve owned (or had a parent who owned) every single PlayStation iteration. I was never on the bleeding edge, I loved the slim versions. The ones that came out a couple of years after the mainline release, and packed the same performance into a much smaller frame. I took my PS2 slim around with me everywhere. I moved my PS4 slim across the world. Then I bought the PS5 slim.

Have you seen the size of that thing? Moreover, a pretty serious knock to my wallet for a single purpose toy.

Enough was enough. As I began my third international move, I resolved to sell my PlayStation 5 Slim (which I could not reasonably fit in a suitcase) and not repurchase it. A huge factor in this decision was that Sony makes it impossible to move an account to a new region. I have games on my British PSN account, and I have games on my Australian one. To continue using this platform, I would need to create a third account, a Canadian one, to be able to continue purchasing things. I have been unable for years to play online, because I cannot attach a working card to my British account. So I’m losing those games. At some point, you just have to accept that you’re losing access to content that you are supposed to own, because the gatekeepers are so actively hostile to their users. You’ve got to make a principled stand some time.

So what now?

A custom gaming PC. I built a PC! Just for games! It’s amazing, I can play the original Witcher games, and I can play the Final Fantasy VII Remake, and I can play just a slew of other titles that were previously inaccessible to me. It’s been a cornucopia of PC-exclusive games for me to play. The fun doesn’t stop there, either. It seems like Sony recently started publishing their PlayStation exclusives onto Steam. I’ve been able to set up PCSX2 and have several very fun strolls down memory lane as I play Sly Cooper or Gran Turismo 3 (still the best one).

It works great. It’s perfect and I love it and it’s small and cool and fast and cute and I learned a lot about computers while I made it. It worked great. Except for one thing: I installed Windows on it. I had to! I wanted to play those PC exclusives, and that’s the way you play them. I accounted for that. I did not account for how hateful Windows is.

Windows 11 is truly horrendous. I couldn’t uninstall any of the crapware it came bundled with. Or rather I could, but they came back with every update, no matter how minor. I think I disabled all of the flags to show me ads on my desktop, on my lock screen, in my start menu, but I really had to hunt. I initially installed Windows without signing into a Microsoft account. That worked perfectly until I signed into Minecraft and it took my entire computer with it. Perhaps I’m the problem. Perhaps I’m holding it wrong. But at every turn, it felt like the operating system was fighting me, like it wasn’t for me.

At the same time, I was consistently hearing the same thing from a number of different sources:

Try Linux!

So I tried Linux. And everything just worked. Thanks to the Steam Deck, you can play most games just fine on Linux. So I installed Fedora 42 KDE on my second drive and I copied everything over and would you believe that it all worked perfectly? Every single game runs just as well as it did on Windows. I’m truly stunned.

To jump back into the real world for just a second, there are two caveats that I feel the need to point out:

• I don’t play any competitive multiplayer games. I suck. Not for any other reason, but it means that I don’t have to worry about kernel level anticheat and things like that. If you suck less, or care less about sucking, this will potentially be an issue for you.
• There is one issue that I’ve found in Timberborn: I use a Logitech MX Master 3S, and had configured the back/forward buttons to rotate the camera, and my thumb scroller to raise and lower the camera. In Linux, under Proton, Timberborn interprets these mouse buttons as the same. My operating system distinguishes them perfectly, and so did Windows. I’ve tried fiddling with Steam input settings, and I truly have no idea what’s going on or what levers I can pull. My hope is that this bug is obsoleted by Timberborn someday supporting the standard “move around in 3d by holding the middle button down”.

This probably isn’t too much of a surprise to anyone who has read even one other article on my blog, but gaming is not all I do on a computer. One of the appeals of building a PC was that I would have a general purpose workstation: something I could use to develop on, something I could use to develop things that I couldn’t develop on a Mac! It turns out that Linux does everything I want to do in this arena better than Windows as well, all without making me curse technology and everything it has wrought.

So I use Linux now. Not full time, because I have my MacBook, but damn close to it for everything I sit at a desk for. For this to work out as splendidly as it did, a few things had to line up for me:

• I have been growing increasingly frustrated and angry at the toll that large tech platforms are taking on my life, and the lives of the people around me.
• I am similarly increasingly keen to avoid the privately-owned algorithms that govern so many of our digital experiences.
• Linux is in a place where it is viable to use it to do all the things I want to do on a computer.
• I had computing hardware that supports Linux well! I’ve run Linux before, and it’s especially tricky now in the wake of Apple Silicon, but using Linux on Macs has always been Linux on hard mode.

I will have a lot more to say in the future about all three of those points, because I have projects underway to reclaim control over my digital presence, to be more deliberate about how I engage with computers, and all of the exciting things I can do now that I have a computer that just runs Linux things with minimal hassle.

On that last part, I’m truly off to the races now. I’m back into tiling window managers, I’ve disabled graphical logins, and I’m charting a course that moves me away from Fedora and onto something where I have even more control over my platform. Right now, I’m evaluating Gentoo. You can expect to see more edits to my dotfiles as I continue on this journey.

It is such a joy to be excited about computers again. It is such a surprise to find myself wanting to play, explore, and experiment. There is utter delight here for me, and I am so grateful that these initiatives exist.

2025-07-26

My body is a mason jar
transparent as a jellyfish
I wish for a heart you can see straight through
for a voice that glows in the dark
and a few really good friends to skip moon rocks to.

From Jellyfish, by Andrea Gibson.

A few weeks ago, Andrea Gibson passed away. Their work touched the lives of many, many people. Today, I want to dedicate a few words to how their work touched my life in particular. It feels denigrating somehow to talk about someone’s art as only their work, when true art demands we place our humanity on display. And so I suppose this is less a discussion of how Andrea Gibson’s work has affected me, but rather how the aspects of their life they chose to share with the world have affected me. And boy howdy, have they affected me profoundly. In strange, surprising, beautiful and delightful ways.

The first Andrea Gibson poem I ever heard was Jellyfish, on 20th September 2012. The second was I Do, later that night. I was 18. I was about to go to university. I was wide-eyed, full of joy, and utterly, perhaps blissfully unaware of what was coming my way. Both of these poems showed me a transformative way to love. To love recklessly and generously, loudly and proudly. But not to love naïvely. To love with one’s eyes as open as one’s heart. It is in that spirit that I write this tribute.

It’s tempting in many ways to look back at the summer of 2012 as a peak, a local maximum between the general ordeal of being a teenager, and the abject isolation of 2013–17. Tempting, I think, but harmful. Andrea Gibson’s poetry helped me see some of the deepest challenges in my life as parts of a story. They did perhaps more than anyone outside of my immediate friends and family to help me integrate the more upsetting experiences of my life into one cohesive Max. These lessons happened over years, in countless poems. Below, I am going to share a few that left particular impacts.

In Rush Hour, Gibson taught me how better to hold an open heart in a closed and callous world; how to have hope when anything we feel we can do is rendered meaningless in the face of such overwhelming cruelty:

In the city where I live
They’ve officially labeled panhandling on medians a crime
because no one wants to drive by and see that shit

…

I remember being five years old
Watching a feather fall from the sky
and praying all day that the bird that lost it could still fly

…

So last night I freed a kite to the wind in hopes of it finding Palestine.
It won’t.
But I’m trying to remember who I used to be.

…

Tomorrow 600,000 feathers are going to fall from the sky.
There’s no way we’re going to catch them all but somebody tell me
We are at least going to try.

In The Nutritionist, Gibson taught me that to suffer is not to be alone; that there are safe spaces in which to share our pain, even in a world that demands of us our best presentation; that it is incumbent upon all of us to build those safe spaces for each other; that we do not necessarily need to be fixed:

So let me tell you I know there are days
it looks like the whole world is dancing in the streets
when you break down like the doors of the looted buildings
You are not alone and wondering who will be convicted
of the crime of insisting you keep loading your grief
into the chamber of your shame
You are not weak just because your heart feels so heavy

if the only thing we have to gain in staying is each other,

my god that’s plenty

my god that’s enough
my god that is so so much for the light to give
each of us at each other’s backs whispering over and over and over

“Live”
“Live”
“Live”

To be taught that we do not necessarily need to be fixed has been something I’ve had to have reinforced time and time again, while having the good fortune to be able to bask in the love of my friends, and my family. I wish it was not such a powerful thing to preach. I wish Andra Gibson had not been so radical. I wish their love was uncontroversial, and regular. But that isn’t our world, yet. If my words have introduced Andrea Gibson to anyone, I hope that their words in turn can teach you something about a radical, hopeful, uncompromising kind of love.

On the day that I met my partner for the first time, she cited an Andrea Gibson poem to me. One of those weird things that makes you feel connected to someone, I guess. The magic of both of us finding that connection in that moment is something that still brings a tear to my eye. The poem Laura had been thinking about was Tincture:

Imagine, when a human dies,
the soul misses the body,
actually grieves the loss of its hands and all they could hold.
Misses the throat closing shy reading out loud on the first day of school.
Imagine the soul misses the stubbed toe, the loose tooth, the funny bone.

…

Nothing in space can imagine it.
No comet, no nebula, no ray of light can fathom the landscape of awe,
the heat of shame.
The fingertips pulling the first gray hair and throwing it away.

I can’t imagine it, the stars say.
Tell us again about goosebumps.
Tell us again about pain.

Andrea Gibson’s poetry has helped me learn how to live, and how to love, and how to grieve. Most of all, I’ve learned how to experience with patience and reverence. They transformed their life, with all their love and pain and sickness and grief, and more love into a gift. Something for all of us.

2025-02-23

Last year, I needed a queue to implement a breadth-first search for an advent of code problem. I was using Elixir, and I wasn’t sure how to implement a queue myself. Luckily, Erlang provides queue, and I was off to the races. I’ve been wondering off and on since then exactly how it works, since the data structure for it was both transparent and also bizarre (to me). So I’m going to read the source and notate it here. Erlang is not a language I know, so hopefully it makes sense to me. We are all learning here.

Basic Queueing Operations

The first thing that we see from the constructor is that a queue is a tuple of two lists. This is further backed up by is_queue/1:

new() -> {[],[]}. %{RearList,FrontList}

%% ...

is_queue({R,F}) when is_list(R), is_list(F) ->
    true;
is_queue(_) ->
    false.

Similar to Elixir, Erlang supports and encourages branching behaviour based on different argument patterns. Essentially, the first clause of the function returns true, but only when the supplied argument is a tuple of two lists. If it is, it is a valid representation of a queue in Erlang! The second clause, with the underscore argument, matches everything else. In general, when set of parameters match multiple function definitions, the first one is taken. I find this an enjoyable and very expressive form of programming. I love me some pattern matching, truly.

We also get a clue in here that the first list is for the rear, and the second one for the front. Just below this, is_empty/1 also confirms that {[], []} is the empty queue.

len/1 shows us that there is no duplication in either queue, meaning it’s memory efficient. If there were duplication, the count of items would not be simply the sum of each list:

len({R,F}) when is_list(R), is_list(F) ->
    length(R)+length(F);
len(Q) ->
    erlang:error(badarg, [Q]).

An example about using to_list/1 shows us the usage of its counterpart from_list/1, and also that the correct way to represent [1, 2, 3, 4, 5] is {[5, 4, 3], [1, 2]}. The big question is “Why?”.

queue:from_list([1,2,3,4,5]).
%% {[5,4,3],[1,2]}

to_list/1 goes on to show that for any queue, its list representation is the second part followed by the first part, backwards:

to_list({In,Out}) when is_list(In), is_list(Out) ->
    Out++lists:reverse(In, []);

Things get really interesting when we look at the source its counterpart from_list/1:

from_list(L) when is_list(L) ->
    f2r(L);

What the hell is f2r/1? Right down at the bottom of the file, there are two internal functions, f2r/1 and r2f/1. They are documented in the code, but come under a header “Internal workers”, and are nowhere to be found in the documentation. They appear relatively simple. Considering a queue {R, F}, where R and F denote Rear and Front lists respectively, f2r/1 moves half of the elements from F to R. Conversely, r2f/1 moves half of the elements of R into F.

Here they are, in their entirety:

%% Move half of elements from R to F, if there are at least three
r2f([]) ->
    {[],[]};
r2f([_]=R) ->
    {[],R};
r2f([Y,X]) ->
    {[Y],[X]};
r2f(List) ->
    {RR,FF} = lists:split(length(List) div 2, List),
    {RR,lists:reverse(FF, [])}.

%% Move half of elements from F to R, if there are enough
f2r([]) ->
    {[],[]};
f2r([_]=F) ->
    {F,[]};
f2r([X,Y]) ->
    {[Y],[X]};
f2r(List) ->
    {FF,RR} = lists:split(length(List) div 2, List),
    {lists:reverse(RR, []),FF}.

So these functions will balance the two ends of the queue for us when they are called. One interesting feature of these two functions is that they take a list, rather than a queue, but return a queue. Meaning that they should only be called when their destination list is empty.

Reading these, it makes sense to me now why R is reversed: prepending to a list is O(1), meaning that enqueueing is similarly cheap. However, if we push n items into our queue, dequeueing them will be O(n). Hence, at some point we wish to rebalance our queue so that ripping items off the front of F is cheap. By reading these functions, it seems that this operation is performed when we run out of items to pop from F, and in that case we take half of the items from R and put them into F. This means, that if a queue contains n items and we drain it, we can expect r2f/1 to be called roughly log(n) times. I now understand the documentation’s meaning when it states that these operations “have an amortized O(1) running time”. Every so often, you get lumped with a worst case of O(n). As long as this is rare, it is not a problem.

There are a few other interesting things about this code:

• I’m pretty sure that f2r/1 is merely a relabelling of r2f/1 with the order of the tuple swapped. I probably would have written it like that.
• If the list has one item only, it is swapped to the other side. That is to say that f2r/1 puts it in R, and r2f/1 puts it in F.

It’s relatively clear to me that we force ourselves to rebalance the queue if we run out of items at either end (out/1 will call r2f/1, and out_r will call f2r/1 to pull from the back. But when else? My instinct is that adding items will never rebalance the queue, because we cannot anticipate how a queue will be drained until the client starts draining it. However, there clearly is some general goal of keeping things ready for anything, since from_list/1 calls f2r/1 to rebalance on creation. That’s how we even got here. Let’s take a look at in/2 to see:

in(X, {[_]=In,[]}) ->
    {[X], In};
in(X, {In,Out}) when is_list(In), is_list(Out) ->
    {[X|In],Out};
in(X, Q) ->
    erlang:error(badarg, [X,Q]).

Well it’s true that it doesn’t call f2r/1, and I would guess that in_r/2 behaves similarly. There are a couple of interesting clauses here, though:

1. The first deals with a queue with only a single item in its rear item, such as a queue that would be generated by creating an empty queue and adding a single item. In this case, the first item is moved to F, and the new item is added to R, leaving the queue balanced.
2. Otherwise, we simply add the new item to the front of R, which represents the back of the queue.

So, if we append 1000 items to an empty queue, we would end up with one item in F, and 999 items in R.

If we turn our attention to out/1, we don’t see anything particularly surprising:

out({[],[]}=Q) ->
    {empty,Q};
out({[V],[]}) ->
    {{value,V},{[],[]}};
out({[Y|In],[]}) ->
    [V|Out] = lists:reverse(In, []),
    {{value,V},{[Y],Out}};
out({In,[V]}) when is_list(In) ->
    {{value,V},r2f(In)};
out({In,[V|Out]}) when is_list(In) ->
    {{value,V},{In,Out}};
out(Q) ->
    erlang:error(badarg, [Q]).

The return value here is a 2-tuple where the first value is either an atom empty, or a tuple with the atom value and the returned value, and the second item is the new queue. Atoms are unique symbols that exist in the Erlang runtime, string constants that are used kind of like global enum values.

My guess above was that r2f/1 would only be called by out/1 iff F was empty. That is not the case. The third clause deals with an empty F, and it does so be emptying all but one element of R into F. However, r2f/1 is called, it is just called when F has 1 element in it, rather than 0. Although, when F has a single item, the queue returned thereafter would indeed have an empty F, so I wasn’t too far off.

As an interesting side-quest, it is not immediately clear to me how we would ever get to the case where F is empty and R has values in it through only these two functions. I know that there are advanced functions for deletions and filtering and mapping and suchlike, so maybe this will be revealed to me later on.

This covers basic queueing! We have in/2, out/1, and even in_r/2 and out_r/1, all powered by an ordered pair of lists, that are kinda balanced out sometimes. But there is more to this module. A lot more, in fact. There are the basic iteration functions, things like all/1, and fold/3. These seem relatively simple to implement, though I wonder if they are converted to a list, or two iterations are run. My bet is the latter. Then, there are the basic functions that seem harder to implement to me. Things like split, join, and delete. I feel like these require some amount of rebalancing and reversing things. Finally, the documentation contains an “Okasaki” API, which links to a 1996 paper (book? It’s 130-odd pages) called “Purely Functional Data Structures”. This is up my alley, but will not be covered in this post. I am already reading a book right now. So, let’s look at splitting and joining.

Splitting

(In the following examples, I am going to omit the degenerate cases that handle errors and empty inputs. They are not algorithmically relevant.)

The API of split doesn’t just halve the queue, it puts the first N items into the first queue, and the remainder into the second queue. As I suspected, the code is a little gnarly, certainly more so than anything hitherto presented:

split(N, {R,F}=Q) when is_integer(N), N >= 1, is_list(R), is_list(F) ->
    Lf = erlang:length(F),
    if  N < Lf -> % Lf >= 2
            [X|F1] = F,
            split_f1_to_r2(N-1, R, F1, [], [X]);
        N > Lf ->
            Lr = length(R),
            M = Lr - (N-Lf),
            if  M < 0 ->
                    erlang:error(badarg, [N,Q]);
                M > 0 ->
                    [X|R1] = R,
                    split_r1_to_f2(M-1, R1, F, [X], []);
                true -> % M == 0
                    {Q,{[],[]}}
            end;
        true -> % N == Lf
            {f2r(F),r2f(R)}
    end;

The first thing is that this is some pretty fun indenting of if statements! The second thing is that this calls two little helper functions that are defined just below:

%% Move N elements from F1 to R2
split_f1_to_r2(0, R1, F1, R2, F2) ->
    {{R2,F2},{R1,F1}};
split_f1_to_r2(N, R1, [X|F1], R2, F2) ->
    split_f1_to_r2(N-1, R1, F1, [X|R2], F2).

%% Move N elements from R1 to F2
split_r1_to_f2(0, R1, F1, R2, F2) ->
    {{R1,F1},{R2,F2}};
split_r1_to_f2(N, [X|R1], F1, R2, F2) ->
    split_r1_to_f2(N-1, R1, F1, R2, [X|F2]).

Now I can actually look at this. This function principally branches on whether or not we can satisfy the first queue solely by pulling elements off the front of F. Namely, that F has at least N elements. There are three possible outcomes:

We have exactly N elements in F

In this case, we take F and R, and return them. We also go to the trouble of rebalancing them.

We have more than N elements in F

So I thought this would be the simple operation of pulling N elements of F, and maybe calling f2r on it. But that’s not what we do here. For reasons surpassing my understanding, we pull one element off the front of F, and pop it into the front of the second queue. Then we grab the remaining N-1 elements, and put them into the rear of the second queue.

This is achieved by that handy helper function split_f1_to_r2/5. This has a very nice recursive definition, but one thing to note is that the base case swaps its arguments around, returning {Q2, Q1}. This is where the swapping is done, ensuring that the front N items go into the front queue.

We have fewer than N elements in F

This means we are going to have to plunder R in order to get to our queue, and that is going to be at least somewhat slow. In this case, we further branch by M. M is rather confusingly written here, I would prefer it be defined as Lr + Lf - N, because double negatives are hard for me. But I’m a big boy and I’m used to not getting what I want at this point.

If M is less than 0, it means we do not have enough elements to move, and we simply bail. This feels cowardly to me, I would probably go for just putting the input queue into the first result, and leaving the second one empty, but I’m not about to come in and tell the standard library authors for a language I don’t even know how to decorate their house. I’m an idiot!

As it happens, though, that is exactly what we do if M is exactly 0, meaning that the input queue contains exactly N items.

In the final case, we actually do some splitting. In this case, we again delegate to a helper function, this time it’s split_r1_to_f2/5. It begins in much the same way, where we take the first element of R and put it into the rear of the second queue, then we ship the remaining M-1 elements into the front of the second queue. Since M is the number of elements left in the queue after removing the first N arguments, the first queue is the first queue according to the API contract, so no swapping occurs.

All of these structures are immutable, but it might be helpful to think of the previous case as moving the first N elements into a new queue, and this case as moving the last M elements into a new queue.

Joining

join({R1,F1}, {R2,F2}) when is_list(R1), is_list(F1), is_list(R2), is_list(F2) ->
    {R2,F1++lists:reverse(R1,F2)};

One thing I’ve learned through this analysis is that there are a number of valid representations for the same notional queue. And nothing throws this idea into quite such sharp relief as this snippet.

This function might seem complicated. It did to me, but it turns out that lists:reverse/2 only reverses the first argument, and just appends the tail.

And with that, I am out of questions about how Erlang’s queue works! There are other functions, and an entire API that I don’t understand (to be fair, it’s the names that make no sense to me, the behaviour seems relatively straightforward). The remaining functions usually involve visiting all elements. If the order is important, that in turn involves reversing one of the lists. In cases where we are searching for a single item and we need to traverse the queue in order (delete/2 et al), then the front portion of the queue will be traversed first, only reversing the rear if needed. Finally, one beautiful feature of this design is that it is totally symmetric. For all relevant functions, there exist _r variants that go from the other end. But what I find particularly pleasing here is that inverting a queue is a simple matter of swapping the front and rear component lists.

2025-02-02

I roll my own colour scheme for any text editor I use. I don’t do this because I’m fussy and no off-the-shelf colour scheme will do (although I am fussy), I do this because I have a very particular manner of highlighting syntax, regardless of the exact colours I’m using. I’ve received a fair bit of inquiry from friends and colleagues about why I do this, and I’ve never seen anyone else do exactly what I do, so I decided I should write it down while I’m thinking about it. (I’m thinking about it because my team and I are trying out Zed for pairing and remote collaboration, so I’ve had recently had to do this dance.) As it happens, I am in fact using my own custom colour scheme as well. I know just enought colour theory to be dangerous, and I had a few specific requirements that other colour schemes weren’t meeting, and I had to do all this anyway, and it’s not that hard. Some might even say it’s fun. But that is a tale for another time. What I want to write about today is not the specific selection of colours I use, but the manner in which I use them.

When I was a kid, I initially learned to code in Notepad. When I first got my hands on some actual editor (I think it was Dreamweaver), I was amazed mostly by syntax highlighting. For the first time, code looked cool. I never understood what the different colours meant, but it did make the entire thing look like a fidget toy, which was extremely up my alley. As I got older, I began to take this coding thing more seriously, and for most of my teenage years I was devouring everything I could find about software development. This included a lot of discussion about tools, and a lot of this really left its mark on me. Sometime around 2009 I started to use Vim for the first time. Some time in 2011 I switched to the Dvorak keyboard layout. I really tried everything. Some of it stuck; some of it didn’t. One thing in particular did take hold in me, though: at some point, I turned off syntax highlighting in Vim.

Exactly when I did this, and exactly what prompted me are sadly lost to the sands of time. I think it was a Damon Conway conference talk about his 1500 line Vimrc, but I could not find such a talk online that predates the initial commit of my Vimrc in my dotfiles, which was over a decade ago. And even by then I had moved on from :syntax off to something more sophisticated. To, as it happens, an early version of the same syntax highlighting scheme that I use to this day. Regardless of where I heard this or who told me, someone advertised that they do not use syntax highlighting, because they find it overwhelming and useless. This resonated with me, because I still didn’t understand what it all meant. So I tried turning it off, and I liked it. But living without syntax highlighting did force me back to first principles, and I ended up pulling a few things back into my life. They haven’t really changed in the last ten years.

Get in the comments

At a similar time, during a debate on the merit of code comments (if this entire blog post were not evidence enough, take it from me directly that I find it difficult to resist discussions as to the colour or colours in which to paint the bikeshed) somebody asked how we are supposed to take comments seriously when our text editors themselves dim them out relative to the background. They are, almost by definition, less unimportant.

This didn’t work on me, of course, my comments looked exactly like the rest of my code. But it did resonate. My general position here is that if somebody goes out of their way to write some prose inside some code, it’s probably important, and so I want to pay attention to it. Moreover, I don’t want to refactor something and miss a comment, left to go stale. In the best case, a stale comment is useless. Somewhere in the middle it is misleading, and in the worst cases can be downright dangerous.

So I made comments green. This was a real bummer for me, because I had to :syntax on, and then I had to build a whole-ass colour scheme in Vim that disabled everything, and then made comments green (ctermfg=2). But I did it, and by then I had eaten from the tree of knowledge of good and evil: syntax highlighting was back on the menu.

A little bit more colour

Since I had the power, it was pretty easy to then make Vim chrome red (status bar, split lines, and so on), and search results blue. The exact colours used here have chopped and changed over the years, but currently search is still blue (it was yellow for a while though), and Vim chrome is purple.

Literally though

I stayed like this for a while, until I had my first (in this story at least) original idea: hardcoding things is bad. There’s nuance here, but in general I consider factoring things into variables good, and not doing so to be vaguely harmful. That’s not the original idea. The original idea was that I could highlight literal expressions in my code, so that a visual scan can find an overuse of them.

Thus, I made scalar expressions cyan. The exact timing of this is not lost to the sands of time, it was almost exactly ten years ago, on 11th January 2015.

Just like that, ten years goes by

And that’s about where we are today. I went from Vim to NeoVim, there was definitely some Jetbrains shenanigans in there. I went from NeoVim to Nova, then to Zed. I liked Zed’s vim mode so much that I switched back to Vim last year, and now I’m somewhere in between Vim and Zed.

But my syntax highlighting, that stays with me. Here it is, in light mode.

2025-01-17

Last year was a pretty big one for me. I turned 30, I rediscovered the joy that I found in programming, and did a lot of other really important growing. As I write this, I am sitting at my mother’s dining table in the UK, over for Christmas in between moving from Sydney, Australia to Vancouver, Canada to live with the love of my life.

I struggle with finishing things off. Really putting a bow on things. At the start of a project, all of my excitement tends to get shovelled into the esoteric details of the thing: one specific mechanism that is particularly tricky, or one particular ornament that is specifically beautiful. This has its pros and cons, of course, and I’ve spent a lot of my career working at keeping the good parts of these tendencies while developing strategies to, you know, actually ever deliver anything. However, this leaves my hobbies in a state of disrepair.

Ever since I really got my life back together in 2018 or so, I’ve thought about — and begun to develop! — several different places to store my writing and share it online. This latest attempt is by far my lowest effort (a nearly default Quarto project) holds a lot of promise for me. So let’s see what happens.

Happy New Year.