Self Hosted

It’s taken about-a-million tiny steps, but as of this moment (2 May, 2024 07:42 UTC-6) this WordPress blog (jjj.blog) and a bunch of other blogs for friends & family are all calling from inside the house running on a cluster of Raspberry Pi 4’s in a server rack in my basement.

Why did I do this?

Aside from the obvious – pride, vanity, education and just-wanting-to – what I really wanted was independence & autonomy. I wanted to break free from paying +/-$100 USD per month in hosting fees for a service that I could provide for nearly $0 (once the network was in place anyways).

I was essentially paying $1200 per year to have a playground of virtual machines to keep 2 of my favorite WordPress plugins stable (WP Multi Network & LudicrousDB) when I could containerize them on my own hardware for nothing.

I wanted to backup & upgrade my servers more easily.

I wanted to own my own infrastructure like I own my own data.

There is a lot that goes into hosting your own website if you want to do it beginning to end, so heeeeeeeere’s:

The current setup

Mail-in-a-Box

I’ve used this for quite a while on a Linode but now it’s running on a dedicated Raspberry Pi 4 with a whopping 8GB of RAM and 1TB SSD. This does a bunch of stuff, but most importantly are Name Servers and Email.

It is responsible for handling all of the necessary DNS, including the authoritative name servers at ns1.jjj.domains and ns2.jjj.domains respectively. All of my domains use these (and the domains of friends I host here).

It also manages all of the ins & outs of email (including alerts) and includes Roundcube which is nice to have for managing accounts that don’t need to be frequently accessed (using IMAP) from a computer or phone.

I use it to easily host some static sites that don’t need WordPress, as well as Nextcloud for slowly weening off of Apple iCloud.

Mail-in-a-Box is mostly python, and doesn’t support ARM64 directly (with the variant of Ubuntu Linux that it requires) so I run a very-slightly modified fork and have contributed a few minor improvements here & there.

Static IP & Reverse DNS

Between March 2018 and January 2024, I ran my own WISP (with help from some friends) so that our neighborhood of about 30 homes could have internet access. That meant I had access to a few static IP addresses, and I used 2 of them to separate the name servers from the web domains being served. (That was an awesome learning experience too that I will blog more about later.)

As of January, our neighborhood now has 1GB up/down symmetrical fiber piped right into our homes from Edge Broadband, and for a little bit extra each month they’ve provided me with a dedicated IP and rDNS to jjj.domains. This took a little bit of work on their end (a new IP address pool, etc…) and it shows (to me) that they understand their customers’ needs are varied & sometimes even a little special.

Because they only offer a single static IP, I have one remaining 1GB Linode (running Alpine Linux) running as a jump server between DNS requests and serving actual content, but that will go away eventually (I hope!)

Unifi Network (Dream Machine Pro SE)

Edge Broadband’s static IP enters the LAN through a UDM Pro SE, and I use some good’ol unsexy port-forwarding to send requests to one of four Raspberry Pi 4’s:

1. Mail-in-a-Box on a bunch of ports
2. GitLab on a few ports (at jjj.software, more on this below)
3. WordPress 1 on a fancy hidden port (at a bunch of domains)
4. WordPress 2 on a different fancy hidden port (at other domains)

There are some firewall rules here to prevent subnets from reaching each other, as well as each Pi running its own instance of UncomplicatedFireWall which I’ll touch more on later too.

I’m a big fan of Unifi gear. We use(d) it for our WISP and our house for WiFi, and I have a separate UNVR Pro running Protect and a UNVR4 setup for network attached storage.

Unifi Network has become an extremely cool piece of software. In the past 5 years they’ve added a ton of pro’ish features while managing to keep the interface feeling light & fast. It’s possible to run it virtually (or on your own hardware) but their hardware is better than anything I’d configure, plus it’s nice not to really need to think about that at all.

Raspberry Pi 4’s

Yes, I’m sure I’ll get 5’s eventually, but they aren’t really necessary yet, at least not for WordPress or the web traffic I normally see.

I have 4 of them side-by-side in a single rack-mounted unit with a total of 6 available. It’s an early-access-only version of what is now Ubiquiti’s Toolless Mini Rack that was never available to the general public for sale, and I also have a dedicated PDU and a 16 port PoE switch that powers the Pi’s and their SSD’s. This is a little bit risky when it comes to available power, but so far so good and I do have some headroom (though not very much).

These Pi’s aren’t setup as a Bramble, even though that’s super cool & compelling. I have mine setup as 4 separate appliances, each with their own Ubuntu installs. Mail-in-a-Box doesn’t support Noble Numbat yet, but I’m certain it will soon.

GitLab EE & WordPress (nightly) are running inside a suite of Docker containers alongside their relative required projects (MySQL, PHP, Memcache, etc…) – GitLab also doesn’t natively support the Pi’s ARM64 architecture, but thankfully someone maintains a fork that does.

GitLab EE

Everything that’s running on the Pi’s is also backed-up & deployed from this self-hosted GitLab. I also keep mirrors of the WordPress plugins I’ve authored and a fork of WordPress that I’m writing this post from inside of there. I hadn’t intended to go to deep into GitLab here, but it really is a key component for me to manage all of this stuff.

And I have nothing against Microsoft or GitHub; I use them everyday for work, more than I use GitLab for home, but remember: independence & autonomy.

WordPress (1 & 2)

A long time ago I decided to run a fork of WordPress for personal use, and named it “Publicious” – I love the name, but it’s a mouthful to pronounce and I got sick of paying for the domains so I’ve retired that idea for now. (Someone out there noticed that and bought “publicious.org” right away, I’m assuming hoping I’d reused an admin password and would blindly try to login to it.)

Anyways, Publicious has been boiled down to:

• a custom PHP boot-loader in front of WordPress (that includes configuration files for MySQL, Memcache, LudicrousDB, and WordPress constants)
• all of the drop-in plugins, specifically sunrise.php for multi-network
• time.php NTP endpoint
• concat.php for combining CSS & JS requests
• cron.php for system events
• a custom index.php
• a local avatar to reduce Gravatar requests
• a buuuuuuuuunch of plugins

Keeping all of this inside of GitLab makes it relatively easy to deploy & rollback, though I’m tempted to ditch it and use vanilla WordPress more here now thanks to all of the work that’s gone into stability & fatal error detection.

I have two Raspberry Pi’s load balanced just in case of a failure, but so far the second one hasn’t been necessary so I may temporarily decommission it to have 1 fewer thing to maintain.

Caddy Server

I’m running Caddy in a few different Docker containers, but ultimately the point of it is best said using their own words:

By default, Caddy automatically obtains and renews TLS certificates for all your sites.

https://caddyserver.com

That means, for me, every domain inside of this WordPress Multi-Site/Network/Domain install will always automatically have TLS deployed for it, and it happens so fast you almost can’t even tell even if you’re looking.

This required setting up a special endpoint that Caddy can use to perform a very-fast lookup of whether or not a domain exists inside of WordPress for that request (inside of the wp_blogs database table using the get_site_by_path() function) which responds with a 200 header if yes and 404 header if no – 200 issues the cert and 404 doesn’t. Easy!

What’s next?

Improved network monitoring. Unifi Network includes some already, and it is fine enough, but I will feel more-better when I have a Raspberry Pi whose only job is to report back and tell me that I did or did-not leave UFW disabled or ports open, etc…

I would like to use ProxMox & network boot to reduce some of the manual’ness of configuring these Pi’s.

I’d like a second static IP to cut out that last Linode.

I need bigger SSD’s to move more stuff into Nextcloud, as well as for all of the media that is stored from all of the WordPress sites in this install.

I might setup a dynamic DNS service for the off-chance this static IP stops being static.

That’s all?

Yep. I just needed to be my own ISP & offer bargain-basement web hosting, fork a bunch of stuff to run custom code on unsupported hardware, and contribute to about 20 different open source projects in my “sPaRe TiMe” over the course of like 10 years, plus convince my new ISP to bend their rules for me to have reverse DNS.

I think/hope that as more people move to self-hosting, more open-source projects will emerge to make this more of a turn-key experience.

It’s liberating – though, risky – and I can feel the collective eyes of friends & colleagues who also (like I do) care immensely about the security of the web and our shared interests not being super fond of the idea of me exposing part of my home network to the web, but… from my perspective… and for me being me, it’s a new twist on a bunch of old problems, and I’m learning & growing & am better because of it.

Also, I really do believe it should be easy & possible for all of us to run our own WordPress (and BuddyPress) on any hardware and have it accessible to everyone everywhere.

I’ll leave comments open for a bit! 💗