Overhauled homelab

I started this “blog-series” to force myself to write more about my set up, and my continuous iterations. But it didn’t happen… until today! This post has been a long time coming, and I have kept the ideas in my head and notes.

I will walk through everything I did, in order:

• Previous setup
  • Network situation
  • What I self-hosted
• Networking
  • Fiber adventure
  • Wiring
  • Wiring the garage
• The new server
• The new setup
  • Debian
  • Dokploy
  • Media
    • Jellyfin
    • Immich
    • Files
  • Monitoring
  • Miscellaneous
    • qBittorrent
    • Linkding
    • Mesh VPN
• Final result
• What’s remaining

Previous setup

Two years ago, I revamped my homelab. I had just bought a new machine to use as my main home server, coming from a 9U server rack with a bunch of machines tangled together. Back then I finally sat down to figure out how I want things. And it has worked great… until it hasn’t.

The new machine back then, a Zimaboard, was set up with NixOS, which seems excellent and worked perfectly until I wanted to do a basic cronjob for backups. I never figured out how to do that, so I basically did my backups manually whenever I had the time. I use Restic, which is awesome.

Network situation

We have a 750/750 Mbps fiber-connection with Bergen Fiber (Altibox), which is more than enough for our needs. It’s a very reliable connection, never slow, absolutely no complaints about the connection.

Earlier this year we moved to a house, where we own the whole thing, it’s amazing! But the network situation wasn’t great. Not a single Ethernet port in the whole house, except where the fiber and modem were located. And guess where that’s located!? Beside the door to the balcony, where the previous owner used to have their TV (we have made a separate TV room). One thing to note, there was a bunch of phone lines throughout the house, and we’ll get to those later.

As you can see, not an ideal placement, especially after we found out that it was the perfect location for the cat to have his food, away from the dog.

Another thing that I particularly didn’t like about the Zimaboard setup I had done, was that I chose to install NixOS directly on the onboard flash storage. Whenever I tried to upgrade the NixOS version, I ran into issues with having enough space. I had to do a bunch of clean ups and micro adjustments every fucking time to get it to work.

My networking situation was fine after moving into the new house, but I was unable to get WiFi in the garage. The coverage seemed fine, but it wasn’t ideal outside or in the TV room. I don’t think having the router on the floor was the best spot, but I didn’t want to mount the thing on the wall of the living room either.

All the WiFi came from an ASUS RT-AX92U router, which worked pretty well, especially on the second floor where it was located. But it was a whole different story on the first floor, and outside.

What I self-hosted

For all my ripped media, TV shows and movies, I used Jellyfin, which is so good, it just works. I also used Jellyfin for audiobooks with a great iOS app, plappa. And for all my Linux ISOs, I used qBittorrent. It runs great as a headless torrenting client, and there’s also a great iOS app for it, qRemote.

I also had a bunch of small Docker containers, for Foundry and some personal projects.

And I used to self-host Nextcloud, but it became cumbersome and just a hassle to maintain. It was bloated for my use-case.

Networking

I bought Ubiquiti UniFi gear, specifically:

• UCG Ultra
• U7 Lite access points x3
• PoE-injectors

I also bought a fresh roll of Ethernet cable, 500 meters, and tools to make my own cables.

UniFi has a lot of respect from homelabbers, and for good reason. It’s not (F)OSS, but it’s excellent hardware with no subscription and no account required to use it.

Buying all the equipment was the easy part, putting it all together, oh man, it was a totally different ordeal.

Fiber adventure

One of the first things I thought about when we bought the house was to move the fiber inlet and modem to the attic, so that it could be out of the way and not be an eyesore. I politely asked my local ISP, Bergen Fiber, if they could do it for me, or if they could provide me with a 5 meter fiber-cable SC/UPC. They said yes to providing me with a cable, and when I followed them up via email asking what I needed to do for them to send me the cable, they replied via SMS that they couldn’t 🙃. They could send me a 2 meter cable though. So I asked around, and spent some time looking for the specific cable online. Most of what I found was the wrong length or way too expensive for a short cable. I managed to find one, from a B2B company that is 5 minutes away from where I live. I asked a plumber-buddy of mine that shops there if he could pick up the cable, and he did for 32 kr (~ $3).

Wiring

Besides the fiber cable looking adventure, I had to wire up the whole house, especially for my new access points. I needed cables going from the attic to the living room, downstairs hallway, and to the garage in a separate building. In addition, I wanted Ethernet in my home office and TV room. How hard can it be?

I asked a buddy of mine to help out figuring out the locations for all the things, and helping me pull cables. I told him it would take a few hours. We spent six hours, and he had to go home, I continued for three more days after that. I thought it was going to be easy…

We started by moving the fiber and modem, which was easy, made a 42 mm hole in the wall from the attic and pushed a conduit down the wall to the living room, then we pulled the fiber. Worked like a charm!

Then we made a hole in the middle of the ceiling in the living room for the PoE access point, that also was easy.

But then came the hardest part, going from the attic, down two floors, without any existing conduit. Like I mentioned, there were phone cables throughout the house, but none of them were wired up, and the conduit didn’t lead anywhere near the attic. We spent the majority of the initial day pulling on the phone cables to figure out where they went and how we could get the wires from the attic. I ended up making a hole above a phone connection box in the living room, so that I could pull conduit from the attic down to the box. Because this box lead to the first floor, in the entrance.

And the actual cable-making? What a fucking mess that was. I knew there was an order to do things in, but the ones I bought, had no documentation, so I just relied on the internet to figure out the order. It took me two hours to get a working cable, because I didn’t notice there was another transparent plastic bag within the transparent plastic bag with “guides” for keeping the twisted pairs in order. I only realized that when I read the bad reviews. I used a super simple cable tester to check that I made the cables right, it was a super worthwhile thing to buy, it would have been a headache without one.

Wiring the garage

My original plan was always to wire Ethernet to the garage, but I didn’t know how to do it. I knew there was conduit going from the house, but I had no idea if there was a room for an Ethernet-cable. Luckily, I found a cut telephone-wire in the garage coming from a yellow conduit, which was the same color as the conduit in the entrance where I had already wired the first floor! I was certain it was the same conduit, so I started pulling the cable to see if it moved the cable from the house, but nothing happened. I did all sorts of shenanigans to make the cable move, but it didn’t flinch, not one bit. I ended up ruining the insulation of the wire, that was how tight it was stuck.

By pure coincidence, I found the old coax and telephone box still attached to the house outside. I opened it up, I loosened all the wires from the contraption, and followed the cables down the conduit on the outside of the house. Lucky for me, the ground outside is gravel, so I just started to dig. And what do you know, I found the fucking conduit, in fact, I found three! One for the house, one for the garage, and one for what I think is the previous ISP/telephone company. I just cut them all, and pulled them out. And now the telephone-wire in the garage came loose like nothing.

I pulled the cable from the house through the conduit and into the garage’s conduit, and voila! We have Ethernet in the garage!

The new server

I managed to get my hands on a huge HPE DL380 Gen10 server as decommissioned hardware. It has two dedicated Intel Xeon CPUs and 288 GB RAM. I got 16x 600 GB enterprise 10k RPM HDDs, I haven’t put in the last 8, but it’s on the to-do list! The 8 that are put in are in RAID6, so I have two that can die before I have to switch them out.

        _,met$$$$$gg.          tim@hailmary
     ,g$$$$$$$$$$$$$$$P.       -------------
   ,g$$P""       """Y$$.".     OS: Debian GNU/Linux 13 (trixie) x86_64
  ,$$P'              `$$$.     Host: ProLiant DL380 Gen10
',$$P       ,ggs.     `$$b:    Kernel: Linux 6.12.94+deb13-amd64
`d$$'     ,$P"'   .    $$$     Uptime: 3 days, 2 hours, 42 mins
 $$P      d$'     ,    $$P     Packages: 406 (dpkg)
 $$:      $$.   -    ,d$$'     Shell: bash 5.2.37
 $$;      Y$b._   _,d$P'       Display (VGA-1): 1024x768 [External]
 Y$$.    `.`"Y$$$$P"'          Terminal: /dev/pts/0
 `$$b      "-.__               CPU: 2 x Intel(R) Xeon(R) Bronze 3204 (12) @ 1.90 GHz
  `Y$$b                        GPU: Matrox Electronics Systems Ltd. MGA G200eH3
   `Y$$.                       Memory: 8.58 GiB / 267.31 GiB (3%)
     `$$b.                     Swap: 22.89 MiB / 127.66 GiB (0%)
       `Y$$b.                  Disk (/): 955.36 GiB / 3.10 TiB (30%) - ext4
         `"Y$b._               Local IP (eno4): 192.168.1.23/24
             `""""             Locale: en_US.UTF-8

My initial plan was to have this server in the attic beside the fiber/modem and router, but after firing the machine up, I quickly realized that I would be able to hear the machine through the ceiling. Good thing I have plenty of room in the garage’s loft.

I never measured the power draw before and after, and honestly it’s not a big concern for me, electricity is fairly cheap here in Norway.

The machine is named hailmary, after the ship in Project Hail Mary. If you look at the neofetch above, you’ll notice it has a butt load of RAM, 288 GB, and I’m barely using any of it. I still have to figure out what to do with it all. I’m thinking about getting a GPU for the server, for running local LLMs on it.

The new setup

Before setting all this up, I moved over everything I wanted to keep from the Zimaboard: all my media (movie and TV show rips, audiobooks), and my photos.

For the first time in my self-hosting career, I have a fucking dashboard for all the stuff that I have!

Let me walk you through everything!

Debian

After having used NixOS for the last two years in my homelab, I wanted to go back to what I know. I could have used Ubuntu server, but I wanted to keep it as minimal as I could, but still familiar.

And remember the cronjob I never managed to set up on NixOS? On Debian that’s just boring, reliable cron again, the tooling is no longer the thing standing between me and automated backups (more on that in What’s remaining).

Dokploy

This is where I throw all my Docker related stuff. We started using this at work recently, so it’s also an excuse to get myself more familiar with how everything works.

It’s useful having a dashboard for all the Docker related stuff for checking if something is running, how resources are being used, or if something is just stuck. I can do this now both in the browser, but also still using the Docker CLI, even from my own machine using Docker context.

Media

Jellyfin

The home of all my media, except photos. Like before, I use it for all my ripped media, TV shows and movies, but also my DRM-free audiobooks. I think the current setup here is a bit of a downgrade in comparison to the Zimaboard because of video-encoding, but I might fix that with a GPU in the server.

Immich

I thought I would be fine without having a dedicated app for all my photos, but I saw that I had space on the server, and more than enough resources. I have heard so much about this piece of software that I had no excuse to not give it a try. It’s really cool, and it’s fast!

Files

I have used Syncthing before, but only between my phone and laptop. Now both my laptop and phone syncs to the new server instead, because it’s always up and available. It’s a sync-server that always — and I mean always — just works.

Monitoring

This time around I’m doing stuff differently, I have monitoring now. But I don’t just have one app/service for that, I have three!

• Uptime Kuma, for checking the uptime of my stuff.
• Beszel, for monitoring the server(s) itself (themselves).
• Grafana + Loki, for having all the logs for everything in one place.

Miscellaneous

qBittorrent

For all of my Linux ISOs. It runs great. I don’t want to keep checking my qBittorrent progress, I just want to know that my Linux ISOs are downloading to the correct location.

Linkding

I have been using Linkding for the last five years, and now it was finally time to have the data on my own machine. It’s the best bookmarking software I have used. I use it daily, and it’s the source of all my links in my monthly posts.

Mesh VPN

After I first heard about NetBird from Jan-Lukas, and that they have a first-party reverse proxy, I had to give them a try. They even have email/password login, no more tying myself to big tech!

I know there are security implications of exposing services via a reverse proxy, so I only expose ones I trust to have good security in the first place. I don’t want to run every little thing I host on a VPS, so NetBird is very convenient when I need to share something with family or friends.

Final result

Like I mentioned in Network situation, our connection is excellent, but now the WiFi-coverage is also excellent. Both floors and the garage have around 700+ Mbps on WiFi! And going from an ASUS router to a Ubiquiti router, wow, what a difference! The UI/UX feels like they want me to use the thing, and doesn’t expect me to look up how to do stuff.

I haven’t done anything with VLANs or network segmentation yet, but it’s something I might play around with in the future now that I have the gear for it.

Remember all the holes I made? Because I have a 3D-printer, I could just fix the holes without doing any painting!

There’s still some work to do with tidying things up, but it works now. And the mess isn’t visible unless you move stuff around.

And now everything is perfectly cable-managed in the attic!

I’m super happy with how all the access points turned out!

What’s remaining

I have now let my new set up soak (Wikipedia) for the last three days, so next up on my list is to turn my old Zimaboard with the HDDs into a backup machine using scheduled Restic jobs. It’ll be named blip-a, since it will be a foreign entity to the hailmary, a separate backup machine (if you know, you know).

And I need to add the remaining 8 HDDs to the server, or maybe not all of them, might be good to have some as a backup.

But for now, I’m just happy it all works. This overhaul was way more involved than I expected when I started, and I learned a ton along the way. Got a homelab horror story about pulling cable through walls, or a service I’m missing? Reply via email, I read everything. See you in the next Kaizen!