Kaizen #4: Overhauled homelab
I started this “blog-series” to force myself to write more about my set up, and
my continuous iterations. But it didn’t happen… until today! This post has been
a long time coming, and I have kept the ideas in my head and notes.
I will walk through everything I did, in order:
Previous setupNetwork situationWhat I self-hosted
Network situation
What I self-hosted
NetworkingFiber adventureWiringWiring the garage
Fiber adventure
Wiring
Wiring the garage
The new server
The new setupDebianDokployMediaJellyfinImmichFilesMonitoringMiscellaneousqBittorrentLinkdingMesh VPN
Debian
Dokploy
MediaJellyfinImmichFiles
Jellyfin
Immich
Files
Monitoring
MiscellaneousqBittorrentLinkdingMesh VPN
qBittorrent
Linkding
Mesh VPN
Final result
What’s remaining
Previous setup
Two years ago, I revamped my homelab. I had just bought a new
machine to use as my main home server, coming from a 9U server rack with a bunch
of machines tangled together. Back then I finally sat down to figure out how I
want things. And it has worked great… until it hasn’t.
The new machine back then, a Zimaboard, was set up with NixOS, which seems
excellent and worked perfectly until I wanted to do a basic cronjob for backups.
I never figured out how to do that, so I basically did my backups manually
whenever I had the time. I use Restic, which is awesome.
Network situation
We have a 750/750 Mbps fiber-connection with Bergen Fiber (Altibox), which is
more than enough for our needs. It’s a very reliable connection, never slow,
absolutely no complaints about the connection.
Earlier this year we moved to a house, where we own the whole thing, it’s
amazing! But the network situation wasn’t great. Not a single Ethernet port in
the whole house, except where the fiber and modem were located. And guess where
that’s located!? Beside the door to the balcony, where the previous owner used
to have their TV (we have made a separate TV room). One thing to note, there was
a bunch of phone lines throughout the house, and we’ll get to those later.
As you can see, not an ideal placement, especially after we found out that it
was the perfect location for the cat to have his food, away from the dog.
Another thing that I particularly didn’t like about the Zimaboard setup I had
done, was that I chose to install NixOS directly on the onboard flash storage.
Whenever I tried to upgrade the NixOS version, I ran into issues with having
enough space. I had to do a bunch of clean ups and micro adjustments every
fucking time to get it to work.
My networking situation was fine after moving into the new house, but I was
unable to get WiFi in the garage. The coverage seemed fine, but it wasn’t ideal
outside or in the TV room. I don’t think having the router on the floor was the
best spot, but I didn’t want to mount the thing on the wall of the living room
either.
All the WiFi came from an ASUS RT-AX92U router, which worked pretty well,
especially on the second floor where it was located. But it was a whole
different story on the first floor, and outside.
What I self-hosted
For all my ripped media, TV shows and movies, I used Jellyfin, which is so good,
it just works. I also used Jellyfin for audiobooks with a great iOS app,
plappa. And for all my Linux ISOs, I used qBittorrent. It
runs great as a headless torrenting client, and there’s also a great iOS app for
it,
qRemote.
I also had a bunch of small Docker containers, for Foundry and some personal
projects.
And I used to self-host Nextcloud, but it became cumbersome and just a hassle to
maintain. It was bloated for my use-case.
Networking
I bought Ubiquiti UniFi gear, specifically:
UCG Ultra
U7 Lite access points x3
PoE-injectors
I also bought a fresh roll of Ethernet cable, 500 meters, and tools to make my
own cables.
UniFi has a lot of respect from homelabbers, and for good reason. It’s not
(F)OSS, but it’s excellent hardware with no subscription and no account required
to use it.
Buying all the equipment was the easy part, putting it all together, oh man, it
was a totally different ordeal.
Fiber adventure
One of the first things I thought about when we bought the house was to move the
fiber inlet and modem to the attic, so that it could be out of the way and not
be an eyesore. I politely asked my local ISP, Bergen Fiber, if they could do it
for me, or if they could provide me with a 5 meter fiber-cable SC/UPC. They said
yes to providing me with a cable, and when I followed them up via email asking
what I needed to do for them to send me the cable, they replied via SMS that
they couldn’t 🙃. They could send me a 2 meter cable though. So I asked around,
and spent some time looking for the specific cable online. Most of what I found
was the wrong length or way too expensive for a short cable. I managed to find
one, from a B2B company that is 5 minutes away from where I live. I asked a
plumber-buddy of mine that shops there if he could pick up the cable, and he did
for 32 kr (~ $3).
Wiring
Besides the fiber cable looking adventure, I had to wire up the whole house,
especially for my new access points. I needed cables going from the attic to the
living room, downstairs hallway, and to the garage in a separate building. In
addition, I wanted Ethernet in my home office and TV room. How hard can it be?
I asked a buddy of mine to help out figuring out the locations for all the
things, and helping me pull cables. I told him it would take a few hours. We
spent six hours, and he had to go home, I continued for three more days after
that. I thought it was going to be easy…
We started by moving the fiber and modem, which was easy, made a 42 mm hole in
the wall from the attic and pushed a conduit down the wall to the living room,
then we pulled the fiber. Worked like a charm!
Then we made a hole in the middle of the ceiling in the living room for the PoE
access point, that also was easy.
But then came the hardest part, going from the attic, down two floors, without
any existing conduit. Like I mentioned, there were phone cables throughout the
house, but none of them were wired up, and the conduit didn’t lead anywhere near
the attic. We spent the majority of the initial day pulling on the phone cables
to figure out where they went and how we could get the wires from the attic. I
ended up making a hole above a phone connection box in the living room, so that
I could pull conduit from the attic down to the box. Because this box lead to
the first floor, in the entrance.
And the actual cable-making? What a fucking mess that was. I knew there was an
order to do things in, but the ones
I bought,
had no documentation, so I just relied on the internet to figure out the order.
It took me two hours to get a working cable, because I didn’t notice there was
another transparent plastic bag within the transparent plastic bag with “guides”
for keeping the twisted pairs in order. I only realized that when I read the bad
reviews. I used a super simple cable tester to check that I made the cables
right, it was a super worthwhile thing to buy, it would have been a headache
without one.
Wiring the garage
My original plan was always to wire Ethernet to the garage, but I didn’t know
how to do it. I knew there was conduit going from the house, but I had no idea
if there was a room for an Ethernet-cable. Luckily, I found a cut telephone-wire
in the garage coming from a yellow conduit, which was the same color as the
conduit in the entrance where I had already wired the first floor! I was certain
it was the same conduit, so I started pulling the cable to see if it moved the
cable from the house, but nothing happened. I did all sorts of shenanigans to
make the cable move, but it didn’t flinch, not one bit. I ended up ruining the
insulation of the wire, that was how tight it was stuck.
By pure coincidence, I found the old coax and telephone box still attached to
the house outside. I opened it up, I loosened all the wires from the
contraption, and followed the cables down the conduit on the outside of the
house. Lucky for me, the ground outside is gravel, so I just started to dig. And
what do you know, I found the fucking conduit, in fact, I found three! One for
the house, one for the garage, and one for what I think is the previous
ISP/telephone company. I just cut them all, and pulled them out. And now the
telephone-wire in the garage came loose like nothing.
I pulled the cable from the house through the conduit and into the garage’s
conduit, and voila! We have Ethernet in the garage!
The new server
I managed to get my hands on a huge HPE DL380 Gen10 server as decommissioned
hardware. It has two dedicated Intel Xeon CPUs and 288 GB RAM. I got 16x 600 GB
enterprise 10k RPM HDDs, I haven’t put in the last 8, but it’s on the to-do
list! The 8 that are put in are in RAID6, so I have two that can die before I
have to switch them out.
My initial plan was to have this server in the attic beside the fiber/modem and
router, but after firing the machine up, I quickly realized that I would be able
to hear the machine through the ceiling. Good thing I have plenty of room in the
garage’s loft.
I never measured the power draw before and after, and honestly it’s not a big
concern for me, electricity is fairly cheap here in Norway.
The machine is named hailmary, after the ship in
Project Hail Mary. If you
look at the neofetch above, you’ll notice it has a butt load of RAM, 288 GB, and
I’m barely using any of it. I still have to figure out what to do with it all.
I’m thinking about getting a GPU for the server, for running local LLMs on it.
The new setup
Before setting all this up, I moved over everything I wanted to keep from the
Zimaboard: all my media (movie and TV show rips, audiobooks), and my photos.
For the first time in my self-hosting career, I have a fucking dashboard for all
the stuff that I have!
Let me walk you through everything!
Debian
After having used NixOS for the last two years in my homelab, I wanted to go
back to what I know. I could have used Ubuntu server, but I wanted to keep it as
minimal as I could, but still familiar.
And remember the cronjob I never managed to set up on NixOS? On Debian that’s
just boring, reliable cron again, the tooling is no longer the thing standing
between me and automated backups (more on that in
What’s remaining).
Dokploy
This is where I throw all my Docker related stuff. We started using this at work
recently, so it’s also an excuse to get myself more familiar with how everything
works.
It’s useful having a dashboard for all the Docker related stuff for checking if
something is running, how resources are being used, or if something is just
stuck. I can do this now both in the browser, but also still using the Docker
CLI, even from my own machine using Docker context.
Media
Jellyfin
The home of all my media, except photos. Like before, I use it for all my ripped
media, TV shows and movies, but also my DRM-free audiobooks. I think the current
setup here is a bit of a downgrade in comparison to the Zimaboard because of
video-encoding, but I might fix that with a GPU in the server.
Immich
I thought I would be fine without having a dedicated app for all my photos, but
I saw that I had space on the server, and more than enough resources. I have
heard so much about this piece of software that I had no excuse to not give it a
try. It’s really cool, and it’s fast!
Files
I have used Syncthing before, but only between my phone and laptop. Now both my
laptop and phone syncs to the new server instead, because it’s always up and
available. It’s a sync-server that always — and I mean always — just works.
Monitoring
This time around I’m doing stuff differently, I have monitoring now. But I don’t
just have one app/service for that, I have three!
Uptime Kuma, for checking the
uptime of my stuff.
Beszel, for monitoring the server(s) itself
(themselves).
Grafana + Loki, for having all the logs for
everything in one place.
Miscellaneous
qBittorrent
For all of my Linux ISOs. It runs great. I don’t want to keep checking my
qBittorrent progress, I just want to know that my Linux ISOs are downloading to
the correct location.
Linkding
I have been using Linkding for the last five years, and now it was finally time
to have the data on my own machine. It’s the best bookmarking software I have
used. I use it daily, and it’s the source of all my links in my monthly posts.
Mesh VPN
After I first heard about NetBird from
Jan-Lukas, and that they have
a first-party reverse proxy, I had to give them a try. They even have
email/password login, no more tying myself to big tech!
I know there are security implications of exposing services via a reverse proxy,
so I only expose ones I trust to have good security in the first place. I don’t
want to run every little thing I host on a VPS, so NetBird is very convenient
when I need to share something with family or friends.
Final result
Like I mentioned in Network situation, our connection is
excellent, but now the WiFi-coverage is also excellent. Both floors and the
garage have around 700+ Mbps on WiFi! And going from an ASUS router to a
Ubiquiti router, wow, what a difference! The UI/UX feels like they want me to
use the thing, and doesn’t expect me to look up how to do stuff.
I haven’t done anything with VLANs or network segmentation yet, but it’s
something I might play around with in the future now that I have the gear for
it.
Remember all the holes I made? Because I have a 3D-printer, I could just fix the
holes without doing any painting!
There’s still some work to do with tidying things up, but it works now. And the
mess isn’t visible unless you move stuff around.
And now everything is perfectly cable-managed in the attic!
I’m super happy with how all the access points turned out!
What’s remaining
I have now let my new set up
soak (Wikipedia) for the last
three days, so next up on my list is to turn my old Zimaboard with the HDDs into
a backup machine using scheduled Restic jobs. It’ll be named blip-a, since it
will be a foreign entity to the hailmary, a separate backup machine (if you
know, you know).
And I need to add the remaining 8 HDDs to the server, or maybe not all of them,
might be good to have some as a backup.
But for now, I’m just happy it all works. This overhaul was way more involved
than I expected when I started, and I learned a ton along the way. Got a homelab
horror story about pulling cable through walls, or a service I’m missing? Reply
via email, I read everything. See you in the next Kaizen!
Tagged with
#kaizen
#homelab