Replacing hand-edited nginx with NixOS (and finding two dead config entries along the way)

I've been running nginx on a box I call VDO for years now - it's the reverse proxy in front of a handful of self-hosted services: VDO.ninja, a Palworld server, Vaultwarden, Leantime, and a few others that have come and gone. Like most homelab nginx setups, it grew the way these things always grow: a sites-enabled/ file added here, a Certbot cron job there, a proxy_pass tweaked at 11pm to fix something that broke. It worked. It also wasn't something I could hand to a future version of myself, let alone anyone else, and reconstruct with any confidence.

So I decided to rebuild it on NixOS, where the whole nginx + TLS setup lives in one declarative configuration.nix instead of a pile of hand-edited files and whatever Certbot happened to do to them over the years. This is the story of that migration - what worked, what broke, and what I found hiding in the config that I didn't expect.

Starting point: auditing what's actually there

Before touching anything, I pulled nginx -T off VDO to get the full, resolved configuration - every included file, flattened. That turned out to be the right first move, because it surfaced two things I would have otherwise just carried forward blindly:

1. A whole stream {} block proxying Palworld's UDP port 8211 through nginx to the game server.
2. A default vhost that silently fell through to my Leantime instance whenever a request didn't match any server_name - not something I'd ever consciously decided, just a side effect of vhost ordering.

Cross-referencing that against my router's port-forwarding table killed the first one immediately: Verizon Fios was port-forwarding Palworld's UDP 8211 straight to the game server's LAN IP, bypassing nginx entirely. The stream{} block had been dead code the whole time - nginx was never actually in that path. I also found a stray port-8080 forward pointing at VDO that didn't correspond to anything in the nginx config at all. A quick ss -tlnp on VDO confirmed nothing was even listening on it. Dead forward, dropped.

Small thing, but worth saying out loud: if you're about to rebuild something from scratch, audit what's actually running before you re-implement what you think is running. I would have faithfully ported over a UDP stream proxy that hadn't done anything in who knows how long.

While I was in there, I also killed the "default vhost falls through to Leantime" behavior and gave Leantime its own real server_name. The new default is a hard return 444 - connection just dies if nothing matches.

The plan: NixOS LXC, built alongside, cut over when ready

Rather than touch the live box, I built the replacement as a brand-new Proxmox LXC running NixOS, using nixos-generators to produce a proxmox-lxc-format template. Everything got built and tested side-by-side with the existing VDO nginx, with DNS still pointed at the old box the whole time. Certs for the new box came via Let's Encrypt DNS-01 challenges against Cloudflare (which already manages my DNS), specifically so the new box didn't need to be internet-facing yet to get real, working certificates. That let me curl --resolve against it and validate every vhost before DNS ever moved.

Where I got stuck (because I will absolutely forget this otherwise)

Nix's flake features aren't on by default. First run of nix run github:nix-community/nixos-generators failed with experimental Nix feature 'nix-command' is disabled. Flakes need both nix-command and flakes enabled - I'd only tried enabling one of them, and I'd also put the flag in the wrong place in the command (after the --, which routes it to the app being run, not to nix itself). Persisting it in ~/.config/nix/nix.conf made the problem go away for good.

A configuration.nix has to actually exist where you point. Obvious in hindsight, but I hit error: file 'nixos-config' was not found in the Nix search path because I hadn't actually put the file at the path I was telling nixos-generators to use yet - I'd mentally filed that as a later step and jumped ahead.

addSSL needs a certificate, even for a vhost whose whole job is dropping connections. My default catch-all vhost errored with services.nginx. virtualHosts._.sslCertificate' was accessed but has no value defined, because telling nginx to also listen on 443 means nginx needs something to present during the TLS handshake, "default" or not. Fix was pointing it at useACMEHost for one of my real certs - the same trick my old hand-written config had been quietly doing all along, I just hadn't carried it forward.

Option names aren't stable across nixpkgs versions, and the error messages are genuinely helpful about it. security.acme.defaults.credentialsFile doesn't exist on the nixpkgs revision I was pinned to - it's environmentFile now. The error output actually suggested the fix directly ("Did you mean ... environmentFile?"), which was a nice surprise after years of cryptic build tool errors elsewhere.

nixos-generators is on its way out. Mid-build I got a deprecation warning - as of NixOS 25.05, this functionality has been folded into nixos-rebuild build-image directly. It still works today and isn't going away suddenly, so I kept going rather than switch tools mid-troubleshoot, but it's worth knowing about if you're starting this today.

First boot has no PATH. Entering the fresh LXC via pct enter and running mkdir got me command not found - the shell exists, the binaries exist in the Nix store, but nothing had wired up PATH yet. export PATH="/run/current-system/sw/bin:$PATH" (or just . /etc/profile) sorted it out.

None of these were hard once I knew what they were. All of them would have been mildly maddening to hit cold.

Sizing the box

For resources, I ended up on 2 vCPUs, 4GB RAM, no swap (disk is on NVMe, so no speed argument for swap - but I bumped RAM up specifically because I dropped swap, since without it there's no cushion if a nixos-rebuild evaluation spikes), and 16GB of disk. That last number isn't about nginx's footprint, which is tiny - it's about giving the Nix store room to hold a few generations before garbage collection kicks in. I added nix.gc to the config to run weekly and prune anything older than 30 days, so that stays bounded going forward instead of creeping.

Where it landed

nixos-rebuild switch on the new box completed clean - no errors, just the expected (and harmless) warning about /boot not existing, which is just NixOS's tooling checking for a bootloader partition that an LXC has no concept of. From there, curl --resolve against every vhost on the new box returned the same content as the equivalent request against the live VDO. Cutover is just a DNS change away at this point.

What I'd tell past-me

• Pull the real, resolved config (nginx -T, not just the files you remember editing) before you start porting anything. You will find dead config you forgot about.
• Cross-reference against what's actually reachable from the outside - your router's port-forwarding table and ss -tlnp on the box will tell you the truth faster than the app config will.
• Build the replacement in parallel and validate with DNS-01 + --resolve before ever touching DNS. There's no reason to take the old thing down to test the new thing.
• NixOS's error messages, when they involve a renamed option, will usually just tell you the new name. Read the whole error before reaching for a search engine.

Create a live Linux USB drive to get started with Linux

 

1. Download a LiveCD from the distribution you choose to try. I am keen on openSUSE so I go to https://get.opensuse.org/tumbleweed/?type=desktop#download and download the KDE LiveCD
   
   
   
   
   
2. Download balenaEtcher from https://etcher.balena.io/
   
   
   
   
   
3. Launch balenaEtcher and choose the ISO and USB disk and click the Flash! button 
   
   
   
   
4. Boot from the USB!

Wiring Up Zabbix in the Homelab: Geomaps, Broken Repos, and the 7.4 Auth Change That Bit Me

I've been rolling Zabbix into the homelab for monitoring, and like most homelab projects, "install the agent and watch the dashboard fill up" turned into an afternoon of chasing three separate problems. None of them were hard once I understood what was actually happening, but none of them were obvious from the error messages either. Writing this up mostly for future-me, but hopefully it saves you a step or two.

Problem 1: I Didn't Want to Hand-Enter Lat/Long for Every Host

Zabbix's Geomap dashboard widget is genuinely nice once it's populated - it plots your hosts on an actual map using the location_lat and location_lon inventory fields. The catch is that nothing populates those fields for you. Add a new host, and it just doesn't show up on the map until you go type in coordinates by hand.

For a homelab where most of my boxes live in the same rack, that's annoying busywork. What I wanted was a template-level default: any host using the template gets a fallback pin, and I can override it later for anything that actually moves (laptops, remote nodes, whatever).

The trick is combining user macros with script items:

Step 1 - Define the macros on the template.

{$DEFAULT_LAT} → 40.6892
{$DEFAULT_LON} → -74.0467

(Coordinates are decimal degrees - latitude -90 to 90, longitude -180 to 180.)

Step 2 - Create a script item for latitude.

• Name/Key: Default Latitude / default.latitude
• Type: Script
• Type of information: Numeric (float)
• Update interval: 1d - it's static data, no reason to poll it constantly
• Script:

return '{$DEFAULT_LAT}';

• Populates host inventory field: Location latitude

Step 3 - Repeat for longitude, swapping in {$DEFAULT_LON} and mapping to Location longitude.

Step 4 - the part that'll trip you up: the script items only get to overwrite inventory data if the host's Inventory mode is set to Automatic. If it's left on Manual or Disabled, the item runs fine, collects the value, and just... doesn't write it anywhere. No error, no warning, it just silently doesn't work. I burned more time than I'd like to admit on this before checking the inventory mode.

One small bonus while I was in the Geomap widget: if you're tired of the map resetting to some default zoom level every time the dashboard loads, pan/zoom to the view you actually want, right-click the map, and pick "Set this view as default." Small thing, but it stuck with me.

Problem 2: Agent Install Failed on Dependencies

Next up, installing zabbix-agent2 on an OpenMediaVault VM (which runs on Proxmox in my case) blew up with:

Depends: libc6 (>= 2.38) but 2.36-9+deb12u9 is to be installed
Depends: libssl3t64 (>= 3.0.13) but it is not installable

The root cause was embarrassingly simple once I looked at it: the VM is on Debian 12 (Bookworm), which tops out at libc6 2.36. I'd grabbed the Zabbix repo package built for Debian 13 (Trixie) / Ubuntu 24.04, which expects newer system libraries that Bookworm doesn't have and isn't going to get. Classic case of downloading the wrong release artifact and not noticing until apt complained.

Fix was a clean purge-and-reinstall with the correct release package:

Purge the bad repo, keys, and cache:

sudo rm -f /etc/apt/sources.list.d/zabbix.list
sudo rm -f /etc/apt/trusted.gpg.d/zabbix*
sudo rm -f /etc/apt/keyrings/zabbix*
sudo apt-get clean

Install the Debian 12–correct release package, then refresh and install:

sudo dpkg -i zabbix-release_latest_7.0+debian12_all.deb
sudo apt update
sudo apt install zabbix-agent2 zabbix-agent2-plugin-*
sudo systemctl enable --now zabbix-agent2

Lesson: always double-check the release package matches the actual OS version on the box, not the OS version you assume it's running. This is exactly the kind of gap that's easy to hit when you're juggling a handful of VMs with slightly different base images.

Problem 3: The Zabbix 7.4 API Auth Change Nobody Warns You About

This one cost me the most confusion. I wanted to query the API directly (Postman/cURL) to pull inventory data, and ran into two separate issues stacked on top of each other.

First: a 404 on the endpoint. Most guides show the API living at /zabbix/api_jsonrpc.php, but depending on how you installed (this applies to a lot of default and Docker-based setups), the frontend is served straight from the domain root. If you're getting a flat 404, check whether your install actually uses the /zabbix/ subdirectory at all before you go further down a rabbit hole.

Second, and the real gotcha: Zabbix 7.4 changed how authentication works. The traditional pattern - sticking "auth": "<token>" inside the JSON-RPC body - is exactly what every older tutorial shows you, and it's exactly what 7.4 now rejects:

{"jsonrpc":"2.0","error":{"code":-32600,"message":"Invalid request.","data":"Invalid parameter \"/\": unexpected parameter \"auth\"."},"id":1}

The error message itself doesn't really tell you why - it just says the parameter is unexpected, which reads like a typo, not a breaking API change. As of 7.4, the API key has to be passed as a standard HTTP header instead of inside the request body:

Authorization: Bearer <your_actual_api_key_here>

And the JSON body gets simplified back down to just the method call:

{
    "jsonrpc": "2.0",
    "method": "host.get",
    "params": {
        "output": ["hostid", "name"],
        "limit": 5
    },
    "id": 1
}

If you're following an older blog post or Postman collection and getting this error, this is almost certainly why - check the Zabbix version before assuming your payload is malformed.

Problem 4: Inventory Fields Come Back Empty

Once the auth was sorted, the API calls worked - but host.get with inventory fields requested (os, software, os_full, etc.) came back mostly blank, except for os, which was a raw, ugly kernel string like Linux version 6.17.2-2-pve.... Not exactly what I wanted for tracking which boxes need OS updates.

Turns out this is by design, not a bug. Zabbix defaults new hosts to Inventory mode: Disabled. Even when a template - like the stock "Linux by Zabbix agent active" template - is actively collecting an item that could populate inventory (system.sw.os in this case), the item's "Populates host inventory field" setting ships blank. Zabbix apparently treats this as a deliberate opt-in, presumably to avoid inventory-table overhead at scale, but for a homelab it just means everything's empty until you flip a few switches.

Three things fixed it:

1. Set the default going forward: Administration > General > Other > Default host inventory mode → Automatic

2. Backfill existing hosts: Data collection > Hosts → select the hosts → Mass update → Inventory tab → set mode to Automatic

3. Actually map the item to a field: Data collection > Templates > Linux by Zabbix agent active > Items → find Operating system (system.sw.os) → set Populates host inventory field to OS (or OS (Full)) → Update

If you want cleaner data than the raw kernel string - something like Ubuntu 24.04 LTS instead of the full uname output - you can add a custom item that regexes it straight out of /etc/os-release:

Key: vfs.file.regexp[/etc/os-release,"PRETTY_NAME=(.*)",,,,\1]
Inventory Field: OS (Full)

Wrap-Up

None of these were individually hard problems, but each one had an unhelpful or misleading surface-level symptom: a silent no-op on the Geomap fields, a dependency error that pointed at the wrong root cause, an auth error that read like a bad payload instead of a breaking version change, and empty inventory that looked like a bug rather than a default setting. If you're standing up Zabbix 7.4 fresh, checking inventory mode and the auth header format up front will save you the loop I went through.

Installing VMware Workstation Pro on openSUSE: A Battle Report

I just wrapped up the VMware Workstation Pro chapter for the book I'm writing, and it was, without question, the longest and most frustrating install I've documented so far. Long enough that I figured it deserved its own post here, separate from the step-by-step version - this one's more about what actually went wrong and what I learned fighting through it.

Why bother with VMware at all?

VMware Workstation Pro is a paid, proprietary hypervisor, which puts it in an odd spot on Linux where free options like KVM/QEMU and libvirt already do most of the job. But Workstation Pro earns its keep with tight kernel integration, a genuinely polished GUI, snapshot/clone workflows that make experimentation cheap, OVF/OVA export for sharing VMs, and support for nested virtualization - handy if you want to run Docker or Kubernetes inside a guest. On openSUSE Tumbleweed specifically, none of that comes easy.

The install itself: death by a thousand dependencies

The first sign this wasn't going to be a zypper install afternoon was the dependency list. VMware needs kernel headers and a C++ toolchain to build its kernel modules against your running kernel - fine, expected. But it also wants a handful of specific library versions, and openSUSE Tumbleweed, being a rolling release, is usually ahead of what VMware's installer expects. In my case, that meant faking an older libxml2 with a symlink:

sudo ln -s /usr/lib64/libxml2.so.16 /usr/lib64/libxml2.so.2
export VMWARE_USE_SHIPPED_LIBS='yes'

That second line matters as much as the symlink - it tells VMware to fall back to the libraries it ships internally rather than fighting your system's newer versions. Without both pieces together, the installer gets confused in ways that are genuinely hard to diagnose from the error messages alone.

The installer finishes... and then quietly fails anyway

Here's the part that really tested my patience. After working through the download (Broadcom now gatekeeps VMware downloads behind a login, a whole separate minor annoyance), making the bundle executable, and running the installer - accepting license agreements, telemetry prompts, all the usual dance - the first launch of VMware itself failed. Not with a helpful error, but with this buried in the logs:

[Apploader] Cannot get library dependencies. (10c)
[AppLoader] Fallback to use all shipped libraries.

The fix wasn't in any VMware documentation I could find. It came down to manually running the setup helper script the installer should have run correctly on its own:

sudo /usr/lib/vmware/bin/vmware-setup-helper -e -o -u yes -c yes

After that, VMware launched cleanly. I want to be honest about how I found this - it wasn't buried deep in a forum thread after hours of searching, it was AI-assisted troubleshooting that got me there fast. I've said this before and I'll say it again here: for these obscure, distro-specific integration failures, an AI that can reason through error messages and suggest specific diagnostic commands is genuinely one of the best tools in your kit now. I don't think I'd have found that exact invocation on my own nearly as quickly.

And then the hardware plot twist

Just when I thought I was through it, I hit a wall that no amount of troubleshooting could fix: my CPU. I'm running an Intel Xeon X5670 in this box, and it turns out every VMware Workstation Pro release after 17.0.2 requires XSAVE instruction set support - which this chip simply doesn't have. Not a driver issue, not a config issue. The silicon itself doesn't support it.

That leaves two options, neither of which is great: switch to different hardware, or downgrade to 17.0.2. And the downgrade path isn't clean either - the kernel modules VMware 17.0.2 expects were built against older kernel APIs, and Tumbleweed's rolling kernel has moved well past that point. So it becomes its own compatibility puzzle, layered on top of everything else.

What I'd tell someone else attempting this

A few honest takeaways, in the spirit of documenting the failures alongside the wins:

• Take VM snapshots liberally while you work through this. I can't overstate how much easier this made recovering from bad states along the way. Without that safety net, this install would have been genuinely miserable rather than just long.
• Check your CPU's instruction set support before you start, not after. lscpu will tell you what you're working with - if XSAVE isn't listed and you're on older enterprise-grade hardware like mine, save yourself the afternoon and either plan on 17.0.2 from the start or reconsider the hardware.
• Rolling-release distros and vendor installers built for "stable" Linux baselines are going to fight each other. This isn't the first time I've hit this in this book - Docker Desktop and a couple of other tools have had their own version of this same friction on Tumbleweed. If you're on a rolling release, budget extra time for exactly this kind of gap.
• Don't be afraid to manually run what the installer should have run for you. The vmware-setup-helper fix wasn't documented anywhere obvious, but the installer's own failure message was specific enough to point toward it once I stopped assuming the GUI installer was the only supported path.

The full step-by-step version of this - with every screenshot and command in sequence - will be in the book. This post is more the "here's what it actually felt like" version. If you're fighting the same install right now: it does end, and it does eventually work. Good luck.