Replacing hand-edited nginx with NixOS (and finding two dead config entries along the way)

I've been running nginx on a box I call VDO for years now - it's the reverse proxy in front of a handful of self-hosted services: VDO.ninja, a Palworld server, Vaultwarden, Leantime, and a few others that have come and gone. Like most homelab nginx setups, it grew the way these things always grow: a sites-enabled/ file added here, a Certbot cron job there, a proxy_pass tweaked at 11pm to fix something that broke. It worked. It also wasn't something I could hand to a future version of myself, let alone anyone else, and reconstruct with any confidence.

So I decided to rebuild it on NixOS, where the whole nginx + TLS setup lives in one declarative configuration.nix instead of a pile of hand-edited files and whatever Certbot happened to do to them over the years. This is the story of that migration - what worked, what broke, and what I found hiding in the config that I didn't expect.

Starting point: auditing what's actually there

Before touching anything, I pulled nginx -T off VDO to get the full, resolved configuration - every included file, flattened. That turned out to be the right first move, because it surfaced two things I would have otherwise just carried forward blindly:

1. A whole stream {} block proxying Palworld's UDP port 8211 through nginx to the game server.
2. A default vhost that silently fell through to my Leantime instance whenever a request didn't match any server_name - not something I'd ever consciously decided, just a side effect of vhost ordering.

Cross-referencing that against my router's port-forwarding table killed the first one immediately: Verizon Fios was port-forwarding Palworld's UDP 8211 straight to the game server's LAN IP, bypassing nginx entirely. The stream{} block had been dead code the whole time - nginx was never actually in that path. I also found a stray port-8080 forward pointing at VDO that didn't correspond to anything in the nginx config at all. A quick ss -tlnp on VDO confirmed nothing was even listening on it. Dead forward, dropped.

Small thing, but worth saying out loud: if you're about to rebuild something from scratch, audit what's actually running before you re-implement what you think is running. I would have faithfully ported over a UDP stream proxy that hadn't done anything in who knows how long.

While I was in there, I also killed the "default vhost falls through to Leantime" behavior and gave Leantime its own real server_name. The new default is a hard return 444 - connection just dies if nothing matches.

The plan: NixOS LXC, built alongside, cut over when ready

Rather than touch the live box, I built the replacement as a brand-new Proxmox LXC running NixOS, using nixos-generators to produce a proxmox-lxc-format template. Everything got built and tested side-by-side with the existing VDO nginx, with DNS still pointed at the old box the whole time. Certs for the new box came via Let's Encrypt DNS-01 challenges against Cloudflare (which already manages my DNS), specifically so the new box didn't need to be internet-facing yet to get real, working certificates. That let me curl --resolve against it and validate every vhost before DNS ever moved.

Where I got stuck (because I will absolutely forget this otherwise)

Nix's flake features aren't on by default. First run of nix run github:nix-community/nixos-generators failed with experimental Nix feature 'nix-command' is disabled. Flakes need both nix-command and flakes enabled - I'd only tried enabling one of them, and I'd also put the flag in the wrong place in the command (after the --, which routes it to the app being run, not to nix itself). Persisting it in ~/.config/nix/nix.conf made the problem go away for good.

A configuration.nix has to actually exist where you point. Obvious in hindsight, but I hit error: file 'nixos-config' was not found in the Nix search path because I hadn't actually put the file at the path I was telling nixos-generators to use yet - I'd mentally filed that as a later step and jumped ahead.

addSSL needs a certificate, even for a vhost whose whole job is dropping connections. My default catch-all vhost errored with services.nginx. virtualHosts._.sslCertificate' was accessed but has no value defined, because telling nginx to also listen on 443 means nginx needs something to present during the TLS handshake, "default" or not. Fix was pointing it at useACMEHost for one of my real certs - the same trick my old hand-written config had been quietly doing all along, I just hadn't carried it forward.

Option names aren't stable across nixpkgs versions, and the error messages are genuinely helpful about it. security.acme.defaults.credentialsFile doesn't exist on the nixpkgs revision I was pinned to - it's environmentFile now. The error output actually suggested the fix directly ("Did you mean ... environmentFile?"), which was a nice surprise after years of cryptic build tool errors elsewhere.

nixos-generators is on its way out. Mid-build I got a deprecation warning - as of NixOS 25.05, this functionality has been folded into nixos-rebuild build-image directly. It still works today and isn't going away suddenly, so I kept going rather than switch tools mid-troubleshoot, but it's worth knowing about if you're starting this today.

First boot has no PATH. Entering the fresh LXC via pct enter and running mkdir got me command not found - the shell exists, the binaries exist in the Nix store, but nothing had wired up PATH yet. export PATH="/run/current-system/sw/bin:$PATH" (or just . /etc/profile) sorted it out.

None of these were hard once I knew what they were. All of them would have been mildly maddening to hit cold.

Sizing the box

For resources, I ended up on 2 vCPUs, 4GB RAM, no swap (disk is on NVMe, so no speed argument for swap - but I bumped RAM up specifically because I dropped swap, since without it there's no cushion if a nixos-rebuild evaluation spikes), and 16GB of disk. That last number isn't about nginx's footprint, which is tiny - it's about giving the Nix store room to hold a few generations before garbage collection kicks in. I added nix.gc to the config to run weekly and prune anything older than 30 days, so that stays bounded going forward instead of creeping.

Where it landed

nixos-rebuild switch on the new box completed clean - no errors, just the expected (and harmless) warning about /boot not existing, which is just NixOS's tooling checking for a bootloader partition that an LXC has no concept of. From there, curl --resolve against every vhost on the new box returned the same content as the equivalent request against the live VDO. Cutover is just a DNS change away at this point.

What I'd tell past-me

• Pull the real, resolved config (nginx -T, not just the files you remember editing) before you start porting anything. You will find dead config you forgot about.
• Cross-reference against what's actually reachable from the outside - your router's port-forwarding table and ss -tlnp on the box will tell you the truth faster than the app config will.
• Build the replacement in parallel and validate with DNS-01 + --resolve before ever touching DNS. There's no reason to take the old thing down to test the new thing.
• NixOS's error messages, when they involve a renamed option, will usually just tell you the new name. Read the whole error before reaching for a search engine.

Create a live Linux USB drive to get started with Linux

 

1. Download a LiveCD from the distribution you choose to try. I am keen on openSUSE so I go to https://get.opensuse.org/tumbleweed/?type=desktop#download and download the KDE LiveCD
   
   
   
   
   
2. Download balenaEtcher from https://etcher.balena.io/
   
   
   
   
   
3. Launch balenaEtcher and choose the ISO and USB disk and click the Flash! button 
   
   
   
   
4. Boot from the USB!

Wiring Up Zabbix in the Homelab: Geomaps, Broken Repos, and the 7.4 Auth Change That Bit Me

I've been rolling Zabbix into the homelab for monitoring, and like most homelab projects, "install the agent and watch the dashboard fill up" turned into an afternoon of chasing three separate problems. None of them were hard once I understood what was actually happening, but none of them were obvious from the error messages either. Writing this up mostly for future-me, but hopefully it saves you a step or two.

Problem 1: I Didn't Want to Hand-Enter Lat/Long for Every Host

Zabbix's Geomap dashboard widget is genuinely nice once it's populated - it plots your hosts on an actual map using the location_lat and location_lon inventory fields. The catch is that nothing populates those fields for you. Add a new host, and it just doesn't show up on the map until you go type in coordinates by hand.

For a homelab where most of my boxes live in the same rack, that's annoying busywork. What I wanted was a template-level default: any host using the template gets a fallback pin, and I can override it later for anything that actually moves (laptops, remote nodes, whatever).

The trick is combining user macros with script items:

Step 1 - Define the macros on the template.

{$DEFAULT_LAT} → 40.6892
{$DEFAULT_LON} → -74.0467

(Coordinates are decimal degrees - latitude -90 to 90, longitude -180 to 180.)

Step 2 - Create a script item for latitude.

• Name/Key: Default Latitude / default.latitude
• Type: Script
• Type of information: Numeric (float)
• Update interval: 1d - it's static data, no reason to poll it constantly
• Script:

return '{$DEFAULT_LAT}';

• Populates host inventory field: Location latitude

Step 3 - Repeat for longitude, swapping in {$DEFAULT_LON} and mapping to Location longitude.

Step 4 - the part that'll trip you up: the script items only get to overwrite inventory data if the host's Inventory mode is set to Automatic. If it's left on Manual or Disabled, the item runs fine, collects the value, and just... doesn't write it anywhere. No error, no warning, it just silently doesn't work. I burned more time than I'd like to admit on this before checking the inventory mode.

One small bonus while I was in the Geomap widget: if you're tired of the map resetting to some default zoom level every time the dashboard loads, pan/zoom to the view you actually want, right-click the map, and pick "Set this view as default." Small thing, but it stuck with me.

Problem 2: Agent Install Failed on Dependencies

Next up, installing zabbix-agent2 on an OpenMediaVault VM (which runs on Proxmox in my case) blew up with:

Depends: libc6 (>= 2.38) but 2.36-9+deb12u9 is to be installed
Depends: libssl3t64 (>= 3.0.13) but it is not installable

The root cause was embarrassingly simple once I looked at it: the VM is on Debian 12 (Bookworm), which tops out at libc6 2.36. I'd grabbed the Zabbix repo package built for Debian 13 (Trixie) / Ubuntu 24.04, which expects newer system libraries that Bookworm doesn't have and isn't going to get. Classic case of downloading the wrong release artifact and not noticing until apt complained.

Fix was a clean purge-and-reinstall with the correct release package:

Purge the bad repo, keys, and cache:

sudo rm -f /etc/apt/sources.list.d/zabbix.list
sudo rm -f /etc/apt/trusted.gpg.d/zabbix*
sudo rm -f /etc/apt/keyrings/zabbix*
sudo apt-get clean

Install the Debian 12–correct release package, then refresh and install:

sudo dpkg -i zabbix-release_latest_7.0+debian12_all.deb
sudo apt update
sudo apt install zabbix-agent2 zabbix-agent2-plugin-*
sudo systemctl enable --now zabbix-agent2

Lesson: always double-check the release package matches the actual OS version on the box, not the OS version you assume it's running. This is exactly the kind of gap that's easy to hit when you're juggling a handful of VMs with slightly different base images.

Problem 3: The Zabbix 7.4 API Auth Change Nobody Warns You About

This one cost me the most confusion. I wanted to query the API directly (Postman/cURL) to pull inventory data, and ran into two separate issues stacked on top of each other.

First: a 404 on the endpoint. Most guides show the API living at /zabbix/api_jsonrpc.php, but depending on how you installed (this applies to a lot of default and Docker-based setups), the frontend is served straight from the domain root. If you're getting a flat 404, check whether your install actually uses the /zabbix/ subdirectory at all before you go further down a rabbit hole.

Second, and the real gotcha: Zabbix 7.4 changed how authentication works. The traditional pattern - sticking "auth": "<token>" inside the JSON-RPC body - is exactly what every older tutorial shows you, and it's exactly what 7.4 now rejects:

{"jsonrpc":"2.0","error":{"code":-32600,"message":"Invalid request.","data":"Invalid parameter \"/\": unexpected parameter \"auth\"."},"id":1}

The error message itself doesn't really tell you why - it just says the parameter is unexpected, which reads like a typo, not a breaking API change. As of 7.4, the API key has to be passed as a standard HTTP header instead of inside the request body:

Authorization: Bearer <your_actual_api_key_here>

And the JSON body gets simplified back down to just the method call:

{
    "jsonrpc": "2.0",
    "method": "host.get",
    "params": {
        "output": ["hostid", "name"],
        "limit": 5
    },
    "id": 1
}

If you're following an older blog post or Postman collection and getting this error, this is almost certainly why - check the Zabbix version before assuming your payload is malformed.

Problem 4: Inventory Fields Come Back Empty

Once the auth was sorted, the API calls worked - but host.get with inventory fields requested (os, software, os_full, etc.) came back mostly blank, except for os, which was a raw, ugly kernel string like Linux version 6.17.2-2-pve.... Not exactly what I wanted for tracking which boxes need OS updates.

Turns out this is by design, not a bug. Zabbix defaults new hosts to Inventory mode: Disabled. Even when a template - like the stock "Linux by Zabbix agent active" template - is actively collecting an item that could populate inventory (system.sw.os in this case), the item's "Populates host inventory field" setting ships blank. Zabbix apparently treats this as a deliberate opt-in, presumably to avoid inventory-table overhead at scale, but for a homelab it just means everything's empty until you flip a few switches.

Three things fixed it:

1. Set the default going forward: Administration > General > Other > Default host inventory mode → Automatic

2. Backfill existing hosts: Data collection > Hosts → select the hosts → Mass update → Inventory tab → set mode to Automatic

3. Actually map the item to a field: Data collection > Templates > Linux by Zabbix agent active > Items → find Operating system (system.sw.os) → set Populates host inventory field to OS (or OS (Full)) → Update

If you want cleaner data than the raw kernel string - something like Ubuntu 24.04 LTS instead of the full uname output - you can add a custom item that regexes it straight out of /etc/os-release:

Key: vfs.file.regexp[/etc/os-release,"PRETTY_NAME=(.*)",,,,\1]
Inventory Field: OS (Full)

Wrap-Up

None of these were individually hard problems, but each one had an unhelpful or misleading surface-level symptom: a silent no-op on the Geomap fields, a dependency error that pointed at the wrong root cause, an auth error that read like a bad payload instead of a breaking version change, and empty inventory that looked like a bug rather than a default setting. If you're standing up Zabbix 7.4 fresh, checking inventory mode and the auth header format up front will save you the loop I went through.

Installing VMware Workstation Pro on openSUSE: A Battle Report

I just wrapped up the VMware Workstation Pro chapter for the book I'm writing, and it was, without question, the longest and most frustrating install I've documented so far. Long enough that I figured it deserved its own post here, separate from the step-by-step version - this one's more about what actually went wrong and what I learned fighting through it.

Why bother with VMware at all?

VMware Workstation Pro is a paid, proprietary hypervisor, which puts it in an odd spot on Linux where free options like KVM/QEMU and libvirt already do most of the job. But Workstation Pro earns its keep with tight kernel integration, a genuinely polished GUI, snapshot/clone workflows that make experimentation cheap, OVF/OVA export for sharing VMs, and support for nested virtualization - handy if you want to run Docker or Kubernetes inside a guest. On openSUSE Tumbleweed specifically, none of that comes easy.

The install itself: death by a thousand dependencies

The first sign this wasn't going to be a zypper install afternoon was the dependency list. VMware needs kernel headers and a C++ toolchain to build its kernel modules against your running kernel - fine, expected. But it also wants a handful of specific library versions, and openSUSE Tumbleweed, being a rolling release, is usually ahead of what VMware's installer expects. In my case, that meant faking an older libxml2 with a symlink:

sudo ln -s /usr/lib64/libxml2.so.16 /usr/lib64/libxml2.so.2
export VMWARE_USE_SHIPPED_LIBS='yes'

That second line matters as much as the symlink - it tells VMware to fall back to the libraries it ships internally rather than fighting your system's newer versions. Without both pieces together, the installer gets confused in ways that are genuinely hard to diagnose from the error messages alone.

The installer finishes... and then quietly fails anyway

Here's the part that really tested my patience. After working through the download (Broadcom now gatekeeps VMware downloads behind a login, a whole separate minor annoyance), making the bundle executable, and running the installer - accepting license agreements, telemetry prompts, all the usual dance - the first launch of VMware itself failed. Not with a helpful error, but with this buried in the logs:

[Apploader] Cannot get library dependencies. (10c)
[AppLoader] Fallback to use all shipped libraries.

The fix wasn't in any VMware documentation I could find. It came down to manually running the setup helper script the installer should have run correctly on its own:

sudo /usr/lib/vmware/bin/vmware-setup-helper -e -o -u yes -c yes

After that, VMware launched cleanly. I want to be honest about how I found this - it wasn't buried deep in a forum thread after hours of searching, it was AI-assisted troubleshooting that got me there fast. I've said this before and I'll say it again here: for these obscure, distro-specific integration failures, an AI that can reason through error messages and suggest specific diagnostic commands is genuinely one of the best tools in your kit now. I don't think I'd have found that exact invocation on my own nearly as quickly.

And then the hardware plot twist

Just when I thought I was through it, I hit a wall that no amount of troubleshooting could fix: my CPU. I'm running an Intel Xeon X5670 in this box, and it turns out every VMware Workstation Pro release after 17.0.2 requires XSAVE instruction set support - which this chip simply doesn't have. Not a driver issue, not a config issue. The silicon itself doesn't support it.

That leaves two options, neither of which is great: switch to different hardware, or downgrade to 17.0.2. And the downgrade path isn't clean either - the kernel modules VMware 17.0.2 expects were built against older kernel APIs, and Tumbleweed's rolling kernel has moved well past that point. So it becomes its own compatibility puzzle, layered on top of everything else.

What I'd tell someone else attempting this

A few honest takeaways, in the spirit of documenting the failures alongside the wins:

• Take VM snapshots liberally while you work through this. I can't overstate how much easier this made recovering from bad states along the way. Without that safety net, this install would have been genuinely miserable rather than just long.
• Check your CPU's instruction set support before you start, not after. lscpu will tell you what you're working with - if XSAVE isn't listed and you're on older enterprise-grade hardware like mine, save yourself the afternoon and either plan on 17.0.2 from the start or reconsider the hardware.
• Rolling-release distros and vendor installers built for "stable" Linux baselines are going to fight each other. This isn't the first time I've hit this in this book - Docker Desktop and a couple of other tools have had their own version of this same friction on Tumbleweed. If you're on a rolling release, budget extra time for exactly this kind of gap.
• Don't be afraid to manually run what the installer should have run for you. The vmware-setup-helper fix wasn't documented anywhere obvious, but the installer's own failure message was specific enough to point toward it once I stopped assuming the GUI installer was the only supported path.

The full step-by-step version of this - with every screenshot and command in sequence - will be in the book. This post is more the "here's what it actually felt like" version. If you're fighting the same install right now: it does end, and it does eventually work. Good luck.

Setting Up a Palworld Dedicated Server (Or: Everything I Got Wrong Along the Way)

I run a homelab, I've got a Proxmox cluster with more hardware than any reasonable person needs, and my kid wanted a Palworld server. This felt like an easy weekend project. It was not an easy weekend project. But it was a genuinely useful one, because almost every mistake I made along the way was the kind of mistake that teaches you something about how your own infrastructure actually behaves - not just how you assume it behaves.

This post is the honest version of that journey, bugs and bad assumptions included.

Step one: SteamCMD doesn't just install on Debian 13

First stumble, and it happened before I'd even touched Palworld. apt install steamcmd on a fresh Debian 13 (trixie) box just fails - no candidate. SteamCMD lives in the non-free component, and it's also a 32-bit binary, so you need i386 architecture support enabled on what's otherwise a clean 64-bit system:

sudo dpkg --add-architecture i386
sudo sed -i '/^Components:/ s/$/ contrib non-free non-free-firmware/' /etc/apt/sources.list.d/debian.sources
sudo apt update
sudo apt install steamcmd

I initially tried chasing this by pointing apt at unstable, which is exactly the wrong move - trixie doesn't have an unstable suite to pull from, and all that did was generate a pile of warnings and waste twenty minutes. Lesson: when a package "doesn't exist," check which component it lives in before you start reaching for bigger hammers.

Auto-starting the server: systemd, not sysctl

Small but worth naming, because I hear people conflate these constantly (I did too, once, years ago): sysctl is for kernel parameters. What you want for "run this on boot and restart it if it dies" is a systemd unit:

[Unit]
Description=Palworld Dedicated Server
After=network.target

[Service]
Type=simple
User=palworld
Group=palworld
WorkingDirectory=/home/palworld/.local/share/Steam/steamapps/common/PalServer
ExecStart=/home/palworld/.local/share/Steam/steamapps/common/PalServer/PalServer.sh
Restart=on-failure
RestartSec=10
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target

That LimitNOFILE bump matters - Palworld servers are known to want a higher open-file limit than the systemd default gives you.

The SMB mount that wasn't allowed to happen

I wanted to pull some save files in from a Samba share on my Proxmox host. Straightforward, I thought - install cifs-utils, mount the share, done. Except every mount attempt came back permission denied, even though smbclient proved the credentials were completely correct.

The tell was that dmesg itself failed with "Operation not permitted," even under sudo. That's the fingerprint of an unprivileged LXC container - it doesn't get the kernel capabilities mount.cifs needs, full stop, no matter how correct your credentials are. No amount of fiddling with vers=, sec=ntlmssp, or credentials file permissions was ever going to fix it, because the problem wasn't the mount command - it was the container.

The right fix was to stop trying to mount inside the container entirely: mount the CIFS share on the Proxmox host, then bind it into the container as a mount point:

# on the host
mount -t cifs //192.168.1.12/public/shared/PalWorld /mnt/palworld-share -o credentials=/etc/samba/creds-palworld

# then
pct set 100 -mp0 /mnt/palworld-share,mp=/mnt/palworld-share
pct reboot 100

No capability loosening on the container needed. This is one of those things where the actual fix is almost embarrassingly simple once you know it, but the error message gives you zero hint that "unprivileged container" is the root cause.

The UDP odyssey (this is the big one)

This is where most of my week went, and it's the part I want to actually walk through honestly, because I got it wrong more than once.

I only have one public IP and one exposed host through my DMZ, so I front everything with nginx and route by hostname. That works great for HTTPS - SNI does the routing for you. It does not work for a raw UDP game port, because there's no hostname to route on at the UDP layer. Palworld talks straight to an IP:port.

nginx's stream module handles this - it's a completely separate config block from the http one, living at the top level of nginx.conf, not inside it:

stream {
    upstream palworld_udp {
        server 192.168.1.90:8211;
    }

    server {
        listen 8211 udp;
        proxy_pass palworld_udp;
        proxy_timeout 60s;
    }
}

I got this working internally almost immediately. Externally, nothing. Classic "works on my machine" - except "my machine" was my own LAN, which is a much easier test than I gave it credit for.

Mistake #1: proxy_timeout 1s. My first pass at this config had a one-second timeout on the UDP proxy session. Fine on a LAN with near-zero latency. Fatal over the real internet, where a one-second gap between packets during the connection handshake is completely normal - nginx would tear the session down before the handshake even finished. Bumped it to 60s. Progress, but still not fixed.

Mistake #2, and the one that actually embarrasses me a little: proxy_responses 0. I'd set this thinking it was a reasonable default. It is not. That directive tells nginx's UDP proxy to not relay responses back to the client - it's meant for fire-and-forget protocols like syslog, where nothing is expected in return. Palworld is a request/response protocol through and through. With that setting in place, nginx would faithfully forward the client's packet to the game server, receive the game server's reply... and just drop it on the floor instead of sending it back out. I confirmed this with tcpdump - three lines of traffic (client in, forward to backend, reply from backend) and then silence where a fourth line should have been.

stream {
    upstream palworld_udp {
        server 192.168.1.90:8211;
    }
    server {
        listen 8211 udp;
        proxy_pass palworld_udp;
        proxy_timeout 60s;
        # no proxy_responses line - the default relays replies, which is what you want
    }
}

Once that was gone, the round trip actually completed. tcpdump is not optional equipment for this kind of debugging - without watching the packets literally stop at one specific hop, I'd have kept guessing at the router.

And a red herring worth naming, because it wasted real time: online "open port checker" tools are TCP-only. Running one against my UDP game port and seeing "closed" told me nothing useful - it wasn't even testing the right protocol. If you're debugging UDP, those tools are close to useless; tcpdump while attempting a real connection is the only test that actually tells you the truth.

The REST API and the setting that wouldn't stick

Palworld ships a REST admin API - player counts, kick/ban, clean shutdown, all over HTTP on port 8212. I wanted it for a simple status page. Enabling it should be a one-line ini edit:

RESTAPIEnabled=True
RESTAPIPort=8212
AdminPassword="something-reasonably-long"

I set it. Restarted the service. Confirmed the file was correct with grep. Still got Unauthorized (AdminPassword is empty) on every request, including with a perfectly valid password. Confirmed the ini's contents byte-for-byte with xxd in case of some invisible character - clean. No duplicate keys. No parse errors. The server was, somehow, just not reading a setting that was sitting right there in the file it had definitely loaded.

Turns out there's a second file: WorldOption.sav. Once a Palworld world has been generated, the server starts reading most world settings - apparently including AdminPassword - from this binary save file instead of the ini, and it silently overrides anything you put in PalWorldSettings.ini from that point forward. The ini edit was completely correct. It just wasn't being consulted anymore.

The fix: stop the server, delete WorldOption.sav, restart. That forces one clean read from the ini, and the server regenerates a fresh .sav from whatever's actually in the ini at that point. Worth knowing this isn't a permanent fix so much as a "you get exactly one clean boot" fix - change AdminPassword again later, and you'll be deleting that file again.

I would not have found this without just searching for the literal error message alongside "PalWorldSettings.ini" - sometimes that's still the fastest path, even after you've checked everything that seems checkable.

DMZ vs. port forwarding - the belated correction

I'd been running everything through my router's DMZ setting, pointed at my nginx box. It works, but it means every port on that box is exposed, whether anything's listening there or not. Once I had the actual list of ports something was really using - 80, 443, and 8211/udp - switching to explicit port-forward rules and turning DMZ off was a five-minute change that meaningfully shrinks what's actually reachable from the internet. I should have done this from day one instead of reaching for DMZ because it was the first thing I found.

The last gotcha: testing over cellular lies to you

Even after all of the above was fixed, I still saw intermittent "connection timed out" while testing from my phone's hotspot - mid-session, after successfully connecting and starting to build a character. Packet capture showed the server responding just fine, continuously, into total silence from the client side.

This one wasn't my infrastructure at all - it's carrier-grade NAT on the mobile network expiring the connection's NAT mapping during a quiet stretch (like sitting on a character-creation screen not sending anything). The server has no way to know the client's public-side mapping just vanished underneath it.

Confirming this took setting up a free WireGuard tunnel through an Oracle Cloud always-free instance and routing my test traffic through that instead of the cellular hotspot. WireGuard's PersistentKeepalive keeps its own NAT mapping alive regardless of game activity, which sidesteps the whole problem. Once I tested through the tunnel, everything was rock solid - confirming the server-side setup had been correct for a while, and cellular hotspots are just an unreliable way to validate a UDP game server.

What I'd tell past-me

• Check which apt component a package lives in before assuming it's missing entirely.
• sysctl and systemd are not the same thing, and confusing them wastes your own time.
• Unprivileged LXC containers can't mount CIFS shares directly - mount on the host, bind into the container.
• For UDP proxying: watch the packets with tcpdump, don't guess. TCP-only port checkers will actively mislead you.
• proxy_responses 0 is not a safe default for anything request/response-shaped.
• If a config value looks right in the file but the app insists otherwise, check whether there's a second source of truth (a save file, a cache, a database) that's silently taking precedence.
• Test UDP game servers from a real home connection, not a cellular hotspot - carrier NAT will produce failures that have nothing to do with your setup.

None of these were exotic problems. Every single one had a boring, findable explanation once I stopped assuming and started watching the actual traffic. That's most of what debugging is, honestly - it's just occasionally really annoying to remember in the moment.

Automating Leantime with C# — Troubles, Trials, and Lessons Learned

I am working on a fun side project - an automation tool to help me manage and maintain my Proxmox home lab. After a few weeks of building out features, I realized my ideas for changes and enhancements were quickly overflowing the amount of mental swap space I was able to commit to the project. I needed a backlog, and I needed it to grow as I worked. I was already using Leantime to manage a project to track plans for wiring my home network and another project to manage the book I am writing, so it was obvious where my tasks needed to live. The problem is, manually adding tasks for an automation project seems counterproductive. It only took me about three days to get code that could manage to automate Leantime with C# code. Those three days were not boring.

The Starting Point

Leantime exposes a JSON-RPC 2.0 API. Not REST, not GraphQL - JSON-RPC. If you have not worked with it before, every call is a POST to a single endpoint with a method name and params in the body:

json
{
  "jsonrpc": "2.0",
  "method": "leantime.rpc.Tickets.Tickets.addTicket",
  "params": { "values": { "headline": "My Task", "type": "task" } },
  "id": 1
}

Authentication is an x-api-key header. Simple enough. I wrote a thin C# client around HttpClient, set up the serialization, and had my first successful call within an hour. That was the last time something worked on the first try.

The Secret 'Secrets' Side Quest

I did not want to paste an API key into a config file. I already had Infisical running in my lab for other secrets, so the plan was to store the Leantime API key there and resolve it at runtime.  But, I wasn't super happy with the Infisical UI so I thought it would be a great time to look at OpenBao (the open-source Vault fork) since it was much cleaner to interact with.  

This turned into its own project within the project. I had previously been using Infisical as my secret provider, so my tool already had a SecretProviderFactory that could resolve secrets by name. Adding OpenBao as a second provider meant abstracting a KV v2 client, writing a scoped read policy, minting a service token, and wiring it all together before I could even start calling the Leantime API:

csharp
var provider = SecretProviderFactory.Create(config);
var apiKey = await provider.GetSecretAsync(config.Leantime.ApiKeySecretName);

Three lines of consuming code. Two days of infrastructure. Worth it for keeping secrets out of source control, but I cannot pretend I did not consider hardcoding the key more than once.

The Update

When I first connected to Leantime and started calling the API, things were inconsistent. Some methods worked, some returned -32601 Method not found, and the responses I got back were not what the documentation suggested. Turns out I was running an older version of Leantime (3.5.0). I updated Leantime to 3.8.0, which fixed some issues and created others - rate limiting appeared where it had not been before, and some method signatures changed. 

Did I mention that Leantime is written in PHP?  I once left a job because they were migrating from SharePoint to Drupal.  Seriously.  But Leantime is a popular project so it shouldn't be that bad.  Little did I know, I was going on a trip through a strange land.  

The update itself was straightforward. SSH into the LXC, pull the release, update the files. But now I had a moving target - code that worked yesterday might not work today because the API contract changed underneath me. There is no published schema or changelog for the RPC layer, so my approach became: try it, read the error, and adapt.  

The Rate Limit Wall

After the update, Leantime started enforcing rate limits - roughly 15 requests per minute. My seeder was trying to create 70 tickets as fast as the network could carry them. The first run got about 4 tickets in before the server started returning HTTP 429.

The fix has two parts. First, the client retries automatically when it sees a 429, respecting the Retry-After header:

csharp
for (int attempt = 0; attempt < 3; attempt++)
{
    var response = await _http.PostAsync(_apiUrl, content);

    if (response.StatusCode == HttpStatusCode.TooManyRequests)
    {
        var retryAfter = response.Headers.RetryAfter?.Delta
            ?? TimeSpan.FromSeconds(60);
        await Task.Delay(retryAfter);
        continue;
    }

    // process response...
}

Second, the seeder introduces a 4500ms delay between ticket creates to stay under the limit proactively rather than constantly hitting the wall and backing off. The deleter uses a shorter 1200ms pace because single-field patches are less expensive on the server. It is not elegant, but it is reliable.

The API Quirks

This is where most of my time went. Leantime's JSON-RPC layer has a personality.

Array-wrapped booleans. Tickets.delete returns [true] - an array containing a boolean - not true. My initial response parser threw a deserialization exception because it expected a scalar. I added unwrapping logic to peel arrays of length 1.

Multiple success formats. Creating a ticket might return 5 (the new ID as an integer), or true (success with no ID), or "5" (the ID as a string), or {"id": 5} (an object), or [5] (an array). I am not exaggerating. The response parser has to handle all of these:

csharp
return el.ValueKind switch
{
    JsonValueKind.Number => el.GetInt32(),
    JsonValueKind.True   => 1,
    JsonValueKind.String => int.TryParse(el.GetString(), out var v) ? v : -1,
    JsonValueKind.Object => TryExtractId(el),
    JsonValueKind.Array  => el[0].GetInt32(),
    _ => -1
};

Parameter wrapping. Some methods expect {"values": {...}} wrapping the actual parameters. Others expect flat params. There is no obvious pattern. I found out by trial and error which methods need the wrapper.

Missing methods. Projects.removeProject returns -32601 Method not found. It is referenced in the Leantime source code but not exposed through the RPC gateway. My delete workflow has to iterate every ticket in a project and remove them one at a time. The project shell still lingers in the UI.

Method name changes. createClient became the method in one version but was create in another. My client tries the new name first and falls back:

csharp
try { return await CallAsync("createClient", new { values }); }
catch (InvalidOperationException ex) when (ex.Message.Contains("-32601")) { }
return await CallAsync("create", new { values });

User response formats. Users.getAll returns an array on some endpoints and a keyed object on others. Same method, different shapes depending on context. I handle both:

csharp
IEnumerable<JsonElement> items = el.ValueKind switch
{
    JsonValueKind.Array  => el.EnumerateArray().ToList(),
    JsonValueKind.Object => el.EnumerateObject().Select(p => p.Value).ToList(),
    _ => []
};

Each one of these cost 15–30 minutes of debugging. Multiply that by a dozen quirks and you have a day gone.  RPC over PHP is...shiny.

Making It Idempotent

I live for repeatability when developing.  I wanted the seeder to be safe to run repeatedly. If a ticket already exists, skip it. If the project is already there, reuse it. The approach is simple - before creating anything, pull all existing tickets and index them by headline:

csharp
var existingByTitle = existingTickets
    .ToDictionary(t => t.Headline.Trim().ToLowerInvariant(), t => t.Id);

if (existingByTitle.TryGetValue(key, out var existingId))
    return existingId;

This means I can evolve the backlog JSON file - add new milestones or tasks - and re-run the seeder without duplicating anything that is already there. Combined with the rate limit pacing, a full seed of 12 milestones and 70 tasks takes about 6 minutes. Not fast, but completely unattended.

Server-Side Configuration via SSH

After I got almost everything working, I decided I should look at removing the rate limit issue that had been slowing me down so much.  I had set myself some rules for interacting with Leantime and one of them required me to manage it's server configuration without logging into the box manually. I needed to set rate limit values and manage an IP whitelist - both stored as environment variables in the .env file on the Leantime LXC without going to the machine's console.  No problem!

The automation tool I am working on already had an SSH client library for managing other Linux nodes, so I extended it to edit the .env file remotely with sed and grep:

csharp
// Read
var (output, _) = await ssh.ExecAsync(host,
    $"grep '^MCP_RATE_LIMIT=' {envFilePath} | cut -d= -f2");

// Write
await ssh.ExecAsync(host,
    $"sed -i 's/^MCP_RATE_LIMIT=.*/MCP_RATE_LIMIT={value}/' {envFilePath}");
await ssh.ExecAsync(host, "systemctl restart apache2");

Straightforward Unix commands behind a C# interface. The host and file path are derived from values already in the config file, so the CLI commands stay clean:

text
leantime rate-limit set 100
leantime whitelist add 192.168.1.103

What I Learned

All learning is good learning. The three days I spent wrestling with Leantime's API were not wasted even if the immediate goal was just "add some tickets to a project tracker." Here is what ended up in my toolbox:

JSON-RPC is not REST. The single-endpoint design means you cannot rely on HTTP verbs or status codes to tell you what happened. Error handling lives entirely in the response body. Your client needs to be more defensive than a typical REST consumer.

Defensive response parsing pays off. Building a parser that handles five different success shapes for the same operation sounds overengineered until you realize the alternative is hard-crashing at 2am when the API returns a string instead of an integer.

Rate limiting is a feature, not a bug. Once I accepted that and built pacing into the workflow, the system became more reliable than it was before the limit existed. The 429 retry logic means transient throttling self-heals without human intervention.

Automate the automation infrastructure. Spending two days on secret management feels excessive until you realize that the API key rotation, secret scoping, and an audit trail all come for free now. Every future integration just calls GetSecretAsync and moves on.

Side quests are the real project. I set out to add backlog tickets to Leantime to help me work on my side project. Along the way I built a generic JSON-RPC client, added OpenBao integration to my secret provider, added SSH-based server management, and learned more about Leantime internals than the documentation will ever tell you. Every one of those pieces is reusable for something else.  Dogfooding helps build higher quality code and much better feature sets.

Having more and varied automation tools in your toolbox is never a bad thing. The specific issues I faced creating my Leantime automation code may not apply to your projects, but the patterns - retry logic, idempotent operations, defensive parsing, secret management, SSH-based configuration - show up everywhere. The next time an API surprises you with inconsistent response shapes or undocumented rate limits, you will already have a playbook and will know that others have felt your pain.