🛰️ Won the Power Outage Battle, Lost to a Ping
📆 2025-08-04 16:40
They said running a homelab was a fun hobby. They didn’t tell me it would turn into a full-time job involving power engineering, DNS sorcery, and the occasional battle with an ancient USB cable. But here we are.
⚡ NUT + UPS + Docker = Apocalypse-Proof?
My setup looks like this:
- One humble APC UPS (aka The Battery Brick of Destiny)
- A Dockerized NUT (Network UPS Tools) server running on Alpine, because I like pain and tiny images
- NUT clients checking UPS status with upsc, like obedient little penguins
- Logic to gracefully shut down services when the battery dips below "oh no" levels
Extras:
- Ping tests to check if my nodes are alive or just lazy
- Wake-on-LAN packets to yell at lazy nodes
- Gotify for shouting notifications to my phone like a panicked toddler
The idea? If the grid fails, my cluster becomes a polite gentleman: it warns everyone, saves its work, and lies down until the lights return.
And it worked! ... until yesterday. My internet went down for more than 3 hours. (Not that I have a lot of traffic and everybody noticed that my sava.rocks services were down).
🚫 The Unexpected Boss Fight: Internet Outage
Power stayed on. But the internet died.
Suddenly, my proud little cluster — which survived power chaos like a champ — sat idle, confused, alone, unable to resolve even a simple DNS query.
It was like watching a bodybuilder forget how to open a peanut jar.
Why? Because while I protected against power failure, I forgot the other lifeblood of modern homelabs: connectivity.
Without internet:
- My Gotify alerts went nowhere.
- DNS? Nah. Local resolution was so last week.
- My cluster, once mighty, was now just a blinking box of existential dread.
📶 Time for Plan B: A Bucket of Wireless Internet
I needed a backup connection. A plan. A dream.
So, what counts as emergency internet?
- A 4G/LTE modem stuffed into the router like a digital EpiPen
- Tethering from an old Android phone named "HotspotOfLastResort"
- A Raspberry Pi trained to scream "INTERNET DOWN!" via text-to-speech
Okay, maybe not the last one. (Or maybe yes.)
Even better: bake internet monitoring into my NUT logic:
🌐 But Wait — What About the Domain?
Keeping your services online isn't just about having internet. It's about being reachable. So I needed to fix DNS too.
🌀 Option 1: Dynamic DNS Like It’s 2009
- Use DuckDNS or similar to point `my-cool-cluster.duckdns.org` at whichever IP is online
- Set up your router or a script to update the DNS record when the connection flips from "cable" to "we found Wi-Fi in the neighbor's mailbox"
- Bonus: add a cron job that checks if your LTE IP changed, and sighs loudly when it does
🧠 Option 2: Fancy DNS Failover
- Use a DNS provider with health checks like Route 53, Bunny, or Cloudflare
- Set up two A records — one for your main line, one for backup
- If the main IP dies, the provider swaps to LTE like a digital stunt double
- Downsides: sometimes DNS propagation is slow enough to brew coffee between flips
🛠️ Option 3: Self-Hosted DNS With Control Issues
- Run `CoreDNS` or `dnsmasq` in your cluster
- Teach it to serve different IPs depending on ping tests, rituals, or moon phases
- Will it work? Yes. Will it break in hilarious ways? Also yes.
📚 Takeaways From My Internet Faceplant
- ⚡ Power is just one piece of the uptime puzzle.
- 🌍 Internet is the other piece. Sometimes the bigger one.
- 🧩 DNS is the glue holding everything together — or exploding spectacularly.
My cluster lived through a lot of power blackouts like a stoic wizard. But a simple internet outage turned it into a confused puppy.
🔮 What’s Next?
I'm now working on:
- A fully automatic failover to LTE
- Smarter DNS routing that doesn’t require divine intervention
- Possibly duct-taping a Starlink antenna to a drone - if I get a drone :)