Self-hosted recovery plans fail when freezes, vendor lockouts, and fragile boot media are treated as separate problems

Most self-hosters have backups somewhere. What they usually lack is a rehearsed path for diagnosing what failed, restoring service from those backups, and then confirming that everything is actually healthy again — before the next Monday-morning freeze.

The Four Failure Modes That Keep Appearing

Recent Reddit threads surface the same cluster of problems across very different stacks:

Proxmox freezes with no visible logs. A mini PC running Proxmox locks up every Monday before dawn. Nothing in the logs. The only recovery path is a physical reboot. The freeze itself is almost a secondary problem — the real gap is that there is no automated watchdog, no remote out-of-band access, and no documented procedure for what to do when the host is unresponsive. Someone has to physically be there.

WordPress vendor lockouts. A site owner hired an SEO vendor who now controls the hosting account and refuses to hand over a backup or restore admin access. The owner has no independent copy of the database, no snapshot they control, and no fallback credentials. This is not a technical failure — it is an access-control failure that backups alone cannot fix.

Home Assistant boot media failures. One user wrote a multi-year summary: SD-card corruption, overheated USB boot drives, missing off-host backups, and a long debugging session assisted by an LLM before reaching stable uptime. Each failure required starting from scratch rather than restoring from a known-good state, because the backups existed locally on the same media that had just failed.

Unraid kernel panics and filesystem corruption. An upgrade triggered kernel panics and data corruption. Recovery required booting from a USB rescue environment and switching the filesystem. The Unraid case is a reminder that upgrade procedures without a pre-upgrade snapshot or rollback target are just optimism.

None of these are exotic edge cases. They are the ordinary failure modes of self-hosted infrastructure, and they share a common root cause: each failure was treated as its own unique incident rather than as an instance of a general recovery pattern.

Why Treating Them Separately Makes Each Worse

When you treat a Proxmox freeze as a "Proxmox problem," you end up researching Proxmox-specific watchdog daemons without also asking whether you have out-of-band access (IPMI, a Raspberry Pi on the same network, a managed PDU) to cover any future unresponsive-host scenario regardless of what is running on it.

When you treat the WordPress lockout as a "vendor problem," you focus on legal remedies rather than also building a policy that your own credentials and your own backup copies are non-negotiable for every hosted property going forward.

When you treat the boot-media failure as an "SD card problem," you upgrade to eMMC or a more durable USB drive — but you may still leave the backup on the same physical host, which means the next failure (a power surge, a dropped NAS, a house fire) hits both the system and the backup simultaneously.

The recovery plan that actually holds up is one that covers the whole class of failure: I cannot access or trust this host. How do I restore service and verify it is healthy?

A tool like ServerCompass helps on the visibility side — it lets you see all your running services at a glance from a single dashboard, so you know immediately which hosts and services are affected when something goes wrong, rather than discovering the scope of an outage incrementally.

Building a Recovery Plan That Holds

A useful recovery plan has three layers: access, restore, and verify.

Access means you can reach the system even when the OS is unresponsive. For physical hardware, that is IPMI or a Pi-based KVM. For a VPS, that is the provider's out-of-band console (Hetzner's rescue system, DigitalOcean's recovery console, etc.). For hosted services, that means you have independent credentials — your own account at the registrar, your own copy of DNS records, your own database export — not just admin access inside a third-party control panel.

Restore means you have a tested, off-host backup that you can actually use. "Off-host" is load-bearing: the Unraid corruption and the Home Assistant boot-media failures both demonstrate that a backup on the same physical device is not a backup for the failure modes that matter most. Off-host means a different machine, a different datacenter, or a cloud object store. Tested means you have run a restore drill at least once and it produced a working system.

Verify means after you restore, you confirm the service is actually functioning — not just running. A process that is up but serving errors is not a recovered service. Verification can be as simple as a curl health-check script and a quick walk through the UI, but it needs to be part of the documented procedure, not an afterthought.

For the Proxmox freeze case specifically: a watchdog daemon (Proxmox has one built in) combined with IPMI or a remote power switch eliminates the "someone has to physically be there" requirement. That is not a Proxmox fix — it is an access-layer fix that applies to any bare-metal host.

Rehearsal Is the Missing Step

The Home Assistant user's multi-year summary is instructive: each failure taught them something, but the learning came from the failure itself rather than from a rehearsal before the failure. By the time they reached stable uptime they had effectively run a recovery drill — just involuntarily.

Scheduling a deliberate restore drill twice a year forces you to discover the gaps before production does. Pick a non-critical VM or a staging environment. Restore from your off-host backup without using anything on the failed host. Verify the result. Document what broke in the procedure. Fix it.

The drill also answers the hardest question in a real incident: not "do I have a backup?" but "how long does a restore actually take, and what do I need to have ready before I start?"

A recovery plan that has never been run is not a plan. It is a note.

Checklist

• Confirm you have out-of-band access to every host (IPMI, provider rescue console, or a network-attached KVM) before the next freeze, not after.
• For every hosted service managed by a vendor, hold your own copy of credentials, DNS records, and database exports — independent of whatever access the vendor has.
• Store at least one backup copy off-host: a different physical machine, a different datacenter, or a cloud object store.
• Before any major upgrade (kernel, Unraid version, Home Assistant major release), take a snapshot or backup you can roll back to, and document the rollback steps.
• Set up a watchdog or health-check for any host that cannot be physically reached quickly — Proxmox has a built-in watchdog; use it.
• Run a restore drill at least twice a year: restore from off-host backup to a clean environment and verify the service is functional, not just running.
• Use a monitoring dashboard (such as ServerCompass) to track service health across hosts so you know the scope of an outage immediately.
• Document the recovery procedure, including estimated time and required resources, so you are not improvising during the incident.