Whole-server recovery

In preview

A backup you have never restored is a hope. This one was restored: a server was backed up and destroyed, and a blank node was rebuilt from the backup and one encrypted bundle, with every item compared.

Built and lab-tested

  • A scheduled server backup writes one encrypted restic repository (a path, S3, SFTP and more) and a recovery bundle with the keys. The bundle is encrypted with a passphrase you hold and is refused if it would sit inside the repository.
  • In the run, a blank node with the panel installed was rebuilt to "every item verified" in 312 seconds, of which the restore script took 200. The lab server held two customers, sites, MariaDB and PostgreSQL databases, a mailbox, DNS zones, a certificate and a Team member.
  • Came back and compared item by item: users with their old passwords and authenticators, accounts and every file (byte-identical), sites, databases with row counts, mailboxes with all messages, DNS zones and DNSSEC keys, the certificate served, DKIM keys, stored secrets such as the mail relay login, plans and settings, and the whole audit log, verifying.
  • A check script lists each item as OK or MISSING and names what a backup never covers. A backup newer than the installed panel, a wrong passphrase, an unreachable repository and a node that already has accounts are all refused before anything changes.
  • If you lose the bundle but know the repository address and password, a data-only restore brings back databases, homes, mailboxes, DKIM keys, certificate files and DNS data; the panel's own settings must be created again.
  • The installer puts the three scripts on every server and refreshes them on upgrade.
  • Per account, without losing a server: restore files to a folder beside the originals, a database to an earlier dump (as a new database by default), one mailbox into a Restored folder, or a whole account. Each shows a plan first.

Not there yet

  • Point-in-time recovery between backups is not built: you get the time of the last server backup, and per database the time of a dump.
  • Does not come back: mail in the queue, signed-in sessions, the licence activation (bound to the old machine), the Store connection, other nodes of a cluster, container volumes and images, anything outside the backed-up paths, and your registrar's DNS records and the new IP address.
  • The audit chain reports that the log was rewritten after a restore; the restore script records that once, with a reason, and the incident stays listed for ever.
  • The run used one Ubuntu 26.04 lab node in a container and an S3 store. Other systems and real machines were not run.

What to keep outside the server

The recovery bundle and its passphrase, in a place that is neither the server nor the backup repository. The vault key is not in the repository on purpose: a backup that held the key next to the data it unlocks would protect nothing.

Every action in this area is in the panel, one keystroke away in the command palette (Ctrl K), on the command line and in the API, and most changes can be previewed as a plan first. Built and tested in our lab; RespireCloud is not released yet.

Modules in this area