Backups

Model

• Target = one restic repository for one account (own random 40-char password in the vault, target/<id>; ADR 0013 §7). Kinds: local (/var/lib/rc-backups/<account>/<id>, root 0700), connection (any storage connection through restic's rclone backend, path below the connection's base path) and s3 (a private bucket + two keys created on the built-in store through the public storage.* actions: a backup key and a prune key, both in the vault). Creating a target installs restic, runs init and fails cleanly (everything created is rolled back) when it cannot write/read.
• Immutability (immutable_days on s3): the bucket is created with S3 object lock (COMPLIANCE, default retention). Proven in the lab: a permanent delete of a snapshot object version with the target's own credentials is AccessDenied, the panel refuses to force-delete the bucket, accounts.deleted keeps the bucket. Honest limit: SeaweedFS has no "write without delete" action, so a plain S3 DELETE (what restic forget does) only adds a delete marker - the data stays recoverable until retention ends but restic no longer sees that snapshot. Prune cannot free space inside the window. Backup, check, snapshots, ls, stats use --no-lock; forget/prune must lock (restic refuses otherwise), so on a locked bucket a lock file may outlive the run (see follow-ups).
• Job = scope (paths relative to the home, excludes as restic globs, databases: export every database first) + schedule (interval >= 15 min, daily, weekly, monthly, account time zone, up to 5 min stable jitter) + retention (keep-last/hourly/ daily/weekly/monthly/yearly, default 7d/4w/3m, daily capped by backups.retention_days) + targets (each gets every run, per-target result). backups.schedule.tick (every minute, system actor) advances next_run before running, so a slow run never repeats, runs at most 5 due jobs per tick, a 5 % restic check per target weekly, a test restore monthly, and raises backups.stale after 48 h without success (once a day).
• Run (backups.job.run, or the tick): per-target busy flag (one backup/restore/check per target, stale after 3 h) -> databases.db.export into <home>/rc-backup/db/<db>.sql.gz (a database that fails to export fails the run: no silent partial backups) -> restic backup <home> --one-file-system --exclude-caches under nice 19 / ionice idle -> 3 attempts with backoff -> stats -> retention (forget + prune with the prune credential, only after success, only snapshots tagged rc of that account; pre-restore safety snapshots are tagged pre-restore and never pruned). Limits: backups.manual_runs_per_day (per target-run), backups.storage_gb (local and built-in S3 targets; skipped run + backups.quota.exceeded, nothing deleted).
• Restore: restore.files (default into <home>/restore-<time>/ with the snapshot's relative tree, ownership/mode/symlinks kept, --verify; inplace over the originals; other into another account on the same server, re-owned), restore.database (replace with a safety dump first, new database, download the dump), restore.account (plan via dry run, typed account name, safety snapshot first, files in place then every database import, rollback within 24 h via restore.rollback). All dry-runnable.
• verify.run (restic check, optional subset), verify.test_restore (random 5 files into a scratch dir, hash verify, cleanup), retention.preview (forget --dry-run), recovery.export (admin, critical: repository locations + passwords + S3 keys; audited).

Backup sources (contract, w11-62)

A module that keeps data of an account outside its home adds it to the account's backup by PROVIDING three actions (declared in its module.yaml, permissions of its own) - there is no registration call, the backups module lists its providers (sourceProviders in cp/sources.go) and declares exactly these three per provider in system_calls (a scheduled run is the system, ADR 0018), so a module cannot add itself and cannot be asked for anything else:

• <module>.backup.sources {account_id} -> {sources: [{id, title, bytes (estimate), consistency: "quiesced" | "crash-consistent", note}]}. Read only. Empty = nothing to add.
• <module>.backup.stage {account_id} -> {staged: [{stack|id, consistency, bytes, note}]}. Copies the sources into <home>/rc-backup/<module>/... as the account's files (run by the provider's agent op, never through a path the account could swap for a symlink), quiescing where the source declares how (stop, copy, resume - resume also on failure).
• <module>.backup.unstage {account_id, path?} removes the copies (after the run, and the restore folder after a restore). Rules: the provider resolves the account itself from account_id with its normal account check (a user can name only their own account, a reseller a customer's, a Team member none) and reads only that account's rows; the backups module passes the target's account and never anything the caller typed. sources that fail with not found / forbidden / precondition (module absent, plan without it, no runtime) are skipped; any other error fails the run (no silent partial backup). A staged copy travels in the files backup of a whole-account job (paths: ["."], not databases-only); a job with scope.no_sources: true leaves them out (the schedule form shows a switch "Include the data of your container stacks" and says what a whole-account backup holds). The run's note says which sources were copied crash-consistent. The Restore tab has a "Containers" kind that calls backups.restore.sources. Restore: backups.restore.sources {target_id, snapshot, stack, into?, confirm?, secrets?} fetches the staged copy from the snapshot into restore-<time>/ in the home, calls <module>.backup.restore, removes the folder. Container stacks (provider containers): the stack's named volumes, one <volume>.tar.gz each + stack.json (compose, non-secret variables, the NAMES of secret variables, app id). A stack is quiesced (stopped for the copy, started again only if it was running) when it declares RC_BACKUP_QUIESCE=stop (compose stacks) or is a catalog app with backup.volumes; otherwise it is copied while running and said so (a database in it recovers as after a power cut). Restore creates a NEW stack (default <stack>-restored) from stack.json; secret values are never in a backup, so they are given again (secrets). Into an EXISTING running stack only when its name is typed as confirm. Extraction is confined to the new volume (os.Root: no .., no symlink out, no hard link out), ownership limited to the dockremap range, setuid/setgid dropped. Costs: the staged copy occupies the home's disk (and its quota) during the run.

Verified in the lab (2026-10-09)

Connect external S3 (fake cloud) -> init repo through rclone -> job with files + MariaDB -> run twice (second run added 4.7 kB of 3 MB) -> delete files and drop a table -> alternate-folder restore (sha256 equal, mode 640 kept), in-place restore, database replace (row data back) -> whole-account restore after deleting www/report.bin and dropping a table, then rollback -> verify 100 %, test restore, retention keep-last 2 removed 2 of 4 snapshots on local and S3 targets -> immutable target proof -> restore into a second account -> scheduler ran the due job -> account deletion removed targets, jobs, local repo, keys and connection config.

Known gaps / next (see handover)

Agent op timeout is 60 s (module SDK cannot set it) so big backups will fail until Host.Agent takes a timeout; no progress/cancel; mail and containers volumes, PITR, server-level repo + DR restore, tar.zst archive download, mailbox restore, domain remap, cron expressions, restoring deletes nothing that is not in the snapshot (a rollback does not remove files the restore added), PostgreSQL "replace" needs a clean import (databases.db.import does not drop existing tables), restic/rclone from the distro (0.16.4 / 1.60.1), per-snapshot delete.