Cluster

Spec docs/specs/cluster.md. Desired state lives in schema m_cluster (node_meta, placement_policies, placements, lb_pools, vips, migrations, sync_links, lb_events); the node list itself (id, roles, status) stays in core and is read with core.node.list / written with core.node.update|retire. Enrolment tokens stay core.node.enrol_token.create.

What exists

• Add a server from a default install (w11-128): a default install keeps the agent channel (NATS, 4222) on loopback. The enrol command and the bus address a new agent gets name the panel's public address (RC_PUBLIC_URL, else the first RC_PANEL_HOSTS) when RC_PANEL_URL is loopback. cluster.panel.bus_status says whether the channel is open; cluster.panel.open_bus (high risk, step-up, preview, cluster.panel.manage) sets RC_NATS_LISTEN=0.0.0.0:4222 and the TLS names through agent op cluster.panel.bus, opens the port through security.firewall.cluster for the addresses entered (none = any), restarts the panel and records whether the channel answers on the public address; cluster.panel.close_bus undoes it and is refused while other servers are enrolled. The addresses live in panel_bus and feed cluster.firewall.requirements.
• Nodes: cluster.node.list|get|refresh|label|cordon|uncordon|roles|drain|undrain|remove, cluster.topology.get. refresh runs cluster.node.inventory (hostname, OS, kernel, CPU, RAM, load, disks, private/public IPs, haproxy/keepalived/nginx active) and caches it; roles previews packages (lb -> haproxy+keepalived+psmisc) and installs them through the allow-listed system.apt path. control can neither be given nor taken away. Removal refuses while accounts, pools, VIPs or sync links remain (force needs confirm=<node id>).
• Placement: policies (least_loaded, spread, pack, pinned; role, plan, labels, zone, priority). simulate shows every candidate and why it was rejected (cordoned, draining, cap, zone, label, no role, offline). assign records the decision and its reason (cluster.placement.get). accounts.created records where an account landed; accounts.deleted forgets it. accounts.create without a node_id asks the internal cluster.placement.decide and records the decision as the placement reason; an explicit node_id wins.
• Load balancers (lbgen renders, the agent validates): HAProxy on lb nodes; a pool is one document (hostnames, port, http/tcp, algorithm, health check with expected status or body regex and interval/rise/fall, sticky cookie-insert / prefix / source / header, TLS terminate / re-encrypt / passthrough with redirect + HTTP/2, maxconn, per-IP connection cap and request rate, allow/deny CIDRs, PROXY protocol to backends, maintenance 503 page, backend weight / backup / drain / disabled). Pools on the same port share a frontend and route by Host. Every user value is validated against a strict grammar before it reaches the file; there is no free-text snippet. create|update|delete|backend|apply all go: render per LB node -> haproxy -c on a scratch copy -> diff -> certs + atomic swap -> graceful reload -> proof that a new worker pid answers (a failed reload leaves the old worker running and systemd still says "Reloaded") -> otherwise the previous generation is restored and the call fails with apply_failed. The pool row is stored only after every LB node accepted it; nodes already changed are put back if a later one fails. Dry run returns a Plan with the real diff per node. Backend drain/disable/remove first set the server to DRAIN over the runtime socket and wait for its sessions to end (pool drain_timeout_sec, max 120 s). cluster.lb.poll (1 min) records UP/DOWN transitions and emits cluster.lb.backend_down|up. Certificates: tls.cert_id uses the PEM the ssl module already deployed on that LB node (/var/lib/respirecloud/ssl/certs/<id>/{fullchain,privkey}.pem, keys never travel over the bus); without it the agent makes a stable self-signed bundle so the pool works at once.
• Floating IPs: keepalived VRRP with unicast peers (works where multicast is blocked), per-node priority, nopreempt unless preempt, a pgrep -x haproxy track script with no weight (w11-19: a failing check puts the instance in FAULT, the address is released and the peer is told at once, whatever the priorities and the preempt setting; the administrator gets cluster.vip.moved, or cluster.vip.holder_unhealthy when no node took the address over; a VIP created before w11-19 gets the new file with its next cluster.vip.update), PASS auth generated and never returned or shown in diffs. state reports MASTER/BACKUP/STOPPED per node by looking for the address on the interface; failover stops keepalived on the holder until the target owns the address, then starts it again as standby.
• Replicated site directories (cluster.sync.*): one-way pull from the source node to targets over the same pinned one-shot TLS stream migrations use, newer files only (tar --newer-mtime), run by cluster.sync.tick every minute and on demand. Chosen over lsyncd / rsync-over-ssh because it needs no SSH keys, no extra daemon and no secret shared between nodes. Consequence: lag is interval_sec (min 60) + up to 1 min, deletes are not propagated, and writes must go to the source node. The zero-lag alternative (shared SeaweedFS filer mount from storage) is not wired.
• Migration / drain (cluster.migrate.*, cluster.node.drain): steps preflight -> dump (databases listed by databases.db.list; mariadb/mysql via mysqldump, postgres via pg_dump, local default socket) -> transfer (source opens a TLS listener on 38400-38499 with a single-use token and a pinned fingerprint, the target pulls; the target home must be empty) -> restore -> verify (file count, bytes and a digest of sorted path+size must match the live source) -> switch -> cleanup (staging removed; the source copy is kept, marked by the agent so a move back replaces it and cluster.migrate.purge_source can remove it). Failure or cancel before the switch removes what was imported on the target; the source is never modified. A drain cordons the node, plans one migration per account with the placement policy (counting accounts already planned), runs them (2 at a time) and flips the node to drained when none are left. Dry run lists the moves with size estimates.
• Firewall: cluster.firewall.requirements lists the ports (4222, 38400-38499, VRRP proto 112, every pool frontend port) for the security module to open; this module does not touch the firewall.

Verified in the lab (PROFILE=cluster, 2026-10-10, HAProxy 2.8.16, keepalived on Ubuntu 24.04)

See docs/tasks/w8/w8-01-cluster-lb.handover.md for the command log: LB pool over cp1+web2 on both nodes, kill a backend -> traffic stays on the other with zero failures, 4 reloads under load with no failed request, sticky cookie, TLS terminate + h2 + redirect, port clash -> automatic rollback, VIP between two LB nodes (hard failure of the master: traffic back in about 3 s in the lab, up to 8 s once measured; HAProxy dying on the holder: about 1.5 s), manual failover, incremental site sync, migration web2 -> cp1 with verified identical copy, rollback on failure and on cancel, drain with cordon.

w9-02 follow-ups (landed)

• accounts.set_node {id, node_id} (internal, system only) moves the row, ensures the user on the target and emits accounts.moved. Migration steps are now preflight, dump, transfer, restore, verify, switch, follow, cleanup. follow calls the idempotent handlers web|ssl|dns|mail|databases.on_account_moved in that order (they are also subscribed to the event). A failing follower leaves the migration awaiting_switch with the reason, emits cluster.migration.follow_failed, and the tick retries every ~90 s; after the switch the copy is never rolled back or cancelled. web re-renders vhosts on the new node then removes them from the old; ssl copies chain+key node to node (cluster.cert.mirror), installs for the web targets and removes the old copy; dns rewrites A/AAAA holding the old node address (records pointing elsewhere, e.g. a pooled zone, stay); databases records the new node and re-applies db/users/grants; mail only reports (mailboxes live on mail servers). Placement follows via cluster.on_account_moved; drain completes when no account is left.
• Pooled sites: a pool spec has site_id; web.sites.set_nodes renders the vhost on every backend node (table m_web.site_nodes) and removes it when a backend leaves; the site's files must exist on each node (use cluster.sync.*).
• LB certificates: for terminate/reencrypt pools ssl.cert.lb_status finds the site's active certificate and ssl.cert.lb_deploy copies it to each LB node (cert_lb_nodes); renewals re-copy and emit ssl.cert.lb_deployed, which makes cluster reload HAProxy. No certificate yet = self-signed fallback.
• Firewall: cluster.firewall.requirements also returns per-node needs with source addresses; security.firewall.cluster applies them as rc-cluster: allow rules through the normal apply/auto-rollback path (only on nodes that already have a panel-managed firewall).

w10-fix-14a

• Move followers are called in order with ordered:true (web renders on the new node, ssl, dns re-points, mail, databases) and a last finalize pass removes the vhosts from the old node after a 12 s grace; the plain accounts.moved event no longer re-points DNS or removes old vhosts.
• Zone records that name "the server" use the node the account lives on (dns accountIPs), not the DNS server's node.
• cluster.node.drain refuses the control node; cluster.node.roles refuses dropping a role the node is serving.
• The agent deletes a home on rollback only when this migration created it (markers/imported-<id>).

w10-fix-16

• Pooling prepares the node (cluster/010): web.sites.set_nodes (called by lb.create|update|delete) first asks accounts (accounts.node.prepare, internal to web) to create the account's Linux user on each new backend node with the same uid/gid as on its home node (system.user.ensure takes optional uid/gid; an existing user with other ids or ids taken by another name are refused). The dry run of an LB call lists these steps. When the account's last site leaves a node accounts.node.release removes the user there again together with its home (only a replica of the home node's files; a leftover home would make a later move to that node refuse). Delete the cluster.sync link of that site first. The home node is never touched.
• Switch grace follows the DNS TTL (cluster/005): the DNS follower answers with the longest TTL of the records it re-pointed; the old node keeps its vhosts for that TTL + 2 s, at least 12 s and at most 10 minutes (moveGraceMax). A resolver that holds an address longer than 10 minutes can still reach the old node afterwards.
• cluster.migrate.purge_source also drops the old node's database copies (cluster/004): the account's databases (databases.db.list) are dropped on the source together with the kept home (agent-checked kept-copy marker, strict name grammar); the dry run lists each one. (Database users: see w10-fix-18.)

Known gaps / contract requests

• A VRRP need is expressed as "allow any protocol from the peer" because the firewall rule model has no raw IP-protocol match (broader than needed).
• Pooled site files are not replicated automatically; create a cluster.sync link for the docroot.
• cluster.parallel_migrations / cluster.lb_pools limit keys are declared but not enforced (admin-only module); the migration semaphore is fixed at 2.
• WireGuard overlay, rolling agent upgrade, SSH enrolment, HTTP/3, OCSP stapling, CrowdSec bouncer, UDP pools, DNS-TTL lowering and mail/DNS repointing during migration are not built.
• Stats are polled per call (no history in stats yet); lb_events rows are per LB node.

w10-fix-18

• One uid/gid per account, cluster-wide (cluster/013): accounts allocates the number once (counter id_counter in its schema, starting above every recorded uid and at least 10000), stores it as accounts.uid/gid and passes it to every system.user.ensure for the account (create, accounts.node.prepare, the move's import via ClusterImport.UID/GID). The agent refuses an id that belongs to another name ("is already used by"); a new account then takes the next number. Accounts from before keep their node's id (gid recorded as the uid); a node where that id is taken refuses the pool/move with both numbers in the message.
• Moves resume (cluster/011): every step is recorded; after a restart the module requeues running moves (OnStart, and cluster.migrate.tick for a stalled one) and the move continues from the recorded step. Before it does, reconcile asks accounts.get where the account lives: on the target = every step up to the switch counts as done; a step that was running runs again (an interrupted transfer first clears the half-imported home, which the agent removes only for the migration that imported it). rollback and the failure path never remove anything on the node the account lives on, and when that cannot be determined the move is paused (status paused): cluster.migrate.retry continues it from the recorded step, cluster.migrate.cancel rolls it back when it never switched (after the switch the copy on the old node stays; moving back is a new move). A failing step after the switch also pauses instead of failing.
• Database users move with their credentials (cluster/012): the dump step asks the source agent (cluster.migrate.dbdump with users) to write each user's authentication string (MariaDB: native hash per host; MySQL: native or caching_sha2; PostgreSQL: the SCRAM verifier) into the migration's staging directory; it travels in the pinned stream with the dumps and the restore step creates the users from it on the target before the switch. No plain password is read, and nothing goes through the control plane or the bus. The databases follower then finds the users and re-applies hosts, TLS, connection limits and grants. cluster.migrate.purge_source drops the old node's database users together with the databases (agent-checked kept-copy marker). A user whose password is not a SCRAM verifier (md5) or whose plugin cannot be carried (e.g. unix_socket) stops the move at the dump step with a message naming the user.

w10-fix-19

• PostgreSQL moves keep owners and grants (cluster/012): the dump keeps owners and ACLs; the target has <db>_own/_rw/_ro first, and after the restore the database's standard grants and default privileges are re-applied. What the app could do on the source it can do on the target (tables, sequences, views).
• Records follow the service (cluster/016): a move re-points the account's web names (apex, www, app names) at the new node. Name-server glue (ns1/ns2 and any NS target inside the zone) stays on the DNS server's node. Mail names (MX targets inside the zone, mail, smtp, imap, pop, autoconfig, autodiscover) stay where the mailboxes live. Mail names (MX targets inside the zone, mail, smtp, imap, pop, autoconfig, autodiscover) follow the node that holds the mailboxes. Mail moves with the account since w11-09 (see below); only when the target runs no mail server does the mail stay where it is, and the plan says so.
• Moving onto a pool replica (cluster/018): if a pool of the account's site lists the target as a backend, the replica home there is replaced by the moved copy (same account, same uid/gid); any other data on the target is still refused. cluster.node.drain on a node that is already draining is the retry (accounts with a failed move get a new one; the answer lists blocked accounts with the reason); cluster.node.undrain cancels the drain.
• HAProxy applies (cluster/014): a port another process holds is refused before anything is written, naming the port and the process; the old configuration keeps serving. A node whose last reload failed ("Reload failed!") is restarted on its last good configuration by the next apply (and by the rollback).
• A failed pool create/update releases what it prepared (cluster/015): the account's user and home on the nodes it had added are removed again.
• The switch grace survives a restart (cluster/017): the end of the DNS-TTL wait is stored in the move row (follow_until).

Mail moves with the account (w11-09)

Two steps in the move, both done by the mail module (mail.move.*, called as the system; declared in system_calls), both repeatable:

• mail_copy (before switch): each of the account's mail domains is copied to the target mail server while mail keeps flowing: the domain's whole directory under /var/vmail/<domain> (maildirs, Dovecot indexes and uid lists, Sieve scripts, quota counters) and its DKIM key files, through the same pinned, single-use TLS stream the home uses (cluster.migrate.export on the source, mail.move.pull on the target; no new port, no secrets on a command line). The target must have a mail server (Mail, Server); otherwise the step says so and the mail stays (as before). A target that already holds mail files for the domain is refused unless they are the kept copy of an earlier move away from it (a move back replaces that copy).
• mail_cutover (after switch, before follow): per domain, recorded phase by phase in m_mail.mail_moves so a restart continues where it stood: copied -> freeze -> close sessions -> frozen -> final pass -> synced -> switch -> switched -> names -> done. The sequence (w11-19), per domain:
  1. Freeze (instant 0): the old server queues new mail for the domain (a transport entry to a closed local port: the sender still gets 250, the message waits deferred; a retry: transport would have made it answer 450 at RCPT) and takes the domain's sign-ins away (nologin in the Dovecot user list: IMAP, POP3, ManageSieve and submission all authenticate against it). Local delivery stops because nothing is delivered from the queue. The step waits until nothing for the domain is being delivered.
  2. Close sessions: doveadm kick *@domain (agent op mail.session.kick) ends every open IMAP / POP3 / ManageSieve session, repeating until doveadm who shows none. A client sees its connection drop, tries to sign in again and is refused until the DNS names point at the new server (wrong-password style failure; it keeps retrying). From here nobody can write into the old copy. Sessions that cannot be closed make the step stop (see 5), they are never ignored.
  3. Final pass: everything changed since the first copy (data or inode change, plus deletions), then both ends are compared by file list and size. Every pass starts by closing sessions again; up to 4 passes, one second apart, so Dovecot can finish its index files. All acknowledged client operations happened before step 2 and are inside this pass.
  4. Switch: mail_domains.node_id moves to the target, which renders domain, mailboxes (hashes, quotas), aliases, forwarders, autoresponders, DKIM; the old server turns the domain into a relay transport smtp:[target]:25 and flushes its queue (the target takes at most 5 parallel deliveries per domain, below Dovecot's 10 per user, so the burst is delivered, not answered 451). Then the names follow (MX, SPF/DKIM/autoconfig, the mail/smtp/imap/pop/autoconfig/autodiscover A records) and the phase is done.
  5. Any failure before the switch (a pass that does not converge, sessions that will not close, an agent that does not answer, the target refusing the domain): the freeze is lifted by the same call (hold entry removed, sign-ins allowed, queue flushed, phase back to copied), the error goes to the panel log and the move's status in plain words ("the mailboxes of X kept changing while they were copied ... Mail is flowing again."), and the move pauses. Retry freezes again and continues; cancel removes the copy. Incoming mail is therefore held for the length of one cutover attempt and never for a paused step. Passwords are the bcrypt hashes in the panel database, so every mailbox signs in with the same password. DKIM keys are the same files, so signatures stay valid.
• cluster.migrate.purge_source also calls mail.move.purge: the old server's mail directories, DKIM key files and relay entries are removed then (and not before). A cancelled or failed move before the switch calls mail.move.abort: the freeze is lifted and the copy on the target removed. The old node's copy stays after a finished move so that moving back works.
• mail.names.repair (dry run = the plan) points mail names of accounts moved before this at the node that holds their mailbox.

What happens to a message that reaches the OLD server

Database replication (w11-20)

One standby copy of one engine's databases on another server, and a way for a person to switch to it. Actions cluster.replication.*, agent ops cluster.replication.apply|status (internal/agent/cluster_repl*.go), table replication_pairs.

What it is, and what it is not (shown on the Replication tab)

• Asynchronous. The primary does not wait for the replica. A commit is on the primary first; the replica has it a moment later. If the primary is lost, everything the replica had not received yet is lost. That is "the loss window": the lag measured at the moment of promotion (seconds, and bytes or transactions). Under steady load it is typically well under a second; after a long outage of the link or a heavy bulk load it can be minutes. A planned switch loses nothing (the primary is fenced and the replica catches up before it is promoted).
• One replica per engine per pair. MariaDB (GTID, MASTER_USE_GTID=slave_pos) and PostgreSQL (physical streaming with a replication slot). MySQL (Oracle) and Galera are not supported in this version and are refused with that reason.
• No automatic failover. A person promotes (step-up). Two primaries must never accept writes, and a machine cannot tell "the primary is dead" from "the link is cut" reliably, so deciding is left to a person who can see both servers.
• A replica is not a backup. A DROP TABLE is replicated within a second. Keep the backups module.
• Whole engine, not single databases: every database and user of the engine on the primary is on the replica (MariaDB: the dump includes the grant tables; PostgreSQL: the whole cluster). The replica server must have the engine installed with the same major version and no customer databases of its own (setup refuses otherwise; a base copy replaces everything).
• Files, mail and web content are not part of this; use cluster.sync for site directories.
• The accounts' connection targets follow a promotion (databases.replication.moved re-records every database and user of the engine on the new node); an application that has the old server's address typed in by hand keeps pointing at the old server and must be changed (or reached through a floating address).

Safety rules (in this order)

1. Fencing. Promote fences the old primary first and refuses to continue when it cannot confirm the fence. Two steps, the same for both engines. Soft fence (before the promotion, the engine keeps running so the replica can fetch the last changes): MariaDB read_only=1 (persisted in the engine config) + every non-system session killed; PostgreSQL default_transaction_read_only=on + every client session terminated. The agent reads the setting back and returns the final position; the replica must have replayed up to it (planned: up to 5 minutes, then the old primary gets its writes back and nothing changed; emergency: 20 seconds, the rest is the loss window). Hard stop (right after the promotion): the service is stopped and masked, so not even a reboot starts it; only rebuild unmasks it. Between the soft fence and the hard stop (a few seconds) a client that explicitly opts back into writes (PostgreSQL SET default_transaction_read_only = off) could still write on the old primary; those writes are never on the new primary. Emergency promote with an unreachable primary: refused with the reason unless the person accepts "the old server may still accept writes if it comes back"; then the pair records fence_pending, and every minute the tick tries again and stops and masks the old primary as soon as it answers (lab: a stopped old primary that booted again was stopped and masked about one minute after it came up, so that is also the window in which it could accept writes). Nothing is promoted twice: the pair is the single record.
2. Secrets. The replication password is generated by the control plane, kept in the vault (replication/<pair id>), sent to the agent inside the typed op only, passed to the engine on stdin or in a 0600 passfile, never in argv, a job log, an event or the audit log (c.Redact). PostgreSQL's standby reads it from a 0600 passfile owned by postgres.
3. Network. Replication traffic flows between the two servers' cluster addresses only (the first private address the node reports), TLS required (a dedicated certificate per engine with the cluster address as an IP SAN; the peer's certificate is pinned: PostgreSQL sslmode=verify-full, MariaDB MASTER_SSL_VERIFY_SERVER_CERT=1). The replication user exists for those two addresses only. cluster.firewall.requirements lists the engine port (3306 / 5432) per node from the peer's address only. The base copy of MariaDB travels over the node-to-node pinned one-shot stream (port 38400-38499, as account moves do); the PostgreSQL base copy is pg_basebackup over the replication connection itself, so no further port is needed.

State machine (replication_pairs.state)

not configured -> base_copy -> streaming <-> lagging
                      |            |  \-> broken   (stopped / error: shown with the engine's own words)
                      v            v
                   failed      promoting -> promoted -> rebuilding -> streaming (roles swapped)

• not configured: no row. setup plans first (dry run) and then runs the job with named steps: check both servers, prepare the primary (config, restart if the engine's own settings changed, replication user, slot), base copy (consistent snapshot: mariadb-dump --single-transaction --gtid / pg_basebackup -X stream), load, start streaming, check.
• streaming: threads running and lag <= threshold (default 30 s). lagging: running but behind more than the threshold, or the replica reports an error that the engine retries. broken: a thread stopped, the link is down for 3 checks, or the engine is not running. The tick (cluster.replication.tick, every minute) reads both ends, stores the reading and raises cluster.replication.unhealthy / .recovered events (administrator notice) on the transitions.
• promoting: planned: fence -> wait until the replica has everything -> promote -> record -> stop the old one. Emergency: skip the wait, accept the loss window. The step reached is stored; a failed promote before promote replica leaves the old primary fenced (read-only) and the pair broken with the reason; running promote again resumes (fence is idempotent, the replica is still a replica). After promote replica the pair is promoted whatever else fails: a person retries databases re-record and the old-primary stop with cluster.replication.promote again (every step is idempotent and checks first).
• promoted: new primary writable, old primary fenced and out of the pair until rebuild.
• rebuilding: rebuild runs setup with the roles swapped, after unfencing and wiping the old primary's data (that data is the divergent copy; whatever the loss window held is gone for good, which the plan says again). A failure leaves the old server stopped and the pair promoted (retry rebuild). Switch back = a planned promote of the rebuilt pair.
• Resumption of setup: the job is idempotent step by step (config compare, CREATE USER IF NOT EXISTS, slot reuse); a failed setup leaves the pair failed with the step; the replica's half-loaded copy is simply replaced by the next attempt.

Verified in the lab (w1120, PROFILE=cluster, Ubuntu 26.04: MariaDB 13.0.2, PostgreSQL 18.6, 2026-10-11)

• MariaDB and PostgreSQL, cp1 -> web2: setup (plan, job, TLS with the peer certificate pinned), write on the primary and read on the replica, lag under load, planned switch under load, the old primary stopped and masked (systemctl start refuses), rebuild of the old primary as the new replica, switch back, emergency promotion with the primary's container stopped (refused first, then accepted with accept_risk), the old primary booting again and being stopped and masked by the tick.
• Measured: setup 63 s (MariaDB, 20 k rows incl. two restarts) / 19 s (PostgreSQL, 200 k rows); lag under 8 parallel bulk-insert clients (60 k-row transactions): 0 s in most samples, peak 4 s / 2.8 MB (MariaDB); 6 clients with 80 k-row batches: peak 3 s (PostgreSQL); planned switch under that load 5.3 s (MariaDB) / 4.8 s (PostgreSQL), idle 1.9 s / 2.2 s; emergency promotion about 1 s once the old primary is down (the 20 s wait only runs when the old primary still answers); rebuild of 200 k rows 5 s (MariaDB), 15.4 M rows 17 s (PostgreSQL).
• Not walked: the Replication tab and the Databases line in a browser at 1440/390 px, light and dark (typechecked, linted, built); MySQL (refused by design); two replicas of one engine.