CDN

Spec docs/specs/cdn.md; handover docs/tasks/w8/w8-02-cdn.handover.md. Manifest edition: cdn (paid add-on key, see licensing below). Desired state lives in schema m_cdn (settings, edges, zones, purges, stats_hourly, integrations). Edges are only ever given the complete rendered configuration (cdngen.Render); apply is idempotent and cdn.reconcile re-applies everything from the database.

How a request flows

client -> GeoDNS (PowerDNS LUA record *.<base domain>) -> nearest healthy edge (nginx proxy_cache) -> origin (site URL or S3 bucket)

• Edge node = a node with the cluster edge role and a row in m_cdn.edges. cdn.edge.enable adds the role through cluster.node.roles, installs nginx.org mainline nginx + nginx-module-njs on a node without nginx (the same build the web module installs); on a node that already has nginx it adds the njs package that matches it (nginx-module-njs for nginx.org's nginx, libnginx-mod-http-js for the distribution's nginx - the wrong one conflicts with the installed nginx in apt) (allow-listed, cdn.edge.install), refuses by name a http_port/https_port/status port another program already holds (before installing anything) and never leaves a restarting unit behind when a first start fails, writes a separate nginx instance (rc-cdn-edge.service, /etc/rc-cdn, cache /var/cache/rc-cdn/<zone>, logs /var/log/rc-cdn) so it never touches the web module's nginx (it can share a web node: pick other http_port/https_port), applies the config, requires /health to answer, and only then adds the node to DNS. A freshly installed distro nginx (default site on :80) is stopped and disabled.
• Zone hostname <zone>.<base_domain> always serves; custom hostnames serve after cdn.hostname.verify (CNAME to the zone hostname or TXT _rc-cdn.<host> = token). A self-signed certificate is made per zone (stable across applies, re-made when the hostname set changes); cert_id uses the PEM the ssl module deployed on that edge node (/var/lib/respirecloud/ssl/certs/<id>), falling back to self-signed with a warning. Real certificates for custom hostnames (w11-26): once a hostname is verified, cdn.cert.ensure (also run by the one-minute edge poll, retried at most every ten minutes per zone) orders one certificate per zone for its verified custom hostnames through ssl.cert.issue (system call, node_id = the first healthy edge, which holds the key), deploys it to every other edge with ssl.cert.deploy (so every renewal follows: the ssl module copies the renewed chain and key, the cdn re-apply every 15 minutes or cdn.reconcile swaps and reloads) and sets the zone's cert_id. HTTP-01 is answered by every zone server on port 80 (/.well-known/acme-challenge/ from /var/lib/rc-acme); an edge without the token file asks the other edges once (X-RC-ACME-Hop), and the HTTPS redirect, hotlink check and signed-URL gate step aside for that path. The host name <zone>.<base domain> keeps a self-signed certificate (a wildcard needs DNS-01 on the base domain). cert_id can no longer be chosen by a customer (it would serve another account's certificate).
• GeoDNS: one LUA A record *.<base domain> (TTL geodns_ttl, default 30 s) in the zone of the same name on the node that runs PowerDNS (dns_node, default the control node). cdn.geodns.apply switches enable-lua-records=yes and enable-lua-record-updates=yes on in its own drop-in rc-cdn.conf (without the second one pdns refuses every REPLACE of the record) and PATCHes the pdns API; a refused update puts the previous record back. A client in a pop's CIDRs (or countries, with geoip on and a GeoIP database configured in pdns) gets that pop's edges; all other clients get any healthy edge (pickclosest with GeoIP, random without). ifportup probes each edge on its own port and drops dead edges (edges on different ports are probed in one call per port and the answer is picked from the union; in that mixed-port form a port group whose edges are all down is still answered, because pdns cannot tell "all down" from "all up" for a selector all, so keep one port across edges in production); the per-pop answer is a list of lists (pop first, everybody else second) so a pop with all edges down falls back to the other pops. Draining edges are left out. Names are validated and the Lua is built from a closed grammar (no customer text reaches it).
• Cache: key = <zone>|<decoded path>[?args]; request collapsing (proxy_cache_lock), background update, stale on error/timeout/5xx, per-zone cache directory (purge-all = delete the directory), X-Cache (HIT/MISS/EXPIRED/BYPASS/STALE) and X-Edge (node id) on every response. Rules are ordered (first match wins; path glob * within a segment, ** across, or extension list) and set TTL, browser TTL, bypass, query handling (ignore/all), ignoring origin Cache-Control, response headers; zone-level cookie bypass, gzip, CORS/security headers, allow/deny CIDRs, hotlink protection, HTTP->HTTPS redirect, origin failover (extra URLs are backup servers), origin host header, custom origin headers, origin timeout. cdn.rule.preview shows which rule a URL hits.
• Cache key and poisoning: the key is <zone>|<decoded path>[?args] and nothing else - no request header is part of it, and the edge sends the origin a fixed Host, an empty Accept-Encoding and the zone's own headers, so a visitor cannot make the origin vary a cached object. Cookies only bypass (never key). A private-bucket zone ignores the query string.
• Soft purge (cdn.purge soft:true): the edge rewrites the validity inside nginx's cache-file header to the past (the file stays); the next request is STALE and nginx revalidates in the background, later requests are HIT. Purge on deploy: cdn.zone.update purge_on_deploy {app_id|site_id, soft} (the caller must be able to see that app or site) makes apps.deploy.finished (status ok) and web.site.deployed purge the whole zone (cdn.on_deploy).
• Purge (cdn.purge): url (exact path, also drops ?query variants), prefix, tag (origin Cache-Tag / Surrogate-Key), all. Done by the agent deleting the cache file (nginx treats a vanished file as a miss; exact URLs are found by the md5 of the key, prefix/tag by reading the key and headers stored in each cache file). Runs on every edge in parallel, returns per-edge counts and partial/failed_edges; rate limit = package limit cdn.purges_per_hour (default 600); each purge is logged with the actor (cdn.purge.history).
• Signed URLs: ?expires=<unix>&token=<hex>[&ip=<ip>], token = HMAC-SHA256(secret, path\nexpires\nip), checked at the edge by an njs function (constant-time compare, keys from the zone's config) before the cache is consulted, so HITs are protected too and the token does not enter the cache key. Keys rotate with an overlap (cdn.token.rotate): the old key keeps validating until now + overlap_hours. cdn.token.sign signs a URL server-side; cdn.token.snippet returns signer code in node/python/php/go/ruby/java/dotnet/curl (verified against the Go implementation in tests and, for python, against a real edge in the lab). Signing secrets are shown once (create, rotate) and never in plans or zone reads.
• Storage zones: origin = a bucket of the built-in S3 (storage). The edge reads anonymously, so the bucket must be public-read (a private bucket is refused with the reason); only GET/HEAD pass the edge; the cache key stays the public path. Writes go to the bucket with the storage module's keys, reads through the CDN; overwrite an object, then purge it. Private buckets (w11-26): for a private bucket the panel makes a read-only key for that bucket alone (storage.key.create, mode read, named cdn-origin-<zone>, as the caller), keeps its secret in the vault and the edges' config, and the edge signs every origin GET with SigV4 in njs (s3auth; the path is built and encoded by njs, $args cleared, Authorization replaced). The bucket stays private (direct anonymous read is 403), the key is revoked with the zone or when the bucket changes. Visitors' Range headers pass through unsigned.
• Origin lockdown: every zone has a random secret sent to the origin as X-RC-CDN-Secret; cdn.zone.get (zone managers only) returns it with a ready nginx snippet.
• SSRF guard: customers cannot point a zone at loopback, link-local or private addresses (literal or resolved at create/update). Admins can (their own sites).
• Stats and metering: cdn.stats.collect (1 min) reads each edge's access log from a saved offset into hourly rows (requests, bytes, hits/misses, status classes, origin bytes, TTFB, top URLs). cdn.stats.get ranges 1 h-90 d, CSV export; cdn.usage.get shows the month against cdn.bandwidth_gb. Events at 80/95/100 % (cdn.bandwidth.warning|exceeded), with setting overage_policy=block zones of that account return 403 until the next month. cdn.origin.failing|recovered from the 5xx share of origin fetches. Rows of a deleted zone are kept 30 days, others 90.
• External CDNs (cdn.integration.*): Cloudflare and Bunny. Token in the vault (cdn/integration/<id>), row keeps ••••last4, test validates scope; zones lists provider zones and the capability matrix; purge (Cloudflare: url/prefix/tag/all; Bunny: url/prefix(wildcard url)/all), setting (allow-listed keys per provider), wire_dns (Cloudflare: provider DNS record; Bunny: CNAME through dns.record.set). An operation the provider lacks is refused with not_supported before any call; the same matrix comes back in list/zones so the UI can grey it out. Only an admin can set an endpoint override (the lab mock).
• Customer API: scopes of an API token are permission ids (cdn.*, or cdn.zone.view + cdn.purge + cdn.stats.view for a deploy script); a token never exceeds its owner or sees another account. Limiting a token to ONE zone is not built (tokens have no resource scope and a Team member cannot create tokens). Every action is a REST call POST /api/v1/a/<action> with an API token that holds the permission (cdn.zone.view, cdn.purge, cdn.stats.view, cdn.zone.manage, cdn.rule.manage; subusers can be scoped to one zone); cdn.api.describe returns the base URL, auth, examples (curl/node/python/php), a deploy-script purge, and for storage zones how to read through the CDN and write to the bucket. ?dry_run=1 previews every mutation as a Plan with the real per-edge config diff (secrets redacted). Actions are also scriptable with respirecloudctl.

Licensing

Manifest edition: cdn (module-wide entitlement key, docs/tasks/w8/w8-04-store-licensing.handover.md). Without the entitlement the host answers not_entitled for every action and the module is "not included" in the Store. Degrades cleanly: the data plane does not depend on the control plane. Edges keep serving the last applied config (nginx), GeoDNS keeps answering, stats lines keep accumulating in the access log until the module is back. Spec says external-CDN integrations ship in the base: a module-wide key cannot split the module, see the handover (per-action edition or a second module id).

Verified in the lab (PROFILE=cluster, FAKENET=full, Ubuntu 24.04, nginx 1.24 + njs 0.8.2, pdns 4.8.3, SeaweedFS 4.48)

See the handover for commands and outputs: pull zone through two edges (MISS/HIT, per-edge cache), purge url/tag/prefix/all, signed URL (valid, tampered, expired, ip-bound, key rotation overlap, python-signed), storage zone (put via presigned URL, serve, overwrite + purge, private bucket refused, PUT refused), stale on origin failure, bypass rule, hotlink, gzip, HTTP->HTTPS redirect, origin lockdown header, GeoDNS per resolver view and failover, edge down/up detection by the scheduler, external CDN integrations against a mock API, stats and usage, dry-run plans.

Not done

Open after w11-26: imgproxy / image optimisation (AC-cdn-20..22), origin shield, per-zone WAF / rate-limit / country rules at the edge (reusing the web module's protection engine and GeoIP), edge access-log streaming to logs, the GeoIP (DB-IP lite) download and refresh, Fastly/CloudFront/KeyCDN/Gcore adapters (external providers: Cloudflare and Bunny are mock-proven only, none real-API-proven), Brotli/zstd, HTTP/3, bandwidth throttle mode, p50/p95 TTFB, an emitter for web.site.deployed, tokens limited to one zone, the UI for certificates / soft purge / deploy link / private bucket, a neutral agent helper file for what cdn shares with cluster / dns.

Rules at the edge (w11-89)

All of it is typed zone settings (cdngen.Security.Countries|RateLimit|WAF, cdngen.Cache.Brotli|Zstd|CompressMin|CompressTypes), validated against closed sets and rendered by cdngen.Render; no customer text reaches nginx. A zone without the settings renders exactly what it did before.

• Country allow / deny (AC-cdn-10): the zone says {mode: allow|deny, codes}; geo/<zone>.conf ships as the marker #@geo <mode> <codes> and the edge's agent replaces it with the address blocks of those countries (cdnExpand, geoip.NetworksOf), because the control plane's modules are separate processes and cannot read the panel's database. Source on the node: the panel's own DB-IP Lite file (<RC_DATA_DIR>/geo/country.mmdb, downloaded by an administrator on Security > Licences with the licence and attribution shown there: the local edge is the panel node), else the security module's database on that node (security.geo.update). With neither the update is refused with a plain message and the Security tab says why the control is off. Cap: 60 countries and 120000 address blocks, refused past it (cdngen.MaxCountries, MaxGeoLines). A visitor from a refused country gets 403; in allow mode an address with no known country is refused too. The expansion happens at every apply, so a changed zone reaches the edge within one apply (seconds); a database that is refreshed later is picked up by the next apply (cdn.reconcile). Nginx geo has the ranges as CIDRs (cdnRangePrefixes). A second edge without the panel's database uses the security module's, so it needs security.geo.update run once.
• Rate limit (AC-cdn-18): limit_req_zone $binary_remote_addr per zone, limit_req ... burst=N nodelay, status 429 with Retry-After: 1.
• Web firewall at the edge (AC-cdn-18): the web module's service (internal/wafd, Coraza + OWASP CRS, detect or block, paranoia 1-4), run as a second instance, rc-cdn-waf.service (user rc-cdn, env RC_WAF_CURRENT|RUN|EVENTS|SECRET), with its own sockets in /run/rc-cdn-waf, generations in /var/lib/rc-cdn-waf and events in /var/log/rc-cdn/waf. A protected zone renders two servers: the public one (TLS, country, rate limit, hotlink, signed URL check, then proxy_pass to the service with X-RC-Site: cdn-<zone>) and an inner one on up-cdn-<zone>.sock that holds the cache and the origin, so every request is inspected BEFORE the cache (a HIT is inspected too) and the service fails open to the inner socket if it is down. The agent compiles the rules (wafd.Validate) before anything goes live, installs the unit on first use, publishes waf/cdn-<zone>.json like the web module publishes its generations, and rolls it back with the nginx config. Blocked requests: cdn.zone.waf_events (the Security tab lists them); the zone's request log (Logs) shows the 403/429 with the cache result taken from the inner server's X-Cache. A zone name over 36 characters gets a short stable id (cdngen.WAFID: cdn-<first 20>-<12 hex of its SHA-256>) so two long names with the same start never collide (w11-98). Since w11-92 the edge runs the same pair as the web module: rc-cdn-waf.service (waf.sock) and rc-cdn-waf-b.service (waf-b.sock), Type=notify, RC_WAF_COMPILE_FIRST=1, restarted one after the other (only when a unit file changed), each copy listed twice in the zone's upstream with the zone's inner server as the last-resort backup. Stand-in proof (two units from the real code, an nginx with the same upstream text, 20 restarts at 100 injections/s): 1479 of 12818 passed uninspected with one Type=simple copy, 0 of 16388 with the pair. A full edge install was not run (see the handover).
• Compression (AC-cdn-09): gzip as before, plus Brotli and zstd on the configured MIME types (default list, or compress_types) above compress_min (default 1024 bytes). The render emits #@mod:brotli ... / #@mod:zstd ... lines and #@LOAD_MODULES@; the node keeps them only when its nginx has the module (ngx_http_brotli_filter_module.so, ngx_http_zstd_filter_module.so under the nginx module directories) and drops them otherwise, and cdn.edge.caps (also in cdn.edge.health) says per edge what exists, so the Caching tab shows "not available on " instead of a dead switch. cdn.edge.install tries the distribution's package for each (libnginx-mod-http-brotli-filter, libnginx-mod-http-zstd; nginx.org: nginx-module-brotli|zstd) and ignores a missing package. Ubuntu 26.04 ships Brotli but no zstd. Range requests: the edge caches whole objects, so the first range request of an uncached large file is passed on as a range (MISS, not cached) and every range is a HIT once one full request has filled the cache (proved with 200 MB). Slices (w11-98): a zone (or a single rule) can turn on cache.slice: nginx slice 1m fetches and caches a large object one 1 MB slice at a time (proxy_set_header Range $slice_range, cache key <zone>|<uri>$slice_range), so the FIRST range request on an uncached large file is answered after one slice (MISS) and every later request for an overlapping range is a HIT; without it the edge fetches the whole file first. The origin must answer Range with 206 (an origin that answers 200 makes the slice request fail with a 502). Off by default. Lab: 200 MB object, Range: bytes=100000000-100000099 MISS, then 100000050-100000199 HIT, the origin saw exactly one request, for 99614720-100663295; with slicing off the origin saw one request for the whole file.

Image optimisation (AC-cdn-20..22, w11-98): imgproxy on each edge

Evaluation (w11-89): nginx image_filter cannot convert formats; imgproxy (MIT, Go + libvips, official container) can, so it is the chosen service (lead's decision, w11-98). Built as follows.

• Install (cdn.images.install, administrator, dry_run shows the plan first: runtime download, the pinned image size (about 81 MB, 250 MB on disk), start, fence, nginx reload, free-disk warning): installs the container runtime through containers.runtime.install (a declared system call) when the node has none, then the agent job cdn.images.apply pulls ghcr.io/imgproxy/imgproxy@sha256:... (the digest is catalog/versions.json component imgproxy, binary image; the agent reads it from the catalog, never from the panel's request), makes the node's key/salt/token, starts the container and waits for its health check. cdn.images.remove refuses while a zone uses it. Which edges run it: setting images_edges (node -> pinned image); cdn.images.get shows it (live per edge for an administrator).
• Where it listens: imgproxy runs as its own user (rc-cdn-img), read-only, no capabilities, 1 GB memory, on the host network and the host's user namespace, bound to 127.0.0.1:18090. A kernel rule (inet rc_cdn_img_fence, unit rc-cdn-img-fence.service) lets only the edge's nginx workers (rc-cdn) and root connect to 18090, and only those plus rc-cdn-img to 18092 (the feeder). imgproxy also answers only signed URLs (HMAC key made on the node) and its allowed source is exactly http://127.0.0.1:18092/. The workers' loopback fence (inet rc_egress) is opened for 127.0.0.1 by the control plane, and the same agent rule closes every other loopback port for them (name resolution excepted), so a customer origin that resolves to 127.0.0.1 gets nothing new. Key, salt and token never leave the node (/var/lib/respirecloud/cdn/img.json, /etc/rc-cdn/imgproxy.conf and .env, mode 0600); a rendered edge configuration contains none of them.
• Request path (cdngen/images.go): on a zone with images.enabled, a GET/HEAD for an image extension with w, h, fit (contain|cover|stretch), q (1-100), fmt (webp|avif|jpeg|png|auto) is validated by cdn.imgVariant (400 with the reason; width/height up to 8192), turned into one canonical variant (fmt=auto resolves from Accept: avif, else webp, else the original format) and answered by @rc_img from the zone's cache (key <zone>|<path>?img=<variant>, so a url purge of the original removes its variants and a second request is a HIT, X-Cache; Vary: Accept). A MISS asks imgproxy with a signed URL. imgproxy fetches the ORIGINAL from the edge's own feeder (<zone>.rcimg servers on 127.0.0.1:18092 behind a secret path), so the origin lock header, the Host header, private buckets and the original's own cache entry work as for any request; the original is never modified. "Auto WebP" (images.auto_webp) turns a JPEG or PNG without parameters into WebP when Accept lists image/webp. Token-protected zones check the signed URL before the image branch.
• Caps (administrator settings img_max_src_mb 25, img_max_mpx 40, img_timeout_sec 10; saving writes them to the container's environment): a source over the size answers 413, over the pixels or not an image or too slow answers 422 (the script maps imgproxy's development-mode messages to its own sentences; a visitor never sees imgproxy's text), an original that does not exist answers 404. The time cap answers 422 because AC-cdn-21 names 413/422 (imgproxy itself says 504).
• Not supported on this version: animated GIF frames beyond the first, SVG, a zone served through the firewall is supported (the branch sits in the inner server).