The read-only standby and the game day (HOST-4)
A correlated Cloudflare failure (risk K2, threat SEC-T15; 2025-06-12, 2025-11-18) takes the RepoDOs, R2 and the edge down together.
A correlated Cloudflare failure (risk K2, threat SEC-T15; 2025-06-12, 2025-11-18) takes the RepoDOs, R2 and the edge down together. HOST-2's pipeline keeps the refs and packs off Cloudflare: every ref update reaches an append-only ref-log at a second provider (RPO ≤ 5 min) and every pack an R2-event copy there (≤ 1 h). The standby is the self-host binary zuppo at that second provider, following those copies and serving reads of them, so every repository stays readable (PC24: the game day reaches the standby in ≤ 4 h).
Cloudflare second provider (one store and host per jurisdiction)
RepoDO ── queue zuppo-refs ──┐
├─▶ DR Worker ─▶ zuppo-drd --jurisdiction eu ─▶ eu store ─▶ zuppo serve --standby (eu) ─▶ reads
R2 zuppo-objects-eu/us ─ R2 events ─┘ └─▶ zuppo-drd --jurisdiction us ─▶ us store ─▶ zuppo serve --standby (us) ─▶ readsA store holds residency.json, the org pins meta/orgs/<org>.json, the ref-log segments reflog/<org>/<repo>/… and the packs repos/<org>/<repo>/packs/…; the standby only reads it.
What it follows, and what it serves
- One jurisdiction. EDGE-3 pins every org to
euorus; the DR copies follow that pin: the DR Worker sends each org's ref-log segments and packs to the DR host of its jurisdiction (zuppo-drd --jurisdiction), which writesresidency.jsononce and an org pinmeta/orgs/<org>.jsonbefore it keeps the org's first object, and refuses copies routed to the wrong host (421). A standby is started with--standby-jurisdiction; it reads nothing of a store whose residency names another jurisdiction and nothing of an org not pinned in its own. The jurisdiction is recorded in the standby's state at the first start and fixed after. - A consistent cut. The sans-IO core (
zuppo_dr::standby::Follower) serves each repository at the last LSN whose ref-log replays with no gap (every part present, every update chained onto the value before it) and whose named packs are all in the store. The ref-log runs ahead of the pack copies (minutes against up to an hour), so a ref is never served before its objects; git's connectivity check on each apply is the backstop. A repository's served LSN never goes back, across restarts too. Details: the module docs ofcrates/zuppo-dr/src/standby.rs; simulation:zuppo-sim run standby --seed N. - Read-only, always. Pushes fail closed (
503,Retry-After), over HTTP and SSH; the API answers only reads (Connect methods markedNO_SIDE_EFFECTS,GET/HEADon REST). Promotion never makes the standby writable: a writable copy would fork history from the RepoDOs that come back after the outage. A standby state cannot be started as a normal forge. - What it holds. Refs in
refs/heads,refs/tagsand pull heads, their objects, the orgs and repositories (from the store's pins and ref-logs). Not the forge's metadata rows (changes, reviews, members: DEP-B3's export is not built), not the cloud's identities: readers sign in to the standby itself (its people sign-in or grants the operator issues). The DR host's canary repository (zuppo-dr/canary,zuppo-dr-eu/canary) moves a ref to random ids that have no objects, so its recovery point always readsfailed(git refuses the ref): the standby serves no ref it cannot back with objects.
Run a standby
# At the second provider, with read-only credentials for the eu DR bucket. AWS_ACCESS_KEY_ID=… AWS_SECRET_ACCESS_KEY=… AWS_REGION=eu-central-003 \ zuppo serve --state /srv/zuppo-standby-eu \ --listen 0.0.0.0:443 --origin https://git.example.dev --tls-cert … --tls-key … \ --standby s3://zuppo-dr-eu --standby-s3-endpoint https://s3.eu-central-003.backblazeb2.com \ --standby-jurisdiction eu --standby-every 10 zuppo admin --state /srv/zuppo-standby-eu standby-status # role, store residency, per-repo recovery points
GET /_standby/health answers 200 only once the standby is promoted: point the DNS failover's health check at it.
The game day (PC24)
scripts/gameday.sh run [OUT] runs the whole drill locally: the Cloudflare stand-in (workerd with zuppo-do, the DR Worker and its fixture), two DR hosts (eu, us) on directory stores, one standby per jurisdiction, and a load of real git pushes to an eu and a us org. Then it kills the Cloudflare stand-in, detects the outage, promotes both standbys, and verifies every repository reads back from its jurisdiction's standby as the last acknowledged push left it (or measures what was lost), with no repository readable from the other jurisdiction's standby. It writes OUT/gameday.json (PC24's check keys: rpo_refs_off_cloudflare, rpo_packs_off_cloudflare, rto_to_readonly_standby) and OUT/gameday.md (the timeline). scripts/gameday.sh check is the short version scripts/check.sh runs.
The quarterly game day in production is the same sequence against the real platform:
- Declare the game day; note T0. Stop Cloudflare traffic for the drill (or wait for a real outage) and note T_kill.
- Confirm the outage from off Cloudflare (the DR host's status page and probes).
- On each jurisdiction's standby:
zuppo admin --state … standby-promote; keep the JSON (the recovery point of every repository). - Move the DNS record of the git and API host to the standby (the provider's API or console; the health check at
/_standby/healthturns green once promoted). - Verify reads:
git ls-remoteand a clone of sample repositories through the public name; note T_served. RTO = T_served − T_kill (≤ 4 h; stretch ≤ 1 h). RPO for refs = the promotion's recovery points against the RepoDOs' last LSNs once Cloudflare is back (≤ 5 min). - Failback: move DNS back,
zuppo admin --state … standby-demote; the standby keeps following.
Fault injection (kill -9 of the DR pipeline)
scripts/chaos.sh run dr_kill_drd dr_kill_workerd (scripts/chaos-dr.sh) kills the real pieces of the local pipeline under a steady load of real git commits and checks the recovery. Each run is a fresh scripts/dr.sh stack plus a standby following the us store; CHAOS_RUNS runs per scenario, CHAOS_SEED replays the kill moments.
dr_kill_drd: theusDR host is killed mid-replication, stays down 1–9 s and restarts on its state while the load goes on. Afterwards the ref-log of every repository replays to the RepoDO's log end with no gap and no conflicting duplicate, the standby serves that LSN, every acknowledged pack is in the store, and every acknowledged commit clones fsck-clean from the promoted standby (zuppo-do-bench gameday verify). The DR host's lag samples give the RPO (PC24: ref-log ≤ 5 min, packs ≤ 1 h).dr_kill_workerd: the whole wrangler/workerd tree (zuppo-do, the DR Worker, its queues) is killed and restarted on the persisted state, then the weekly reconcile runs once (it copies the packs whose R2 events were lost). Miniflare's queue broker keeps its messages in memory, while Cloudflare Queues are durable, so commits acknowledged within 3 s of the last ref-log write before the kill may never reach the ref-log: the run counts them (acked_missing_from_reflog) and holds every older acknowledgement, and every load repository's log, to the same invariants. Queue durability itself is not checked locally.- Not covered: real Cloudflare (Queues, R2 event notifications, Durable Object eviction), an S3 store, power loss (no fsync is tested);
CHAOS_DR_KILLS=Nkills the DR host N times a run.