DocsOpen app

Operator runbook (alpha)

Start with self-host installation. Commands act on the running instance via its local mode-0600 socket; use its OS account and exact state path.

Start with self-host installation. Commands act on the running instance via its local mode-0600 socket; use its OS account and exact state path. Keep tokens, login links, KMS credentials and raw transcripts out of incident tickets and shell traces. Back up before changing identity or key configuration. The API contract determines request shapes and permissions.

Monitoring

sh
zuppo health --url "$ZUPPO_HEALTH_URL"
zuppo admin --state "$ZUPPO_STATE" status | jq -e '.version and .origin and (.uptime_s >= 0)'

Probe from outside the service network as well as locally. A health 200 proves a listener, not that every repo/object/identity provider is usable. Add a least-privilege synthetic API read and git fetch of a canary repo; alert on failures, latency and 429/503 rates separately. Preserve stderr with timestamps and rotation; native tracing writes there (RUST_LOG selects verbosity). Watch process RSS/CPU, git children, open FDs, free state/scratch disk, Postgres availability/lock connection and object-store failures. Do not expose internal gitd diagnostics publicly: repository names and cross-org counters are not an anonymous metrics endpoint.

Cloud/DR monitoring must remain reachable off Cloudflare. Inspect DR-host status and the standby's standby-status: per-repo served LSN/recovery point, refused orgs and missing packs are more useful than a green process. /_standby/health returns 200 only after promotion; its failure before promotion is expected. Alert on the off-platform ref/pack copy ages (targets 5 min / 1 h), audit-sink lag (5 min), and recovery-point gaps. These are targets: local game days do not prove real-cloud availability or a 30-day 99.9% SLO.

Common failures

SymptomInspect / act
State already locked / another serverFind the owning instance; stop it gracefully through its service manager. Do not delete the lock or start a second writer
Team lock lostRepair Postgres connectivity and let the exited writer be replaced; one server per database is intentional
Solo network filesystem refusedMove a stopped, complete state to local block storage; do not bypass the check
Downgrade refusedRoll forward to the last writer's version or restore a matching backup; never edit the format/version stamp
Origin mismatchRestore the recorded origin; changing only DNS/--origin does not migrate grants/policies
Sign-in 503 but API worksHTTPS or http://localhost is required; verify origin/proxy TLS and provider settings
401 / repeated refresh failuresCheck expiry, revocation and serialized refresh-token rotation; obtain fresh person consent, not a reused refresh token
403 / envelope-breachInspect current grant, sponsor role, RFC hashes and requested paths/refs; the grant and protected-path rules both apply
429Honour Retry-After; inspect the named grant/sponsor/org bucket; lower concurrent work or obtain consent to raise the budget
Stale base / etag mismatchRe-read and reconcile the structured conflict; never force the old write
Missing objects / not our refPreserve server/client logs, ref/LSN, pack inventory and compaction timing; verify a fresh clone and store consistency before any cleanup
SSH host key changedStop and compare the preserved identity with the backup/DB; never train users to disable host-key checking
Webhook not deliveredConfirm signature secret, public DNS/address policy, timeout and non-2xx response; redirects are not success
Standby missing a recent refIt waits for a contiguous ref-log and every named pack. Check copy lag/residency rather than promoting writes

Load and known failures

Prior shared-Mac runs hit search-sidecar 10-second timeouts at extreme host load (~150), listener/socket shutdown races and (fixed: the web e2e mock server now takes a free port) a Playwright port-4399 collision with another lane. Those are recorded harness/load observations, not permission to ignore a failed production request. Run isolated checks with bounded workers and unique ports; save the original failure before retrying. Never stop other lanes' servers or broaden timeouts merely to manufacture green evidence.

A prior BEN-2 replay intermittently returned upload-pack: not our ref during concurrent fetch and compaction, while pushes and final clone connectivity remained clean. Record that as a real failed fetch, not a successful replay. Later fixes may supersede the report; diagnose the version actually deployed. A health check alone cannot rule it out. Dedicated throughput measurements are required before capacity/SLO promises: the single local workerd process missed PC3/PC8 at high concurrency; PC33's dense workset miss batches remained seconds, not its one-second target. The capacity table deliberately labels its historical footprint and incomplete 24-hour measurement.

If disk is exhausted, stop new work, preserve durable data and free only identified rebuildable caches/scratch owned by this instance after stopping relevant work. Never remove packs, key rows, SQLite WAL files or database rows as a space-recovery shortcut. Keep backup/restore capacity outside the forge's failure domain.

Fault injection (crash safety of the real binaries)

scripts/chaos.sh kills, starves and mis-clocks the real binaries and checks the invariants after recovery: no acknowledged push is lost or changed, refs and objects are consistent (git fsck on a fetch and on the object store rebuilt from the ref store's own registry), the merge queue never lands a group head CI was not asked about, GitHub sync converges with no drift, and the DR replica is consistent. It is beyond the deterministic simulator: it runs zuppo serve on a real state directory, kill -9 at pseudo-random moments (CHAOS_SEED replays them), a 96 MiB disk image for disk-full (never the host's disk), a clock shim (scripts/chaos/clock.c) for clock steps, and on the local cloud stand-in workerd, the edge, gitd-receive and the engine with a proxy that cuts the connection between edge and engine.

sh
scripts/chaos.sh list                       # the scenarios
CHAOS_RUNS=10 scripts/chaos.sh run          # every scenario ten times; summary in target/chaos/summary.md
scripts/chaos.sh run clock_jump             # one scenario; a failed run keeps its server log and ledger

A local run on a laptop is not a cloud pass and not a power-loss test: kill -9 loses no page-cache data, so durability against a host crash or a sudden power cut is not shown by it.

Rotate credentials and keys

  • TLS certificates: install new certificate/key atomically with restrictive ownership, then restart gracefully; validate the chain and external hostname. SSH host identity is different: preserve it across upgrades, publish a deliberate replacement fingerprint out of band.
  • Registered SSH user/agent keys: add the replacement public key and test fetch/push under its intended grant, then remove the old fingerprint with ssh-key-remove. Removal revokes its grant. An agent key also needs --agent, --session and its envelope detail; do not register it as an unrestricted person key.
  • OAuth: refresh tokens rotate on use. Replace/revoke a compromised grant via GrantService, issue new consent, then verify old-token rejection. Do not copy a person token into an agent process. One-time bootstrap secrets are returned in bootstrap-tokens.json or admin apply; retain only in the secret store, never in the repo.
  • S3/Postgres/provider credentials: provision a replacement with least privilege, update the deployment's secret and roll the single writer, test storage/sign-in, then revoke the old one. Chart-owned Secrets trigger a rollout; externally managed ones need an explicit rollout.
  • Tenant encryption: zuppo-keys separates a random data key from the platform wrapping KEK. Rewrap preserves the data key under the new KEK; retain previous KEKs until every stored row and required backup can unwrap. There is no `zuppo admin rotate-keys` command in the open binary. Use the adapter/keyring's provisioned tooling, not a made-up serve flag. Crypto-shred is irreversible once key copies/old KEKs are gone; backups can prolong recoverability.

SAML and SCIM setup

The open single binary has no enterprise engine attached; its enterprise RPCs return NOT_IMPLEMENTED after auth checks. Enterprise runs zuppo-apid-ee (the zuppo-ee-host package) with the API adapter's serve flags, not zuppo-enterprise-dev (memory-only, test endpoints). The engine keeps state under <state>/ee; include this and its sealed credentials in backups. ZUPPO_EE_ENDPOINTS defaults to strict; custom is an explicit emulator/private-endpoint choice, not a production fix for failed endpoint validation. ZUPPO_EE_CA_FILE adds a private KMS CA. Do not expose testbed/dev endpoints or assume the local stand-in is a production IdP.

Use an owner/admin person on the Business plan; agents are always refused. The Settings UI uses the same SsoService and ScimService as clients:

  • Get SSO config and save its ETag. Set exactly one of IdP metadata XML or URL with UpdateSsoConfig, ifMatch and a unique requestId; keep requireSso=false.
  • Give the IdP the returned spEntityId, acsUrl and spMetadataUrl. Verify signing-key fingerprints through a trusted channel. Do not guess ACS/audience URLs.
  • StartSsoTest, open the returned login URL, complete a real IdP login, then re-read until lastTest is PASSED/current. A changed provider clears this proof and enforcement.
  • Add domains, publish the returned _zuppo-verify TXT records and verify them. Re-read the ETag before enabling requireSso. Enforcement requires the current successful test and ends members' grants except the caller's own; plan for re-consent. Keep a tested recovery account.
  • With SAML configured, rotate/create the SCIM token through RotateScimToken; store the once-shown token directly in the IdP and use its returned base URL. Rotation replaces the old token. Map SCIM userName to the SAML subject, then test provisioning and deactivation: removal/deactivation must revoke membership and sponsored grants (target within one minute).
  • During IdP signing-certificate rotation update metadata, re-test and re-enable deliberately. Deleting SSO is refused while enforcement is on, and also removes its SCIM configuration.

BYOK

Enterprise Settings/ByokService binds AWS KMS, Google Cloud KMS or Vault/OpenBao Transit. Get config/ETag, provision least-privilege KMS credentials, then BindByok with the key, write-only credentials, ifMatch and requestId. Binding probes encrypt/decrypt and rewraps the same org data key. CheckByok records the KMS result; watch lastCheck and audit records. Use returned revocationTtlSeconds (default 300): disabling KMS access is not instantaneous while an open key is cached. There is no fallback to the platform KEK on KMS failure.

For rotation, keep old keys decrypt-capable while binding/checking the new key and verifying old rows/backups; config reports previousKeys. Unbind deliberately rewraps back to the platform KEK; it is not a workaround for customer revocation. Destroying the KMS key can permanently lose data. Local KMS emulators in scripts/byok-kms.sh establish adapter behavior, not real IAM or cloud KMS reachability. Do not confuse this encryption BYOK with a hosted reviewer's model API key.

Audit export and streaming

AuditService is available without enterprise, but only to owner/admin people. List newest-first for incidents; export oldest-first as JSONL (default) or CSV. ExportAuditLog takes afterSeq, limit (default 5,000, maximum 20,000), filters and requestId. Its JSON response carries base64 data, filename/type, chain hashes, chainVerified and nextAfterSeq. Persist the response/proof with the decoded file; while truncated, request the next chunk using nextAfterSeq and a new request id. Export itself creates audit.exported; freeze a time range to avoid chasing your own exports. A failed chain verification is an incident, never discard/rebuild audit rows to hide it.

The enterprise zuppo-audit-stream adapter sends verified segments at least once to S3, signed webhooks, syslog TCP/TLS or OTLP/HTTP. Run/status/replay use its own config/control socket; this is not the forge event-webhook protocol. Cursor advances only after sink acknowledgement, so receivers deduplicate tenant:seq. A chain break halts delivery; refused sinks pause with an alert; lag above five minutes alerts. Repair the sink/credential/chain cause, then replay from the last durably accepted sequence. Preserve cursors and the original signed segments in backups.

Evidence boundaries

These instructions derive from zuppo/zuppo-axi help, the proto/OpenAPI contract and recorded HOST-1/HOST-3/HOST-4/HOST-5, SAML/BYOK/audit, remeasurement and load-failure reports. The short runnable smoke checks open-server operator/API/agent paths; it does not claim a live Kubernetes cluster, real IdP/KMS, cloud DR DNS cutover, a week-long LTS soak or production latency.