DocsOpen app

Code search (zuppo-searchd)

zuppo-searchd answers "where does this text occur in this repository at this commit" for the web UI, the API and agents (SRV-4; ARC-18, ARC-43; criterion PC13…

zuppo-searchd answers "where does this text occur in this repository at this commit" for the web UI, the API and agents (SRV-4; ARC-18, ARC-43; criterion PC13: p95 <= 300 ms end to end, and a path an agent changed is searchable at once).

It is the sans-IO core crates/zuppo-search (routing by query kind, Zoekt budgets, the freshness overlay, index scheduling) driven by crates/zuppo-searchd with real processes:

text
 edge / API ──HTTP──▶ zuppo-searchd ──▶ zoekt-webserver sidecar (shards: default branch + index slots)
                           │        ├─▶ in process (gix): the overlay's blobs, the changed paths
                           │        └─▶ git ls-tree / grep / cat-file / rev-parse (pinned git)
                           └─ reads gitd's caches:  <cache>/<id>.git  (bare, reftable)

Zoekt indexes the default branch of each repository, and the branches searched by ref in a few index slots. Everything else (an agent's change tip, a push that the indexer has not caught up with) is answered by reading just the paths that differ from the indexed commit (blobs read in process, matched in Rust), so the answer is current without reindexing per change.

Run it

sh
bash scripts/zoekt.sh                       # pinned Zoekt -> ~/.cache/zuppo-tools/zoekt-<rev>/bin
zuppo-searchd serve \
  --cache /srv/gitd/cache --state /srv/searchd \
  --listen 10.0.0.7:7070 \
  --zoekt-bin-dir ~/.cache/zuppo-tools/zoekt-d64299b4420d/bin \
  --git /usr/local/bin/git-2.56.0 \
  --token-file /run/secrets/searchd-token

The first line on stdout, once the service is ready, is listening on 10.0.0.7:7070 (scripts and tests wait for it). Logging is RUST_LOG (default info); requests are logged at debug without the pattern, and no pattern is logged above debug.

flagmeaning
--cache DIRgitd's cache root: repository org/name is the bare reftable repository DIR/org/name.git
--state DIRindex sources (src/, hard-linked packs Zoekt can read) and, by default, the shards (shards/)
--shards DIRshard directory the sidecar serves (default <state>/shards)
--listen ADDRdefault 127.0.0.1:0
--git PATHthe git executable (env ZUPPO_GIT; the pinned 2.56.0 in production)
--zoekt-bin-dir DIRholds zoekt-webserver and zoekt-git-index (env ZUPPO_ZOEKT_BIN_DIR)
--token T / --token-file Frequire Authorization: Bearer T on everything except /healthz
--poll-ms Nhow often the cache is scanned for new repositories and moved tips (default 1000)
--search-timeout-ms Na search answers (truncated) or fails this long after it started (default 2000)
--processes Nmost line searches and diffs at once (default 2 x CPUs)
--tune KEY=VALUEa knob of the core, repeatable; see Tuning

SIGINT and SIGTERM stop accepting, let open requests finish (up to 10 s), stop the driver and kill the Zoekt child, then exit 0.

Deployment recipe

  • gitd serves fetches from <cache>/<id>.git and keeps those caches (packs only; the ref store is reftable). searchd reads the same directory; it never writes to it. Run both on one host or one volume (hard links of packs need one filesystem; across filesystems searchd copies the packs, which costs disk and time for large repositories).
  • searchd next to gitd, with its own --state directory (disk: the shards and one set of packs per indexed repository; it is derived data and can be deleted, the next start rebuilds). It starts and supervises one zoekt-webserver on a free 127.0.0.1 port; a crashed sidecar is restarted with backoff (250 ms doubling to 10 s) and GET /healthz is 503 meanwhile.
  • Network. searchd speaks plain HTTP/1.1 (and h2c) and has no TLS: it belongs on a private network behind the edge or the API, never on the internet. --token/--token-file is the check that a caller is the edge; it is compared in constant time. /healthz needs no token (for the orchestrator's probe).
  • The API (or edge) authorizes a user's access to repo first and then calls searchd; searchd itself does no per-user authorization.

On restart searchd asks Zoekt which commit the shards on disk were built from and adopts them (no rebuild); a repository is indexed once its default branch tip has been quiet for the debounce.

Wire

Errors are JSON: {"error": <code>, "message": <text>}. The codes are invalid_query (400), unauthorized (401), not_found (404), method_not_allowed (405), payload_too_large (413), backend (502; Zoekt or git failed), timeout (504; the search deadline passed with nothing to return; this one has no message).

POST /v1/search

Body (JSON, at most 64 KiB, unknown fields are rejected):

field
reporepository id (org/name), required
refa full ref name (refs/changes/1), a branch name (feature/x, read as refs/heads/feature/x) or HEAD (the default: the cache's default branch)
commita full lowercase hex commit id; wins over ref
patternthe text or regex, required, at most 4096 bytes, no line breaks
modeauto (default: text unless it has regex metacharacters, then a regex), literal, regex (Rust regex syntax, the same on every path: index, overlay and unindexed; Zoekt's RE2 agrees on what it accepts)
casesmart (default: insensitive unless the pattern has an upper-case letter), sensitive, insensitive
pathonly paths matching this regex
langonly files of this language (rust, python, ...; see Limits)
limitmost matching lines returned, 1..=1000 (default 50)
contextlines of context around each match, 0..=5

The target is commit if given (it must be a full id that exists as a commit), else ref. Unknown repository, ref or commit: 404. A bad regex, an unknown field or a malformed commit: 400.

Response 200:

json
{
  "files": [
    {"origin": "overlay", "path": "src/lib.rs", "score": 0.0,
     "lines": [{"number": 2, "text": "fn fresh() { needle_fresh(); }", "ranges": [[11, 23]],
                "before": [], "after": []}]},
    {"origin": "index", "path": "src/extra.rs", "score": 5000000000.01,
     "lines": [{"number": 1, "text": "needle_fresh again", "ranges": [[0, 12]],
                "before": [], "after": []}]}
  ],
  "truncated": false,
  "stats": {"kind": "identifier", "freshness": "overlay",
            "target": "7c0f…", "indexed": "a3d9…",
            "overlay_paths": 2, "overlay_dropped": 0, "index_lag_ms": 412}
}

ranges are byte offsets into text. truncated is true when more matches exist than limit or a backend stopped early (a Zoekt cap or wall budget, the git grep budget, the search deadline); a truncated answer is still a 200. Files come overlay-first (the freshest; by path), then the index's by score, then path.

GET /v1/stats

Counters since start and the state of every repository:

json
{"zoekt": {"health": "up", "reason": null, "restarts": 0, "forced_restarts": 0},
 "searches": {"in_flight": 0,
              "by_kind": {"identifier": 12, "regex_scan": 1},
              "by_freshness": {"exact": 9, "overlay": 3, "unindexed": 1},
              "errors": {"invalid_query": 2}},
 "replans": 0,
 "repos": {"acme/tool": {"branch": "main", "tip": "7c0f…", "indexed": "7c0f…",
                         "index_lag_ms": 0, "in_flight": 0, "shard_bytes": 20480}}}

index_lag_ms is how long the default branch has been ahead of the index (PC13: <= 1 min target). replans counts searches the core re-planned because Zoekt served another commit than planned (an index swap mid-query). zoekt.forced_restarts counts webserver restarts searchd forced because Zoekt missed new shards (see below); zoekt.restarts counts deaths only.

Watcher misses. zoekt-webserver finds new shards through a directory watcher (fsnotify), which can miss an event (seen with kqueue on macOS: six shards written, never loaded). searchd does not trust it: after zoekt-git-index it polls /api/list for 3 s (watch_grace); if the new commit is not served it logs a warning, restarts the webserver (a fresh process reads the directory) and waits for the commit again, up to the 30 s serve timeout. Publishes of several repositories behind one miss share one restart. Searches in flight during the restart fail with 502 backend or are re-planned by the core; new ones wait up to 2 s for the new process.

POST /v1/refresh

{"repo": "org/name"}: read that repository's default branch tip now instead of at the next poll (use it after a push when the next second matters); 202 {"status":"accepted"}, 404 for an unknown repository. Searches never need it: the target is resolved from the cache per request.

GET /healthz

200 ok while the Zoekt sidecar is up; 503 with the reason (zoekt is starting, zoekt is down: <exit status and the tail of its stderr>) otherwise. Never needs the token. While Zoekt is down, searches planned against an index fail with 502 backend (the core reports the Zoekt failure; it does not guess), searches of repositories without an index still work (git grep or the Rust scan; see Limits), and the sidecar restarts on its own.

Routing by kind

The core classifies every pattern and gives each kind its own Zoekt budget (spike 4: literals 3.9 ms p95, unanchored regexes 397 ms p95 on the same index). shard cap and total cap are multiples of limit; wall is Zoekt's MaxWallTime.

stats.kindpatternshard captotal capwall
literalfixed text, not identifier-shaped (foo bar, a.b with mode: literal)24250 ms
identifier[A-Za-z_][A-Za-z0-9_]*24250 ms
regex_literala regex with a required literal of >= 3 bytes (fn\s+parse_\w+): Zoekt's trigrams narrow it1250 ms
regex_scana regex without one (a.c), and fixed text shorter than a trigram1250 ms

Small character classes narrow a regex_scan pattern: a run of literal characters and printable-ASCII classes of at most 16 characters (ba[rz], [qxz]{3}) becomes trigram OR-sets of at most 64 trigrams ((content:"bar" or content:"baz")) ANDed into Zoekt's query, which Zoekt answers from its trigram index (its own extraction drops classes). Their matches can add lines of files the pattern matches, which the per-line re-check drops; because they also count against Zoekt's match caps, a regex_literal pattern (already narrowed by its literal) gets none.

Other budgets: the overlay (changed paths, at most 2000 of them, uncapped for a branch in an index slot) gets 250 ms; the whole-tree search of an unindexed repository 1500 ms; the whole search 2000 ms. When a budget stops a backend the answer is truncated, not an error. Once the overlay alone holds limit lines (its files come first) the search answers without waiting for Zoekt, truncated.

Freshness

stats.freshness says how current the answer is relative to the target:

valuemeaning
exactthe index is at the target commit; Zoekt alone answered
overlaythe index is at another commit (behind the default branch, or the target is an agent's change); the paths that differ were searched at the target (blobs read in process with gix and kept per searched commit, lines matched with the Rust regex crate), Zoekt's hits for them were dropped; everything else is identical in both commits, so Zoekt's answer for it is the target's answer
overlay_truncatedmore differing paths than the overlay budget (2000): hits of the uncovered paths were dropped, some changed files may be missing (stats.overlay_dropped); truncated is true
unindexedno index: a new repository before its first index run, or one Zoekt cannot read; the whole tree is searched under its wall budget

Overlay-first ordering: files from the overlay (origin overlay) precede the index's, because they are the changed ones, and a changed file never appears from the index. Renames are a delete plus an add; a deleted path is never searched.

Index scheduling: a moved default branch tip waits for debounce_ms of quiet (default 2 s) but never longer than max_wait_ms (30 s) after the first unindexed change; one index run per repository at a time; a failed run is retried with backoff (1 s doubling to 5 min). A run is zoekt-git-index over a files-backend source repository that hard-links the cache's packs (Zoekt cannot open reftable repositories) and returns once the webserver serves the new shards, so searches see the old commit, then the new one, never a half-written shard.

Index slots: a search by ref of a branch other than the default (refs/heads/x, an agent's refs/changes/n) whose tip differs from the default branch asks for one of index_slots (4) slots; the least recently searched branch gives its slot up. The next index run (same debounce and bounds) builds the default branch and every slot in one zoekt-git-index -branches run: files the branches share are indexed once, and since the branch set never changes, every run after the first is a -delta build of the files that changed on some branch (a full build again once 32 delta shards stack up). Until the slot serves the branch its searches are the overlay against the default branch's index; then they read the slot (exact, or the overlay against the slot's commit after a push, without the 2000-path cap). A search by commit never uses a slot.

Tuning

--tune key=value, repeatable; values are non-negative integers. docs/plan PC13 tunes these.

keydefault
<kind>.shard_per_limit2 / 2 / 1 / 1Zoekt ShardMaxMatchCount = limit x this (literal, identifier, regex_literal, regex_scan)
<kind>.total_per_limit4 / 4 / 2 / 2TotalMaxMatchCount = limit x this
<kind>.wall_ms250 / 250 / 50 / 50Zoekt MaxWallTime
search_timeout_ms2000the whole search (same as --search-timeout-ms)
overlay_budget2000most overlay paths searched
overlay_wall_ms250wall budget of the overlay
unindexed_wall_ms1500wall budget of the whole-tree search
exclude_max1000most changed paths excluded in Zoekt's query (-f:); beyond it only the post-filter drops stale hits
diff_entries, diff_paths128, 200000memoised diffs: entries, and paths over all entries
replans2re-plans against the commit Zoekt served before 502 index churn
debounce_ms, max_wait_ms2000, 30000index scheduling
backoff_base_ms, backoff_max_ms1000, 300000index retry backoff
index_slots4branches besides the default one indexed at once (0..=63; 0: the default branch only)

Example: --tune regex_scan.wall_ms=100 --tune literal.total_per_limit=8 --tune debounce_ms=500.

Limits

  • SHA-256 repositories cannot be indexed (Zoekt reads SHA-1 repositories only). They are searched unindexed under the whole-tree wall budget, and the index run is attempted once and never retried.
  • The default branch and `index_slots` branches are indexed. Any other ref (a branch past the slots, a commit searched by id) is the overlay against the default branch's index; a target far from it (thousands of differing paths) degrades to overlay_truncated. Each slot can add up to a repository's index size of shards (a branch far from the default one), so the disk bound is about (1 + index_slots) times the default branch's shards. A default branch whose name contains zuppo-slot- is not indexed (a slot query matches branch names by substring).
  • Non-UTF-8 paths are read lossily (replacement characters), so a query cannot name them and their hits may carry a mangled path. Non-UTF-8 line text is replaced the same way; lines over 64 KiB are cut at a character boundary. Binary files (a NUL in the first 8000 bytes, as git grep -I), symlinks and submodules are skipped.
  • One regex engine, with size limits (SEC-8). Every user regex (the pattern and the path filter) is compiled by the Rust regex crate (linear time, no unbounded backtracking) under a 1 MiB compiled-size ceiling (a pattern that needs more is 400) and a 1 MiB lazy-DFA cache. No regex is handed to git: git grep -E is libc's POSIX ERE, which differs in dialect (no \d, (?i), lazy quantifiers or \pL; on macOS no escapes at all) and blows up on counted repetitions (((a{1,20}){1,20}){1,20} ran past 30 s in Apple git over a 244 KB file; the crate compiles it in 2 ms). So every regex search of an unindexed repository streams blobs through one git cat-file --batch and matches each line in Rust, and the overlay reads its blobs in process (only the first 2 MiB of a line are matched, 64 KiB of it returned). git grep -F is kept only for a fixed string over the whole tree of an unindexed repository, and only where it provably answers the same: not empty and, case-insensitively, free of non-ASCII text, k and s (Unicode folding would differ). Searching an overlay at a commit that does not exist yields no hits rather than an error; the whole-tree scan fails.
  • `lang` is a fixed extension table (rust, python, typescript, c++, ... see zuppo_search::lang); .h is C. An unknown language is invalid_query.
  • Files Zoekt did not index (too large, too few trigrams) appear in the index as a NOT-INDEXED: <reason> placeholder; searchd drops those lines, and such a file is found only through the overlay, if it is among the changed paths.
  • Matches never span lines: a regex runs per line (the core re-checks every returned line).
  • One searchd serves one cache. There is no sharding across hosts and no per-user access check.

Behind CodeService.Grep (zuppo serve --searchd)

zuppo serve --searchd http://127.0.0.1:7070 (env ZUPPO_SEARCHD; --searchd-token / ZUPPO_SEARCHD_TOKEN when searchd has a token) makes the forge's Grep ask this service first, for the commit the forge resolved the ref to, with context, limit (max_matches) and the request's case rule. searchd must read the forge's own caches, <state>/cache. The answer is git grep's whenever the index cannot give it exactly, so these fall back to git grep (the engine runs it as before, never an error): searchd is down or slow (3 s), answers an error, or its answer is overlay_truncated or was cut short of the limit by a budget; a pattern over 4096 bytes, context over 5, a case-insensitive pattern with non-ASCII text (Zoekt folds case differently from the matcher), or a path glob this service's path regex cannot say ([...], \, a ** that is not a whole path segment; the rest are translated and equal git's :(glob) pathspecs, checked against git grep in zuppo-apid's searchd tests).

What differs from git grep when the index answers: a search that stops at its limit returns some of the matches (by score), not the first ones in path order, and total_matches is then the number found (a lower bound; truncated is set); and a file the index did not index (over 1 MiB, or with too many distinct trigrams: the NOT-INDEXED placeholders above) is searched only while it is among the paths changed since the indexed commit. Everything else (lines, line numbers, context, sorting by path) is equal, and searchd/tests/forge.rs checks it against a forge without an index, also right after a push.

The Cloudflare engine Container

The engine Container (deploy/cloudflare/engine) runs searchd and the pinned Zoekt beside zuppo serve (its start.sh; scripts/search-image.sh builds their image, which up.sh does after the engine base), over the container's own disk: the index is derived data, rebuilt after a fresh start, and until then searches scan the tree. Nothing in this service needs a Cloudflare account. What stays open is the measurement on the real platform (deploy/cloudflare/RUNBOOK.md), and the PC13 measurement on the standard-4 container (bench/pc13) that fixes the defaults in Tuning.