Code search (zuppo-searchd)
zuppo-searchd answers "where does this text occur in this repository at this commit" for the web UI, the API and agents (SRV-4; ARC-18, ARC-43; criterion PC13…
zuppo-searchd answers "where does this text occur in this repository at this commit" for the web UI, the API and agents (SRV-4; ARC-18, ARC-43; criterion PC13: p95 <= 300 ms end to end, and a path an agent changed is searchable at once).
It is the sans-IO core crates/zuppo-search (routing by query kind, Zoekt budgets, the freshness overlay, index scheduling) driven by crates/zuppo-searchd with real processes:
edge / API ──HTTP──▶ zuppo-searchd ──▶ zoekt-webserver sidecar (shards: default branch + index slots)
│ ├─▶ in process (gix): the overlay's blobs, the changed paths
│ └─▶ git ls-tree / grep / cat-file / rev-parse (pinned git)
└─ reads gitd's caches: <cache>/<id>.git (bare, reftable)Zoekt indexes the default branch of each repository, and the branches searched by ref in a few index slots. Everything else (an agent's change tip, a push that the indexer has not caught up with) is answered by reading just the paths that differ from the indexed commit (blobs read in process, matched in Rust), so the answer is current without reindexing per change.
Run it
bash scripts/zoekt.sh # pinned Zoekt -> ~/.cache/zuppo-tools/zoekt-<rev>/bin zuppo-searchd serve \ --cache /srv/gitd/cache --state /srv/searchd \ --listen 10.0.0.7:7070 \ --zoekt-bin-dir ~/.cache/zuppo-tools/zoekt-d64299b4420d/bin \ --git /usr/local/bin/git-2.56.0 \ --token-file /run/secrets/searchd-token
The first line on stdout, once the service is ready, is listening on 10.0.0.7:7070 (scripts and tests wait for it). Logging is RUST_LOG (default info); requests are logged at debug without the pattern, and no pattern is logged above debug.
| flag | meaning |
|---|---|
--cache DIR | gitd's cache root: repository org/name is the bare reftable repository DIR/org/name.git |
--state DIR | index sources (src/, hard-linked packs Zoekt can read) and, by default, the shards (shards/) |
--shards DIR | shard directory the sidecar serves (default <state>/shards) |
--listen ADDR | default 127.0.0.1:0 |
--git PATH | the git executable (env ZUPPO_GIT; the pinned 2.56.0 in production) |
--zoekt-bin-dir DIR | holds zoekt-webserver and zoekt-git-index (env ZUPPO_ZOEKT_BIN_DIR) |
--token T / --token-file F | require Authorization: Bearer T on everything except /healthz |
--poll-ms N | how often the cache is scanned for new repositories and moved tips (default 1000) |
--search-timeout-ms N | a search answers (truncated) or fails this long after it started (default 2000) |
--processes N | most line searches and diffs at once (default 2 x CPUs) |
--tune KEY=VALUE | a knob of the core, repeatable; see Tuning |
SIGINT and SIGTERM stop accepting, let open requests finish (up to 10 s), stop the driver and kill the Zoekt child, then exit 0.
Deployment recipe
- gitd serves fetches from
<cache>/<id>.gitand keeps those caches (packs only; the ref store is reftable). searchd reads the same directory; it never writes to it. Run both on one host or one volume (hard links of packs need one filesystem; across filesystems searchd copies the packs, which costs disk and time for large repositories). - searchd next to gitd, with its own
--statedirectory (disk: the shards and one set of packs per indexed repository; it is derived data and can be deleted, the next start rebuilds). It starts and supervises onezoekt-webserveron a free127.0.0.1port; a crashed sidecar is restarted with backoff (250 ms doubling to 10 s) andGET /healthzis 503 meanwhile. - Network. searchd speaks plain HTTP/1.1 (and h2c) and has no TLS: it belongs on a private network behind the edge or the API, never on the internet.
--token/--token-fileis the check that a caller is the edge; it is compared in constant time./healthzneeds no token (for the orchestrator's probe). - The API (or edge) authorizes a user's access to
repofirst and then calls searchd; searchd itself does no per-user authorization.
On restart searchd asks Zoekt which commit the shards on disk were built from and adopts them (no rebuild); a repository is indexed once its default branch tip has been quiet for the debounce.
Wire
Errors are JSON: {"error": <code>, "message": <text>}. The codes are invalid_query (400), unauthorized (401), not_found (404), method_not_allowed (405), payload_too_large (413), backend (502; Zoekt or git failed), timeout (504; the search deadline passed with nothing to return; this one has no message).
POST /v1/search
Body (JSON, at most 64 KiB, unknown fields are rejected):
| field | |
|---|---|
repo | repository id (org/name), required |
ref | a full ref name (refs/changes/1), a branch name (feature/x, read as refs/heads/feature/x) or HEAD (the default: the cache's default branch) |
commit | a full lowercase hex commit id; wins over ref |
pattern | the text or regex, required, at most 4096 bytes, no line breaks |
mode | auto (default: text unless it has regex metacharacters, then a regex), literal, regex (Rust regex syntax, the same on every path: index, overlay and unindexed; Zoekt's RE2 agrees on what it accepts) |
case | smart (default: insensitive unless the pattern has an upper-case letter), sensitive, insensitive |
path | only paths matching this regex |
lang | only files of this language (rust, python, ...; see Limits) |
limit | most matching lines returned, 1..=1000 (default 50) |
context | lines of context around each match, 0..=5 |
The target is commit if given (it must be a full id that exists as a commit), else ref. Unknown repository, ref or commit: 404. A bad regex, an unknown field or a malformed commit: 400.
Response 200:
{
"files": [
{"origin": "overlay", "path": "src/lib.rs", "score": 0.0,
"lines": [{"number": 2, "text": "fn fresh() { needle_fresh(); }", "ranges": [[11, 23]],
"before": [], "after": []}]},
{"origin": "index", "path": "src/extra.rs", "score": 5000000000.01,
"lines": [{"number": 1, "text": "needle_fresh again", "ranges": [[0, 12]],
"before": [], "after": []}]}
],
"truncated": false,
"stats": {"kind": "identifier", "freshness": "overlay",
"target": "7c0f…", "indexed": "a3d9…",
"overlay_paths": 2, "overlay_dropped": 0, "index_lag_ms": 412}
}ranges are byte offsets into text. truncated is true when more matches exist than limit or a backend stopped early (a Zoekt cap or wall budget, the git grep budget, the search deadline); a truncated answer is still a 200. Files come overlay-first (the freshest; by path), then the index's by score, then path.
GET /v1/stats
Counters since start and the state of every repository:
{"zoekt": {"health": "up", "reason": null, "restarts": 0, "forced_restarts": 0},
"searches": {"in_flight": 0,
"by_kind": {"identifier": 12, "regex_scan": 1},
"by_freshness": {"exact": 9, "overlay": 3, "unindexed": 1},
"errors": {"invalid_query": 2}},
"replans": 0,
"repos": {"acme/tool": {"branch": "main", "tip": "7c0f…", "indexed": "7c0f…",
"index_lag_ms": 0, "in_flight": 0, "shard_bytes": 20480}}}index_lag_ms is how long the default branch has been ahead of the index (PC13: <= 1 min target). replans counts searches the core re-planned because Zoekt served another commit than planned (an index swap mid-query). zoekt.forced_restarts counts webserver restarts searchd forced because Zoekt missed new shards (see below); zoekt.restarts counts deaths only.
Watcher misses. zoekt-webserver finds new shards through a directory watcher (fsnotify), which can miss an event (seen with kqueue on macOS: six shards written, never loaded). searchd does not trust it: after zoekt-git-index it polls /api/list for 3 s (watch_grace); if the new commit is not served it logs a warning, restarts the webserver (a fresh process reads the directory) and waits for the commit again, up to the 30 s serve timeout. Publishes of several repositories behind one miss share one restart. Searches in flight during the restart fail with 502 backend or are re-planned by the core; new ones wait up to 2 s for the new process.
POST /v1/refresh
{"repo": "org/name"}: read that repository's default branch tip now instead of at the next poll (use it after a push when the next second matters); 202 {"status":"accepted"}, 404 for an unknown repository. Searches never need it: the target is resolved from the cache per request.
GET /healthz
200 ok while the Zoekt sidecar is up; 503 with the reason (zoekt is starting, zoekt is down: <exit status and the tail of its stderr>) otherwise. Never needs the token. While Zoekt is down, searches planned against an index fail with 502 backend (the core reports the Zoekt failure; it does not guess), searches of repositories without an index still work (git grep or the Rust scan; see Limits), and the sidecar restarts on its own.
Routing by kind
The core classifies every pattern and gives each kind its own Zoekt budget (spike 4: literals 3.9 ms p95, unanchored regexes 397 ms p95 on the same index). shard cap and total cap are multiples of limit; wall is Zoekt's MaxWallTime.
| stats.kind | pattern | shard cap | total cap | wall |
|---|---|---|---|---|
literal | fixed text, not identifier-shaped (foo bar, a.b with mode: literal) | 2 | 4 | 250 ms |
identifier | [A-Za-z_][A-Za-z0-9_]* | 2 | 4 | 250 ms |
regex_literal | a regex with a required literal of >= 3 bytes (fn\s+parse_\w+): Zoekt's trigrams narrow it | 1 | 2 | 50 ms |
regex_scan | a regex without one (a.c), and fixed text shorter than a trigram | 1 | 2 | 50 ms |
Small character classes narrow a regex_scan pattern: a run of literal characters and printable-ASCII classes of at most 16 characters (ba[rz], [qxz]{3}) becomes trigram OR-sets of at most 64 trigrams ((content:"bar" or content:"baz")) ANDed into Zoekt's query, which Zoekt answers from its trigram index (its own extraction drops classes). Their matches can add lines of files the pattern matches, which the per-line re-check drops; because they also count against Zoekt's match caps, a regex_literal pattern (already narrowed by its literal) gets none.
Other budgets: the overlay (changed paths, at most 2000 of them, uncapped for a branch in an index slot) gets 250 ms; the whole-tree search of an unindexed repository 1500 ms; the whole search 2000 ms. When a budget stops a backend the answer is truncated, not an error. Once the overlay alone holds limit lines (its files come first) the search answers without waiting for Zoekt, truncated.
Freshness
stats.freshness says how current the answer is relative to the target:
| value | meaning |
|---|---|
exact | the index is at the target commit; Zoekt alone answered |
overlay | the index is at another commit (behind the default branch, or the target is an agent's change); the paths that differ were searched at the target (blobs read in process with gix and kept per searched commit, lines matched with the Rust regex crate), Zoekt's hits for them were dropped; everything else is identical in both commits, so Zoekt's answer for it is the target's answer |
overlay_truncated | more differing paths than the overlay budget (2000): hits of the uncovered paths were dropped, some changed files may be missing (stats.overlay_dropped); truncated is true |
unindexed | no index: a new repository before its first index run, or one Zoekt cannot read; the whole tree is searched under its wall budget |
Overlay-first ordering: files from the overlay (origin overlay) precede the index's, because they are the changed ones, and a changed file never appears from the index. Renames are a delete plus an add; a deleted path is never searched.
Index scheduling: a moved default branch tip waits for debounce_ms of quiet (default 2 s) but never longer than max_wait_ms (30 s) after the first unindexed change; one index run per repository at a time; a failed run is retried with backoff (1 s doubling to 5 min). A run is zoekt-git-index over a files-backend source repository that hard-links the cache's packs (Zoekt cannot open reftable repositories) and returns once the webserver serves the new shards, so searches see the old commit, then the new one, never a half-written shard.
Index slots: a search by ref of a branch other than the default (refs/heads/x, an agent's refs/changes/n) whose tip differs from the default branch asks for one of index_slots (4) slots; the least recently searched branch gives its slot up. The next index run (same debounce and bounds) builds the default branch and every slot in one zoekt-git-index -branches run: files the branches share are indexed once, and since the branch set never changes, every run after the first is a -delta build of the files that changed on some branch (a full build again once 32 delta shards stack up). Until the slot serves the branch its searches are the overlay against the default branch's index; then they read the slot (exact, or the overlay against the slot's commit after a push, without the 2000-path cap). A search by commit never uses a slot.
Tuning
--tune key=value, repeatable; values are non-negative integers. docs/plan PC13 tunes these.
| key | default | |
|---|---|---|
<kind>.shard_per_limit | 2 / 2 / 1 / 1 | Zoekt ShardMaxMatchCount = limit x this (literal, identifier, regex_literal, regex_scan) |
<kind>.total_per_limit | 4 / 4 / 2 / 2 | TotalMaxMatchCount = limit x this |
<kind>.wall_ms | 250 / 250 / 50 / 50 | Zoekt MaxWallTime |
search_timeout_ms | 2000 | the whole search (same as --search-timeout-ms) |
overlay_budget | 2000 | most overlay paths searched |
overlay_wall_ms | 250 | wall budget of the overlay |
unindexed_wall_ms | 1500 | wall budget of the whole-tree search |
exclude_max | 1000 | most changed paths excluded in Zoekt's query (-f:); beyond it only the post-filter drops stale hits |
diff_entries, diff_paths | 128, 200000 | memoised diffs: entries, and paths over all entries |
replans | 2 | re-plans against the commit Zoekt served before 502 index churn |
debounce_ms, max_wait_ms | 2000, 30000 | index scheduling |
backoff_base_ms, backoff_max_ms | 1000, 300000 | index retry backoff |
index_slots | 4 | branches besides the default one indexed at once (0..=63; 0: the default branch only) |
Example: --tune regex_scan.wall_ms=100 --tune literal.total_per_limit=8 --tune debounce_ms=500.
Limits
- SHA-256 repositories cannot be indexed (Zoekt reads SHA-1 repositories only). They are searched
unindexedunder the whole-tree wall budget, and the index run is attempted once and never retried. - The default branch and `index_slots` branches are indexed. Any other ref (a branch past the slots, a commit searched by id) is the overlay against the default branch's index; a target far from it (thousands of differing paths) degrades to
overlay_truncated. Each slot can add up to a repository's index size of shards (a branch far from the default one), so the disk bound is about(1 + index_slots)times the default branch's shards. A default branch whose name containszuppo-slot-is not indexed (a slot query matches branch names by substring). - Non-UTF-8 paths are read lossily (replacement characters), so a query cannot name them and their hits may carry a mangled path. Non-UTF-8 line text is replaced the same way; lines over 64 KiB are cut at a character boundary. Binary files (a NUL in the first 8000 bytes, as
git grep -I), symlinks and submodules are skipped. - One regex engine, with size limits (SEC-8). Every user regex (the pattern and the
pathfilter) is compiled by the Rustregexcrate (linear time, no unbounded backtracking) under a 1 MiB compiled-size ceiling (a pattern that needs more is400) and a 1 MiB lazy-DFA cache. No regex is handed to git:git grep -Eis libc's POSIX ERE, which differs in dialect (no\d,(?i), lazy quantifiers or\pL; on macOS no escapes at all) and blows up on counted repetitions (((a{1,20}){1,20}){1,20}ran past 30 s in Apple git over a 244 KB file; the crate compiles it in 2 ms). So every regex search of an unindexed repository streams blobs through onegit cat-file --batchand matches each line in Rust, and the overlay reads its blobs in process (only the first 2 MiB of a line are matched, 64 KiB of it returned).git grep -Fis kept only for a fixed string over the whole tree of an unindexed repository, and only where it provably answers the same: not empty and, case-insensitively, free of non-ASCII text,kands(Unicode folding would differ). Searching an overlay at a commit that does not exist yields no hits rather than an error; the whole-tree scan fails. - `lang` is a fixed extension table (
rust,python,typescript,c++, ... seezuppo_search::lang);.his C. An unknown language isinvalid_query. - Files Zoekt did not index (too large, too few trigrams) appear in the index as a
NOT-INDEXED: <reason>placeholder; searchd drops those lines, and such a file is found only through the overlay, if it is among the changed paths. - Matches never span lines: a regex runs per line (the core re-checks every returned line).
- One searchd serves one cache. There is no sharding across hosts and no per-user access check.
Behind CodeService.Grep (zuppo serve --searchd)
zuppo serve --searchd http://127.0.0.1:7070 (env ZUPPO_SEARCHD; --searchd-token / ZUPPO_SEARCHD_TOKEN when searchd has a token) makes the forge's Grep ask this service first, for the commit the forge resolved the ref to, with context, limit (max_matches) and the request's case rule. searchd must read the forge's own caches, <state>/cache. The answer is git grep's whenever the index cannot give it exactly, so these fall back to git grep (the engine runs it as before, never an error): searchd is down or slow (3 s), answers an error, or its answer is overlay_truncated or was cut short of the limit by a budget; a pattern over 4096 bytes, context over 5, a case-insensitive pattern with non-ASCII text (Zoekt folds case differently from the matcher), or a path glob this service's path regex cannot say ([...], \, a ** that is not a whole path segment; the rest are translated and equal git's :(glob) pathspecs, checked against git grep in zuppo-apid's searchd tests).
What differs from git grep when the index answers: a search that stops at its limit returns some of the matches (by score), not the first ones in path order, and total_matches is then the number found (a lower bound; truncated is set); and a file the index did not index (over 1 MiB, or with too many distinct trigrams: the NOT-INDEXED placeholders above) is searched only while it is among the paths changed since the indexed commit. Everything else (lines, line numbers, context, sorting by path) is equal, and searchd/tests/forge.rs checks it against a forge without an index, also right after a push.
The Cloudflare engine Container
The engine Container (deploy/cloudflare/engine) runs searchd and the pinned Zoekt beside zuppo serve (its start.sh; scripts/search-image.sh builds their image, which up.sh does after the engine base), over the container's own disk: the index is derived data, rebuilt after a fresh start, and until then searches scan the tree. Nothing in this service needs a Cloudflare account. What stays open is the measurement on the real platform (deploy/cloudflare/RUNBOOK.md), and the PC13 measurement on the standard-4 container (bench/pc13) that fixes the defaults in Tuning.