Slow embedding backend reads as dead, and each recovery costs a full re-collect

[SK-DEV-AI]

Slow embedding backend reads as dead, and each recovery costs a full re-collect

Binary 0.55.1, project ~10K files / 31K semantic entries, remote embedding backend (NVIDIA nemotron-3-embed-1b).

Three things I watched in aft-plugin.log today that look linked.

1. The backend is slow, not dead, but AFT calls it dead.

embedding backend appears down; suspending active retries, will resume on next change fired 13 times today. I called the endpoint directly each time I checked: HTTP 200, correct 2048-dim vectors, but 2.3 seconds for a two-word probe. So the probe cannot tell "slow" from "dead", and every verdict of dead arms a resume-on-next-change retry.

2. Each cycle re-collects the entire corpus with nothing changed.

12:07:42 semantic collect: 41082 chunks from 2615 files in 1629 ms
12:11:10 semantic collect: 41082 chunks from 2615 files in 1445 ms
12:15:10 semantic collect: 41082 chunks from 2615 files in 1471 ms
12:17:59 semantic collect: 41082 chunks from 2615 files in 1642 ms

The perf ticks between these show ingested:0, paths:0. No file changes, yet the full 41K-chunk parse plus extract (~15s of threaded CPU) reruns every few minutes. If resume only needs the un-embedded dirty set, it should not cost a full collect. This is the CPU ramp felt on the machine.

3. The allocator never gives the cycle memory back.

allocator slack relief: released=0 slack_before=3946097488 rss_before=1018249216 rss_after=833642496
allocator slack relief: released=24576 slack_before=4132255456 slack_after=4132231888

~4GB slack retained, relief releases 0 to 24KB. Resident oscillates roughly 470MB to 1.4GB with each collect cycle. The process never returns to its settled size because each recovery round trips through the same surge.

Separately, a watcher overflow earlier forced a full rescan (9776 files, 228520 trigrams, 7111 ms, pool=6), which is the expected path and not the complaint. The complaint is only the steady-state loop above: slow backend misread as dead, full re-collect per recovery, unreturned memory per round.

Happy to pull any other log slice that helps. I did not read the source for the probe timeout or the resume path, so the above is behavior only.

[aft-alfonso]

Thanks — the three are one loop, and the log shape you pasted matches the source exactly.

The mechanism: background embedding requests use one fixed deadline for the whole batch (semantic.timeout_ms, default 25 s, over a batch of up to semantic.max_batch_size = 64 chunks). Against a backend that answers a two-word probe in 2.3 s, a 64-chunk batch of code cannot finish inside 25 s, so reqwest reports a timeout; the client classifies a timeout as transient and retries the same batch at the same deadline three times; three timeouts fail the corpus refresh, which trips the breaker and logs "appears down". The recovery "probe" is not a small request — it re-sends the whole corpus refresh, whose first step is the full collect you see (41082 chunks each time, nothing changed), and its first big batch times out again. Slow-but-alive and dead are indistinguishable to that ladder, and each round pays the collect. The memory oscillation is that collect's transient allocations; the 4 GB "slack" figure is virtual heap already returned to the OS lazily, so relief releasing 0–24 KB is expected rather than a leak, but the surge itself should not recur.

Fix in progress on both halves: per-request deadlines that scale with batch size, with the batch halved on timeout so a slow backend converges instead of failing (a one-chunk request that still times out is the real "dead" signal), and resume that re-embeds only the un-embedded set the failed refresh already knew about, never a fresh corpus collect. Until that ships, on 0.55.1 you can make the loop stop today from ~/.config/cortexkit/aft.jsonc: "semantic": { "timeout_ms": 180000, "max_batch_size": 8 } — the batch then fits the deadline on that backend. Changing those keys does not rebuild the index.

[aft-alfonso]

Shipped in v0.56.0: embed deadlines scale with batch size and measured per-item latency, a timeout halves the batch instead of retrying it, only a one-chunk timeout counts as the backend being down, and recovery re-embeds exactly the failed file set without re-collecting the corpus. Thanks for the detailed timings.