(SWA radix cache) Prompt committed beneath a running request's expired prefix frees the tree's own slots at finish #683

Before you start

• I have read the FAQ and my problem is not answered there.
• I have read the Roadmap and this is not already planned there.
• I have searched existing issues and found no duplicate.
• I am on the latest release, or on a freshly rebuilt main when building from source.

What happened

Two ordinary /v1/completions requests kill the scheduler worker within two seconds on a sliding-window model: a long prompt (longer than the window) that is still prefilling/decoding, and a short prompt that shares its first few tokens with it and decodes at least two tokens beside it. When both have ended and
the scheduler goes idle, check_integrity fires

AssertionError: SWA-slot leak/double-free: free(6208) + tree(349) != capacity(6553)
  File "freetoken/scheduler/cache.py", line 540, in check_integrity
  File "freetoken/scheduler/scheduler.py", line 157, in run_when_idle
Backend supervisor: backend worker freetoken-TP0-scheduler exited
Backend worker is gone and cannot be restarted; stopping the API server

API server crashes ten seconds later. This is the assertion #204 reports after 10-20 minutes of concurrent tool-calling traffic. The reduction here needs no tools, no reasoning, no cache warm-up and no MoE offload; it reproduced 9/9 on main (377a3bc, 0.1.3+21) on an RTX 4090 with openai/gpt-oss-20b (12 sliding-window layers, window 128, 12 full layers), both --moe-strategy fused and offload.

Reproduction
repro_swa_double_free.py

ft serve --model openai/gpt-oss-20b --num-tokens 32768 --max-prefill-length 2048 --memory-ratio 0.8 \
         --enable-cache-report --moe-strategy fused          # cache_type resolves to swa_radix, page_size 1
python3 repro_swa_double_free.py --stage side --no-warm --side-max-tokens 4   # attached; httpx only

The script sends:

1. request A: a ~2.8k-token prompt ("<nonce>r0 " + 400 nonce words), streamed, max_tokens 200;
2. 20 ms later request B: an 8-token prompt that starts with the same nonce ("<nonce>probe0 hi",
   5 shared tokens), non-streamed, max_tokens 4;
3. waits for both, then polls /health.

Exit 1 = /health reports {"status":"error","message":"backend worker freetoken-TP0-scheduler exited"} within seconds of the last reply. The server log shows A's second prefill chunk and B's prefill in one batch (Prefill batch, #new-seq: 2, #new-token: 771, #cached-token: 0), then the assertion right after Scheduler is idle, waiting for new reqs....

Some ablations (one fresh server per row)

In short it needs the SWA radix cache, a tombstoned (out-of-window) region of the tree that B's prompt descends into, B committing its prompt while A still holds the lock on that path, and B decoding at least one step after that commit.

Below are some Claude slob that might be helpful for fixing:

Where the four slots go (one-off instrumentation, not in the attached repro)

Printing every node at the failing check (tree_dump_diag2_mt4.log):

[DIAG-TREE] evictable=349 protected=0 free=6208 cap=6553
  depth=1 pos=0    len=5    tomb=True  ref=0 swa_live=0   swa_sentinel=5     <- shared head, A's slots
  depth=2 pos=5    len=1914 tomb=True  ...                                   <- A's out-of-window prompt
  depth=3 pos=1919 len=748  tomb=True  ...
  depth=4..7       A's retained window + generated tokens, all live          <- 343 live, consistent
  depth=2 pos=5    len=3    tomb=False ref=0 swa_live=0   swa_sentinel=3     <- B's prompt after the head
  depth=3 pos=8    len=1    tomb=False ref=0 swa_live=0   swa_sentinel=1     <- B's first generated token
  depth=4 pos=9    len=2    tomb=False ref=0 swa_live=2   swa_sentinel=0     <- B's last two tokens

The swa free-list holds no duplicate (6208 entries, 6208 unique), the free-list and the tree do not
overlap, and mapped + free = 6553 exactly: the pool is consistent. The tree's swa_evictable counts
349 tokens as live, but four of them, all in B's nodes, map to the sentinel 0 -- their swa slots were
returned to the pool while the nodes stayed non-tombstone. The same freed tensor also returned those
nodes' full-pool pages to free_slots while the tree still references them, which is the
"duplicate pages in the full free list" #204 measured but could not attribute.

Why (reading scheduler/cache.py and kvcache/swa_radix_cache.py at 377a3bc)

1. B's prefill shares a batch with A's final chunk. Draining it commits A's prompt first (_cache_req_swa(finished=False), cache.py:421-): A's out-of-window prefix becomes a tombstone node and A locks the whole path (inc_lock walks to the root, so the shared head has ref_count > 0).
2. B's prompt commit then runs insert(B[:8], swa_evicted_seqlen=0, update_kv_after_len=0) (swa_radix_cache.py:137-): the walk matches the tombstoned shared head, which is full-locked, so Branch 3 keeps the tombstone and frees B's five duplicate head slots (correct), and adds B's remaining prompt tokens as a live leaf under the tombstone node (B's own slots become the tree's).
3. The commit continues with m = self.prefix_cache.match_prefix(req.input_ids[:insert_len]) (cache.py, the else: branch of _cache_req_swa). match_prefix (swa_radix_cache.py:88-135) counts the live run since the last tombstone -- B's 3 tokens -- and because it is shorter than the window (128) it returns cached_len = 0. B therefore gets SWACacheHandle(0, root, []), the page table is not re-pointed (if m.cached_len > 0), and B holds no lock on its own committed nodes.
4. B keeps decoding on its page-table row, whose slots are now also the tree's.
5. At B's finish, _cache_req_swa(finished=True) calls insert(B[:cached_len], ..., update_kv_after_len=old_handle.cached_len) with update_kv_after_len = 0. Every matched node is now in the "fresh" region; B's own prompt node is live, so the walk takes the # Matched a live node -> tree slots are canonical; drop the request's dup branch and appends seg_kv -- the request's slots for that span -- to freed. But they are the very slots the node holds. self._free_swa(freed) resets their mapping to the sentinel and self._free(freed) returns their full pages to free_slots, while the node stays live with swa_evictable unchanged.
6. At idle, free + tree exceeds the capacity by the length of those nodes and check_integrity raises; the frontend stops accepting and exits (api_server.py:1002-1012).

The design assumption that breaks is in step 3: an unfinished commit assumes that the committed prefix
is either reusable (cached_len == insert_len, handle locks it, row re-pointed) or absent; the window
rule in match_prefix produces a third state -- committed into the tree but refused as a prefix -- that
the finish-time insert then treats as someone else's slots. Every variation in the table above follows:
without a tombstone on B's path (distinct, two_short, sequential: A's head is revived by Branch 1
once A's lock is gone) match_prefix starts at +inf and returns the full prefix; with
max_tokens 1 there is no unfinished commit; with --max-running-requests 1 B never commits beside A.

The same defect's other face

Under generic concurrent traffic (the fuzzer's first 10-iteration run, 252 requests whose prompts share marker heads) the idle check fails on the FULL pool first, CacheManager integrity check failed: free_pages(7868) + cache_pages(25187) != num_pages(32768) (+287 pages): the pages step 5 returned to free_slots are still referenced by the tree, so the next requests are handed pages the tree owns. That is the duplicate-page count #204 measured (free_slots holding 2125 pages twice).

Not inherited from SGLang

FreeToken's SWA radix cache is modelled on SGLang's (the tombstone / revive branches cite sglang's _insert_helper), so the same two-request sequence was run against stock SGLang 0.5.21 on the same card and model (gpt-oss-20b, RTX 4090; 12 arms, one fresh server each, including a round with --chunked-prefill-size 256 where B was batched with A's last prefill chunk exactly as in FreeToken's log, plus --swa-full-tokens-ratio 0.1): 0 of 12 died, and SGLang's own idle pool-conservation check (invariant_checker._check_all_pools, which warns on a leak or a double free) logged nothing. SGLang's UnifiedRadixCache does not hit this with the same two-request sequence; the defect is in what FreeToken added -- the window rule in match_prefix returning cached_len 0 for a committed-but-short live run, and the finish-time insert freeing seg_kv for a node that already holds those slots. (Evidence: the SGLang session's campaign_logs/local/u79_shape_on_sglang_probe_20261007/README.md, commit f9207ca20 on faultline.)

How did you install FreeToken

Built from source

FreeToken version

ft --version: freetoken version 0.1.3 git rev-parse --short HEAD: 377a3bc (main, 2026-10-07)

OS

Arch Linux

OS details

Ubuntu 20.04.6 LTS, kernel 5.15.0-138-generic, Python 3.12.8.

GPU and driver

NVIDIA GeForce RTX 4090 24 GB (sm_89), driver 580.126.09, CUDA 13.0.

CPU and system RAM

AMD Ryzen Threadripper PRO 5955WX (16 cores / 32 threads), 440 GiB RAM

Checkpoint

openai/gpt-oss-20b (Hugging Face, revision 6cee5e81ee83917806bbde320786a8fb61efebee)

Command

Full log

Anything else

No response