fix(huggingface-transformers): close HFT concurrency race (refcount TOCTOU + AbortController key collision)

[sroussey]

Summary

Two high-severity concurrency bugs surfaced during the HFT pipeline hardening in #634. Both are latent under enough parallel load that they were not caught by the eviction tests added there.

H1 — Refcount-after-lookup TOCTOU (19 run-fn files)

Every HFT run-fn except HFT_Chat.ts awaited getPipeline() first and then acquired the in-use refcount:

const pipeline = await getPipeline(model!, emit, {}, signal);
await withHftPipelineInUse(getPipelineCacheKey(model!), async () => { ... });

Between the two awaits the refcount for the just-returned entry is zero. A sibling loader that pushes the cache over MAX_CACHED_PIPELINES and runs enforcePipelineCacheCap can pick that entry as an LRU-non-in-use candidate and call model.dispose(). The subsequent inference then invokes a disposed ONNX/WASM session, crashing the worker or (worse) reading whatever memory the WASM runtime handed back.

HFT_Chat.ts already had the correct order (refcount, then load) — used as the model.

Fix: move getPipeline inside withHftPipelineInUse in all 19 run-fns. No signature changes. Zero-shot / regular branches in HFT_TextClassification, HFT_ImageClassification, HFT_ObjectDetection are each wrapped independently. Logger pipeline ready lines in HFT_TextEmbedding and HFT_ImageEmbedding moved inside the callback so the log stays truthful.

H2 — modelAbortControllers key collision (HFT_Pipeline.ts)

modelAbortControllers was a Map<string, AbortController> keyed on bare model_path. Two concurrent getPipeline calls for the same model but different pipeline / dtype / device combinations produce different cacheKeys — the load dedupe in pipelineLoadPromises misses, so both loads run to completion. Both called .set(modelPath, ...), which overwrote the first controller. The first caller's abort() then fired on an orphaned controller and never reached the in-flight fetch, so cancelling a load did nothing.

Compounding the problem, abortableFetch used a substring pathname match (pathname.includes(/${modelPath}/)), so Xenova/bge-reranker-base collided with Xenova/bge-reranker-base-v2 — aborting one would cancel the other's fetches.

Fix:

• Map<string, Set<AbortController>> keyed by model_path; concurrent loads coexist under one key.
• abortableFetch collects every matching controller's signal and combines them via AbortSignal.any(...) (falls back to the first match when unavailable).
• New pathMatchesModelSegment(pathname, modelPath) does segment-based matching so -v2 does not match base.
• removeModelController(modelPath, controller) helper replaces both .delete(modelPath) sites (early-abort branch + finally), and deletes the map entry when its set becomes empty so abortableFetch doesn't walk stale keys.

Test plan

• packages/test/src/test/ai-provider-hft/HFT_Pipeline.race.test.ts — H1 regression:
  • a cache-hit pipeline is not disposed while a caller awaits between lookup and refcount, and
  • concurrent inference across N run-fns never disposes an in-use pipeline (N = cap + 2 seeded, two pinned in-use).
• packages/test/src/test/ai-provider-hft/HFT_AbortControllers.test.ts — H2 regression:
  • two concurrent loads for the same model_path do not orphan an AbortController (set size 2 → 1 → 0 with key cleanup), and
  • pathMatchesModelSegment table: rejects substring collisions (Xenova/bge-reranker-base-v2 vs Xenova/bge-reranker-base, foobert vs bert) and accepts full-segment matches.
• bun scripts/test.ts provider-hft vitest — all 59 vitest tests pass (5 skipped are pre-existing integration skips).
• bun run build:types — all 40 TS build tasks succeed.

modelAbortControllers and pathMatchesModelSegment are exported with @internal JSDoc from HFT_Pipeline.ts for the test to reach (mirroring the existing pipelines export convention).

Context

Follows up #634, which introduced the bounded-LRU pipeline cache + refcount system that these bugs sit inside. The refcount plumbing (withHftPipelineInUse, isHftPipelineInUse, removeCachedPipeline in-use skip) was already in place — only the ordering at the call sites was wrong for H1, and the map value type / key comparator was wrong for H2.

Commits

• fix(huggingface-transformers): hold pipeline refcount before getPipeline lookup (19 run-fns)
• fix(huggingface-transformers): key model AbortControllers per-controller to survive concurrent loads (HFT_Pipeline.ts)
• test(huggingface-transformers): regressions for H1 (refcount race) and H2 (AbortController key)

Generated with Claude Code

https://claude.ai/code/session_01YY8Dv6snTc9maxiV5sbHwF

---

Generated by Claude Code

[github-actions]

Coverage Report

Status	Category	Percentage	Covered / Total
🔵	Lines	62.94%	27145 / 43128
🔵	Statements	62.78%	28112 / 44775
🔵	Functions	63.3%	5160 / 8151
🔵	Branches	51.56%	13353 / 25893

File Coverage

File	Stmts	Branches	Functions	Lines	Uncovered Lines
Changed Files
providers/huggingface-transformers/src/ai/common/HFT_BackgroundRemoval.ts	8.33%	0%	0%	8.33%	19-25, 33-46
providers/huggingface-transformers/src/ai/common/HFT_ImageClassification.ts	5%	0%	0%	5%	30-75
providers/huggingface-transformers/src/ai/common/HFT_ImageEmbedding.ts	5%	0%	0%	5%	23-58
providers/huggingface-transformers/src/ai/common/HFT_ImageSegmentation.ts	3.12%	0%	0%	3.22%	20-59, 67-90
providers/huggingface-transformers/src/ai/common/HFT_ImageToText.ts	14.28%	0%	0%	14.28%	19-35
providers/huggingface-transformers/src/ai/common/HFT_ObjectDetection.ts	5.88%	0%	0%	5.88%	30-74
providers/huggingface-transformers/src/ai/common/HFT_Pipeline.ts	5.26%	2.85%	4.76%	5.26%	26-47, 72-233, 251, 254-267, 283-309, 342-375, 416-568, 580-742
providers/huggingface-transformers/src/ai/common/HFT_StructuredGeneration.ts	1.81%	0%	0%	1.85%	24-63, 72-150
providers/huggingface-transformers/src/ai/common/HFT_TextClassification.ts	6.66%	0%	0%	6.66%	24-76
providers/huggingface-transformers/src/ai/common/HFT_TextEmbedding.ts	3.57%	0%	0%	3.57%	27-101
providers/huggingface-transformers/src/ai/common/HFT_TextFillMask.ts	16.66%	100%	0%	16.66%	17-28
providers/huggingface-transformers/src/ai/common/HFT_TextGeneration.ts	4.16%	0%	0%	4.34%	30-76
providers/huggingface-transformers/src/ai/common/HFT_TextLanguageDetection.ts	16.66%	0%	0%	16.66%	21-38
providers/huggingface-transformers/src/ai/common/HFT_TextNamedEntityRecognition.ts	16.66%	100%	0%	16.66%	21-39
providers/huggingface-transformers/src/ai/common/HFT_TextQuestionAnswer.ts	7.69%	0%	0%	8.33%	30-57
providers/huggingface-transformers/src/ai/common/HFT_TextReranker.ts	2.7%	0%	0%	2.94%	21-80, 104-138
providers/huggingface-transformers/src/ai/common/HFT_TextRewriter.ts	7.69%	0%	0%	8.33%	23-42
providers/huggingface-transformers/src/ai/common/HFT_TextSummary.ts	8.33%	0%	0%	9.09%	23-40
providers/huggingface-transformers/src/ai/common/HFT_TextTranslation.ts	8.33%	0%	0%	9.09%	27-46
providers/huggingface-transformers/src/ai/common/HFT_ToolCalling.ts	0.72%	0%	0%	0.78%	45-296, 308-407

Generated in workflow #2727 for commit d8569cd by the Vitest Coverage Report Action