When a diff times out, every coalesced follower re-runs the same diff-tree: 10 concurrent requests start 10 processes, and each follower waits up to 2×DiffTimeout

[matt-edmondson]

What's wrong

In DiffCache.GetAsync (GitBranchStateCache/Diffs/DiffCache.cs:59-81), a follower waits for the leader with WaitForLeaderAsync(DiffTimeout). If the leader's result is not true and published, the follower calls ComputeAsync itself. The leader reports only a bool, ticket.Complete(outcome.Succeeded), and a timeout counts as false. So a follower cannot tell "the leader ran this exact diff and it timed out" from "the leader was abandoned". Failures are also never cached, so every client heartbeat repeats the whole cycle.

MirrorFetcher makes the opposite decision for the same situation (Mirrors/MirrorFetcher.cs:111-116): "Trying again here would multiply exactly the load the coalescer exists to prevent."

Failure scenario

A pathological merge-base..tip diff (a huge vendor drop or a generated-file rewrite) runs past DiffTimeout (default 2 min). N clients are asking about that branch:

1. The leader times out at 2 min.
2. All N followers, having waited up to 2 min, each start their own git diff-tree on exactly the diff already known to be too slow. That is N more processes, each running up to another 2 min.
3. Every follower's response takes about 4 min and still ends in timeout.
4. On the next heartbeat (30 s) the cache is still empty, so the stampede repeats.

Reproduction

With the repo's test harness (ServiceFixture + ScriptedGit with DiffTimesOut = true and a 1 s delay on diff-tree), 10 concurrent requests for the same uncached diff produced 10 diff-tree invocations. All returned 200 with the branch marked partial/timeout.

Suggested fix

• Carry the outcome kind through IWorkTicket, e.g. Succeeded / TimedOut / Abandoned, instead of a bool.
• When the leader timed out, followers return DiffOutcome.Timeout() without recomputing. They recompute only when the leader was abandoned (cancelled, or the process died).
• Optionally cache a timeout per DiffKey for a short period, e.g. one heartbeat interval, so repeated heartbeats don't re-run a diff known to be too slow.

Acceptance criteria

• With a diff that times out, N concurrent requests produce exactly one diff-tree invocation, and every follower gets a timeout outcome within about DiffTimeout.
• A leader that is cancelled or abandoned still lets a follower compute, which keeps the existing fallback behaviour.

Related, not duplicates: #50 (a non-UTF-8 path stalls a diff) and #44 (clone leader cancellation, on the mirror side).

[matt-edmondson]

Triage

• Category: Bug
• Priority: Medium. It only happens with a pathological diff that exceeds DiffTimeout, but then it multiplies load N-fold and repeats on every heartbeat, which is what the coalescer is supposed to prevent. Responses still end in a timeout, so no data is wrong.
• Area / suggested assignment: diff coalescing: DiffCache.GetAsync and IWorkTicket (outcome kind rather than bool). Optionally add short-lived negative caching per DiffKey.
• Duplicates: none. It is related to A diff containing one non-UTF-8 path stalls for the full DiffTimeout and is reported as a timeout instead of "not valid UTF-8" #50 (non-UTF-8 path stalls a diff) and When the request cloning a repository disconnects, the clone is killed and every waiting request gets 503, so a clone longer than the client timeout never finishes #44 (clone leader cancellation). MirrorFetcher already follows the policy this issue asks for.
• In progress: no matching open PR. The open PRs Make --allow replace an upstream's configured allow-list #60 and Sweep staging directories left by clones that never finished #61 cover the allow-list and staging-directory cleanup, which are unrelated.

---

Generated by Claude Code