Skip to content

When the request cloning a repository disconnects, the clone is killed and every waiting request gets 503, so a clone longer than the client timeout never finishes #44

Description

@matt-edmondson

What's wrong

BranchStateHandler (GitBranchStateCache/Endpoints/BranchStateHandler.cs:91) takes context.RequestAborted and passes it to fetcher.EnsureCurrentAsync (lines 156 and 362). That token then goes all the way down to the git process:

  • MirrorFetcher.EnsureCurrentAsync has the leader run CloneAsync / FetchAsync with the leader request's own cancellation token.
  • GitRunner.RunAsync (Git/GitRunner.cs:84-91) kills the git process tree and rethrows OperationCanceledException when that token fires.
  • The leader's ticket is disposed without Complete(true), so FollowAsync (Mirrors/MirrorFetcher.cs:~100-120) sees a failed leader. With no mirror on disk yet, it returns Unavailable("…the request that was creating one did not finish.") and does not try again.

So one client hanging up cancels shared work that every coalesced request is waiting on.

Failure scenario

  1. Request A (leader) starts git clone for a repository that isn't mirrored yet. The repository is large, so the clone takes longer than A's client HTTP timeout.
  2. Request B (follower) arrives for the same repository and waits on the flight.
  3. A's client times out and disconnects. The clone is killed and the staging directory is discarded.
  4. B's client is still connected, but it gets 503 no-mirror.
  5. The next request becomes leader, starts the clone from zero, and hits the same timeout.

For any repository whose clone takes longer than the clients' timeout, the mirror is never created. The same thing on a fetch hands followers Stale instead of Current.

This was reproduced with a scratch test using a fake runner that behaves like GitRunner on cancellation (the existing GitRunnerTests.RunAsync_WhenTheRequestIsAbandoned_KillsTheTreeAndPropagatesTheCancellation pins that behaviour):

leader: cancelled (client went away)
follower (still connected, never cancelled): Unavailable / No mirror exists for this repository and the request that was creating one did not finish.
clone invocations: 1

Suggested fix

  • Run clone and fetch under a token owned by the service, linked to IHostApplicationLifetime.ApplicationStopping plus FetchTimeout, instead of the request's token.
  • Leader and followers each await the shared task with their own request token (.WaitAsync(requestToken)). A disconnecting caller then stops waiting, but the work continues.
  • Or, as a minimum, when the leader was only cancelled (not failed), let a still-connected follower take over as leader instead of returning Unavailable.

Acceptance criteria

  • A test where the leader's token is cancelled mid-clone while a follower waits ends with the clone completing and the follower getting Current.
  • The git process is still killed on host shutdown and after FetchTimeout.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingreadyFully specified; implement as written

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions