Relay transfers uncached when no staging file can be opened - #68
Merged
Merged
Conversation
Both object paths called ObjectStore.OpenStaging unguarded, so a store that stopped accepting writes after the startup probe (a permissions change, a read-only remount, inode exhaustion, a stray file where the staging directory belongs) turned every pull miss and every push into a 500, even though upstream was healthy. A staging file that cannot be opened now costs a cold cache rather than a failed transfer. A download is streamed from upstream without storing it, and an upload is relayed to upstream without teeing it into the store. Each fallback logs StagingUnavailable (event 2012) and increments gitlfscache.staging_failures, so the degraded state stays visible. Fixes #57 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015u9u95HXjY2mDagnzGzxK2
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Fixes #57
What was wrong
Both object paths in
ObjectRouteHandlercalledstore.OpenStaging(...)with no error handling.OpenStagingrunsDirectory.CreateDirectoryandFileStream.New(..., FileMode.CreateNew, ...), and the startup writability probe only runs once.So if the store stopped accepting writes later, every pull miss and every push failed with a 500, even though upstream was healthy. Causes include a permissions change, a read-only remount, inode exhaustion, or a stray file at
{root}/{upstream}/staging.On a download, upstream's status and headers had already been copied to the response when the exception escaped.
Change
ObjectRouteHandler.TryOpenStagingcatchesIOException,UnauthorizedAccessExceptionandArgumentExceptionfromOpenStaging, which are the three the issue lists. On failure it:StagingUnavailable(event 2012, warning) with the exception, oid and upstream;gitlfscache.staging_failures, tagged by upstream;ReadTeeStreamwithStream.Nullas the sink, soRecordUploadstill reports the relayed byte count. Nothing is published.One interaction to be aware of: catching
ArgumentExceptionalso covers the invalid-upstream-key throw inOpenStaging. That is the symptom described in #56, where a dot in the upstream key gives a 500 on every object transfer. After this change those transfers succeed uncached, and each one logs a warning and counts a staging failure, instead of returning 500. #56 still needs its own fix so those objects can be cached.Tests
Two new integration tests in
ProxyFlowTests. Each puts a file where{root}/github/stagingbelongs, following the repro in the issue:Download_StagingCannotBeOpened_IsServedFromUpstreamWithoutCaching: the client gets 200 with the full object, upstream is fetched once, nothing is stored, and the metric is 1.Upload_StagingCannotBeOpened_IsStillRelayedUpstream: the client gets 200, upstream receives the bytes, nothing is stored, and the metric is 1.Checked both directions:
TryOpenStagingreduced to a barestore.OpenStaging(upstream), both tests fail.🤖 Generated with Claude Code
https://claude.ai/code/session_015u9u95HXjY2mDagnzGzxK2
Generated by Claude Code