Skip to content

Relay transfers uncached when no staging file can be opened - #68

Merged
matt-edmondson merged 1 commit into
mainfrom
fix/staging-open-failure-bypasses-cache
Sep 28, 2026
Merged

matt-edmondson merged 1 commit into
mainfrom
fix/staging-open-failure-bypasses-cache

Conversation

@matt-edmondson

Copy link
Copy Markdown
Contributor

Fixes #57

What was wrong

Both object paths in ObjectRouteHandler called store.OpenStaging(...) with no error handling. OpenStaging runs Directory.CreateDirectory and FileStream.New(..., FileMode.CreateNew, ...), and the startup writability probe only runs once.

So if the store stopped accepting writes later, every pull miss and every push failed with a 500, even though upstream was healthy. Causes include a permissions change, a read-only remount, inode exhaustion, or a stray file at {root}/{upstream}/staging.

On a download, upstream's status and headers had already been copied to the response when the exception escaped.

Change

  • New ObjectRouteHandler.TryOpenStaging catches IOException, UnauthorizedAccessException and ArgumentException from OpenStaging, which are the three the issue lists. On failure it:
    • logs StagingUnavailable (event 2012, warning) with the exception, oid and upstream;
    • increments a new counter, gitlfscache.staging_failures, tagged by upstream;
    • returns null.
  • Download miss: when there is no staging file, the object is streamed from upstream to the client without storing it. This uses the same branch that range requests already take.
  • Upload: when there is no staging file, the body is still relayed to upstream through ReadTeeStream with Stream.Null as the sink, so RecordUpload still reports the relayed byte count. Nothing is published.

One interaction to be aware of: catching ArgumentException also covers the invalid-upstream-key throw in OpenStaging. That is the symptom described in #56, where a dot in the upstream key gives a 500 on every object transfer. After this change those transfers succeed uncached, and each one logs a warning and counts a staging failure, instead of returning 500. #56 still needs its own fix so those objects can be cached.

Tests

Two new integration tests in ProxyFlowTests. Each puts a file where {root}/github/staging belongs, following the repro in the issue:

  • Download_StagingCannotBeOpened_IsServedFromUpstreamWithoutCaching: the client gets 200 with the full object, upstream is fetched once, nothing is stored, and the metric is 1.
  • Upload_StagingCannotBeOpened_IsStillRelayedUpstream: the client gets 200, upstream receives the bytes, nothing is stored, and the metric is 1.

Checked both directions:

  • With TryOpenStaging reduced to a bare store.OpenStaging(upstream), both tests fail.
  • With the fix, the full suite passes: 332/332.

🤖 Generated with Claude Code

https://claude.ai/code/session_015u9u95HXjY2mDagnzGzxK2


Generated by Claude Code

Both object paths called ObjectStore.OpenStaging unguarded, so a store
that stopped accepting writes after the startup probe (a permissions
change, a read-only remount, inode exhaustion, a stray file where the
staging directory belongs) turned every pull miss and every push into a
500, even though upstream was healthy.

A staging file that cannot be opened now costs a cold cache rather than
a failed transfer. A download is streamed from upstream without storing
it, and an upload is relayed to upstream without teeing it into the
store. Each fallback logs StagingUnavailable (event 2012) and increments
gitlfscache.staging_failures, so the degraded state stays visible.

Fixes #57

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015u9u95HXjY2mDagnzGzxK2
@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

When the staging file can't be created, pulls and pushes fail with 500 instead of bypassing the cache

2 participants