Skip to content

Reposition the temp file when seeking to offset zero - #363

Merged
funkyshu merged 7 commits into
C2FO:mainfrom
youdie006:os-seek-start-overwrite
Sep 22, 2026
Merged

funkyshu merged 7 commits into
C2FO:mainfrom
youdie006:os-seek-start-overwrite

Conversation

@youdie006

Copy link
Copy Markdown
Contributor

On the os backend, Seek(0, io.SeekStart) followed by Write appends instead of overwriting from
the start. mem gets it right.

os   Seek(0,0)+Write("HELLO") -> "hello worldHELLO"   want "HELLO world"   FAIL
os   Seek(6,0)+Write("there") -> "hello there"        want "hello there"   PASS

mem  Seek(0,0)+Write("HELLO") -> "HELLO world"
mem  Seek(6,0)+Write("there") -> "hello there"

Offset 6 is the case the conformance suite already covers, which is why this has gone unnoticed.

Cause

copyToLocalTempReader buffers writes into a temp file. When the file exists and Seek/Read ran
first, it copies the original in with io.Copy — leaving the temp file's cursor at EOF — and then
repositions, but only when the cursor is past the start (backend/os/file.go:512):

if f.cursorPos > 0 {
    if _, err := tmpFile.Seek(f.cursorPos, 0); err != nil {

For cursorPos == 0 the reposition is skipped, so the write lands at EOF.

Change

Drop the > 0 guard — the seek is needed for offset zero precisely because io.Copy moved the
cursor.

Tests

Two, both red with only backend/os/file.go reverted:

  • A case in the shared DefaultIOTestCases, so mem defines the expected value rather than me
    asserting it. Only os fails:
    --- FAIL: TestIOConformance/Seek_to_start,_Write,_Close,_file_exists
        Seek to start, Write, Close, file exists: expected results that text but got some textthat
    
    mem passes it before and after.
  • A table-driven case in the osFileTest suite covering offset 0 and offset 6, since the
    conformance run is behind the vfsintegration tag and would not count toward the coverage gate.

go test ./backend/os/ ./backend/mem/ ./backend/testsuite/ is green, as is
go test -tags vfsintegration -run TestIOConformance ./backend/os/ ./backend/mem/. gofmt -l lists
none of the changed files. CHANGELOG entry added under Unreleased.

Scope

sftp and azure delegate Seek straight through and may share this shape, but I could not
exercise them without credentials, so this PR is limited to os.

Separately and not included: os, azure and sftp do not validate whence (Linux accepts 3
for SEEK_DATA) while mem, s3, gs and ftp return vfs.ErrSeekInvalidWhence
(errors.go:25). Happy to send that separately if you want it.


Disclosure: prepared with AI assistance; I verified the backend comparison and both red/green runs
myself.

copyToLocalTempReader copies the existing file in with io.Copy, which
leaves the temp file at EOF, then seeks back only when cursorPos > 0. So
Seek(0, io.SeekStart) followed by Write appended instead of overwriting
from the start. The mem backend already behaves correctly.
The minio/minio repository has been removed from Docker Hub, so the
testcontainers conformance CI job failed to provision the s3 backend
container:

  pull access denied for minio/minio, repository does not exist or
  may require 'docker login': denied: requested access to the
  resource is denied

quay.io/minio/minio publishes the identical pinned tag
(RELEASE.2025-09-07T16-13-09Z), so this is a drop-in replacement.

Verified locally with:
  TESTCONTAINERS_RYUK_DISABLED=true go test -tags=vfsintegration -race -timeout 30m ./...
The minio container now provisions correctly.
…ntent

initWriters gated the download-existing-content step on cursorPos != 0,
so Seek(0, io.SeekStart) followed by Write skipped downloading the
existing object entirely. On Close, the object was replaced with only
the newly written bytes, silently discarding everything else -
matching os's bug fixed above, but via a different mechanism (the
existing-object download is skipped outright here, rather than left at
the wrong cursor position after a copy).

The gate now checks whether Seek or Read was called before this first
Write, not whether the resulting cursor happens to be zero - mirroring
the os backend's fix and the vfs.File contract (Write-first always
overwrites; Seek/Read-first preserves untouched content).

Fixes C2FO#365.

Added regression tests for both backends:
- s3: TestSeekToStartThenWriteDownloadsExistingContent, mirroring the
  existing TestSeekThenWriteDownloadsExistingContent (offset 6) but at
  offset 0.
- gs: TestSeekThenWrite, mirroring the os backend's table-driven test
  (offset 0 and offset 6) using the in-process fakestorage server.

Both new tests were verified to fail against the pre-fix code (exact
symptom: object ends up containing only the newly-written bytes) and
pass with the fix.

Also fixes a test-hygiene issue found in review of the os fix: the
existing os TestSeekThenWrite reset a shared fixture file
(test_files/test.txt) with a non-deferred call at the end of each
subtest, so a failed assertion earlier in a subtest would leave that
shared file mutated for later subtests/suite tests. Switched to a
fixture-local file per subtest with a deferred cleanup instead.

Verified locally end-to-end with the testcontainers integration suite
(TESTCONTAINERS_RYUK_DISABLED=true go test -tags=vfsintegration
./...): the shared 'Seek to start, Write, Close, file exists' IO case
now passes for os, mem, sftp, azure, gs, and s3 (both SSE variants).

Note: the same shared IO conformance case also fails for ftp, but
ftp's Write path (DataConn/OpenWrite) is a structurally different
mechanism with no cursorPos/initWriters concept, so that is a separate
bug outside the scope of C2FO#365 and this fix - flagging for a follow-up
issue rather than addressing here.
@c2fo-cibot c2fo-cibot Bot added size/L Denotes a PR that changes 100-499 lines and removed size/M Denotes a PR that changes 30-99 lines labels Sep 19, 2026
The previous commit's per-subtest fixture rework built the fixture
path via path.Join(s.tmploc.Path(), ...) - a vfs-style path (e.g.
/C:/Temp/...) - and passed it straight to the stdlib os.WriteFile to
seed the fixture, and to fileSystem.NewFile to open it. vfs.File
operations translate that leading-slash-before-drive-letter form to a
native path internally (toNativeOSPath), but raw os.* calls do not, so
os.WriteFile failed on Windows CI:

  open /C:/Temp/.../test_files/seek_then_write_rewind.txt: The
  filename, directory name, or volume label syntax is incorrect.

Rebuilt the fixture entirely through vfs.File (Location.NewFile +
File.Write/Seek/Close), never touching the raw os package with a vfs
path, so path translation stays consistent across platforms. Verified
locally (macOS); Windows behavior inferred from reading toNativeOSPath
directly since no Windows machine is available here - will confirm
via the next CI run on this PR.
FTP writes streamed new bytes directly to the server via STOR with a
REST offset, relying entirely on the server to preserve any existing
content the write doesn't cover. Against a live vsftpd server (the
testcontainers module), this held for writes that reach or extend past
the original EOF, but Seek(0, io.SeekStart) followed by a shorter
Write replaced the whole file with only the newly written bytes,
losing everything after them - the ftp analog of C2FO#365 (fixed for
s3/gs), surfaced by the same shared conformance case added in the
first commit here.

Fix: when Seek or Read happens before the first Write on an existing
file, Write now downloads the full existing content into a local temp
file first, patches it at the current offset, and defers the actual
upload to Close, which uploads the merged result in a single plain
STOR - rather than streaming straight through STOR+REST and trusting
the server to keep the rest of the file intact. Fresh writes (no prior
Seek/Read) keep the existing direct-streaming path unchanged.

Read and Seek redirect to the buffered temp file once a merge is in
progress, and Close always resets the seek/read/write tracking flags.

Added TestSeekToStartThenWritePreservesExistingContent, verified to
fail against the pre-fix code (mock expectations unmet - Write went
straight to the live dataconn instead of buffering) and pass with the
fix.

Verified end-to-end with the real local testcontainers integration
suite (TESTCONTAINERS_RYUK_DISABLED=true go test -tags=vfsintegration
./...): the shared 'Seek to start, Write, Close, file exists' IO case
now passes for os, mem, sftp, azure, gs, s3 (both SSE variants), and
ftp.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

The FTP regression test has an incorrect close expectation, and deleting a file with an active merge buffer can recreate it on a later Close.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 2 High severity · 2 Low severity

Open (4)
What changed in this PR

Fixes offset-zero writes after seeking, extending the behavior across local and remote backends and updating integration-test infrastructure.

Changes:

  • Repositions OS temp files at offset zero.
  • Preserves untouched content for S3, GS, and FTP writes after seek/read.
  • Adds regression tests, conformance coverage, changelog entries, and updates the MinIO image registry.
File Description
backend/​os/​file.go Fixes temp-file positioning.
backend/​os/​file_test.go Tests offset-zero and middle seeks.
backend/​s3/​file.go Preserves existing objects after seek/read.
backend/​s3/​file_test.go Adds S3 regression coverage.
backend/​gs/​file.go Preserves existing objects after seek/read.
backend/​gs/​file_test.go Adds GCS regression coverage.
backend/​ftp/​file.go Adds local merge buffering for FTP writes.
backend/​ftp/​file_test.go Adds FTP regression coverage.
backend/​testsuite/​io_conformance.go Adds shared offset-zero conformance coverage.
testcontainers/​containers_test.go Switches MinIO to Quay.
CHANGELOG.md Documents the fixes and image update.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread backend/ftp/file.go
Comment thread backend/ftp/file_test.go
Comment thread backend/s3/file.go
Comment thread testcontainers/containers_test.go
…lexity

- Delete now calls Close() first so a write-after-seek/read merge buffered
  in f.tempFile is flushed and cleaned up before the remote file is removed.
  Previously, deleting mid-write left the local temp file orphaned on disk,
  and a later Close call would still upload the buffered content,
  resurrecting the file Delete had just removed. Mirrors the existing
  s3/gs Delete behavior.
- Extracted the tempFile-repositioning branch of Seek into a seekTempFile
  helper to bring Seek's cyclomatic complexity back under the golangci-lint
  threshold (16 -> under 15). only-new-issues in CI didn't catch this because
  the reported line is the unchanged function signature.
- Added regression tests: TestDeleteFlushesBufferedWriteBeforeRemoving,
  TestReadThenWritePreservesExistingContent (Read, not just Seek, before
  Write triggers the same preserve-content merge), and
  TestSeekWithinBufferedWrite (covers seekTempFile's success and error
  paths, previously untested at 0% coverage).

Co-authored-by: Cursor <cursoragent@cursor.com>
@c2fo-cibot c2fo-cibot Bot added size/XL Denotes a PR that changes 500-999 lines and removed size/L Denotes a PR that changes 100-499 lines labels Sep 20, 2026
…ding close comment

- testcontainers/README.md and doc.go still documented the s3 backend's
  conformance image as minio/minio after containers_test.go was pinned to
  quay.io/minio/minio (minio/minio no longer exists on Docker Hub).
- The comment above Write's defensive dataconn close in ftp/file.go claimed
  it was closing the stale Seek-opened dataconn, but Exists' own SingleOp
  DataConn call already closes that one as a side effect of its mode
  mismatch handling; this close is actually a no-op on the fresh SingleOp
  connection Exists leaves behind. Corrected the comment to describe what
  actually happens; no behavior change.

Co-authored-by: Cursor <cursoragent@cursor.com>
@funkyshu
funkyshu merged commit cb748ec into C2FO:main Sep 22, 2026
44 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/XL Denotes a PR that changes 500-999 lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants