grab: transfer every download as a series of byte ranges - #114
Open
cavaliercoder wants to merge 9 commits into
Open
grab: transfer every download as a series of byte ranges#114cavaliercoder wants to merge 9 commits into
cavaliercoder wants to merge 9 commits into
Conversation
Adds Request.RangeSize and Request.Concurrency. Setting RangeSize splits
a download into ranges of that size, fetched with separate Range
requests; Concurrency bounds how many are in flight.
There is one transfer implementation rather than two. A download that is
not split is the single range covering whatever remains, and a fresh
download of a file of unknown length is that range requested with no
Range header at all - byte for byte the request grab has always made.
The tests pin those headers, because building every transfer out of
ranges must not change what an unsplit one puts on the wire.
Ranges are dispatched in ascending order so the destination fills from
the front, and each is written at its offset with WriteAt. The
destination is therefore never opened O_APPEND, which on some systems
forces every write to the end of the file whatever offset it was given.
Request.NoStore writes to an in-memory WriterAt so it takes the same
path as everything else.
A split transfer writes its progress to a checkpoint file beside the
destination, so an interrupted one resumes without refetching. The
record carries the URL, size, range size and the remote file's
validators, and is discarded unless they all still match - a stronger
guarantee than an unsplit resume can make, since that has no choice but
to assume the remote file is unchanged. Workers report partial progress
as they write, so what an interruption costs is bounded by the
checkpoint interval rather than by RangeSize. The destination is flushed
before the checkpoint naming it is renamed into place, so a checkpoint
can never claim data the filesystem has not committed.
The size of the remote file is read from Content-Range where the server
offers it rather than inferred, which also lets a range be checked
against the file it came from: a server reporting a different total part
way through has given us a different file, and one answering a Range
request with the whole file has given us a response starting somewhere
other than where we asked.
Supporting changes:
- grabtest serves byte ranges, Content-Range and validators, records
the ranges it served, and rejects the forms grab never sends
- benchmarks for what a transfer costs and what tuning it is worth,
split into `make bench` for regressions and `make bench-network` for
the shape of the trade-offs
- the default client is built on http.DefaultTransport, which it was
not before, so it now has connection, TLS handshake and idle
timeouts at all; MaxIdleConnsPerHost is raised since a split
transfer makes many requests to one host
- cmd/grab gains -range-size, -concurrency, -http1, -o and -batch, and
reports each file's rate and negotiated protocol
- docs/ describes the architecture and its invariants, how to reason
about range size and concurrency, and how to test and benchmark,
with the numbers measured against real mirrors from two clients
Based on the design and prototype in #102, which implemented this as a
fixed count of chunks rather than a fixed chunk size, and which had no
way to resume a transfer it interrupted.
refs #86
Co-authored-by: Justin Israel <justinisrael@gmail.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A split transfer writes ranges at their offset, so an interrupted one can leave a file that is already full length with parts of it never written. validateLocal only accounted for this when the transfer that found the file would itself split, so a later transfer with no RangeSize resumed from the file's length - and where the file was already full length, skipped the transfer altogether and reported success on a file with a hole in it. The checkpoint beside the file is what marks it as written out of order, whatever the transfer that finds it is configured to do. Treat its presence as authoritative in both paths: an unsplit transfer starts over rather than resuming by length, and discards the checkpoint it will not maintain. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Setting RangeSize made every split transfer record its progress, and flushing the destination before each record is what holds a split transfer to the rate the device accepts data rather than the rate the network delivers it. Not every caller wants to pay that: a download that is cheap to repeat would rather have the bandwidth. Durable makes it a choice. Set it and the transfer records what it has written about once a second, as before. Leave it unset and the transfer never flushes, and an interruption costs everything it had written. The checkpoint file itself is still written for every split transfer, once, before any of the destination is. Its presence rather than its contents is what marks the file as written out of order, and without that marker nothing distinguishes a file abandoned mid-split from a resumable or a finished one - a later transfer would resume from a length that means nothing, or take a file that reached full length with holes in it for a complete one. So recording no progress costs a single write at the start of the transfer, while recording it costs a flush per second. BenchmarkTransfer gains durable as an axis, since the gap between the two is now the thing worth watching. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Durable promised that Response.Err waits for the data, but only a split transfer flushed. An unsplit one had no checkpointer to do it, and a split one skipped its final flush on success, since the checkpoint it would have written was about to be deleted anyway. Both left the last of the transfer for the operating system to write out afterwards. Give a transfer that is not split a flusher, which does on the same interval what the checkpointer does around each record. Flushing throughout rather than once at the end is what keeps the reported rate honest: a single flush on completion would let a transfer run at the speed of memory and then stall, at 100% complete, for as long as the device needed to catch up. And have the checkpointer flush whenever it stops, whether or not it writes a final record, so a durable transfer that finished has its last second of data on the disk before Err returns. The checksum needs no change. It is computed by re-reading the destination rather than from the bytes as they passed through, so under Durable it already reads what has been flushed, and a write that failed is caught before the checksum rather than after it. That read may still be served from the page cache, which the docs now say rather than claim the medium was verified. BenchmarkTransfer now covers unsplit transfers with and without Durable: on an SSD with an 8 MiB file, 2505 against 1004 MB/s. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
hasCheckpoint appended the checkpoint suffix to the destination file name without checking there was one, so a transfer whose destination is not yet known asked about ".grab" relative to the process working directory. planTransfer acts on the answer by removing that file, so an unrelated file of that name was deleted. Reaching it needs nothing unusual: a destination directory leaves Response.Filename empty until the name is resolved from the response, and a server that rejects HEAD - the case the HEAD fallback exists for - reaches planTransfer before that happens. A transfer with no destination has no checkpoint. Say so in hasCheckpoint, where the empty name becomes a path, so that every caller that reads or removes one is covered rather than just this caller. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
planSplitTransfer returned checksumFile as soon as the checkpoint left nothing missing, three lines before it assigns Response.checkpoint. The checkpoint is removed by copyFile, which that path never reaches, so a transfer that found its work already done left the record behind for good. The stale file then marks a finished download as written out of order: every later transfer of that path takes the checkpoint branch, and an unsplit one discards the record and downloads the complete file again. Reaching it takes a durable split transfer cancelled as its last range lands, which records every range as complete before it stops. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
store flushed the destination and the temporary checkpoint, then renamed the checkpoint into place. The rename is atomic, but the directory entry it creates is not durable until the directory itself is flushed, so a crash could leave a destination whose data reached the disk beside a checkpoint that no longer existed. That is the one state the checkpoint is there to prevent. Its presence, not its contents, is what marks a file as written out of order; without it a later transfer resumes from a length that means nothing, or takes a file that reached full length with holes in it for a complete one. Only the first store needs the flush. Losing a later checkpoint reverts to an earlier one, which understates what was written and costs a refetch, so the once-per-second store is unchanged and a transfer pays one extra flush in total. Windows has no equivalent: FlushFileBuffers rejects the handle that opening a directory yields, so syncDir is a documented no-op there and the durability of a rename is left to the file system. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
byteSize.Set multiplied the parsed number by its unit without checking the result fits an int64, so -range-size 10000000000GB wrapped to a negative size. The check for a negative value runs before the multiply and does not catch it. A negative RangeSize does not split the transfer, so the download ran as a single request while reporting no error and appearing to be configured for ranges. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
RangeSize and Concurrency both have guards for values that make no sense - a Concurrency below one runs a single worker, and a RangeSize that is zero, negative or no smaller than the file leaves the transfer unsplit - but nothing held them in place. Cover them, adapting the negative chunk count from #102. Its separate RangeRequestMinSize has no equivalent here and needs none: RangeSize is itself the floor, as a file no larger than one range is never split. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements HTTP Range support, redesigned from #102 by @justinfx. Refs #86.
The ranged path is the only path: an unsplit download is the degenerate case
of one range covering the file, owned by one goroutine. That removes branching
rather than adding it.
API
Three fields on
Request:RangeSize int64Concurrency intDurable boolResponse.Errdoes not report success until the data is on the disk. Where the transfer is split, also record which ranges are complete so an interruption resumes.Durableis the axis that decides what a split transfer costs. Without it thetransfer never flushes and an interruption loses everything it wrote; with it
the file is flushed about once a second, which holds the transfer to the rate
the device accepts data rather than the rate the network delivers it. Flushing
throughout rather than once at the end keeps the reported rate honest — a
single flush on completion would run at the speed of memory and then stall at
100% while the device caught up.
A split transfer always writes its checkpoint file once, before any of the
destination. Its presence rather than its contents is what marks the file as
written out of order; without that marker nothing distinguishes a file
abandoned mid-split from a resumable or a finished one.
Commits
60fa7f524ca6638fc5e86Request.Durable9aa5afea856c5e.grabin the working directory for a checkpointa6cb85fb1545fd97fcd0dThe last four came out of a review pass over the first four; each has a
regression test verified to fail against the unfixed code.
Measured
Real downloads from public mirrors, checksums verified:
Over HTTP/2 the ranges multiplex onto one connection and share its congestion
window, so concurrency does not multiply bandwidth — RFC 9113 §9.1 asks clients
not to open a second connection. Past a point concurrency is all downside and a
little bit rude: 64 workers earned a
503from one mirror.Checkpointed throughput tracks the disk's sync rate about 1:1 with no library
overhead — gp3 at 125 MB/s provisioned (135 MB/s measured by
dd) gave132–177 MB/s; at 1000 (378 MB/s) it gave 347–399.
docs/range-requests.mdcarries the reasoning and the full results.Notes
NewClientnow setsMaxIdleConnsPerHost = MaxIdleConns. At the default of two, concurrent ranges overflow the idle pool and perfectly good connections get closed and re-handshaked.TCP_NODELAYchange is not carried over: grab is almost purely a receiver, and itsconn.(*net.TCPConn)assertion panics on any non-TCP network.🤖 Generated with Claude Code