use multithreaded zstd compression for big chunks, #9961

[ThomasWaldmann]

Fixes #9961.

libzstd can compress a single chunk with a thread pool, but borg never asked it to: zstd.compress() was called without options, so nb_workers stayed 0.

Setting nb_workers alone would not have helped either - zstd sizes its jobs from the window log, and that default swallows any chunk borg produces whole, so the workers never engage (measured at 0.97x, i.e. marginally slower). job_size has to be set explicitly. libzstd refuses a job below 512KiB, so we ask for exactly that: most jobs, most parallelism.

Why the 768KiB cutoff

Not because smaller chunks cannot be split - a 768KiB chunk is happily cut into 512KiB + 256KiB. The problem is that the split is uneven: every thread waits for the one full-size job, which costs about as much as compressing the whole chunk single-threaded, and thread overhead comes on top.

Measured on a 12-core machine (Apple M3 Pro), speedup vs single-threaded:

chunk	L1	L3	L6	L10
512K	0.98x	1.00x	1.01x	1.06x
544K	0.80x	0.84x	0.86x	0.91x
608K	0.93x	0.90x	0.99x	1.07x
640K	1.02x	1.02x	1.09x	1.11x
704K	1.10x	1.11x	1.20x	1.27x
768K	1.23x	1.21x	1.31x	1.38x

Break-even is ~640KiB. The cutoff is 768KiB (= 1.5 jobs) to keep margin, since the thread overhead is machine dependent.

This also answers the "job_size is probably level-dependent" question from #9961: it is not. Both the 512KiB job floor and the break-even are the same at levels 1, 3, 6 and 10 - they follow from libzstd's job size minimum, not from the window size.

Results

10GiB of compressible data in a single file, zstd,3, -e aes256-ocb -i sha256, median of 2:

	create	MB/s	cpu	repo size
single-threaded	55.0s	186	53.9s	3,005,680,422 B
this PR (12 workers)	33.8s	303	65.7s	3,012,993,516 B

1.63x on borg create. The compressor alone gets 2.61x for a 2MiB chunk and 4.16x for an 8MiB one; the end-to-end figure is lower because compression is only part of create.

Costs

• Compression ratio: +0.24% on that run (+0.05% for 1MiB chunks at zstd,3, +0.64% for 8MiB ones, more at higher levels which rely on longer match history). A job only sees its own data.
• CPU time: +22%. It buys wallclock with otherwise-idle cores.
• BORG_ZSTD_MT_WORKERS=1 turns it off; the default is os.cpu_count().

Points for review

• On by default. That is the "optimise for speed" choice, but it changes behaviour for every zstd user (+22% cpu, +0.24% archive size). Easy to flip to opt-in - it is the default in get_zstd_mt_workers().
• Compressed bytes now depend on the worker count. The same chunk compressed on machines with different core counts gives different (equally valid) output. Chunk ids are unaffected, they are computed on the plaintext before compression, but pack contents and therefore pack ids vary. Worth confirming nothing relies on byte-reproducible compression output.
• No test exercises the MT path. Correctness of multi-threaded zstd is the zstd project's to test, so there is no roundtrip test here - but that also means CI only ever runs the single-threaded branch, since the existing compressor tests use data far below 768KiB. A mistake in borg's options dict would not be caught. A single test compressing one 768KiB buffer would close that gap if you want it.
• The env var is read at compress time and cached, like BORG_BLAKE3_MT_THRESHOLD, so an invalid value is reported by borg's normal error handling rather than as an import traceback.
• Measured on one machine, one data shape, mostly zstd,3. A low-core machine will see much less.

Tests: compress_test.py 60 passed (incl. a new one for the env var), create_cmd_test.py clean. black and ruff clean.

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.70%. Comparing base (198c751) to head (a50eb6f).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #9962   +/-   ##
=======================================
  Coverage   85.70%   85.70%           
=======================================
  Files          95       95           
  Lines       16815    16815           
  Branches     2577     2577           
=======================================
  Hits        14411    14411           
  Misses       1668     1668           
  Partials      736      736           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.