Repository navigation
Archives from separate cargo invocations can bundle different tokio builds off one Cargo.lock; the link guard catches it but two agents hit it today in unrelated work #10671
Description
Activity
Third sighting today, in a third piece of unrelated work: the agent fixing #10437 hit it compiling
pgend-to-end —perry-ext-netandperry-ext-pgwrapper archives built in separate invocations, resolved the same way (one cargo invocation covering all of them).That agent attributed it to #507 and #7629. Having checked, it is neither — both are closed and describe different failure modes: #507 is LTO dead-stripping tokio's
CONTEXTstatics, #7629 is a deterministic SIGABRT in one gap test. Same neighbourhood (tokio inside ext-crate archives), different mechanism. This issue — archives from separate cargo invocations carrying different tokio builds off oneCargo.lock— remains distinct and open.Three independent hits in one day, each by someone who had read the warnings, is the argument for fixing it in the build system rather than documenting it harder.
Concrete reproduction of this failure mode, found while re-validating four packages after their original compile blockers landed (mongodb #10758, ethers #10757, node-fetch #10759, nodemailer #10798 — all closed).
All four need both the
http-client-pumpwell-known recipe (fornode:http/node:https) andperry-ext-net. WithPERRY_NO_AUTO_OPTIMIZE=1, none of them can link:Error: error: the wrapper archive(s) below bundle a DIFFERENT tokio compilation than the stdlib archive they would be linked with. target/perry-no-auto-http-pump/release/libperry_stdlib.a bundles tokio-1200aeae2e77614b libperry_ext_net.a bundles tokio-ef1329cc84fee118Root cause is exactly what the error message + this issue title describe:
crates/perry/src/commands/compile/optimized_libs/no_auto.rs'sbuild_http_client_pump_stdlibalways runscargo build --release -p perry-stdlib-static -p perry-ext-http --features perry-stdlib/external-http-client-pump— deliberately not including
perry-ext-net(per its own doc comment, to avoid forcing every no-auto program to carrylibperry_ext_http.a). That's fine when a program only needshttp-client-pump. It's not fine when the program also needsnet(or any other well-known routing) in the same binary:perry-ext-netcomes from a separately-built archive (either the prebuilttarget/release/libperry_ext_net.a, orperry's ownwell-known: routing net → rebuilding perry-ext-net with shared tokiostep), and feature-unification differences between that build and the narrowhttp-client-pumprecipe's build produce a different tokio content hash even at the same pinned version — so the link-time guard (correctly) refuses every time, deterministically, not just as a caching artifact. I confirmed this by deletingtarget/perry-no-auto-http-pumpand rebuilding from scratch (same hash1200aeae2e77614bboth times), and separately by manually pre-building a fully coherentstdlib+http+net+...set directly into that target dir — Perry's ownhttp-client-pump (no-auto)step re-invokes its narrower recipe anyway on the next compile and clobbers it back to the incoherent state.Workaround that does work: unset
PERRY_NO_AUTO_OPTIMIZEand let the fullauto-optimizecoherent rebuild run. Slower, but it unifies tokio across the whole feature set in one invocation and all four packages linked and ran (hitting separate, further-downstream bugs — #11043, #11044, #10809, #11045 — none of them tokio-coherence related).If a fix lands here, the direct check is: compile any of mongodb/ethers/node-fetch/nodemailer with
PERRY_NO_AUTO_OPTIMIZE=1and confirm it links (not just that the tokio hashes match — confirm the binary also runs past its first socket operation, since a silently-adopted stale archive would still "link").Re-tested after turnloop landed (
b77aba6343, tokio removed fromperry-runtime). This still reproduces, and the guard now names three tokio compilations instead of two.Tested on
origin/main=2f854511e9(v0.5.1640), compiling real-sourcemongodb@7.5.0withPERRY_NO_AUTO_OPTIMIZE=1. Fails after 13m43s with the same coherence guard, naming:libperry_stdlib.a bundles tokio-1200aeae libperry_ext_mongodb.a bundles tokio-d03f425a libperry_ext_net.a bundles tokio-2cbb3241So turnloop removed tokio from
perry-runtime, but tokio is still separately bundled per cargo invocation intoperry-stdliband eachperry-ext-*wrapper, andno_auto.rsstill never builds them in one invocation. The workaround remains necessary: leavePERRY_NO_AUTO_OPTIMIZEunset so auto-optimize produces a coherent set.Worth stating plainly because the natural assumption after turnloop is the opposite: tokio leaving the runtime does not fix this, and the count of distinct compilations went up, since
perry-ext-netis now a separate participant rather than sharing the runtime's copy.The guard itself is behaving correctly — it is refusing a binary that would abort at its first socket rather than shipping it. The defect is that
no_auto.rs's recipe cannot produce a coherent set for any package needing both thehttp-client-pumprecipe andperry-ext-net, which is at least mongodb, ethers, node-fetch and nodemailer.Why this matters more now than before
Removing the
ioredis,mongodband http/net bindings — the remaining work in the binding-removal campaign — deletes three of the six crates that still carry tokio. So the campaign and shrinking the tokio surface are largely the same work, and this issue is what makes those packages expensive to validate in the meantime: every probe of them must either use auto-optimize or hit this guard.
Two independent agents hit this today, hours apart, in unrelated work. Filing it as a build-system issue
because it is not a one-off and it produces a confusing failure.
The problem
Two static archives built in separate
cargoinvocations can carry different tokio compilations even off anidentical
Cargo.lock. Linking them together then fails.Perry's own linker detects and refuses this (
crates/perry/src/commands/compile/library_search.rs,runtime_compat.rslink guard), which is good — it fails loudly rather than producing a subtly broken binary. Butthe error surfaces far from the cause, and the fix is non-obvious unless you already know the rule.
The two sightings
pgauto-triggered aperry-ext-pgbuild (it bridges tosqlx::postgres+ tokio) in its own cargo invocation; the resultinglibperry_ext_pg.abundled a differenttokio than the existing
libperry_stdlib.a. Resolved by rebuilding coherently:cargo build --profile perry-dev -p perry -p perry-runtime-static -p perry-stdlib-static -p perry-ext-pgperry-stdlib-staticwithoutperry-ext-http, tripped theruntime_compat.rsguard, and had to rebuild both together in one invocation.Why it is worth fixing rather than documenting
CLAUDE.md already warns that dropping a
-pchanges cargo feature unification, and the fix-agent brief tellsagents to keep the package set identical across builds. Both sightings happened anyway, to agents that had read
those warnings — because the failure mode is triggered by an auto-triggered build (case 1) or by a reasonable
partial rebuild (case 2), not by carelessness.
Options worth weighing: have the ext-crate auto-build join the parent invocation rather than spawning its own;
have the link guard's error name the required
cargo build -p ...line explicitly; or record a build fingerprintin each archive so the mismatch is reported as "built in a different invocation" rather than a tokio symbol clash.
Related
The same family as the stale-
.atrap CLAUDE.md documents under "Verifying a runtime change" — where rebuildingthe rlib crates without the
-staticwrappers silently links a stale archive and makes both arms of an A/B behaveidentically. That one produces a vacuous pass; this one produces a confusing failure. Both come from archives
and invocations not lining up.