fix: vendor the C sources as submodules so docs.rs can build - #4
Merged
protocolstardust merged 8 commits intoAug 27, 2026
Merged
Conversation
rayforce-sys/build.rs clones the RayforceDB core and the rayforce-q client
from GitHub at build time when no local checkout is present. docs.rs builds
in a network-isolated sandbox, so both crates fail there with:
fatal: unable to access 'https://github.com/RayforceDB/rayforce.git/':
Could not resolve host: github.com
Cargo packages submodule contents into the published .crate as long as the
submodule sits inside the package directory, so carrying the sources this way
makes the build work offline for docs.rs and for any firewalled consumer.
This commit only adds the submodules; build.rs starts using them next.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… time Removes clone_pinned() and the RAYFORCE_REPO/RAYFORCE_REF (and _Q_) constants; the submodule commit is now the only pin, so build.rs and release.yml can no longer disagree the way they had (build.rs said v2.5.8, release.yml v2.5.1). Source resolution is now RAYFORCE_SRC, else vendor/<name>. The undocumented ~/rayforce probe is gone — it could silently shadow the pinned submodule. Cargo.toml gains an `include` whitelist. The vendored repos carry test suites and a website; `make lib` needs only src/*/*.c, include/ and the Makefile, which keeps the .crate at 1.3 MiB. The .c/.h extension filters are load-bearing: an `include` list is matched against the filesystem rather than git, so a bare `src/**` packages the .o/.d output of a local build despite the core's .gitignore. The vendored core is staged into OUT_DIR and built there, because the core's Makefile builds strictly in-tree — running it in place would write into the shared registry cache for a crates.io consumer, and it makes `cargo package` fail its verify step on "files added". A RAYFORCE_SRC checkout is still built in place; it belongs to the caller. Three things follow from building under OUT_DIR: - RAY_VERSION and GIT_HASH must be passed explicitly. The Makefile resolves them with `git describe` / `git rev-parse`, and git searches upward, so an unset value does not degrade to "unknown" — it reports the HEAD of whatever repository encloses the build directory. - Objects are dropped when the stamped flags change, since the Makefile tracks header dependencies but not flag changes. - Sources removed upstream are pruned from the staging area, so a core bump cannot leave a stale .c that $(wildcard src/*/*.c) still picks up. Also passes -j$NUM_JOBS (make ran serially) and drops -Werror from the vendored build: consumers compile this with whatever toolchain they have, and a new diagnostic from a future compiler should not be a hard failure in someone else's dependency tree. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both workflows checked out RayforceDB/rayforce and RayforceDB/rayforce-q into separate paths and pointed RAYFORCE_SRC / RAYFORCE_Q_SRC at them. That is now one `submodules: recursive` checkout, which also fixes what CI was testing: ci.yml pinned no ref at all, so it built against core master rather than the version consumers get. Adds two guards: - scripts/check-vendored-pin.sh asserts the submodule's tag and commit match CORE_VERSION / CORE_COMMIT in build.rs. Those constants are stamped into librayforce.a because a crate unpacked from crates.io has no git history to read them from, so a mismatch means published crates report a version they were not built from. - A packaging check in both workflows. A .crate missing its vendored sources builds fine in CI, where the submodule is on disk, and then fails for every consumer and on docs.rs — release is the last point at which that is catchable. The comment above "Locate libclang" claimed .cargo/config.toml sets a macOS LIBCLANG_PATH; it only sets RUST_TEST_THREADS. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The README told readers to clone the core to ~/rayforce and export RAYFORCE_SRC. Neither is needed now — the C sources ship with the crate — and the ~/rayforce default no longer exists. Documents cloning with --recurse-submodules, and keeps RAYFORCE_SRC / RAYFORCE_Q_SRC as the override for working on the core itself. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The installation page still described the old model — "builds against a local checkout", RAYFORCE_SRC defaulting to ~/rayforce — none of which is true now that the core ships as a submodule. Adds the procedure for moving the pin: the submodule and the CORE_VERSION / CORE_COMMIT constants in build.rs have to move together, because the core's Makefile normally reads its version from git and a crate unpacked from crates.io has no git history. check-vendored-pin.sh is what catches a mismatch. Also covers RAYFORCE_SRC / RAYFORCE_Q_SRC for building against a core you are changing, and flags that a bump may require updating the symbols wrapper.h hand-declares. Fixes the dependency snippets on that page too: they read 0.1, the crate is at 1.0.x. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The "Check vendored core pin" step failed on every PR run with "core submodule is at <not on a tag>, but build.rs CORE_VERSION expects v2.5.8", while the CORE_COMMIT half of the same check passed — the submodule was pinned correctly all along. actions/checkout clones submodules with `git submodule update --depth=1`, and a shallow clone carries no tags at all, so `git describe --tags --exact-match` has nothing to name HEAD with. The check was asserting something CI could never observe. Resolves refs/tags/v$CORE_VERSION and compares it to HEAD instead, fetching that single ref when it is absent (~1s, and only where the clone is already shallow, so a full local checkout keeps its history). This is also the stronger assertion: it fails if the tag does not exist upstream at all, which `describe` silently treated as "not on a tag". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…headers wrapper.h hand-copied 13 declarations for core symbols absent from the public rayforce.h. That was the only option when the core was cloned at build time, but 840ffba vendored it as a submodule, so the headers those signatures came from now ship in the crate and can be read directly. The copies were not merely redundant. C linkage matches on name alone and bindgen only ever saw the copy, so a signature that drifted from the core was undefined behaviour with no diagnostic. 3fecff2 was the manual fix after the poll API moved into the public header and changed its return type; there is no longer a way for that to go unnoticed. What kept wrapper.h alive was the allowlist, not header availability: `ray_.*` was safe only because bindgen's sole include path was include/. Bounding generation to the public header instead lets the private ones onto the path without dragging in ~550 internal functions. Two wrinkles the vendored headers bring, both handled at the bindgen boundary only — the core is still built by its own Makefile: - lang/internal.h is the only header declaring the splayed/parted builtins, and it reaches mem/heap.h:442's _Atomic global. bindgen 0.70 aborts the whole parse there, before any allowlist applies. Defining the keyword away is inert: both atomics in the parse are unallowlisted file-scope globals, and rayforce.h never says _Atomic, so no emitted layout can shift. - core/runtime.h:114 completes ray_runtime_s, which rayforce.h:656 leaves opaque. Left alone bindgen would publish ray_vm_t and friends, ~67 KB of private layout churning on every core bump. Generated bindings are unchanged but for ray_runtime_s, now a 40-byte opaque blob rather than a zero-sized one. Only ever held by pointer (rayforce/src/runtime.rs:22), and the old size was wrong regardless. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
wrapper.h is gone, so the warning telling maintainers to update its declarations no longer applies. Signatures now need no maintenance at all, but a bump that renames or relocates one of the internal symbols fails the build instead — name the two lists in build.rs that fix it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Collaborator
|
Approved, thanks for the effort! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
docs.rsbuilds with networking disabled, andrayforce-sys/build.rscloned the C core from GitHub at build time — so bothrayforceandrayforce-sys1.0.1 fail there withCould not resolve host: github.com. The same failure hits any firewalled consumer.The core and
rayforce-qnow ship as git submodules underrayforce-sys/vendor/. Cargo packs submodule contents into the published.crate, so the build needs no network at all.Changes
v2.5.8/ q2.0.0.clone_pinned()and theRAYFORCE_REPO/RAYFORCE_REFconstants are gone — the submodule commit is the only pin, sobuild.rsandrelease.ymlcan't disagree again (they already had:v2.5.8vsv2.5.1).includewhitelist inrayforce-sys/Cargo.toml. The vendored repos carry test suites and a website;make libneeds onlysrc/*/*.c,include/and the Makefile. Packaged crate: 183 files, 1.4 MiB compressed.OUT_DIRfirst — otherwise the build writes into the shared registry cache, andcargo packagefails its verify step. ARAYFORCE_SRCcheckout is still built in place.RAY_VERSION/GIT_HASHfrom git, and git searches upward from the build dir — unset, it stamped an unrelated repository's HEAD intolibrayforce.a. Both are now passed explicitly and checked byscripts/check-vendored-pin.sh.rayforce-sys/wrapper.hdeleted. It hand-copied 13 declarations for core symbols missing from the publicrayforce.h— the only option while the core was cloned at build time. Now that the private headers ship, bindgen reads them directly, so a signature can no longer drift from the core silently: C linkage matches on name alone, and bindgen only ever saw the copy. That already cost a manual fix in3fecff2, when the poll API moved into the public header and changed its return type.CORE_PRIVATE_HEADERSandINTERNAL_FNSinbuild.rsreplace the file. What had kept it alive was the allowlist rather than header availability —ray_.*was safe only because bindgen's sole include path wasinclude/; bounding generation to the public header lets the private ones onto the path without pulling in ~550 internal functions.lang/internal.his the only header declaring the splayed/parted builtins and it reachesmem/heap.h:442's_Atomicglobal, which bindgen 0.70 cannot resolve; it aborts the whole parse before any allowlist applies, so the keyword is defined away for the parse (both atomics involved are unallowlisted file-scope globals, andrayforce.hnever says_Atomic, so no emitted layout can shift). Andcore/runtime.h:114completesray_runtime_s, whichrayforce.h:656leaves opaque — left alone bindgen would publishray_vm_tand friends, ~67 KB of private layout that would churn on every core bump.submodules: recursivecheckout replaces the three separate ones. This also fixes what CI tested —ci.ymlpinned no ref, so it built against coremaster. Adds the pin check and a guard that the.crateactually carries the vendored sources.-j$NUM_JOBStomake(it ran serially) and drops-Werrorfrom the vendored build, so a future compiler's new diagnostic isn't a hard failure in someone else's dependency tree.Verified
.git, emptyHOME, noRAYFORCE_SRC, and rancargo rustdoc --lib --offline— i.e. what docs.rs does. Succeeds.git clone --recurse-submodulesbuilds and passes all 15 test binaries;fmtandclippy -D warningsclean.librayforce.areportsrayforce 2.5.8 (f0d4bb4), matching the pinned submodule.wrapper.houtput except forray_runtime_s, now a 40-byte opaque blob rather than zero-sized. It is only ever held by pointer (rayforce/src/runtime.rs:22), and the old size was wrong regardless. 84 tests pass; the packaged.cratecarries nowrapper.hand no build artifacts.docs.rs won't retry 1.0.1, so this needs a
1.0.2release to take effect.🤖 Generated with Claude Code