Skip to content

fix: vendor the C sources as submodules so docs.rs can build - #4

Merged
protocolstardust merged 8 commits into
RayforceDB:masterfrom
ihrfv:fix/vendor-core-as-submodules
Aug 27, 2026
Merged

protocolstardust merged 8 commits into
RayforceDB:masterfrom
ihrfv:fix/vendor-core-as-submodules

Conversation

@ihrfv

@ihrfv ihrfv commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

docs.rs builds with networking disabled, and rayforce-sys/build.rs cloned the C core from GitHub at build time — so both rayforce and rayforce-sys 1.0.1 fail there with Could not resolve host: github.com. The same failure hits any firewalled consumer.

The core and rayforce-q now ship as git submodules under rayforce-sys/vendor/. Cargo packs submodule contents into the published .crate, so the build needs no network at all.

Changes

  • Submodules pinned at core v2.5.8 / q 2.0.0. clone_pinned() and the RAYFORCE_REPO/RAYFORCE_REF constants are gone — the submodule commit is the only pin, so build.rs and release.yml can't disagree again (they already had: v2.5.8 vs v2.5.1).
  • include whitelist in rayforce-sys/Cargo.toml. The vendored repos carry test suites and a website; make lib needs only src/*/*.c, include/ and the Makefile. Packaged crate: 183 files, 1.4 MiB compressed.
  • Staged build. The core's Makefile builds in-tree, so the vendored copy is mirrored into OUT_DIR first — otherwise the build writes into the shared registry cache, and cargo package fails its verify step. A RAYFORCE_SRC checkout is still built in place.
  • Explicit version stamping. The Makefile reads RAY_VERSION/GIT_HASH from git, and git searches upward from the build dir — unset, it stamped an unrelated repository's HEAD into librayforce.a. Both are now passed explicitly and checked by scripts/check-vendored-pin.sh.
  • rayforce-sys/wrapper.h deleted. It hand-copied 13 declarations for core symbols missing from the public rayforce.h — the only option while the core was cloned at build time. Now that the private headers ship, bindgen reads them directly, so a signature can no longer drift from the core silently: C linkage matches on name alone, and bindgen only ever saw the copy. That already cost a manual fix in 3fecff2, when the poll API moved into the public header and changed its return type. CORE_PRIVATE_HEADERS and INTERNAL_FNS in build.rs replace the file. What had kept it alive was the allowlist rather than header availability — ray_.* was safe only because bindgen's sole include path was include/; bounding generation to the public header lets the private ones onto the path without pulling in ~550 internal functions.
  • Two constraints that surfaced, both pinned at the bindgen boundary only — the core is still compiled by its own Makefile. lang/internal.h is the only header declaring the splayed/parted builtins and it reaches mem/heap.h:442's _Atomic global, which bindgen 0.70 cannot resolve; it aborts the whole parse before any allowlist applies, so the keyword is defined away for the parse (both atomics involved are unallowlisted file-scope globals, and rayforce.h never says _Atomic, so no emitted layout can shift). And core/runtime.h:114 completes ray_runtime_s, which rayforce.h:656 leaves opaque — left alone bindgen would publish ray_vm_t and friends, ~67 KB of private layout that would churn on every core bump.
  • CI: one submodules: recursive checkout replaces the three separate ones. This also fixes what CI tested — ci.yml pinned no ref, so it built against core master. Adds the pin check and a guard that the .crate actually carries the vendored sources.
  • Passes -j$NUM_JOBS to make (it ran serially) and drops -Werror from the vendored build, so a future compiler's new diagnostic isn't a hard failure in someone else's dependency tree.

Verified

  • Packaged the crate, extracted it somewhere with no .git, empty HOME, no RAYFORCE_SRC, and ran cargo rustdoc --lib --offline — i.e. what docs.rs does. Succeeds.
  • Fresh git clone --recurse-submodules builds and passes all 15 test binaries; fmt and clippy -D warnings clean.
  • librayforce.a reports rayforce 2.5.8 (f0d4bb4), matching the pinned submodule.
  • Bindings regenerated from scratch match the previous wrapper.h output except for ray_runtime_s, now a 40-byte opaque blob rather than zero-sized. It is only ever held by pointer (rayforce/src/runtime.rs:22), and the old size was wrong regardless. 84 tests pass; the packaged .crate carries no wrapper.h and no build artifacts.

docs.rs won't retry 1.0.1, so this needs a 1.0.2 release to take effect.

🤖 Generated with Claude Code

ihrfv and others added 8 commits August 26, 2026 23:10
rayforce-sys/build.rs clones the RayforceDB core and the rayforce-q client
from GitHub at build time when no local checkout is present. docs.rs builds
in a network-isolated sandbox, so both crates fail there with:

    fatal: unable to access 'https://github.com/RayforceDB/rayforce.git/':
    Could not resolve host: github.com

Cargo packages submodule contents into the published .crate as long as the
submodule sits inside the package directory, so carrying the sources this way
makes the build work offline for docs.rs and for any firewalled consumer.

This commit only adds the submodules; build.rs starts using them next.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… time

Removes clone_pinned() and the RAYFORCE_REPO/RAYFORCE_REF (and _Q_) constants;
the submodule commit is now the only pin, so build.rs and release.yml can no
longer disagree the way they had (build.rs said v2.5.8, release.yml v2.5.1).
Source resolution is now RAYFORCE_SRC, else vendor/<name>. The undocumented
~/rayforce probe is gone — it could silently shadow the pinned submodule.

Cargo.toml gains an `include` whitelist. The vendored repos carry test suites
and a website; `make lib` needs only src/*/*.c, include/ and the Makefile, which
keeps the .crate at 1.3 MiB. The .c/.h extension filters are load-bearing: an
`include` list is matched against the filesystem rather than git, so a bare
`src/**` packages the .o/.d output of a local build despite the core's
.gitignore.

The vendored core is staged into OUT_DIR and built there, because the core's
Makefile builds strictly in-tree — running it in place would write into the
shared registry cache for a crates.io consumer, and it makes `cargo package`
fail its verify step on "files added". A RAYFORCE_SRC checkout is still built
in place; it belongs to the caller.

Three things follow from building under OUT_DIR:

- RAY_VERSION and GIT_HASH must be passed explicitly. The Makefile resolves
  them with `git describe` / `git rev-parse`, and git searches upward, so an
  unset value does not degrade to "unknown" — it reports the HEAD of whatever
  repository encloses the build directory.
- Objects are dropped when the stamped flags change, since the Makefile tracks
  header dependencies but not flag changes.
- Sources removed upstream are pruned from the staging area, so a core bump
  cannot leave a stale .c that $(wildcard src/*/*.c) still picks up.

Also passes -j$NUM_JOBS (make ran serially) and drops -Werror from the vendored
build: consumers compile this with whatever toolchain they have, and a new
diagnostic from a future compiler should not be a hard failure in someone
else's dependency tree.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both workflows checked out RayforceDB/rayforce and RayforceDB/rayforce-q into
separate paths and pointed RAYFORCE_SRC / RAYFORCE_Q_SRC at them. That is now
one `submodules: recursive` checkout, which also fixes what CI was testing:
ci.yml pinned no ref at all, so it built against core master rather than the
version consumers get.

Adds two guards:

- scripts/check-vendored-pin.sh asserts the submodule's tag and commit match
  CORE_VERSION / CORE_COMMIT in build.rs. Those constants are stamped into
  librayforce.a because a crate unpacked from crates.io has no git history to
  read them from, so a mismatch means published crates report a version they
  were not built from.
- A packaging check in both workflows. A .crate missing its vendored sources
  builds fine in CI, where the submodule is on disk, and then fails for every
  consumer and on docs.rs — release is the last point at which that is
  catchable.

The comment above "Locate libclang" claimed .cargo/config.toml sets a macOS
LIBCLANG_PATH; it only sets RUST_TEST_THREADS.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The README told readers to clone the core to ~/rayforce and export
RAYFORCE_SRC. Neither is needed now — the C sources ship with the crate — and
the ~/rayforce default no longer exists. Documents cloning with
--recurse-submodules, and keeps RAYFORCE_SRC / RAYFORCE_Q_SRC as the override
for working on the core itself.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The installation page still described the old model — "builds against a local
checkout", RAYFORCE_SRC defaulting to ~/rayforce — none of which is true now
that the core ships as a submodule.

Adds the procedure for moving the pin: the submodule and the CORE_VERSION /
CORE_COMMIT constants in build.rs have to move together, because the core's
Makefile normally reads its version from git and a crate unpacked from
crates.io has no git history. check-vendored-pin.sh is what catches a
mismatch. Also covers RAYFORCE_SRC / RAYFORCE_Q_SRC for building against a
core you are changing, and flags that a bump may require updating the symbols
wrapper.h hand-declares.

Fixes the dependency snippets on that page too: they read 0.1, the crate is
at 1.0.x.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The "Check vendored core pin" step failed on every PR run with "core submodule
is at <not on a tag>, but build.rs CORE_VERSION expects v2.5.8", while the
CORE_COMMIT half of the same check passed — the submodule was pinned correctly
all along.

actions/checkout clones submodules with `git submodule update --depth=1`, and
a shallow clone carries no tags at all, so `git describe --tags --exact-match`
has nothing to name HEAD with. The check was asserting something CI could
never observe.

Resolves refs/tags/v$CORE_VERSION and compares it to HEAD instead, fetching
that single ref when it is absent (~1s, and only where the clone is already
shallow, so a full local checkout keeps its history). This is also the
stronger assertion: it fails if the tag does not exist upstream at all, which
`describe` silently treated as "not on a tag".

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…headers

wrapper.h hand-copied 13 declarations for core symbols absent from the
public rayforce.h. That was the only option when the core was cloned at
build time, but 840ffba vendored it as a submodule, so the headers those
signatures came from now ship in the crate and can be read directly.

The copies were not merely redundant. C linkage matches on name alone and
bindgen only ever saw the copy, so a signature that drifted from the core
was undefined behaviour with no diagnostic. 3fecff2 was the manual fix
after the poll API moved into the public header and changed its return
type; there is no longer a way for that to go unnoticed.

What kept wrapper.h alive was the allowlist, not header availability:
`ray_.*` was safe only because bindgen's sole include path was
include/. Bounding generation to the public header instead lets the
private ones onto the path without dragging in ~550 internal functions.

Two wrinkles the vendored headers bring, both handled at the bindgen
boundary only — the core is still built by its own Makefile:

  - lang/internal.h is the only header declaring the splayed/parted
    builtins, and it reaches mem/heap.h:442's _Atomic global. bindgen
    0.70 aborts the whole parse there, before any allowlist applies.
    Defining the keyword away is inert: both atomics in the parse are
    unallowlisted file-scope globals, and rayforce.h never says _Atomic,
    so no emitted layout can shift.

  - core/runtime.h:114 completes ray_runtime_s, which rayforce.h:656
    leaves opaque. Left alone bindgen would publish ray_vm_t and friends,
    ~67 KB of private layout churning on every core bump.

Generated bindings are unchanged but for ray_runtime_s, now a 40-byte
opaque blob rather than a zero-sized one. Only ever held by pointer
(rayforce/src/runtime.rs:22), and the old size was wrong regardless.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
wrapper.h is gone, so the warning telling maintainers to update its
declarations no longer applies. Signatures now need no maintenance at
all, but a bump that renames or relocates one of the internal symbols
fails the build instead — name the two lists in build.rs that fix it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@protocolstardust

Copy link
Copy Markdown
Collaborator

Approved, thanks for the effort!

@protocolstardust
protocolstardust merged commit 82f2b34 into RayforceDB:master Aug 27, 2026
1 check passed
@ihrfv
ihrfv deleted the fix/vendor-core-as-submodules branch August 27, 2026 13:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants