feat(packager): bundle the module introspector and its compiler - #22
Draft
TomChv wants to merge 2 commits into
Draft
feat(packager): bundle the module introspector and its compiler#22TomChv wants to merge 2 commits into
TomChv wants to merge 2 commits into
Conversation
Entrypoint generation scans a module's TypeScript with the compiler API, so the scanner and the compiler have to come from somewhere. The engine ships both in its image; we build the scanner from the vendored source and carry the compiler next to it. The scanner is a plain `bun build`, not the `--compile`d binary the engine ships: that is ~100MB and per-platform, where this is portable JS. The compiler stays external to it and is trimmed to package.json + lib/typescript.js, which resolves because TypeScript 6 declares no exports map and points main straight at that file. Verified before anything depends on it: with the compiler mounted at node_modules/typescript, the bundled scanner runs over a fixture module and emits a typedef.json carrying the per-declaration source locations the entrypoint renderer consumes. index.ts and telemetry.ts join them — the module-facing wrappers that make sdk/ resolvable as @dagger.io/dagger. They are taken from the v1.0.0-beta.9 tag, not from upstream's working tree, which re-exports an `agent` symbol this engine's core.js does not have; every name they re-export is checked against the bundle. The bundle now costs ~18MB committed, over half of it the compiler. Signed-off-by: Tom Chauveau <tom@dagger.io>
…ng it The compiler is a build-time dependency of the scanner, not something a user's module consumes: the scanner calls the API of the version the vendored library locks (6.0.3), while a module declares 5.9.3 for its own runtime. Carrying 9.1MB of third-party blob in git, re-committed on every engine bump, to serve one exec is the wrong trade — codegen installs it, pinned, so the layer is content-addressed on the version alone and shared across every module's generate. The version is derived rather than hand-written: the packager reads it off the resolved install into bundle/typescript-version.txt, so re-vendoring cannot silently move the scanner onto a compiler API it was not written against. Verified the scan still works with the compiler installed from that pin rather than copied in. The bundle drops from ~18MB to ~9.5MB, and everything left in it is built from the vendored source. The cost is a registry fetch on a cold cache; committing the compiler again is one line if offline generation ever matters. Signed-off-by: Tom Chauveau <tom@dagger.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Entrypoint generation scans a module's TypeScript with the compiler API, so the
scanner and the compiler have to come from somewhere. The engine ships both in
its image; we build the scanner from the vendored source and carry the compiler
next to it.
The scanner is a plain
bun build, not the--compiled binary the engineships: that is ~100MB and per-platform, where this is portable JS. The compiler
stays external to it and is trimmed to package.json + lib/typescript.js, which
resolves because TypeScript 6 declares no exports map and points main straight
at that file.
Verified before anything depends on it: with the compiler mounted at
node_modules/typescript, the bundled scanner runs over a fixture module and
emits a typedef.json carrying the per-declaration source locations the
entrypoint renderer consumes.
index.ts and telemetry.ts join them — the module-facing wrappers that make sdk/
resolvable as @dagger.io/dagger. They are taken from the v1.0.0-beta.9 tag, not
from upstream's working tree, which re-exports an
agentsymbol this engine'score.js does not have; every name they re-export is checked against the bundle.
The bundle now costs ~18MB committed, over half of it the compiler.
Signed-off-by: Tom Chauveau tom@dagger.io
Stack created with GitHub Stacks CLI • Give Feedback 💬