Skip to content

[miniflare] Add opt-in local Artifacts simulator [2/3] - #16177

Open
leostera wants to merge 16 commits into
cloudflare:mainfrom
leostera:lostera/artifacts-miniflare-simulator
Open

leostera wants to merge 16 commits into
cloudflare:mainfrom
leostera:lostera/artifacts-miniflare-simulator

Conversation

@leostera

@leostera leostera commented Oct 9, 2026 •

Copy link
Copy Markdown
Collaborator

Artifacts local-development stack — 2 of 3

Part 2/3; the native Git client (#16176) is merged. This branch is rebased onto upstream main, so its PR diff shows only the opt-in simulator layer. The development-tools integration is PR #16186 (3/3) (incremental 2/3 → 3/3 diff); merge this PR first, then restack 3/3 onto main and revalidate its exact head. #16092 is a closed full-stack reference, not another PR to merge.

What changes in this layer

  • Add the Miniflare binding Worker, durable metadata, Node Git sidecar, authenticated Git smart HTTP, namespace persistence, and lifecycle management.
  • Expose the 15 public binding methods via real workerd RPC, with direct Git fetch/clone and push.
  • At this intermediate stage, local storage requires explicit dev.remote: false. An omitted setting retains existing remote behavior; the final development-tools layer changes the default.
  • Include an internal architecture README and a Miniflare changeset. The public Artifacts docs PR remains draft until the final behavior and release are approved.

The simulator builds on the original prototype by @GregBrimble. Focused local tests establish tested behavior, not parity with the deployed service. The unused live-comparison harness was removed from this PR; no deployed comparison has been run. A future private comparison needs owner authorization for an exclusively reserved disposable namespace. Explicit HTTPS imports can contact the supplied host, including private-network destinations, and do not follow HTTP redirects.

Validation

  • pnpm -r --filter miniflare... build
  • pnpm --filter miniflare exec vitest run test/plugins/artifacts test/config/v4-convert.spec.ts — 109 passed, 1 platform-specific test skipped on macOS (including inherited Git-client tests)
  • Miniflare build, typecheck, type-aware lint, and format passed after the latest review batch; earlier changeset validation passed. Exact-head CI/rereview pending.

@changeset-bot

changeset-bot Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 580c8ee

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 8 packages
Name Type
miniflare Minor
@cloudflare/deploy-helpers Patch
@cloudflare/pages-shared Patch
@cloudflare/remote-bindings Patch
@cloudflare/runtime-types Patch
@cloudflare/vite-plugin Patch
@cloudflare/vitest-plugin Patch
wrangler Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@workers-devprod
workers-devprod requested review from a team and jamesopstad and removed request for a team October 9, 2026 12:21
@workers-devprod

workers-devprod commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Codeowners approval required for this PR:

  • @cloudflare/wrangler
Show detailed file reviewers
  • .changeset/artifacts-opt-in-simulator.md: [@cloudflare/wrangler]
  • packages/miniflare/scripts/build.mjs: [@cloudflare/wrangler]
  • packages/miniflare/src/config/v4-convert.ts: [@cloudflare/wrangler]
  • packages/miniflare/src/index.ts: [@cloudflare/wrangler]
  • packages/miniflare/src/plugins/artifacts/README.md: [@cloudflare/wrangler]
  • packages/miniflare/src/plugins/artifacts/controller.ts: [@cloudflare/wrangler]
  • packages/miniflare/src/plugins/artifacts/git-client.ts: [@cloudflare/wrangler]
  • packages/miniflare/src/plugins/artifacts/git-sidecar.ts: [@cloudflare/wrangler]
  • packages/miniflare/src/plugins/artifacts/index.ts: [@cloudflare/wrangler]
  • packages/miniflare/src/plugins/artifacts/storage.ts: [@cloudflare/wrangler]
  • packages/miniflare/src/plugins/shared/index.ts: [@cloudflare/wrangler]
  • packages/miniflare/src/workers/artifacts/binding.worker.ts: [@cloudflare/wrangler]
  • packages/miniflare/test/config/v4-convert.spec.ts: [@cloudflare/wrangler]
  • packages/miniflare/test/plugins/artifacts/contract.spec.ts: [@cloudflare/wrangler]
  • packages/miniflare/test/plugins/artifacts/contract/README.md: [@cloudflare/wrangler]
  • packages/miniflare/test/plugins/artifacts/contract/scenarios.ts: [@cloudflare/wrangler]
  • packages/miniflare/test/plugins/artifacts/https-import.spec.ts: [@cloudflare/wrangler]
  • packages/miniflare/test/plugins/artifacts/index.spec.ts: [@cloudflare/wrangler]
  • packages/miniflare/test/plugins/artifacts/lifecycle.spec.ts: [@cloudflare/wrangler]
  • packages/miniflare/test/plugins/artifacts/storage.spec.ts: [@cloudflare/wrangler]

@devin-ai-integration devin-ai-integration Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Newer findings are available below. Devin Review posted a newer report on this PR, in addition to the findings presented here.

Devin Review found 7 potential issues.

Devin Review

Comment thread packages/miniflare/src/plugins/artifacts/git-sidecar.ts Outdated
Comment thread packages/miniflare/src/plugins/artifacts/controller.ts
Comment thread packages/miniflare/src/workers/artifacts/binding.worker.ts Outdated
Comment thread packages/miniflare/src/plugins/artifacts/git-client.ts Outdated
Comment on lines +7 to +26
test("Artifacts defaults to local and only an explicit connection is remote", ({
expect,
}) => {
const remoteProxyConnectionString = new URL(
"http://127.0.0.1:8787"
) as RemoteProxyConnectionString;
const converted = convertV4MiniflareOptions({
name: "worker",
script: "export default {};",
artifacts: {
LOCAL: { namespace: "local" },
REMOTE: { namespace: "remote", remoteProxyConnectionString },
},
});
expect(converted.workers[0].config.env?.LOCAL).toMatchObject({
dev: { remote: false },
});
expect(converted.workers[0].config.env?.REMOTE).toMatchObject({
dev: { remote: true },
});

@devin-ai-integration devin-ai-integration Bot Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Legacy Artifacts settings select local storage

The v4 converter sets dev.remote: false without an explicit opt-in, selecting the local simulator. That differs from the stated intermediate-stage default for omitted settings.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed at 00bd9ad: legacy conversion preserves omitted dev.remote instead of synthesizing false. This intermediate layer remains remote by default; local still requires explicit V5 dev.remote: false. The local-default switch is reserved for 3/3.

Comment on lines +942 to +948
if (sourceUrl.protocol !== "https:") {
throw new ArtifactsError(
"INVALID_INPUT",
"Invalid source.url: must be an HTTPS URL."
);
}
return { source, target, sourceUrl };

@devin-ai-integration devin-ai-integration Bot Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟥 Git import reaches private host networks

A Worker can pass an internal HTTPS URL to import. Host Git connects outside workerd's network restrictions, allowing the Worker to reach private services.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a deliberate local-only boundary: an explicit HTTPS import contacts its specified host from the developer machine, which may include a private-network host. Import redirects are disabled, and the internal README now warns against untrusted URLs. We are not claiming workerd network isolation for host Git; a broader private-destination policy needs an owner decision.

Comment thread packages/miniflare/src/plugins/artifacts/git-sidecar.ts Outdated
@pkg-pr-new

pkg-pr-new Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
@cloudflare/autoconfig

npm i https://pkg.pr.new/@cloudflare/autoconfig@16177

@cloudflare/build-output-utils

npm i https://pkg.pr.new/@cloudflare/build-output-utils@16177

@cloudflare/codemods

npm i https://pkg.pr.new/@cloudflare/codemods@16177

@cloudflare/config

npm i https://pkg.pr.new/@cloudflare/config@16177

@cloudflare/containers-shared

npm i https://pkg.pr.new/@cloudflare/containers-shared@16177

create-cloudflare

npm i https://pkg.pr.new/create-cloudflare@16177

@cloudflare/deploy-helpers

npm i https://pkg.pr.new/@cloudflare/deploy-helpers@16177

@cloudflare/kv-asset-handler

npm i https://pkg.pr.new/@cloudflare/kv-asset-handler@16177

miniflare

npm i https://pkg.pr.new/miniflare@16177

@cloudflare/pages-functions

npm i https://pkg.pr.new/@cloudflare/pages-functions@16177

@cloudflare/pages-shared

npm i https://pkg.pr.new/@cloudflare/pages-shared@16177

@cloudflare/runtime-types

npm i https://pkg.pr.new/@cloudflare/runtime-types@16177

@cloudflare/unenv-preset

npm i https://pkg.pr.new/@cloudflare/unenv-preset@16177

@cloudflare/vite-plugin

npm i https://pkg.pr.new/@cloudflare/vite-plugin@16177

@cloudflare/vitest-plugin

npm i https://pkg.pr.new/@cloudflare/vitest-plugin@16177

@cloudflare/workers-auth

npm i https://pkg.pr.new/@cloudflare/workers-auth@16177

@cloudflare/workers-editor-shared

npm i https://pkg.pr.new/@cloudflare/workers-editor-shared@16177

@cloudflare/workers-utils

npm i https://pkg.pr.new/@cloudflare/workers-utils@16177

wrangler

npm i https://pkg.pr.new/wrangler@16177

commit: 580c8ee

devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

@leostera
leostera force-pushed the lostera/artifacts-miniflare-simulator branch from 925153c to 5902a96 Compare October 9, 2026 20:48
devin-ai-integration[bot]

This comment was marked as resolved.

@edmundhung
edmundhung requested review from edmundhung and removed request for jamesopstad October 12, 2026 10:44
Comment thread packages/miniflare/src/plugins/artifacts/controller.ts
Comment thread packages/miniflare/src/workers/artifacts/binding.worker.ts Outdated
Comment thread packages/miniflare/src/workers/artifacts/binding.worker.ts Outdated
Comment thread packages/miniflare/test/plugins/artifacts/contract/live.ts Outdated
Comment thread packages/miniflare/test/plugins/artifacts/index.spec.ts Outdated
Comment thread packages/miniflare/test/plugins/artifacts/https-import.spec.ts
@leostera
leostera force-pushed the lostera/artifacts-miniflare-simulator branch from 93a4ff3 to 9e58655 Compare October 12, 2026 22:49
@leostera
leostera requested a review from edmundhung October 13, 2026 15:08
@leostera

Copy link
Copy Markdown
Collaborator Author

Hey folks, here’s a small example if you want to try the local Artifacts simulator quickly: https://github.com/leostera/workers-sdk-playground

It uses the Wrangler preview from the stacked 3/3 PR with an Artifacts binding set to remote: false. Run npm i, then npm run dev and npm test in separate terminals.

@workers-devprod

Copy link
Copy Markdown
Contributor

Codeowners approval required for this PR:

  • @cloudflare/wrangler
Show detailed file reviewers
  • .changeset/artifacts-opt-in-simulator.md: [@cloudflare/wrangler]
  • packages/miniflare/scripts/build.mjs: [@cloudflare/wrangler]
  • packages/miniflare/src/config/v4-convert.ts: [@cloudflare/wrangler]
  • packages/miniflare/src/index.ts: [@cloudflare/wrangler]
  • packages/miniflare/src/plugins/artifacts/README.md: [@cloudflare/wrangler]
  • packages/miniflare/src/plugins/artifacts/controller.ts: [@cloudflare/wrangler]
  • packages/miniflare/src/plugins/artifacts/git-client.ts: [@cloudflare/wrangler]
  • packages/miniflare/src/plugins/artifacts/git-sidecar.ts: [@cloudflare/wrangler]
  • packages/miniflare/src/plugins/artifacts/index.ts: [@cloudflare/wrangler]
  • packages/miniflare/src/plugins/artifacts/storage.ts: [@cloudflare/wrangler]
  • packages/miniflare/src/plugins/shared/index.ts: [@cloudflare/wrangler]
  • packages/miniflare/src/workers/artifacts/binding.worker.ts: [@cloudflare/wrangler]
  • packages/miniflare/test/config/v4-convert.spec.ts: [@cloudflare/wrangler]
  • packages/miniflare/test/plugins/artifacts/contract.spec.ts: [@cloudflare/wrangler]
  • packages/miniflare/test/plugins/artifacts/contract/README.md: [@cloudflare/wrangler]
  • packages/miniflare/test/plugins/artifacts/contract/scenarios.ts: [@cloudflare/wrangler]
  • packages/miniflare/test/plugins/artifacts/https-import.spec.ts: [@cloudflare/wrangler]
  • packages/miniflare/test/plugins/artifacts/index.spec.ts: [@cloudflare/wrangler]
  • packages/miniflare/test/plugins/artifacts/lifecycle.spec.ts: [@cloudflare/wrangler]
  • packages/miniflare/test/plugins/artifacts/storage.spec.ts: [@cloudflare/wrangler]

if (!this.#pendingRoots) {
throw new Error("Artifacts backend requested outside runtime update");
}
this.#pendingRoots.add(root);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: should we normalize the path before caching the backend? Relative and absolute paths to the same folder currently start separate backends, and the second one can delete temporary files from an import still running in the first.

@leostera
leostera requested a review from zebp October 16, 2026 14:45
@leostera

Copy link
Copy Markdown
Collaborator Author

Addressed the requested PR 2/3 changes in 580c8eeaf6c50c4e2acd3bf394c3bfe631283cd6: deletion now fails closed on unreadable ownership, streamed Git uploads enforce the size limit, saved token records are pruned, sidecar warnings use the Miniflare logger, and the contract README is cleaned up. Added regressions for the first three. Local Miniflare build, Artifacts suite (90 passed, 1 skipped), and pnpm check pass. Exact-head CI is pending.

@workers-devprod

Copy link
Copy Markdown
Contributor

Codeowners approval required for this PR:

  • @cloudflare/wrangler
Show detailed file reviewers
  • .changeset/artifacts-opt-in-simulator.md: [@cloudflare/wrangler]
  • packages/miniflare/scripts/build.mjs: [@cloudflare/wrangler]
  • packages/miniflare/src/config/v4-convert.ts: [@cloudflare/wrangler]
  • packages/miniflare/src/index.ts: [@cloudflare/wrangler]
  • packages/miniflare/src/plugins/artifacts/README.md: [@cloudflare/wrangler]
  • packages/miniflare/src/plugins/artifacts/controller.ts: [@cloudflare/wrangler]
  • packages/miniflare/src/plugins/artifacts/git-client.ts: [@cloudflare/wrangler]
  • packages/miniflare/src/plugins/artifacts/git-sidecar.ts: [@cloudflare/wrangler]
  • packages/miniflare/src/plugins/artifacts/index.ts: [@cloudflare/wrangler]
  • packages/miniflare/src/plugins/artifacts/storage.ts: [@cloudflare/wrangler]
  • packages/miniflare/src/plugins/shared/index.ts: [@cloudflare/wrangler]
  • packages/miniflare/src/workers/artifacts/binding.worker.ts: [@cloudflare/wrangler]
  • packages/miniflare/test/config/v4-convert.spec.ts: [@cloudflare/wrangler]
  • packages/miniflare/test/plugins/artifacts/contract.spec.ts: [@cloudflare/wrangler]
  • packages/miniflare/test/plugins/artifacts/contract/README.md: [@cloudflare/wrangler]
  • packages/miniflare/test/plugins/artifacts/contract/scenarios.ts: [@cloudflare/wrangler]
  • packages/miniflare/test/plugins/artifacts/https-import.spec.ts: [@cloudflare/wrangler]
  • packages/miniflare/test/plugins/artifacts/index.spec.ts: [@cloudflare/wrangler]
  • packages/miniflare/test/plugins/artifacts/lifecycle.spec.ts: [@cloudflare/wrangler]
  • packages/miniflare/test/plugins/artifacts/storage.spec.ts: [@cloudflare/wrangler]

@clanky-the-clanker

Copy link
Copy Markdown

💬 Approved with comments — The opt-in simulator is well structured, stays behind explicit dev.remote: false and has thorough tests. Only two optional cleanup suggestions remain in the sidecar shutdown and Git process handling.

Findings: 2 suggestions

The decision is advisory: this comment never approves, blocks or requests changes on the pull request.

🔧 Fix with your agent

Paste the following into your coding agent to investigate and fix these issues.

Fix the following code review issues found in pull request #16177.

Before making changes, review each issue and present a brief summary table:
- For each finding, state whether you agree, disagree, or need clarification
- If you disagree (e.g., the fix would require disproportionate changes
  for minimal ROI, or the finding is factually incorrect), explain why
- If you need to clarify something before deciding, ask those questions
- Then share your plan for which issues to tackle and in what order

## SUGGESTION (2)

### [Code Quality] Sidecar `close()` awaits `once(worker, "exit")`, which rejects if the worker emits `error` first. A thread that did exit is then reported as a failed shutdown (warning only).
- **File:** packages/miniflare/src/plugins/artifacts/controller.ts:103

### [Code Quality] `runGit`'s `error` handler does not remove the `abort` listener from the long-lived shutdown signal. A failed spawn can leave a listener attached.
- **File:** packages/miniflare/src/plugins/artifacts/git-client.ts:176

📋 Code Quality: 💬 2 suggestions

Well structured and heavily tested; two small cleanup points in the sidecar shutdown and Git process handling.

Recommendation: 💬 Optional Nits

  • [SUGGESTION] Sidecar close() awaits once(worker, "exit"), which rejects if the worker emits error first. A thread that did exit is then reported as a failed shutdown (warning only). (packages/miniflare/src/plugins/artifacts/controller.ts:103)
  • [SUGGESTION] runGit's error handler does not remove the abort listener from the long-lived shutdown signal. A failed spawn can leave a listener attached. (packages/miniflare/src/plugins/artifacts/git-client.ts:176)
View full review

Findings

  • packages/miniflare/src/plugins/artifacts/controller.ts:103-105: close() awaits once(worker, "exit"). events.once rejects if the worker emits error before exit, for example after an uncaught exception in the worker thread. Shutdown is then reported as "Unable to stop a retired Artifacts Git backend" even though the thread did exit. #closeExcept absorbs the rejection through allSettled, so the only effect is a misleading warning. To fix it, resolve on either exit or error.

  • packages/miniflare/src/plugins/artifacts/git-client.ts:176-179: the error handler in runGit clears the timeout and rejects, but it does not remove the abort listener added on options.signal. Only the close handler removes it. If spawning fails (for example with ENOENT) and close never fires, one listener stays on the long-lived shutdown signal. The leak is small. Either remove the listener in the error path too, or note why the manual listener is needed next to spawn's own signal option. Node's built-in abort kills only the child, not its process group.

Checked, no issue

  • The beginUpdate / commitUpdate / abortUpdate lifecycle in src/index.ts is wired correctly, including the early return on dispose.
  • Every runGit call site passes the cancellation signal.
  • build.mjs emits git-sidecar.js at the path controller.ts expects.
  • V4 convert, the plugin and the changeset all keep remote behaviour unless dev.remote === false, and v4-convert.spec.ts covers this.
  • The compatibility date is supported by the pinned workerd.

✅ All other checks passed

🔒 Security: ✅ LGTM

The Git sidecar is the main new trust boundary. These points held up:

  • It listens only on 127.0.0.1.
  • Every request must carry a random 32-byte secret in the x-local-artifacts-backend header, or it gets a 403.
  • Namespace and repository names are validated against componentPattern.
  • Admin payloads are validated with zod.
  • Request bodies are capped at 256 MiB and Git processes time out after 300 seconds.
  • Git is spawned with an argument array, not a shell string, and with hooks disabled.

HTTPS import can contact private-network hosts. The pull request description says so, and the simulator is a local, opt-in development tool.

The security review covered the sidecar, the Git client and the controller. It did not cover src/workers/artifacts/binding.worker.ts or the plugin index.ts in depth.


📖 AGENTS.md: ✅ LGTM

No AGENTS.md changes are needed for this pull request.


⚡ Performance: ✅ LGTM

No measurable performance regressions:

  • Abort listeners on the shared signal are removed when each child process closes.
  • Uploads are capped, Git processes have timeouts, and packfiles are streamed with backpressure instead of buffered.
  • Repository locks are cleaned up when released.
  • The one extra git config process per HTTP request costs little in a local development simulator.

📝 Documentation: ✅ LGTM

The internal README and the changeset describe the opt-in behaviour accurately.


Reviewed commit: 580c8eea

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Untriaged

Development

Successfully merging this pull request may close these issues.

3 participants