Skip to content

[miniflare] Add local Artifacts binding simulator - #16092

Closed
leostera wants to merge 21 commits into
cloudflare:mainfrom
leostera:lostera/artifacts-local-mode
Closed

leostera wants to merge 21 commits into
cloudflare:mainfrom
leostera:lostera/artifacts-local-mode

Conversation

@leostera

@leostera leostera commented Oct 6, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Add a Miniflare simulator for the public Artifacts binding contract, backed by native Git. It implements namespace/repository RPC, persistent namespace-scoped metadata and Git repositories, authenticated Git smart HTTP, and a per-Miniflare sidecar worker thread so synchronous Node binding proxies do not deadlock. Local development does not require Cloudflare credentials. An omitted remote or remote: false uses local storage; remote: true explicitly selects remote bindings.

This is a draft for contract and migration review, not a claim of production Git-engine or API parity. Native Git is a host dependency. An explicit HTTPS repository import can contact its specified Git source; local binding selection never implicitly synchronizes with production. The architecture diagrams and local setup guide are in packages/miniflare/src/plugins/artifacts/.

Original implementation by @GregBrimble. Adapted and integrated into Miniflare with additional contract and routing tests.

Local API surface

The simulator implements all 15 methods exposed by the Artifacts Worker binding. This is a surface inventory, not a claim of deployed-service behavioral parity.

Artifacts service surface Miniflare simulator coverage
Worker binding — namespace: create, get, list, import, delete All five methods implemented
Worker binding — repository metadata and tokens: info, createToken, listTokens, revokeToken All four methods implemented
Worker binding — repository reads: readBlob, readTree, readCommit, readFile, log All five methods implemented
Worker binding — repository fork: fork Implemented
Git smart HTTP: discovery, git-upload-pack, git-receive-pack Authenticated discovery, clone/fetch, and push

Validation checklist

Completed locally (macOS)

  • Real workerd/RPC namespace and repository operations: names, pagination, invalid inputs, missing objects, blobs/trees/commits/files and logs; namespace isolation, shared bindings, persistence, fork variants, and Git protocol-v2 clone/fetch of a depth-1 import.
  • Native Git HTTP authorization: read/write scopes, token metadata and revocation, wrong-repository access, unauthenticated access, and an offline Git environment. The sidecar shuts down with Miniflare; missing Git fails during startup.
  • Public-binding HTTPS import against an isolated, authenticated loopback Git server: full and shallow/branch imports, Basic authentication and errors, missing repository/branch, duplicate target, untrusted TLS, interrupted transfer cleanup and retry. A test-scoped CA is trusted without disabling verification. Import credentials are not retained in public metadata or the bare repository's Git config.
  • Workerd-controlled test clock exercises valid-before/exact/after token expiry, Git read/write authorization, revocation and expiry across restart; concurrent create/fork/import variants and a committed operation surviving abrupt process exit. This does not prove atomic recovery from a crash during an in-flight Git operation.
  • Shared observable-contract scenarios run against the local binding, with a field-level comparator and an opt-in, guarded live-comparison harness. No deployed-service comparison has been run.
  • Real local entry-point fixtures for Wrangler dev, the Vite plugin, and Workers Vitest (two tests each for omitted remote and remote: false). Wrangler and Vite exercise loopback Git push and file reads; Wrangler checks reload, restart and listener shutdown. Workers Vitest checks create/get/list and a missing-file read. Explicit remote: true routing remains covered by mocked tests, not live-service E2E.
  • Miniflare Artifacts focused suites: 45 passed locally. Related Miniflare v4 conversion, workers-utils display/routing, remote-bindings and Wrangler binding-display suites passed in earlier local validation. Focused builds, package typechecks, Wrangler test typecheck, lint and formatting passed locally. A manual macOS Wrangler-dev smoke test also exercised create, authenticated clone, commit/push, and reading updated files.

To do

  • Checks CI: the Wrangler test typecheck passes after moving dev teardown into a helper (TypeScript previously narrowed the callback to never). The Vite fixture lint errors—shadowing fileURLToPath and using recursive rmSync instead of the shared cleanup helper—are also fixed; the Checks job passed.
  • Fixture CI reruns: Linux interactive-dev container rebuild failures and a macOS NODE_ENV wait timeout did not recur on the next full run. No unrelated fixture code was changed.
  • Vite on Windows: both local Artifacts fixtures loaded their Worker entry and pushed/read a Git file in Windows CI. Moving the fixture out of the Windows 8.3 temp path resolved the /index.js failure.
  • Windows Miniflare storage and Git: all 45 focused Artifacts tests passed on Windows, including storage paths, legacy-layout detection, import, persistence/restart, Git clone/push/fetch and teardown. Both Wrangler dev Artifacts tests also passed; the earlier Git filename-too-long failure did not recur.
  • macOS packages-and-tools: the full job passes on the current head. An earlier ECONNRESET in a production-limit Local Explorer email fixture did not recur; its focused test also passes locally. No unrelated fixture code was changed.
  • Workers Vitest on Windows: the two local Artifacts fixture tests pass locally on macOS, but the Windows packages-and-tools job did not execute this targeted Vitest-plugin file. Add Windows coverage or run it there before claiming this entry point is verified on Windows.
  • In-flight crash recovery: test a process exit during a native Git write/import and fix any incomplete repository or metadata state. The existing abrupt-exit test covers only a previously committed operation.
  • Deployed-service contract: obtain an authorized, exclusively reserved disposable namespace and run the opt-in comparison. Resolve observable differences in error responses, Blob values, retained repository handles, HEAD/refs/paths, pagination and Git protocol variants; the local comparator alone does not establish parity.
  • Default routing and Git dependency: decide how existing implicit-remote users migrate to local by default, and whether requiring native git is acceptable for the initial release; adjust behavior and documentation to match those decisions.
  • Public docs: the in-repository local guide now covers checkout setup, Git token prompts, dynamic dev ports, Windows CI coverage, import safety, and limitations. Publish external docs for explicit remote opt-in, native Git, HTTPS import access and local/remote separation once the default-routing and Git-dependency decisions are settled.

REST/dashboard emulation, billing/events, SSH and Git LFS are outside this PR's scope.


@changeset-bot

changeset-bot Bot commented Oct 6, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: bb926c2

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 13 packages
Name Type
miniflare Minor
wrangler Minor
@cloudflare/workers-utils Minor
@cloudflare/deploy-helpers Patch
@cloudflare/pages-shared Patch
@cloudflare/remote-bindings Patch
@cloudflare/runtime-types Patch
@cloudflare/vite-plugin Patch
@cloudflare/vitest-plugin Patch
@cloudflare/autoconfig Patch
@cloudflare/cli-shared-helpers Patch
@cloudflare/containers-shared Patch
@cloudflare/workers-auth Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@pkg-pr-new

pkg-pr-new Bot commented Oct 6, 2026 •

Copy link
Copy Markdown
@cloudflare/autoconfig

npm i https://pkg.pr.new/@cloudflare/autoconfig@16092

@cloudflare/build-output-utils

npm i https://pkg.pr.new/@cloudflare/build-output-utils@16092

@cloudflare/codemods

npm i https://pkg.pr.new/@cloudflare/codemods@16092

@cloudflare/config

npm i https://pkg.pr.new/@cloudflare/config@16092

@cloudflare/containers-shared

npm i https://pkg.pr.new/@cloudflare/containers-shared@16092

create-cloudflare

npm i https://pkg.pr.new/create-cloudflare@16092

@cloudflare/deploy-helpers

npm i https://pkg.pr.new/@cloudflare/deploy-helpers@16092

@cloudflare/kv-asset-handler

npm i https://pkg.pr.new/@cloudflare/kv-asset-handler@16092

miniflare

npm i https://pkg.pr.new/miniflare@16092

@cloudflare/pages-functions

npm i https://pkg.pr.new/@cloudflare/pages-functions@16092

@cloudflare/pages-shared

npm i https://pkg.pr.new/@cloudflare/pages-shared@16092

@cloudflare/runtime-types

npm i https://pkg.pr.new/@cloudflare/runtime-types@16092

@cloudflare/unenv-preset

npm i https://pkg.pr.new/@cloudflare/unenv-preset@16092

@cloudflare/vite-plugin

npm i https://pkg.pr.new/@cloudflare/vite-plugin@16092

@cloudflare/vitest-plugin

npm i https://pkg.pr.new/@cloudflare/vitest-plugin@16092

@cloudflare/workers-auth

npm i https://pkg.pr.new/@cloudflare/workers-auth@16092

@cloudflare/workers-editor-shared

npm i https://pkg.pr.new/@cloudflare/workers-editor-shared@16092

@cloudflare/workers-utils

npm i https://pkg.pr.new/@cloudflare/workers-utils@16092

wrangler

npm i https://pkg.pr.new/wrangler@16092

commit: bb926c2

@leostera
leostera force-pushed the lostera/artifacts-local-mode branch 4 times, most recently from 686d9f1 to 64a5a33 Compare October 7, 2026 21:51
@leostera
leostera marked this pull request as ready for review October 8, 2026 12:52
@workers-devprod
workers-devprod requested review from a team and dario-piotrowicz and removed request for a team October 8, 2026 12:52
@workers-devprod

Copy link
Copy Markdown
Contributor

Codeowners approval required for this PR:

  • @cloudflare/wrangler
Show detailed file reviewers
  • .changeset/local-artifacts-binding.md: [@cloudflare/wrangler]
  • packages/miniflare/scripts/build.mjs: [@cloudflare/wrangler]
  • packages/miniflare/src/index.ts: [@cloudflare/wrangler]
  • packages/miniflare/src/plugins/artifacts/LOCAL_DEVELOPMENT.md: [@cloudflare/wrangler]
  • packages/miniflare/src/plugins/artifacts/README.md: [@cloudflare/wrangler]
  • packages/miniflare/src/plugins/artifacts/controller.ts: [@cloudflare/wrangler]
  • packages/miniflare/src/plugins/artifacts/git-client.ts: [@cloudflare/wrangler]
  • packages/miniflare/src/plugins/artifacts/git-sidecar.ts: [@cloudflare/wrangler]
  • packages/miniflare/src/plugins/artifacts/index.ts: [@cloudflare/wrangler]
  • packages/miniflare/src/plugins/artifacts/storage.ts: [@cloudflare/wrangler]
  • packages/miniflare/src/plugins/shared/index.ts: [@cloudflare/wrangler]
  • packages/miniflare/src/workers/artifacts/binding.worker.ts: [@cloudflare/wrangler]
  • packages/miniflare/test/config/v4-convert.spec.ts: [@cloudflare/wrangler]
  • packages/miniflare/test/plugins/artifacts/contract.spec.ts: [@cloudflare/wrangler]
  • packages/miniflare/test/plugins/artifacts/contract/README.md: [@cloudflare/wrangler]
  • packages/miniflare/test/plugins/artifacts/contract/compare.ts: [@cloudflare/wrangler]
  • packages/miniflare/test/plugins/artifacts/contract/live.ts: [@cloudflare/wrangler]
  • packages/miniflare/test/plugins/artifacts/contract/scenarios.ts: [@cloudflare/wrangler]
  • packages/miniflare/test/plugins/artifacts/https-import.spec.ts: [@cloudflare/wrangler]
  • packages/miniflare/test/plugins/artifacts/index.spec.ts: [@cloudflare/wrangler]
  • packages/miniflare/test/plugins/artifacts/lifecycle.spec.ts: [@cloudflare/wrangler]
  • packages/miniflare/test/plugins/artifacts/storage.spec.ts: [@cloudflare/wrangler]
  • packages/remote-bindings/src/maybe-start-or-update-session.test.ts: [@cloudflare/wrangler]
  • packages/vite-plugin-cloudflare/src/tests/artifacts-local.spec.ts: [@cloudflare/wrangler]
  • packages/vitest-plugin/test/artifacts-local.test.ts: [@cloudflare/wrangler]
  • packages/workers-utils/src/config/binding-local-support.ts: [@cloudflare/wrangler]
  • packages/workers-utils/src/print-bindings.ts: [@cloudflare/wrangler]
  • packages/workers-utils/tests/config/binding-local-support.test.ts: [@cloudflare/wrangler]
  • packages/workers-utils/tests/print-bindings.test.ts: [@cloudflare/wrangler]
  • packages/wrangler/src/tests/dev/artifacts-local.test.ts: [@cloudflare/wrangler]
  • packages/wrangler/src/tests/dev/remote-bindings.test.ts: [@cloudflare/wrangler]
  • packages/wrangler/src/tests/print-bindings.test.ts: [@cloudflare/wrangler]
  • packages/wrangler/src/dev/miniflare/index.ts: [@cloudflare/wrangler]
  • turbo.json: [@cloudflare/wrangler]

@leostera
leostera force-pushed the lostera/artifacts-local-mode branch from 264a046 to d64199e Compare October 8, 2026 12:54

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 9 potential issues.

Devin Review

Comment on lines +404 to +406
await this.backend(repository, { action: "delete" });
delete repositories[name.toLowerCase()];
await this.saveRepos(repositories);

@devin-ai-integration devin-ai-integration Bot Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Failed cleanup strands a deleted repository

If Git cleanup fails, delete rejects after removing the repository from persistent metadata. A retry returns false, leaving its Git data inaccessible.

Learn more

The namespace stores repository metadata in a Durable Object, while a separate Git sidecar owns the repository directory. Saving the metadata removal before calling the sidecar makes an I/O failure in retireRepository leave the two stores out of sync. A failed rename leaves the Git directory at its original location, while a failed recursive removal can leave it in a retired directory. Either way, the caller receives an error but delete no longer finds the name for a retry.

Example: A Windows process temporarily holds a handle to repo.git; directory retirement fails with EPERM. The delete call rejects, but the next delete("repo") returns false and get("repo") fails while the original Git directory remains.

Recommended fix: Define a recoverable deletion state in LocalArtifactsNamespaceObject, or restore the metadata when the sidecar fails before renaming. For failures after a successful rename, finish deletion asynchronously or persist a pending-cleanup record so retries and startup can complete it without reporting an unsuccessful deletion of an already removed repository.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 9c03e0d: deletion commits metadata removal before touching the Git directory, so a failed metadata write leaves both metadata and contents intact. Git cleanup atomically retires the directory before removing files; restart cleans an interrupted retirement, and same-name creation handles an orphan left before retirement. Added metadata-failure and interrupted-cleanup regressions.

Comment thread packages/miniflare/src/plugins/artifacts/controller.ts
Comment on lines +630 to +637
function createRepoHandle(
namespace: LocalArtifactsNamespaceObject,
name: string
): LocalArtifactsRepo {
const read = <T>(request: BackendRequest): Promise<T> =>
publicCall(async () =>
namespace.backend<T>(await namespace.getRepo(name), request)
);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Deleted repository handles revive after recreation

After deletion and recreation under the same name, createRepoHandle resolves the replacement by name. A retained handle now reads or mutates the new repository.

Learn more

A repository handle stores only its name, and its operations call getRepo each time. Deleting the repository invalidates the handle temporarily, but recreating the name gives it access to the replacement with a different ID. This changes the identity of an existing object without a new get() call.

Example: Keep a handle for demo, delete demo, then create a fresh demo. Calling oldHandle.info() returns the new repository and oldHandle.createToken() creates a token for it.

Recommended fix: Bind each handle to the repository ID returned at get() time and check the ID on every operation; reject it after deletion or recreation.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 9c03e0d: a repository handle now captures its repository ID at get() and checks that ID for every read and mutation. The regression test retains a handle across delete/recreate and verifies it cannot inspect or mint tokens for the replacement.

Comment thread packages/miniflare/src/plugins/artifacts/git-client.ts
Comment thread packages/miniflare/src/plugins/artifacts/git-client.ts Outdated
Comment thread packages/miniflare/src/plugins/artifacts/git-sidecar.ts Outdated
Comment on lines +923 to +929
if (sourceUrl.protocol !== "https:") {
throw new ArtifactsError(
"INVALID_INPUT",
"Invalid source.url: must be an HTTPS URL."
);
}
return { source, target, sourceUrl };

@devin-ai-integration devin-ai-integration Bot Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟥 Git imports reach private developer-network hosts

When a Worker supplies a private HTTPS URL, import() passes it to native Git without network restrictions. The Worker can probe services reachable from the developer's machine.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an explicit HTTPS import requested by Worker code running in local development; private HTTPS Git origins are supported, and binding selection itself does not make a network request. We clarified that Git contacts hosts reachable from the developer machine, including private-network hosts, in the local development guide (SDK commit 9c03e0d and companion draft docs PR #34106). A general private-destination policy would need a separate product/security decision; this comment is not being represented as an automatic block or as deployed-service parity.

Comment thread packages/miniflare/src/plugins/artifacts/git-client.ts
Comment thread packages/miniflare/src/plugins/artifacts/git-client.ts
@leostera

leostera commented Oct 9, 2026

Copy link
Copy Markdown
Collaborator Author

Closing this full-stack PR for now to avoid duplicate review while the split stack is reviewed. Part 1 is #16176; the simulator and development-tools branches remain pushed and will be proposed against cloudflare/workers-sdk:main separately. This PR and its review history are preserved and can be reopened if needed. No code was merged from this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants