Skip to content

feat(blob): optional blob event capture to files (Event Grid schema) - #2714

Open
The3G wants to merge 19 commits into
Azure:mainfrom
The3G:feature/blob-event-capture
Open

feat(blob): optional blob event capture to files (Event Grid schema)#2714
The3G wants to merge 19 commits into
Azure:mainfrom
The3G:feature/blob-event-capture

Conversation

@The3G

@The3G The3G commented Aug 6, 2026

Copy link
Copy Markdown

Summary

Adds an opt-in feature that captures mutating blob operations as Azure Event Grid–schema JSON files on disk, so they can be processed later by external tooling. Controlled entirely by a command-line / configuration switch and off by default.

  • New --blobEventCapture boolean flag (off by default) plus optional --blobEventCapturePath <dir> (defaults to <location>/__blobevents__); mirrored by VS Code settings azurite.blobEventCapture / azurite.blobEventCapturePath.
  • Captures all mutating blob/container operations (container create/delete, put/delete blob, put block list, page/append writes, …), emitting one JSON file per event named ${eventTime}-${id}.json using the Event Grid storage event schema (eventType, subject, data.api, data.url, data.eTag, data.contentLength, data.blobType, sequencer, …).
  • Writes are fire-and-forget and never fail a storage operation; the sink self-disables if it cannot initialize (mirrors the existing Telemetry precedent).
  • Wired through both backends (LokiJS BlobServer and SqlBlobServer) and both entry points (CLI BlobServerFactory and the VS Code extension).

Implementation notes

  • Event files are published atomically (write <name>.json.tmp then rename()), so consumers never observe a partially written file.
  • The captured data.url has its query string stripped, so SAS credentials are never persisted to disk.
  • Event filename segments are sanitized to prevent path traversal.

Test plan

  • npm run test:blob (LokiJS suite): 543 passing, 0 failing, 3 pending
  • New unit tests: BlobEventFactory (schema + SAS stripping), FileBlobEventSink (atomic write + self-disable), resolveBlobEventCapturePath
  • New end-to-end test tests/blob/apis/eventCapture.test.ts: container create/delete, put/delete blob, put block + block list, and disabled-by-default (folder never created)
  • npx eslint src/**/*.ts: clean
  • README updated with the new CLI flags and VS Code settings

Notes

The feature is entirely off unless explicitly enabled, so existing behaviour is unchanged by default. Configuring a capture path without enabling capture is a no-op (a warning is logged and the path is ignored).

The3G and others added 18 commits August 6, 2026 11:35
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Thread IBlobEventSink from BlobServer/SqlBlobServer through
BlobRequestListenerFactory into all five emitting handlers
(AppendBlob, BlockBlob, Blob, PageBlob, Container); add eventSink
lifecycle (init/close) to beforeStart/afterClose in both servers.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A SAS-authenticated request carries its credential in the query string
(e.g. `?...&sig=...`). BlobEventFactory persisted `request.getUrl()`
verbatim, which for Express is path + full query, so the SAS signature
could be written to a plaintext event file on disk. Strip the query
before storing: this removes the credential-leak-to-disk risk and also
matches real Azure Storage events, whose `data.url` is the bare blob URL.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The captured event files are meant to be consumed by an external
processor watching the folder. A bare writeFile creates the directory
entry before its contents are flushed, so a consumer (or a fast poller)
can read an empty/partial *.json file. Write to a <name>.json.tmp
first and rename it into place; rename within one filesystem is atomic,
so a *.json file is only ever observed complete.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Poll until the expected event appears instead of a single fixed 200ms
  sleep, so the tests are reliable under load and on the slower @SQL path.
- Name-scope the ContainerCreated filter (was order-dependent) and assert
  the matching ContainerDeleted event.
- Assert data.url is present and query-free (end-to-end complement to the
  SAS query-strip fix).
- Skip not-yet-complete files in the folder reader defensively.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Document --blobEventCapture / --blobEventCapturePath in the command line
options section and the azurite.blobEventCapture / azurite.blobEventCapturePath
VS Code settings, matching the existing house style.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The azurite.blobEventCapture / azurite.blobEventCapturePath settings were
declared in package.json, documented, and implemented in VSCEnvironment,
but VSCServerManagerBlob never passed them to BlobConfiguration, so the
VS Code toggle was a silent no-op. Thread them through, and extract the
CLI factory's path-resolution into a shared, unit-tested helper
(resolveBlobEventCapturePath) so both entry points behave identically.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings August 6, 2026 10:36

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds an opt-in “blob event capture” feature that records mutating blob/container operations as Azure Event Grid–schema JSON files on disk, wired through both CLI and VS Code entry points and both LokiJS/SQL backends.

Changes:

  • Introduces Event Grid–shaped blob event model + factory and a file-based sink with atomic publish semantics.
  • Plumbs an optional event sink through the blob request pipeline and emits events from mutating handlers.
  • Adds unit/E2E tests plus CLI/VS Code configuration surface and documentation updates.

Reviewed changes

Copilot reviewed 30 out of 30 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
tests/BlobTestServerFactory.ts Extends test server factory to enable/route blob event capture in tests.
tests/blob/resolveBlobEventCapturePath.test.ts Unit tests for capture-path resolution behavior.
tests/blob/FileBlobEventSink.test.ts Unit tests for file sink atomic writes, filename safety, and self-disable.
tests/blob/BlobEventFactory.test.ts Unit tests for event envelope/schema, SAS query stripping, and sequencer behavior.
tests/blob/apis/eventCapture.test.ts E2E coverage validating events are written when enabled and not written by default.
src/common/VSCServerManagerBlob.ts Wires VS Code settings into blob server configuration for event capture.
src/common/VSCEnvironment.ts Adds VS Code settings accessors for blob event capture flags/path.
src/common/Environment.ts Adds CLI flags for blob event capture and capture path.
src/blob/utils/constants.ts Adds default __blobevents__ folder constant.
src/blob/SqlBlobServer.ts Creates/initializes/closes the optional file event sink in SQL backend.
src/blob/SqlBlobConfiguration.ts Extends SQL config to carry capture enable/path settings.
src/blob/IBlobEnvironment.ts Adds environment interface accessors for capture enable/path.
src/blob/handlers/PageBlobHandler.ts Emits events for page blob create/page writes; plumbs sink into handler base.
src/blob/handlers/ContainerHandler.ts Emits container create/delete events; plumbs sink into handler base.
src/blob/handlers/BlockBlobHandler.ts Emits events for PutBlob/PutBlock/PutBlockList operations.
src/blob/handlers/BlobHandler.ts Emits blob delete events; plumbs sink into handler base.
src/blob/handlers/BaseHandler.ts Adds common emitBlobEvent() helper to safely publish events.
src/blob/handlers/AppendBlobHandler.ts Emits events for append blob create/append writes.
src/blob/events/resolveBlobEventCapturePath.ts Shared logic to resolve effective capture path for CLI + VS Code.
src/blob/events/IBlobEventSink.ts Defines sink interface (init/emit/close) with non-throwing contract.
src/blob/events/IBlobEvent.ts Defines Event Grid–shaped event types and payload structure.
src/blob/events/FileBlobEventSink.ts Implements file sink with sanitization + temp-write + rename publish.
src/blob/events/BlobEventFactory.ts Builds Event Grid–shaped events and strips SAS query strings from URLs.
src/blob/BlobServerFactory.ts Wires CLI flags into blob configuration; resolves default capture path and warns on misconfig.
src/blob/BlobServer.ts Creates/initializes/closes the optional file event sink in LokiJS backend.
src/blob/BlobRequestListenerFactory.ts Plumbs the optional sink into handler construction.
src/blob/BlobEnvironment.ts Adds blob-specific CLI flags accessors for capture enable/path.
src/blob/BlobConfiguration.ts Extends blob config to carry capture enable/path settings.
README.md Documents new CLI flags and VS Code settings and the on-disk schema/behavior.
package.json Adds VS Code extension settings azurite.blobEventCapture and azurite.blobEventCapturePath.
Suppressed comments (1)

src/blob/BlobRequestListenerFactory.ts:122

  • ServiceHandler is created without the eventSink, so ServiceSubmitBatch (which delegates to BlobBatchHandler) will not emit captured events even when blob event capture is enabled. This breaks the stated goal of capturing mutating blob operations, since batch sub-requests can include create/delete/put operations.
      serviceHandler: new ServiceHandler(
        this.accountDataStore,
        this.oauth,
        this.metadataStore,
        this.extentStore,

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/common/VSCServerManagerBlob.ts Outdated
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings August 6, 2026 11:13

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 30 out of 30 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/blob/events/BlobEventFactory.ts:17

  • sequencerCounter is a number, so after ~9e15 events it will lose integer precision and nextSequencer() can stop being strictly monotonic. Since sequencer is intended to reflect ordering, using bigint avoids precision loss in long-running/high-throughput scenarios.
let sequencerCounter = 0;
function nextSequencer(): string {
  sequencerCounter += 1;
  return sequencerCounter.toString(16).padStart(64, "0");
}

@The3G

The3G commented Aug 6, 2026

Copy link
Copy Markdown
Author

@microsoft-github-policy-service agree

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants