feat(blob): optional blob event capture to files (Event Grid schema) - #2714
Open
The3G wants to merge 19 commits into
Open
feat(blob): optional blob event capture to files (Event Grid schema)#2714The3G wants to merge 19 commits into
The3G wants to merge 19 commits into
Conversation
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Thread IBlobEventSink from BlobServer/SqlBlobServer through BlobRequestListenerFactory into all five emitting handlers (AppendBlob, BlockBlob, Blob, PageBlob, Container); add eventSink lifecycle (init/close) to beforeStart/afterClose in both servers. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A SAS-authenticated request carries its credential in the query string (e.g. `?...&sig=...`). BlobEventFactory persisted `request.getUrl()` verbatim, which for Express is path + full query, so the SAS signature could be written to a plaintext event file on disk. Strip the query before storing: this removes the credential-leak-to-disk risk and also matches real Azure Storage events, whose `data.url` is the bare blob URL. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The captured event files are meant to be consumed by an external processor watching the folder. A bare writeFile creates the directory entry before its contents are flushed, so a consumer (or a fast poller) can read an empty/partial *.json file. Write to a <name>.json.tmp first and rename it into place; rename within one filesystem is atomic, so a *.json file is only ever observed complete. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Poll until the expected event appears instead of a single fixed 200ms sleep, so the tests are reliable under load and on the slower @SQL path. - Name-scope the ContainerCreated filter (was order-dependent) and assert the matching ContainerDeleted event. - Assert data.url is present and query-free (end-to-end complement to the SAS query-strip fix). - Skip not-yet-complete files in the folder reader defensively. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Document --blobEventCapture / --blobEventCapturePath in the command line options section and the azurite.blobEventCapture / azurite.blobEventCapturePath VS Code settings, matching the existing house style. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The azurite.blobEventCapture / azurite.blobEventCapturePath settings were declared in package.json, documented, and implemented in VSCEnvironment, but VSCServerManagerBlob never passed them to BlobConfiguration, so the VS Code toggle was a silent no-op. Thread them through, and extract the CLI factory's path-resolution into a shared, unit-tested helper (resolveBlobEventCapturePath) so both entry points behave identically. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds an opt-in “blob event capture” feature that records mutating blob/container operations as Azure Event Grid–schema JSON files on disk, wired through both CLI and VS Code entry points and both LokiJS/SQL backends.
Changes:
- Introduces Event Grid–shaped blob event model + factory and a file-based sink with atomic publish semantics.
- Plumbs an optional event sink through the blob request pipeline and emits events from mutating handlers.
- Adds unit/E2E tests plus CLI/VS Code configuration surface and documentation updates.
Reviewed changes
Copilot reviewed 30 out of 30 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tests/BlobTestServerFactory.ts | Extends test server factory to enable/route blob event capture in tests. |
| tests/blob/resolveBlobEventCapturePath.test.ts | Unit tests for capture-path resolution behavior. |
| tests/blob/FileBlobEventSink.test.ts | Unit tests for file sink atomic writes, filename safety, and self-disable. |
| tests/blob/BlobEventFactory.test.ts | Unit tests for event envelope/schema, SAS query stripping, and sequencer behavior. |
| tests/blob/apis/eventCapture.test.ts | E2E coverage validating events are written when enabled and not written by default. |
| src/common/VSCServerManagerBlob.ts | Wires VS Code settings into blob server configuration for event capture. |
| src/common/VSCEnvironment.ts | Adds VS Code settings accessors for blob event capture flags/path. |
| src/common/Environment.ts | Adds CLI flags for blob event capture and capture path. |
| src/blob/utils/constants.ts | Adds default __blobevents__ folder constant. |
| src/blob/SqlBlobServer.ts | Creates/initializes/closes the optional file event sink in SQL backend. |
| src/blob/SqlBlobConfiguration.ts | Extends SQL config to carry capture enable/path settings. |
| src/blob/IBlobEnvironment.ts | Adds environment interface accessors for capture enable/path. |
| src/blob/handlers/PageBlobHandler.ts | Emits events for page blob create/page writes; plumbs sink into handler base. |
| src/blob/handlers/ContainerHandler.ts | Emits container create/delete events; plumbs sink into handler base. |
| src/blob/handlers/BlockBlobHandler.ts | Emits events for PutBlob/PutBlock/PutBlockList operations. |
| src/blob/handlers/BlobHandler.ts | Emits blob delete events; plumbs sink into handler base. |
| src/blob/handlers/BaseHandler.ts | Adds common emitBlobEvent() helper to safely publish events. |
| src/blob/handlers/AppendBlobHandler.ts | Emits events for append blob create/append writes. |
| src/blob/events/resolveBlobEventCapturePath.ts | Shared logic to resolve effective capture path for CLI + VS Code. |
| src/blob/events/IBlobEventSink.ts | Defines sink interface (init/emit/close) with non-throwing contract. |
| src/blob/events/IBlobEvent.ts | Defines Event Grid–shaped event types and payload structure. |
| src/blob/events/FileBlobEventSink.ts | Implements file sink with sanitization + temp-write + rename publish. |
| src/blob/events/BlobEventFactory.ts | Builds Event Grid–shaped events and strips SAS query strings from URLs. |
| src/blob/BlobServerFactory.ts | Wires CLI flags into blob configuration; resolves default capture path and warns on misconfig. |
| src/blob/BlobServer.ts | Creates/initializes/closes the optional file event sink in LokiJS backend. |
| src/blob/BlobRequestListenerFactory.ts | Plumbs the optional sink into handler construction. |
| src/blob/BlobEnvironment.ts | Adds blob-specific CLI flags accessors for capture enable/path. |
| src/blob/BlobConfiguration.ts | Extends blob config to carry capture enable/path settings. |
| README.md | Documents new CLI flags and VS Code settings and the on-disk schema/behavior. |
| package.json | Adds VS Code extension settings azurite.blobEventCapture and azurite.blobEventCapturePath. |
Suppressed comments (1)
src/blob/BlobRequestListenerFactory.ts:122
ServiceHandleris created without theeventSink, soServiceSubmitBatch(which delegates toBlobBatchHandler) will not emit captured events even when blob event capture is enabled. This breaks the stated goal of capturing mutating blob operations, since batch sub-requests can include create/delete/put operations.
serviceHandler: new ServiceHandler(
this.accountDataStore,
this.oauth,
this.metadataStore,
this.extentStore,
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 30 out of 30 changed files in this pull request and generated no new comments.
Suppressed comments (1)
src/blob/events/BlobEventFactory.ts:17
sequencerCounteris anumber, so after ~9e15 events it will lose integer precision andnextSequencer()can stop being strictly monotonic. Sincesequenceris intended to reflect ordering, usingbigintavoids precision loss in long-running/high-throughput scenarios.
let sequencerCounter = 0;
function nextSequencer(): string {
sequencerCounter += 1;
return sequencerCounter.toString(16).padStart(64, "0");
}
Author
|
@microsoft-github-policy-service agree |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an opt-in feature that captures mutating blob operations as Azure Event Grid–schema JSON files on disk, so they can be processed later by external tooling. Controlled entirely by a command-line / configuration switch and off by default.
--blobEventCaptureboolean flag (off by default) plus optional--blobEventCapturePath <dir>(defaults to<location>/__blobevents__); mirrored by VS Code settingsazurite.blobEventCapture/azurite.blobEventCapturePath.${eventTime}-${id}.jsonusing the Event Grid storage event schema (eventType,subject,data.api,data.url,data.eTag,data.contentLength,data.blobType,sequencer, …).BlobServerandSqlBlobServer) and both entry points (CLIBlobServerFactoryand the VS Code extension).Implementation notes
<name>.json.tmpthenrename()), so consumers never observe a partially written file.data.urlhas its query string stripped, so SAS credentials are never persisted to disk.Test plan
npm run test:blob(LokiJS suite): 543 passing, 0 failing, 3 pendingBlobEventFactory(schema + SAS stripping),FileBlobEventSink(atomic write + self-disable),resolveBlobEventCapturePathtests/blob/apis/eventCapture.test.ts: container create/delete, put/delete blob, put block + block list, and disabled-by-default (folder never created)npx eslint src/**/*.ts: cleanNotes
The feature is entirely off unless explicitly enabled, so existing behaviour is unchanged by default. Configuring a capture path without enabling capture is a no-op (a warning is logged and the path is ignored).