Conversation
The stream is reachable over gRPC next to the workflow service: ids, names, offsets and page sizes are bounded on the frontend, every method declares its authorization scope and rate-limit priority, and a call for a namespace active elsewhere is forwarded like a workflow RPC.
This was referenced Sep 25, 2026
Registering the service turned the whole surface on everywhere, so it is off by default now and the calls that destroy other workflows' data need an operator. A workflow's owned streams share one byte budget, because their per-stream budgets multiplied out well past the mutable state limit. The notify task no longer strands its coalescing flag or drops a consumer's pin on a transport error.
…3-stream-service # Conflicts: # service/frontend/service.go
Main now builds under Go 1.27, where go fix rewrites this shape, so make fmt was left dirty on these two files.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR puts the stream behind a gRPC service on the frontend.
StreamServiceis served next toWorkflowService, so an outside producer or reader reaches a stream the same way it reaches a workflow. The history side hosts the component and the two side-effect tasks; the frontend forwards.What changed?
chasm/lib/stream/proto/v1/{service,request_response}.protoand their generated Go: fifteen RPCs covering standalone streams, streams attached to a workflow, and the two calls history makes on itself while resolving a subscription.chasm/lib/stream/service/: the history-side handler, the frontend forwarder, the CHASM library registration, and the retention and notify-consumers task handlers.chasm/lib/stream/gen/streampb/v1/namespace.goexposes each routed request's namespace to the interceptors. A test drives that off the descriptor, so a new RPC fails there instead of silently opting out of rate limits, validation, authorization and redirection.common/api/metadata.go: every method declares namespace scope with read, write or admin access. Without an entry the default authorizer denies every stream RPC and the no-op one admits any namespace caller. The two internal calls are admin so no namespace-level role reaches them through a frontend.service/frontend/configs/quotas.go: a rate-limit priority per method, the two polls counted against concurrent long polls,ListStreamson the visibility limiter.service/frontend/fx.goregisters the stream methods as redirectable, so a call for a namespace active elsewhere is forwarded like a workflow RPC.chasm/lib/workflow/workflow.gogains theStreamsandStreamCursorsmaps and the accessors the service needs to reach a stream attached to a workflow. The commands that use them come in the next layer.cmd/tools/protogen/main.godeletes only.pb.gofiles from agendirectory instead of wiping it.namespace.gohas to sit in the generated package, because a method must be declared in the package of the type it is on.tests/stream_test.go: the RPC surface end to end against a real cluster.Why?
The service is the whole external surface. Splitting it from the component keeps the review of "what can a caller do, and who is allowed to" separate from "what does the component guarantee".
How did you test it?
All green.
Series
Server 3 of 6. Replaces #2.
Previous: #4
moe/AI-198-srv-2-stream-component. Next: #6moe/AI-198-srv-4-workflow-commands.Review round
The service is behind
stream.enabled, a namespace setting that defaults to off. Every RPC resolves its namespace through one place, so the gate covers the whole surface, and the functional suites turn it on.TruncateStreamandDeleteStreamareAccessAdminnow: both destroy records another workflow's History depends on, which is whyDeleteWorkflowExecutionneeds an operator.Admission got the rest of what it was missing. Streams one execution owns share a byte budget,
stream.ownedStreamsMaxBytesPerWorkflow, because the per-stream budget multiplied by the stream count comes to far more thanlimit.mutableStateSize.error, and exceeding that terminates the workflow rather than refusing an append.producer_idandworkflow_idare length-checked like the stream id and name.CreateStreamsettles the lifecycle: retention left unset takes the namespace's, a non-positive one is refused, and one over the namespace's is refused, so a closed stream can no longer sit in the database for good.SubscribeWorkflowrefuses a request that sets both a stream id and a stream name.RegisterStreamConsumeranswersstream_absentinstead of a bareNotFoundwhen no execution holds the stream, so the subscribe fallback fires on that one condition and not on a registry miss or a shard that has moved. A workflow that owns a stream namedxcan no longer also consume a standalone stream with idx: they share one cursor keyspace, and the collision is refused rather than silently handed the wrong cursor.AdvanceConsumerHeadonly matches an external cursor for the same reason.The notify task no longer strands anything.
Validateaccepts unconditionally, since turning the task down left the coalescing flag up with nothing to lower it and no later append would schedule another, so a consumer that subscribed afterwards was never woken.Discardlowers the flag too. The fan-out runs through a bounded worker pool with a per-call deadline instead of one serial loop, the routed request carries its namespace, and a transportNotFoundno longer releases a consumer's replay floor. A truncate refused by a pin probes the runs holding it and tries again, which is the only way a stream nobody appends to finds out that a pin belongs to a finished run.ListStreamsrefuses a query namingTemporalNamespaceDivision. The archetype is a predicate the caller's query replaces rather than one it is anded with, so such a query would list any archetype in the namespace through a stream-scoped read-only RPC.Known limits, argued rather than fixed:
stream.maxConsumeItemsPerTaskis clamped to 1000, the largest page a single stream read returns. The setting's description says so.Review round: api surface
The api branch renamed
AppendStreamRecordsCommandAttributes.stream_idtostream_nameandSubscribeStreamCommandAttributes.stream_idtostream_name_or_id, and gaveWorkflowStreamRecordsAppendedEventAttributesafrom_offsetand ato_offsetin place of a first offset and a count. Field numbers are unchanged, so the wire does not move. The service speaks its own protos and names streams throughStreamNameandStreamIdfields of its own, so this branch carries the merge and needs no adaptation.Brought onto current main (
d8f9c6d86b2c) by merge; the biggest adaptation was taking the stream handler and its tasks toerrors.AsType, plus resolving the stream service wiring inservice/frontend/service.go.