Skip to content

Added the CHASM stream component. - #4

Open
moedash wants to merge 7 commits into
moe/AI-198-srv-1-api-pinfrom
moe/AI-198-srv-2-stream-component
Open

moedash wants to merge 7 commits into
moe/AI-198-srv-1-api-pinfrom
moe/AI-198-srv-2-stream-component

Conversation

@moedash

@moedash moedash commented Sep 25, 2026 •

Copy link
Copy Markdown
Owner

This PR adds the CHASM stream component.

A stream is a CHASM component holding records in batches, with a frontier, producer cursors and consumer pins. It lives next to the workflow that owns it, which is what lets a publish ride the workflow task's own commit. Nothing wires it up yet: there is no service, no command and no delivery here, only the component and its unit tests.

What changed?

  • chasm/lib/stream/stream.go: the component. Appends, batching, the frontier, producer dedup by (producer, attempt, sequence), closing, truncation with a floor no active consumer can be truncated past, the cap that reclaims the oldest batches, and the read window a consumer pages through. A filtered read advances only past the messages it examined, so a page whose batches did not reach the window end cannot skip unexamined offsets.
  • chasm/lib/stream/cursor.go: the consumer cursor, held by the reader rather than the stream, so folding in a delivered range commits with the event that records it.
  • chasm/lib/stream/config.go: the caps as namespace-scoped dynamic config, with the old constants as defaults, plus a byte and item budget for a stream a workflow owns that sits well under the mutable state limit.
  • chasm/lib/stream/messages.go: the byte and count caps on what one read returns.
  • chasm/lib/stream/proto/v1/{message,stream_state,tasks}.proto and their generated Go: the record, the batch, the component's state and the two side-effect tasks.
  • Unit tests for all of the above.

Why?

The payload lives in the component, not in History and not in a log table. That is the property everything above this layer depends on, so it is worth reading before the wiring.

How did you test it?

go build ./...
make proto            # tree stays clean, generated files match the three protos
make fmt              # tree stays clean
go test ./chasm/lib/stream/... -count=1

All green.

Series

Server 2 of 6. Replaces #2.

Previous: #3 moe/AI-198-srv-1-api-pin. Next: #5 moe/AI-198-srv-3-stream-service.

Review round

checkBudget compared an absolute head offset against an item count, so a stream whose floor had moved, or one created above zero, read as over budget the moment it existed. It measures held records now, which is head minus base. The close reason on Terminate goes through payload.EncodeString, so an SDK data converter can read it, and the record kind is settled on the copy the store serializes rather than through the caller's protos.

Known limits, argued rather than fixed:

  • While a workflow consumer is registered, StreamLifecycle.max_items stops being a rolling window. The consumer's floor sits where it subscribed, because replay re-reads every range its History recorded back to that point, so the cap behaves as a lifetime quota and appends past it are refused instead of reclaiming behind the consumer. The two promises cannot both hold and the one kept is that a workflow can replay a decision it already made. Deregistering the consumer releases it.
  • The producer dedup table keeps one entry per producer id, so a producer has one append in flight at a time. Pipelining needs a distinct id per lane. The stale-sequence error says so.
  • ForgetConsumer releases the floor when a run closes. A closed run takes no further task, but it can still be queried and it can be the base of a reset, and neither is served once truncation has taken the bytes. Both are refused with the offset the stream now starts at.
  • Every limit in Limits treats zero as "use the default", so none can be configured to zero. Turning streams off for a namespace is what stream.enabled is for.

Review round: api surface

The public StreamRecord.sequence no longer reserves -1 for a record whose producer does not number its records, because an unset int64 reads as zero. The stored record mirrors the public one field for field, so its comment and the tests that leaned on the sentinel follow.

Brought onto current main (d8f9c6d86b2c) by merge, which came through without conflicts and needed no code change on this layer.

Records live in the component next to the workflow that owns them, so a publish rides the same commit. Retention, truncation and the consumer floor are the component's own state, and the caps that bound resource use come from namespace-scoped dynamic config.
The item budget compared an absolute head offset against a count, so a stream
whose floor had moved, or one created above zero, was over budget the moment
it existed. The close reason now carries its encoding, and the record kind is
settled on a copy rather than through the caller's protos.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant