Conversation
`make fmt` left these two files dirty, which fails the `fmt` job. The proto also carried a truncated comment for a field that is no longer there.
The stream log methods on `ExecutionStore` had no fault-injection or telemetry wrapper, so `make go-generate` left the tree dirty.
Both were rewritten to point at `add_stream_log`, which dropped `nexus_incoming_services` from the Cassandra upgrade path and `tasks_v2` from the SQLite one. A released version has to stay what it was.
Cassandra goes to 1.14, SQLite to 0.12, MySQL and PostgreSQL to 1.20. The MySQL and PostgreSQL stores already query `stream_log`, so their base schema needed the table too.
It holds one cluster per arm for the whole run, which outlasts the dedicated pool and parks the functional suite until the 35 minute timeout.
Three placeholders carried a trailing digit. MySQL connections interpolate parameters, so the digit was appended to each value and a read returned batches past the offset it asked for.
The orphan comment dropped from the `.proto` still rode the generated field, which leaves `make proto` dirty.
The `api-go` branch was rebased onto its upstream `main`, which pulls in newer gRPC and OpenTelemetry requirements through the module graph.
The new api pin ships the nexusoperation proto, so the generated import map has to list it.
The module pins its own grpc version and has to track the root module after the api pin bumped it.
The mixed brain tests live in their own module, so the root go test invocation cannot reach them.
The metric SDK returns a nil slice when no exemplar was recorded and an empty one when the reservoir was already allocated, which made the comparison flaky.
The log is appended to, so a failure from an earlier run kept failing every later verification.
Taken from upstream temporalio/temporal temporalio#11906. The api version this branch pins rejects a system payload that is not binary/protobuf, so the endpoint has to refuse one instead of labelling it.
Payloads live in the CHASM component, so the table, the store methods, the four schema bumps and the scavenger marker had no caller on the serving path. The internal protos reserve the field numbers the log addressing used.
The completion path reached the stream with the request context's engine, which only serves shards this host owns. Registration now resolves its start offset in one transition, and routed calls made under the workflow lock carry a bounded deadline.
Cold replay walks every page of history for the recorded ranges instead of stopping at the first. A range that is truncated, deleted or over the replay budget fails the task with its own cause rather than handing matching a retry. Slices now name the run that holds the stream.
A refused publish or subscribe came back as a plain error, which failed the completion call and left the worker retrying the same command with nothing in History saying why.
The publish commits with the task, so a task that fails after publishing has to leave no event, no offset and no message behind.
Delivery to a workflow drops anything that is not data, so a producer leaving the field unset got an offset for a message no subscriber ever saw.
The batch and the frontier commit in one transition and the execution serializes transitions, so the pin only made an outside producer fail when the workflow published in between. The stripe lock and the separate create step went with it.
A pin keyed by workflow id alone outlived its run, refused a later run of the same id and kept the notify task pushing at closed workflows. The push now reports a closed or continued run, so the stream drops or re-keys the pin, and appends coalesce into one outstanding notify task.
Retention now probes the consumers of a closed stream and re-arms while one is running, and DeleteStream refuses unless force is set. The truncation test also asserts the floor from the moment of subscribing, which is when it is taken.
The page-level fallback moved the reader to the end of a window whose batches did not reach it. The message loop already advances past every filtered message it examined, which is all the advance a filtered page is owed.
A stream id becomes an execution's business id and a stream name a key in mutable state, so both are bounded like workflow ids before the request is routed.
Hand-written lines now fit in 100 characters, the stream handlers all tag their context with the caller, the comments describe the current behaviour without history, and the SDK validation host is opt-in like the other tooling.
The api names an entry a record and carries its kind, producer, attempt and sequence on the wire, so the store keeps those fields and hands them back on the Workflow Task instead of dropping everything but the body. The workflow's own publish clears the producer id, which is how a reader tells its records from an outside producer's.
A workflow reads a topic before anything has been written to it, and the name it subscribes with is the stream an outside producer later appends to. A subscribe that names neither an owned stream nor a standalone one now creates the owned stream instead of failing the task.
…ages meet. The importas rule reserves the pb suffix for go.temporal.io/api packages, so the internal chasm package takes the streamlib alias the rest of the tree already uses for it.
The race test asks the env for a context from eight goroutines at once, and the unguarded append was a data race. Under the race detector that fails the test, and every parallel test running at that moment fails with it, which is how the standalone Nexus and activity parity tests went red on the same shard.
…ervice. Without an entry in the method table the default authorizer denied every stream RPC and the no-op one admitted any namespace caller. Each method now declares namespace scope with read, write or admin access, has a rate-limit priority, and is forwarded to the namespace's active cell like a workflow RPC.
A query dispatched straight through matching is built without RecordWorkflowTaskStarted, so nothing re-supplied the ranges its history recorded and a cold worker could not replay a consuming workflow to answer. Matching now asks History for them through a new internal RPC after it has fetched the history.
A reset run is rebuilt from events, and the cursors were CHASM state the events never touched, so the new run had no subscription and could not replay the ranges its copied history records. The rebuild now recreates cursors from the events, the reset copies what only the base run knows and gives the new run streams of its own from the inherited offset, and replay reads each range from the run that holds it.
…odes. State-based replication carries the stream, its batches and the cursor to the standby, and the workflow goes on consuming and publishing there. Event-based replication rebuilds only the cursor from the recorded ranges, so the second suite pins that the stream does not follow the namespace yet.
A reset run is rebuilt from events, and the cursors were CHASM state the events never touched, so the new run had no subscription. The rebuild now recreates cursors from the events, the reset copies what only the base run knows, and replay reads each range from the run that holds it.
State-based replication carries the stream, its batches and the cursor to the standby, and the workflow goes on there. Event-based replication rebuilds only the cursor from the recorded ranges, so the second suite pins that the stream does not follow the namespace yet.
The merge adds no changes; it gives the chain the original head as an ancestor so downstream pins and merges reconcile.
This was referenced Sep 25, 2026
Its stream continues the base run's offset space, which the item budget read as a stream already full. The event-based failover test now runs a task on the new active cluster and asserts what that task sees, and the publish side of a reset is characterized rather than left unstated.
The event bounds its batch with a from and a to offset like every other range in this API.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR carries stream subscriptions through a workflow reset and characterizes a stream under namespace failover.
A reset run is rebuilt from events, and the cursors were CHASM state the events never touched, so the new run had no subscription and could not replay the ranges its copied history records.
What changed?
InheritStreamsOnResetcopies what only the base run knows. A subscription the events never mentioned starts where it started in the base run. A cursor on a stream in another execution keeps reading that stream and is given the frontier the base run last knew. A cursor on a stream the base run owned gets a stream of this run's own, starting at the offset the cursor stands at, so the offset space continues and a range in this run's History is unambiguous about which run holds it.tests/xdc/stream_failover_test.gopins what happens under namespace failover in both replication modes. State-based replication carries the stream, its batches and the cursor to the standby, and the workflow goes on consuming and publishing there. Event-based replication rebuilds only the cursor from the recorded ranges, so the second suite pins that the stream does not follow the namespace yet.19b4452be899, the head of Added server-side streams for workflows and external clients. #2. It adds no tree change. It gives this chain the original head as an ancestor, so the downstream PRs that pin or branch off that sha keep correct diffs and merge cleanly once this chain lands.Why?
Reset is where the two halves of a subscription, the events and the CHASM state, have to be reconciled, and failover is where the honest answer is that state-based replication carries a stream and event-based does not. Both are characterizations, so they read better after the behaviour they characterize.
How did you test it?
Series
Server 6 of 6. Replaces #2.
Previous: #7
moe/AI-198-srv-5-delivery-replay. Next: none, this is the tip of the server chain.Review round
A reset run's stream continues the base run's offset space, so it is born with a head offset well above zero while holding nothing. The item budget read that head as a stream already full, which refused every publish on the reset run once the inherited cursor stood past
stream.ownedStreamMaxItems. The budget measures held records now, fixed in #4, andTestAnInheritedStreamIsNotBornOverItsItemBudgetcovers the reset shape directly.TestStreamDoesNotReplicateUnderEventBasedReplicationYetused to assert that the records did not replicate and stop there, while its comment claimed something about the workflow's first task on the new cluster. It now drives that task and asserts what it sees: aWorkflowTaskFailedwithSTREAM_RANGE_UNAVAILABLEnaming the cursor whose stream the cluster does not hold. The cause comes from the #7 change; before it the task failed with a bare error and matching retried forever with nothing in History.TestResetOfAPublisherRestartsItsStreamAtZerocharacterizes the publish side of a reset, which had no coverage: a workflow that only publishes holds no cursor, so its reset run gets a stream starting at offset zero while the copied history still names the base run's offsets. The two appended events then describe different records at the same offset.Known limits, argued rather than fixed:
InheritStreamsOnResetsays so.Review round: api surface
The reset test reads the appended event's range through
GetFromOffset(). Everything else on this branch comes up through the merge.Brought onto current main (
d8f9c6d86b2c) by merge, which came through without conflicts and needed no code change on this layer.