Skip to content

fix(actors): surface unregistered actor startup errors - #5512

Open
NathanFlurry wants to merge 1 commit into
stack/fix-rivetkit-protect-local-serverless-handler-ownership-qvtxsxknfrom
stack/fix-actors-surface-unregistered-actor-startup-errors-sxrxmrpm
Open

fix(actors): surface unregistered actor startup errors#5512
NathanFlurry wants to merge 1 commit into
stack/fix-rivetkit-protect-local-serverless-handler-ownership-qvtxsxknfrom
stack/fix-actors-surface-unregistered-actor-startup-errors-sxrxmrpm

Conversation

@NathanFlurry

@NathanFlurry NathanFlurry commented Jul 30, 2026

Copy link
Copy Markdown
Member

No description provided.

@NathanFlurry

Copy link
Copy Markdown
Member Author

@claude

claude Bot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Review: surface unregistered actor startup errors

Overview

When an actor fails to start because its factory isn't registered (e.g. removed from a serverless handler), guard previously just saw the actor stop and retried the wake signal up to 8 times before eventually returning a generic ActorWakeRetriesExceeded/timeout error. This PR plumbs the real reason through so guard can fail fast with a clear actor.not_registered error instead:

  • rivet-envoy-protocol: adds ActorErrorEnvelope + encode_actor_error/decode_actor_error, a tiny JSON envelope smuggled through the existing free-text ActorStateStopped.message string via a rivet-error: prefix.
  • envoy-client: when on_actor_start fails with a RivetError anywhere in the anyhow chain, encodes it via the envelope instead of the plain format!("{error:#}"); falls back to the old plain formatting for non-RivetError causes, so ordinary crash messages are unaffected.
  • pegboard: adds Actor::NotRegistered { actor_name } (actor.not_registered).
  • guard: in the ready-wait loop, when the actor stops, decodes the crash message and short-circuits to Actor::NotRegistered instead of burning through wake retries.

Overall this is a well-scoped, minimal change with good unit tests on the new parsing logic.

Strengths

  • Fallback path (format!("{error:#}")) for non-RivetError causes is preserved, so existing crash-message behavior for ordinary actor errors is unchanged.
  • Avoids wasting up to 8 rewake round-trips on a failure mode that can never succeed, replacing a generic timeout error with an actionable one.
  • Reuses the existing "failed" metrics outcome label rather than adding a new one (keeps ROUTE_PEGBOARD_READY_WAIT_DURATION label cardinality bounded, per the metrics conventions in CLAUDE.md).
  • New Cargo.toml dependency additions (rivet-envoy-protocol in guard, rivet-error in envoy-client, serde_json in envoy-protocol) are minimal and each directly justified by the new code path.
  • Confirmed the encode side is only reachable via a genuine downcast_ref::<RivetError>() match (a real Rust type check), not string sniffing, so a user actor's own thrown/anyhow error text can't spoof the rivet-error: envelope and get mis-decoded as a structured not_registered error.

Suggestions

  1. No timeout/metric bound on the new op call (moderate). actor_not_registered_name issues a pegboard::ops::actor::get call (a UDB transaction + get_workflows) unconditionally on every stopped_sub event in the ready-wait loop. Every other async op in this function (the initial actor fetch, the wake signal) is wrapped in phase_timeout(...) with its own metric and a dedicated timeout error. This new call isn't, so a slow/hanging UDB read here would stall the ready-wait loop with no timeout guard, inconsistent with the rest of the function's discipline.

  2. New failure path for a diagnostic check (minor). actor_not_registered_name(ctx, actor_id).await? propagates any op error (even a transient/unrelated one) straight out of handle_actor_v2, short-circuiting the existing wake-retry logic entirely. That may be intentional given this file's existing convention of propagating op errors directly elsewhere, but it's worth confirming that a transient failure of this purely-diagnostic lookup should be allowed to fail the whole ready-wait instead of falling back to the pre-existing retry behavior.

  3. Duplicated chain-walk logic (nice-to-have). actor_start_error_message's error.chain().find_map(|cause| cause.downcast_ref::<RivetError>()) re-implements a piece of what rivet_error::RivetError::extract already does. Using extract directly wasn't appropriate here since it sanitizes non-RivetError causes into a generic internal error (losing today's useful raw crash text for ordinary actor crashes), so the manual walk looks like a deliberate choice, just flagging it in case it wasn't.

  4. Test coverage. The new unit tests cover the envelope round-trip and the not_registered extraction logic well. There's no test exercising the end-to-end behavior change in handle_actor_v2 (short-circuiting the wake-retry loop). That's likely hard to unit test without a fuller harness, so this may be an acceptable gap, but flagging in case there's an existing integration fixture this could hook into.

None of the above are blocking; this looks like a solid, focused fix.

@NathanFlurry
NathanFlurry force-pushed the stack/fix-rivetkit-protect-local-serverless-handler-ownership-qvtxsxkn branch from c8d20ec to 8e9291e Compare July 30, 2026 21:14
@NathanFlurry
NathanFlurry force-pushed the stack/fix-actors-surface-unregistered-actor-startup-errors-sxrxmrpm branch from 4cda149 to 747c01e Compare July 30, 2026 21:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant