Port #3752 to main: answer callers whose agent moved

[kmatasfp]

Ports #3752 to main.

The worker-service half is already on main: the idempotency-key mint happens in normalize_agent_invocation_identity, above the retry loop, and RecordingWorkerClient already expects a key. What is ported is the type-level half, WorkerClient::invoke_agent taking IdempotencyKey rather than Option<IdempotencyKey>, so the broken shape cannot be expressed again. main's invoke_agent recomputes freshness_disposition_for_dispatch per attempt, so 1.5.x's "build the whole request once outside the closure" refactor does not carry over.

a_supplied_idempotency_key_reaches_the_executor_unchanged is not ported; main's ephemeral_lookup_accepts_the_final_invocation_identity already covers it. keyless_invocations_are_each_given_their_own_key is rewritten against main's invocations() recorder.

The interval's doc comment is corrected rather than copied: it said the re-check "clones the agent's last known status", which stopped being true on main with #3846.

Verified by neutralising the InvalidShardId arm on this branch: a_caller_is_answered_when_its_agents_shard_is_taken_away then fails with caller parked in invoke_and_await was never answered, and a_result_that_lands_while_ownership_is_being_checked_still_reaches_the_caller with a status timeout.

Review found that the re-check could be starved: biased polled the subscription first, and a receiver that has fallen behind the bus is ready at once with Lagged, so under sustained lag the deadline arm was never polled. The deadline arm now comes first. a_caller_is_answered_when_its_agents_shard_is_taken_away_while_the_event_bus_lags shrinks the bus to 16 and floods it with 64 unrelated events every 50ms; it times out with the old arm order and is answered on the first tick with the new one. #3752 on 1.5.x has the same starvation and needs this ported back.

[netlify]

✅ Deploy Preview for golemcloud canceled.

Name	Link
🔨 Latest commit	c1c4b21
🔍 Latest deploy log	https://app.netlify.com/projects/golemcloud/deploys/6aa30e664809790008f52cf3

[vigoo]

Found one more thing compared to the original review:

The ownership recheck can still be starved under sustained broadcast lag.

In golem-worker-executor/src/worker/mod.rs:5466–5469, biased; prioritizes waiting over the timer—even when waiting returns RecvError::Lagged rather than a completion. The lag handler then sleeps 100 ms and restarts. If unrelated events overflow the configured buffer during each sleep, the event branch keeps winning and the expired ownership timer never runs. A caller whose agent moved can therefore remain stuck.

I reproduced these selection/retry semantics in an isolated Tokio harness: a buffer capacity of 16, 32 unrelated events every 50 ms, and the same five-second ownership deadline resulted in a 20-second timeout with zero ownership checks and 201 lagged restarts. A deadline-first control performed the check at five seconds. This demonstrates the overflow case, not its frequency with the production default buffer.

Could we ensure overdue ownership checks run despite repeated Lagged results, while preserving the final result lookup before rerouting, and add a sustained-lag regression test? The existing four ownership tests pass but do not exercise broadcast overflow.

This behavior was already present in #3752 and carried over unchanged; it is an incomplete-fix edge case, not a port-specific regression.

[kmatasfp]

@vigoo your feedback should be addressed