Retry apiserver 429s instead of failing the mission

[Jonathan-Eid]

What

• Adds ApiRateLimit.ThrottleRetryHandler, a DelegatingHandler that retries apiserver 429s with exponential backoff.
• Installs it at the single new k8s.Kubernetes(...) site, so all ~46 apiserver calls in the library are covered without touching any call site.
• DELETE is exempt; only 429 is retried.

Why

• One 429 on a list events inside WaitForAllReplicasReady killed a parallel catchup mission. That path has no retry, so the exception propagated through MakeFormation and ended the mission.
• Throttling is routine, not exceptional: we served 36,784 429s in the week of 2026-08-17, peaking at 63/s. Every unretried call site is a coin flip.

Testing

Unit (FSLibrary.Tests, 3 new tests, full suite 31/31 green, fantomas --check clean):

• rides out 429s and returns the eventual success
• leaves DELETE alone
• gives up at the deadline and surfaces the 429
• never starts an attempt the budget cannot pay for (2 attempts, one 500ms wait, stops rather than sleeping out a 750ms budget)

End-to-end on test cluster, SimplePayment mission (re-run against the final commit after review changes):

• unfixed + 429 → exit 134, stack trace identical to failed parallel catchup mission
• injected 429s → 6 absorbed, 3/3 replicas ready
• real APF 429s → 6 absorbed mid-mission, wait loop passed
• unbounded 429s → 7 retries, stopping at 46s elapsed because the 8th wait would not fit the budget, then a clean HttpOperationException carrying the 429, no TaskCanceledException
• 429'd DELETEs → 12 rejected, 0 retries, teardown in 2s

Log output

One WRN per retry, naming the verb, the path, the budget consumed so far, and the next wait. Under real APF throttling on ssc-test:

[13:51:03 WRN] apiserver throttled GET /apis/apps/v1/.../statefulsets (00:00:17.3568953 elapsed); retrying in 4000 ms
[13:51:38 WRN] apiserver throttled GET /apis/apps/v1/.../deployments (00:00:15.1216046 elapsed); retrying in 2000 ms
[13:52:03 WRN] apiserver throttled GET /apis/networking.k8s.io/v1/.../ingresses (00:00:15.1916866 elapsed); retrying in 16000 ms
[13:52:35 WRN] apiserver throttled GET /apis/networking.k8s.io/v1/.../ingresses (00:00:46.4712969 elapsed); retrying in 32000 ms
[13:53:37 WRN] apiserver throttled GET /apis/gateway.networking.k8s.io/v1/.../httproutes (00:00:15.1033545 elapsed); retrying in 32000 ms
[13:54:28 WRN] apiserver throttled GET /apis/batch/v1/.../jobs (00:00:15.1495252 elapsed); retrying in 32000 ms
[13:55:37 INF] StatefulSet stellar-supercluster/ssc-1755z-da14be-sts-core: 3/3 replicas ready

Giving up at the deadline needs no new logging -- the retries above it show the budget climbing, and the existing error path carries the 429 body and stack trace:

[14:52:00 WRN] apiserver throttled GET /...events (00:00:46.0362192 elapsed); retrying in 15000 ms
[14:52:17 ERR] Exception thrown. Message = Operation returned an invalid status code 'TooManyRequests',
               response body {"kind":"Status",...,"reason":"TooManyRequests","code":429}

Known limitations

• DELETE exemption means more orphaned objects during a throttling incident; the orphan sweep is the backstop.
• One LogWarn per retry is loud during a throttling storm.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Adds centralized Kubernetes API retry handling for HTTP 429 responses while exempting DELETE requests.

Changes:

• Adds exponential-backoff retry handling with Retry-After support.
• Installs the handler for all Kubernetes client calls.
• Adds tests for success, DELETE exemption, and deadline behavior.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
src/FSLibrary/ApiRateLimit.fs	Implements the retry handler.
src/FSLibrary/StellarSupercluster.fs	Installs the handler on the Kubernetes client.
src/FSLibrary.Tests/Tests.fs	Adds retry-handler tests.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/FSLibrary.Tests/Tests.fs:778

• This wall-clock upper bound can fail even when the handler is correct: the test intentionally waits 500 ms, leaving only 250 ms for scheduler or GC pauses. Please verify the no-extra-delay behavior with an injectable delay/clock instead, so loaded CI workers cannot make this test flaky.

    Assert.True(sw.Elapsed < System.TimeSpan.FromMilliseconds 750.0, sprintf "took %O" sw.Elapsed)