Add Ymir infrastructure outage handling research - #238
lbarcziova wants to merge 1 commit into
Conversation
Design spike covering error classification, cost-aware retry, and delayed replay for Ymir agent workflows during external service outages. Fixes: https://redhat.atlassian.net/browse/PACKIT-5137 Assisted-by: Claude Opus 4.6
TomasTomecek
left a comment
TomasTomecek
left a comment
There was a problem hiding this comment.
Truly exceptional research; I just have a few clarifying questions
| 4. Optional: per-dependency flags, configurable limits, Slack | ||
| notification when a dependency is paused. |
There was a problem hiding this comment.
I think the notification here is truly important. Our monitoring skill should do a @channel message when it detects an outage, especially early in the rollout so we can see well how the system behaves.
| | Maximum total postponement | Wall-clock time from first outage-caused delay to final give-up | 2 hours | | ||
| | Reproducer delay | How long to wait before retrying a reproducer task (Testing Farm is slow/expensive) | 30 minutes | | ||
| | Provider retries during provider outage | LLM provider-specific retries after a provider 429/failure is detected | 0 | | ||
| | After all limits exhausted | What happens when a task hits the ceiling | One `ERROR_LIST` entry; stop automation | |
There was a problem hiding this comment.
will we track these tasks somehow? will we know how many of them will be dropped i.e. when the limit is reached?
|
|
||
| | Limit | Meaning | Starting value | | ||
| | --------------------------------------- | ------------------------------------------------------------------------------------------- | --------------------------------------: | | ||
| | LLM calls while dependency is paused | How many new agent/model runs to allow for tasks needing a paused service | 0 | |
There was a problem hiding this comment.
I do understand this requirement that we want to spend $0 dollars when we know an outage is happening and a job cannot succeed. At the same time I wonder if we could have a simple prompt and run it i.e. once every 30 minutes to check if the outage is still on and impacting us
3 participants
Design spike covering error classification, cost-aware retry, and delayed replay for Ymir agent workflows during external service outages.
Fixes: https://redhat.atlassian.net/browse/PACKIT-5137
Assisted-by: Claude Opus 4.6