[DX-5514] recovering state

[Tofel]

Handle Recovering state so that we don't fail with:

collect evidence after 2 poll(s): poll rule "[Dev]Griddle App Test Service No Heartbeat" (cfx5gk8k0wqv4c): gave up after 6 sequential failures: transport error: parse rule state: state response: group "one-minute" (folder "devex-cicd"): rule 0: rule "cfx5gk8k0wqv4c": instance 0: unrecognized instance state "Recovering (NoData)" 

[github-actions]

👋 Tofel, thanks for creating this pull request!

To help reviewers, please consider creating future PRs as drafts first. This allows you to self-review and make any final changes before notifying the team.

Once you're ready, you can mark it as "Ready for review" to request feedback. Thanks!

[github-actions]

📊 API Diff Results

No changes detected for module github.com/smartcontractkit/chainlink-testing-framework/grafana-alertcheck

View full report

[Copilot]

Copilot review overview

🟡 Changes recommended

Recovery handling can incorrectly pass failures or skip necessary observation.

Review effort: Balanced
Findings: 4

Open (4)

• Read appended recovery polls before selecting recovery instances · New
• Enforce individual recovery deadlines during recording and classification · New
• Require pre-window onset evidence for recovery classification · New
• Retire recovery eligibility after refires, pauses, and vanished instances · New

What changed in this PR

Adds Grafana Recovering-state support to alert checking, including observation beyond the normal window.

Changes:

• Parses Recovering states and records recovery durations.
• Adds recovery observation and classification logic with tests.
• Updates CLI defaults, documentation, and release notes.

File	Description
grafana-alertcheck/​internal/​gate/​parse_state.go	Parses Recovering and recovery durations.
grafana-alertcheck/​internal/​gate/​parse_state_test.go	Tests parsing additions.
grafana-alertcheck/​internal/​gate/​log.go	Records recovery durations per poll.
grafana-alertcheck/​internal/​gate/​coverage.go	Shares window-membership helpers.
grafana-alertcheck/​internal/​gate/​classify.go	Classifies recovery episodes and extension evidence.
grafana-alertcheck/​internal/​gate/​classify_test.go	Tests recovery classification.
grafana-alertcheck/​internal/​gate/​check.go	Adds bounded recovery observation.
grafana-alertcheck/​internal/​gate/​check_test.go	Tests direct and recorded recovery observation.
grafana-alertcheck/​docs/​reference/​cli.md	Documents updated state options.
grafana-alertcheck/​docs/​how-alerts-are-evaluated.md	Explains recovery behavior and timing.
grafana-alertcheck/​cmd/​common.go	Accepts recovering in state selection.
grafana-alertcheck/​cmd/​check.go	Updates default-state help text.
grafana-alertcheck/​.changeset/​v0.1.9.md	Announces Recovering support.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Copilot review overview

🟡 Changes recommended

Recovery observation can lose timely clears during draining, and stale clock calculations can exceed recovery deadlines.

Review effort: Balanced
Findings: 3 · 1

Open (4)

• Recovery polling loses timely drain observations · New
• Retire recovery eligibility after refires, pauses, and vanished instances
• Require pre-window onset evidence for recovery classification
• Stale clock causes recovery deadline overruns · New

Resolved since last review (2)

• Enforce individual recovery deadlines during recording and classification
• Read appended recovery polls before selecting recovery instances

[Copilot]

Copilot review overview

🟡 Changes recommended

Recovery classification can forgive in-window failures after interval changes, and recovery polling can amplify requests beyond the planned budget.

Review effort: Balanced
Findings: 1

Open (1)

• Interval changes can misclassify in-window failures · New

Resolved since last review (4)

• Recovery polling loses timely drain observations
• Retire recovery eligibility after refires, pauses, and vanished instances
• Require pre-window onset evidence for recovery classification
• Stale clock causes recovery deadline overruns

Previously missed (1)

In code that hasn't changed since last review

Single-step mode over-polls slow recovery rules

grafana-alertcheck/​internal/​gate/​check.go:1047

In single-step mode, each iteration sends every pending UID to directRecoverySource, then waits only for the shortest rule cadence. With recovering rules evaluating every 10s and 300s, the slow rules are requested roughly every 5s instead of every 150s (assuming short requests). During a long recovery period, this multiplies Grafana requests and queues fast rules behind work that the startup per-rule budget did not account for. Track per-rule due times and pass only due UIDs to the direct source; log-tail reads can keep a shared cadence. Add coverage for mixed evaluation intervals.