[DX-5521] non-Grafana managed alerts

[Tofel]

Things backed by Promehetus-like backends, e.g. Promehetus or VictoriaMetrics

[github-actions]

👋 Tofel, thanks for creating this pull request!

To help reviewers, please consider creating future PRs as drafts first. This allows you to self-review and make any final changes before notifying the team.

Once you're ready, you can mark it as "Ready for review" to request feedback. Thanks!

[github-actions]

📊 API Diff Results

No changes detected for module github.com/smartcontractkit/chainlink-testing-framework/grafana-alertcheck

View full report

[Copilot]

Copilot review overview

🟡 Changes recommended

Datasource identity collisions can silently select the wrong Prometheus rule, alongside smaller diagnostics and performance issues.

Review effort: Balanced
Findings: 1 · 3 · 1

Open (5)

• Rule key collisions overwrite state for duplicate alert definitions · New
• Label selection displays empty UIDs for datasource rules · New
• Exact UID selectors trigger unnecessary bulk datasource fetches · New
• Datasource lastError is lost during state parsing · New
• Explanation misstates the manageAlerts filter behavior · New

What changed in this PR

Adds support for discovering, recording, and evaluating Prometheus-compatible datasource-managed alerts.

Changes:

• Adds datasource discovery, parsing, identity, and polling.
• Extends logs, verdicts, CLI output, and documentation.
• Adds datasource-specific tests and recovery semantics.

File	Description
grafana-alertcheck/​internal/​gate/​watch.go	Records datasource rules by key.
grafana-alertcheck/​internal/​gate/​watch_test.go	Updates recorder tests for rule references.
grafana-alertcheck/​internal/​gate/​watch_daemon_test.go	Mocks datasource discovery.
grafana-alertcheck/​internal/​gate/​testdata/​ds_rules.json	Adds datasource API fixture.
grafana-alertcheck/​internal/​gate/​terminal.go	Adds rule keys to early termination.
grafana-alertcheck/​internal/​gate/​source.go	Implements datasource discovery and polling.
grafana-alertcheck/​internal/​gate/​source_test.go	Updates source interface tests.
grafana-alertcheck/​internal/​gate/​source_fake_test.go	Extends the fake datasource source.
grafana-alertcheck/​internal/​gate/​source_ds_test.go	Tests datasource source behavior.
grafana-alertcheck/​internal/​gate/​schedule.go	Schedules rules by key.
grafana-alertcheck/​internal/​gate/​resolve.go	Resolves datasource names and keys.
grafana-alertcheck/​internal/​gate/​resolve_test.go	Tests datasource resolution forms.
grafana-alertcheck/​internal/​gate/​parse_state.go	Generalizes parsed rule state.
grafana-alertcheck/​internal/​gate/​parse_ruler.go	Extends definition metadata.
grafana-alertcheck/​internal/​gate/​parse_datasource.go	Parses Prometheus-compatible rules.
grafana-alertcheck/​internal/​gate/​parse_datasource_test.go	Tests datasource parsing.
grafana-alertcheck/​internal/​gate/​log.go	Persists keys and datasource metadata.
grafana-alertcheck/​internal/​gate/​log_ds_test.go	Tests datasource log compatibility.
grafana-alertcheck/​internal/​gate/​load.go	Loads discovered datasource definitions.
grafana-alertcheck/​internal/​gate/​labels.go	Enables datasource label selection.
grafana-alertcheck/​internal/​gate/​identity.go	Defines cross-source rule keys.
grafana-alertcheck/​internal/​gate/​identity_test.go	Tests key generation and fallback.
grafana-alertcheck/​internal/​gate/​handoff.go	Uses keys during recorder handoff.
grafana-alertcheck/​internal/​gate/​datasource_semantics_test.go	Tests datasource evaluation semantics.
grafana-alertcheck/​internal/​gate/​coverage.go	Adds datasource coverage caveats.
grafana-alertcheck/​internal/​gate/​classify.go	Classifies and reports by rule key.
grafana-alertcheck/​internal/​gate/​check.go	Integrates datasource rules into checks.
grafana-alertcheck/​internal/​gate/​check_test.go	Updates check source fixtures.
grafana-alertcheck/​internal/​gate/​check_ds_test.go	Adds an end-to-end datasource check.
grafana-alertcheck/​docs/​reference/​log-format.md	Documents new log fields.
grafana-alertcheck/​docs/​reference/​cli.md	Documents datasource CLI behavior.
grafana-alertcheck/​docs/​index.md	Documents additional permissions.
grafana-alertcheck/​docs/​how-alerts-are-evaluated.md	Explains datasource semantics.
grafana-alertcheck/​docs/​architecture.md	Documents key-based architecture.
grafana-alertcheck/​docs/​advanced.md	Documents discovery and request costs.
grafana-alertcheck/​cmd/​table.go	Adds source-aware result rendering.
grafana-alertcheck/​cmd/​table_test.go	Tests table additions.
grafana-alertcheck/​cmd/​list.go	Lists datasource rules and keys.
grafana-alertcheck/​cmd/​list_test.go	Tests datasource listing.
grafana-alertcheck/​.changeset/​v0.1.10.md	Summarizes the release changes.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Copilot review overview

🟡 Changes recommended

Label-narrowed duplicate identities can silently poll and classify the wrong datasource rule.

Review effort: Balanced
Findings: 1

Open (1)

• Label-filtered duplicate rules can select the wrong sibling · New

Resolved since last review (5)

• Rule key collisions overwrite state for duplicate alert definitions
• Datasource lastError is lost during state parsing
• Exact UID selectors trigger unnecessary bulk datasource fetches
• Label selection displays empty UIDs for datasource rules
• Explanation misstates the manageAlerts filter behavior

[Copilot]

Copilot review overview

🟡 Changes recommended

Selector ambiguity and log-mode identity collisions can cause the wrong datasource rule to be monitored.

Review effort: Balanced
Findings: 1

Open (1)

• Log mode fails to reject duplicate datasource keys · New

Resolved since last review (1)

• Label-filtered duplicate rules can select the wrong sibling

Previously missed (3)

In code that hasn't changed since last review

Ambiguous selectors resolve to the wrong datasource rule

grafana-alertcheck/​internal/​gate/​resolve.go:103

Trying the whole string as a datasource title and returning immediately makes valid segmented selectors silently choose the wrong rule. For example, if one datasource rule is literally titled Platform/HighErrorRate and a Grafana rule is Platform/HighErrorRate, the documented Platform/HighErrorRate selector resolves only to the datasource rule instead of reporting ambiguity. Collect candidates from both interpretations, deduplicate by key, and require an unambiguous result before returning.

Slash-containing datasource suggestions are not resolvable

grafana-alertcheck/​internal/​gate/​resolve.go:269

For the newly supported datasource titles containing /, this “copyable” suggestion is not resolvable: prefixing datasource and group creates more than three segments, so parseNameForm rejects it. Return the exact key: selector for slash-containing titles so the suggested value can actually be used.

Probe failure omits required datasource query permission

grafana-alertcheck/​internal/​gate/​source.go:273

When /api/datasources succeeds but this probe returns 403 because datasource query permission is missing, the resulting error only says the API is “unusable” and does not identify the required permission. Add the permission to this probe-failure message so operators can act on the failure.

[Copilot]

Copilot review overview

🟡 Changes recommended

Log identity validation and mixed alerting/recording key collisions can cause incorrect rule classification.

Review effort: Balanced
Findings: 2 · 1

Open (3)

• Reject logged alerts whose UID resolves to a recording rule · New
• Detect identity collisions between alerting and recording rules · New
• Use key selectors for datasource paths containing slashes · New

Resolved since last review (1)

• Log mode fails to reject duplicate datasource keys

[Copilot]

Copilot review overview

🟡 Changes recommended

Datasource parsing silently ignores unknown rule types and misses Prometheus’s camelCase keepFiringFor field.

Review effort: Balanced
Findings: 1 · 1

Open (2)

• Support Prometheus camelCase keepFiringFor field · New
• Reject unknown rule types instead of silently dropping them · New

Resolved since last review (3)

• Detect identity collisions between alerting and recording rules
• Reject logged alerts whose UID resolves to a recording rule
• Use key selectors for datasource paths containing slashes

[Copilot]

Copilot review overview

🔵 Needs a closer look

The core datasource recorder-log resolution workflow lacks end-to-end regression coverage.

Review effort: Balanced
Findings: None

Resolved since last review (2)

• Support Prometheus camelCase keepFiringFor field
• Reject unknown rule types instead of silently dropping them

Previously missed (1)

In code that hasn't changed since last review

Add datasource log-resolution recorder-mode coverage

grafana-alertcheck/​internal/​gate/​check.go:606

The new datasource log-resolution path is not exercised by the datasource tests: check_ds_test.go covers only single-step mode, while log_ds_test.go stops at childSchedule/timing derivation. Please add a recorder-mode check test that reads a header containing a datasource rule, re-fetches it through DatasourceDefinitions, and classifies its rule_key; otherwise the core watch → check --log workflow can regress without detection.