Skip to content

fix(observability): pin the Grafana datasource uid - #54

Merged
PhilippTheServer merged 1 commit into
OpenTaberna:mainfrom
PhilippTheServer:pl/grafana-datasource-uid
Aug 26, 2026
Merged

PhilippTheServer merged 1 commit into
OpenTaberna:mainfrom
PhilippTheServer:pl/grafana-datasource-uid

Conversation

@PhilippTheServer

Copy link
Copy Markdown
Contributor

Closes #53

Every panel on the dashboard rendered "No data" while Grafana logged nothing wrong.

The datasource file declared no uid, so Grafana generated a random one
(PBFA97CFB590B2093 here). Every panel references the fixed string "prometheus", which
resolved to a datasource that does not exist.

The dashboard and the datasource are provisioned from separate files, and that string is the
only thing tying them together. Nothing checked they agree.

Why this one is nasty

Both files provision successfully. The log says finished to provision dashboards. The
result — eleven empty panels — looks exactly like a shop with no traffic. Nothing anywhere
says "that datasource doesn't exist".

Why my verification missed it

I ran each panel's PromQL against Prometheus directly and reported 11/11 returning data.
That was true and irrelevant: querying the datastore cannot see a broken datasource
reference. Only a query through Grafana can.

Re-checked properly, through /api/ds/query — the request a panel actually makes:

OK  Outbox pending                       points=1
OK  Outbox failed                        points=1
OK  Outbox dead                          points=1
OK  Webhooks unprocessed                 points=1
OK  Awaiting shipment                    points=1
OK  Request rate by route                points=10
OK  Error rate (5xx)                     points=1
OK  Latency p95 by route                 points=10
OK  Latency p99                          points=1
OK  Database connection pool             points=2
OK  Requests in flight                   points=1

11 panels return data through Grafana, 0 do not

The check

tests/test_grafana_provisioning.py — static, no running stack needed. Asserts a datasource
declares a uid, that every panel points at one that is actually provisioned, that dashboards
carry a stable uid and title, and that no panel ships without a query.

Against the state that shipped:

E  assert set()
E  assert not {'prometheus'}
FAILED test_the_datasource_declares_an_explicit_uid
FAILED test_every_panel_points_at_a_datasource_that_is_provisioned[opentaberna-health.json]

After the fix: 5 passed.

Every panel on the provisioned dashboard rendered "No data" while Grafana
logged nothing wrong.

The datasource file declared no uid, so Grafana generated a random one on first
provision. Every panel references the fixed string "prometheus", which therefore
resolved to a datasource that does not exist. The dashboard and the datasource
come from separate files and that string is the only thing tying them together.

The failure mode is the problem: both files provision successfully, the log says
so, and eleven empty panels look exactly like a shop with no traffic.

It slipped through verification because that verification ran each panel's
PromQL against Prometheus directly. Querying the datastore cannot see a broken
datasource reference — only a query through Grafana can, and re-checking that
way now shows all eleven panels returning data.

Adds static checks over the provisioning files: that a datasource declares a
uid, that every panel points at one that is actually provisioned, and that no
panel ships without a query. They need no running stack, and both fail against
the state that shipped.

Closes #53

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YY1ekLLeFLkAU2kvdQ8Ey4
@PhilippTheServer
PhilippTheServer merged commit f88efdf into OpenTaberna:main Aug 26, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Every Grafana panel shows "No data": the datasource uid was never pinned

1 participant