fix(grafana): stop requiring the internet to start

[Gerrrt]

Found while running the whole-stack restore rehearsal in #164.

Note

Stacked on #164, because the runbook section this updates only exists there.
Base is fix/backup-volumes-verified; retarget to main once #164 merges.
The only files touched here are compose.yaml and the runbook.

The bug

GF_INSTALL_PLUGINS made Grafana's background installer contact grafana.com
on every start, and a failure there was fatal to the process:

starting module plugin.backgroundinstaller: ... failed to install plugin
grafana-piechart-panel@: Get "https://grafana.com/api/plugins/..."

Grafana crash-looped, even though both plugins were already in grafana-data.
A host that had lost its uplink — an ordinary disaster — would restore
successfully and then fail to bring Grafana up. Every other service starts
offline fine.

Why removal rather than GF_PLUGINS_PREINSTALL

The task offered "drop piechart, migrate the rest". Investigating, neither
plugin was doing anything, so there is nothing to migrate:

• Unused. No dashboard under grafana/dashboards/ uses either panel type —
  every panel across all five is a core type, and the only mention of either
  plugin anywhere in the repo was the compose line itself. No UI-created
  dashboards exist to worry about either: the dashboard table is empty in the
  live database.
• grafana-piechart-panel never worked. It is Angular, which this Grafana
  refuses to initialise — downloaded and then rejected on every start with
  Plugin validation failed ... angular plugins are not supported. Grafana has
  had a built-in piechart since v8 anyway.
• The variable is deprecated in favour of GF_PLUGINS_PREINSTALL.

Removing the declaration fixes all three at once.

Verification

As the task specified — scratch project, internal: true on the compose network
so there is no route off the host, both cases that matter:

Case	Result
Fresh volumes, no egress	grafana healthy, 0 restarts, database: ok, whole stack up
Restored volumes, no egress	grafana healthy, 0 restarts, database: ok, whole stack up, restored admin row (created 2026-08-17) and 3 datasources intact

The mechanism is visible in the logs: Plugins installed plugins=[], while
Grafana's own bundled apps (pyroscope, exploretraces, metricsdrilldown,
lokiexplore, elasticsearch) still register from inside the image. On restored
volumes those apps do log Failed to get plugin info against grafana.com —
but unlike a GF_INSTALL_PLUGINS entry, that failure is not fatal; Grafana
keeps the version it already has and carries on. That distinction is the whole
fix.

Docs

The runbook's CAUTION becomes a NOTE: the stack starting with no internet is
now a tested property worth knowing during a recovery, rather than a trap. The
finding is kept in the rehearsal write-up, in the past tense.

Loose end, deliberately not done here

The grafana-piechart-panel directory is still inside the live grafana-data
volume, so the Angular rejection is still logged once per start. It is cosmetic,
no Loki rule matches it, and deleting files out of a production volume isn't
something to fold into this change. To clear it when convenient:

docker compose -f stacks/observability/compose.yaml stop grafana
docker run --rm -v observability_grafana-data:/d "$(./scripts/image-for.sh archiver)" \
  rm -rf /d/plugins/grafana-piechart-panel /d/plugins/grafana-clock-panel
docker compose -f stacks/observability/compose.yaml start grafana

🤖 Generated with Claude Code