From 35fee7c499be3cfb3b53fa5153964c3f7f21f744 Mon Sep 17 00:00:00 2001 From: Garrett Allen <98648590+Gerrrt@users.noreply.github.com> Date: Sat, 29 Aug 2026 13:43:08 +0000 Subject: [PATCH] fix(grafana): stop requiring the internet to start MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit GF_INSTALL_PLUGINS made Grafana's background installer contact grafana.com on every start, and a failure there was fatal to the process: starting module plugin.backgroundinstaller: ... failed to install plugin grafana-piechart-panel@: Get "https://grafana.com/api/plugins/..." Grafana crash-looped, even though both plugins were already present in grafana-data. So a host that had lost its uplink — an entirely ordinary disaster — would restore successfully and then fail to bring Grafana up. Every other service in the stack starts offline. Found by running the whole-stack restore rehearsal in docs/runbooks/restore-the-stack.md. Both plugins are removed rather than migrated to GF_PLUGINS_PREINSTALL, because neither was doing anything: - No dashboard under grafana/dashboards/ uses either panel type. Every panel across the five dashboards is a core type, and the only mention of either plugin anywhere in the repository was the compose line itself. There are no UI-created dashboards to worry about either — the dashboard table is empty in the live database. - grafana-piechart-panel is an Angular plugin, and this Grafana refuses to initialise those. It was downloaded and then rejected on every single start: "Plugin validation failed ... angular plugins are not supported". Grafana has had a built-in piechart since v8 regardless. - GF_INSTALL_PLUGINS is itself deprecated in favour of GF_PLUGINS_PREINSTALL. Removing the declaration fixes all three at once and leaves nothing to migrate. Verified as the task asked, on a scratch project with `internal: true` on the compose network so there is no route off the host, in both cases that matter: fresh volumes grafana healthy, 0 restarts, database ok, whole stack up restored volumes grafana healthy, 0 restarts, database ok, whole stack up with the restored admin row and datasources intact The mechanism is now visible in the logs: "Plugins installed plugins=[]", while Grafana's own bundled apps (pyroscope, exploretraces, metricsdrilldown, lokiexplore, elasticsearch) still register from inside the image. On restored volumes those apps do log "Failed to get plugin info" against grafana.com, but unlike a GF_INSTALL_PLUGINS entry that failure is not fatal — Grafana keeps the version it already has and carries on. The runbook's CAUTION becomes a NOTE: the stack starting with no internet is now a tested property worth knowing during a recovery, rather than a trap. The finding is kept in the rehearsal write-up, in the past tense. One loose end, deliberately not done here: the grafana-piechart-panel directory is still inside the live grafana-data volume, so the Angular rejection is still logged once per start. It is cosmetic, no alert rule matches it, and deleting files out of a production volume is not something to fold into this change. Co-Authored-By: Claude Opus 5 --- docs/runbooks/restore-the-stack.md | 17 ++++++++++------- stacks/observability/compose.yaml | 23 ++++++++++++++++++++++- 2 files changed, 32 insertions(+), 8 deletions(-) diff --git a/docs/runbooks/restore-the-stack.md b/docs/runbooks/restore-the-stack.md index ecb2213..b4e1f4f 100644 --- a/docs/runbooks/restore-the-stack.md +++ b/docs/runbooks/restore-the-stack.md @@ -287,13 +287,16 @@ started on the result; §4 was run against it, including the negative assertions **What it found, which per-volume testing had not.** -- **Grafana will not start without internet access.** `GF_INSTALL_PLUGINS` makes - the background installer contact `grafana.com` on every start, and a failure - there is fatal — Grafana crash-loops, even though both plugins are already - present in the restored volume. On a host that has lost its uplink, which is a - perfectly ordinary disaster, the restore succeeds and Grafana still will not - come up. Clearing that variable is the workaround; the plugins in the volume - are used regardless. +- **Grafana would not start without internet access.** `GF_INSTALL_PLUGINS` + made the background installer contact `grafana.com` on every start, and a + failure there was fatal — Grafana crash-looped, even though both plugins were + already present in the restored volume. On a host that has lost its uplink, + which is a perfectly ordinary disaster, the restore succeeded and Grafana + still would not come up. Neither plugin was used by any dashboard, and one of + them was an Angular plugin this Grafana refuses to load anyway, so the + declaration was removed rather than repaired. Grafana's own bundled apps ship + inside the image and need no network, which is why an offline start works now + — verified on both fresh and restored volumes. - **Alloy replays, and now there is a number for it.** Restoring `alloy-data` put the log positions back to their offsets at the stamp, and Alloy re-read from there and re-shipped the lines with their original timestamps — about diff --git a/stacks/observability/compose.yaml b/stacks/observability/compose.yaml index 83bd7f3..d966e2b 100644 --- a/stacks/observability/compose.yaml +++ b/stacks/observability/compose.yaml @@ -180,7 +180,28 @@ services: GF_AUTH_ANONYMOUS_ENABLED: "false" GF_ANALYTICS_REPORTING_ENABLED: "false" GF_ANALYTICS_CHECK_FOR_UPDATES: "false" - GF_INSTALL_PLUGINS: grafana-clock-panel,grafana-piechart-panel + # No plugins are installed here, deliberately. This was + # GF_INSTALL_PLUGINS: grafana-clock-panel,grafana-piechart-panel and both + # were dead weight — no dashboard under grafana/dashboards/ uses either + # panel type, and grafana-piechart-panel is an Angular plugin, which this + # Grafana refuses to initialise: it was downloaded and then rejected on + # every single start. + # + # It was also an availability bug, which is the reason it is gone rather + # than merely tidied. The background installer contacts grafana.com on + # every start, and a failure there is FATAL to the process — Grafana + # crash-looped on a host with no uplink, even though both plugins were + # already present in grafana-data. That turns an ordinary outage into a + # failed disaster recovery, and it was found by doing one: see the last + # section of docs/runbooks/restore-the-stack.md. + # + # Grafana still preinstalls its own bundled apps (pyroscope, exploretraces, + # metricsdrilldown, lokiexplore, elasticsearch). Those ship inside the + # image and need no network, which is what makes an offline start work. + # + # If a plugin is ever genuinely needed, add GF_PLUGINS_PREINSTALL — + # GF_INSTALL_PLUGINS is deprecated — and re-test the offline start before + # committing it. GF_PATHS_PROVISIONING: /etc/grafana/provisioning PROMETHEUS_URL: http://prometheus:9090 LOKI_URL: http://loki:3100