Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 10 additions & 7 deletions docs/runbooks/restore-the-stack.md
Original file line number Diff line number Diff line change
Expand Up @@ -287,13 +287,16 @@ started on the result; §4 was run against it, including the negative assertions

**What it found, which per-volume testing had not.**

- **Grafana will not start without internet access.** `GF_INSTALL_PLUGINS` makes
the background installer contact `grafana.com` on every start, and a failure
there is fatal — Grafana crash-loops, even though both plugins are already
present in the restored volume. On a host that has lost its uplink, which is a
perfectly ordinary disaster, the restore succeeds and Grafana still will not
come up. Clearing that variable is the workaround; the plugins in the volume
are used regardless.
- **Grafana would not start without internet access.** `GF_INSTALL_PLUGINS`
made the background installer contact `grafana.com` on every start, and a
failure there was fatal — Grafana crash-looped, even though both plugins were
already present in the restored volume. On a host that has lost its uplink,
which is a perfectly ordinary disaster, the restore succeeded and Grafana
still would not come up. Neither plugin was used by any dashboard, and one of
them was an Angular plugin this Grafana refuses to load anyway, so the
declaration was removed rather than repaired. Grafana's own bundled apps ship
inside the image and need no network, which is why an offline start works now
— verified on both fresh and restored volumes.
- **Alloy replays, and now there is a number for it.** Restoring `alloy-data`
put the log positions back to their offsets at the stamp, and Alloy re-read
from there and re-shipped the lines with their original timestamps — about
Expand Down
23 changes: 22 additions & 1 deletion stacks/observability/compose.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -180,7 +180,28 @@ services:
GF_AUTH_ANONYMOUS_ENABLED: "false"
GF_ANALYTICS_REPORTING_ENABLED: "false"
GF_ANALYTICS_CHECK_FOR_UPDATES: "false"
GF_INSTALL_PLUGINS: grafana-clock-panel,grafana-piechart-panel
# No plugins are installed here, deliberately. This was
# GF_INSTALL_PLUGINS: grafana-clock-panel,grafana-piechart-panel and both
# were dead weight — no dashboard under grafana/dashboards/ uses either
# panel type, and grafana-piechart-panel is an Angular plugin, which this
# Grafana refuses to initialise: it was downloaded and then rejected on
# every single start.
#
# It was also an availability bug, which is the reason it is gone rather
# than merely tidied. The background installer contacts grafana.com on
# every start, and a failure there is FATAL to the process — Grafana
# crash-looped on a host with no uplink, even though both plugins were
# already present in grafana-data. That turns an ordinary outage into a
# failed disaster recovery, and it was found by doing one: see the last
# section of docs/runbooks/restore-the-stack.md.
#
# Grafana still preinstalls its own bundled apps (pyroscope, exploretraces,
# metricsdrilldown, lokiexplore, elasticsearch). Those ship inside the
# image and need no network, which is what makes an offline start work.
#
# If a plugin is ever genuinely needed, add GF_PLUGINS_PREINSTALL —
# GF_INSTALL_PLUGINS is deprecated — and re-test the offline start before
# committing it.
GF_PATHS_PROVISIONING: /etc/grafana/provisioning
PROMETHEUS_URL: http://prometheus:9090
LOKI_URL: http://loki:3100
Expand Down
Loading