Add loki logs, update Fabric dashboard to include port/phy state over time, add mysql SlurmDB accounting server - #148
Open
brennentsmith-runpod wants to merge 3 commits into
Open
Add loki logs, update Fabric dashboard to include port/phy state over time, add mysql SlurmDB accounting server#148brennentsmith-runpod wants to merge 3 commits into
brennentsmith-runpod wants to merge 3 commits into
Conversation
Loki (head-node log store) and Grafana Alloy (per-node shipper) collect Slurm logs from every node into Loki on node-0. Alloy parses each line into low-cardinality level/component labels plus job_id structured metadata and re-stamps entries with their event time. - Loki 3.7.3 + Alloy 1.17.1 (pinned + SHA-256 verified) in the Dockerfile - supervisor programs: alloy (every node), loki (node-0 only) - post_start starts loki on the head node - Grafana Loki datasource + "Slurm Logs" dashboard (+ home nav link) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Integrate the externally-authored SemiAnalysis "Fabric & Interconnect" dashboard, converted from Grafana's v2 (Scenes) schema to the repo's classic schemaVersion 39 (uid preserved so the home nav link still works). Regroups the panels into InfiniBand/RoCE, PCIe (DCGM), Ethernet Fabric, TCP Health, and Clock Sync sections. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Install MariaDB (server + client) and run it on node-0 only as the Slurm accounting (slurmdbd) backend. On the head node, post_start initialises the datadir, starts mariadbd, and ensures the slurm_acct_db database + slurm user exist (slurmdbd creates the tables itself). - datadir scoped per cluster instance: /workspace/slurm-acct-db/$RUNPOD_POD_ID (no stable per-cluster id exists; avoids two clusters sharing a network volume corrupting one InnoDB datadir) - adaptive run-user: mysql normally, root when /workspace forbids chown - clears a partial datadir left by a failed init; loopback bind only - SchedMD-recommended InnoDB tuning; db/user/pass overridable via env Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Yhlong00
approved these changes
Jul 17, 2026
velaraptor-runpod
approved these changes
Jul 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.