[HWORKS-3351] Split the Helm chart values reference into per-subchart pages (#683)

.claude/docs/design-system.md

@@ -237,6 +237,9 @@ Three rules keep them inside the column:
 - Material floors every header at 5rem; that floor is removed, so a narrow column (`#`, a flag) takes only what it needs.
 - Content tabs are linked (`content.tabs.link`): picking Java on one set switches every set on the page and the choice persists. Tab strips are compact segmented controls sitting 0.35rem above their block.
 
+Where the chips themselves are too long for any column, use a list, not a table.
+The generated Helm values pages (`setup_installation/common/helm_chart_values/`) carry keys up to 90 characters, so each value is a definition-list entry inside `.hops-values`: the key chip on a hairline-separated row, its type, default and description under it at table type size.
+
 ## Theme features
 
 Set in `mkdocs.yml` under `theme.features`. Current set and why:

.github/workflows/mkdocs-test.yml

@@ -31,6 +31,25 @@ jobs:
       - name: Install Python dependencies
         run: uv sync --extra cli
 
+      # The Helm values pages are filled by gen-helm-values at build time and
+      # never committed. A committed copy is published as-is whenever a build
+      # leaves the page alone (no matching chart in Nexus yet), so the check
+      # resets every page to its placeholder and fails if that changed one.
+      - name: Check the Helm values pages hold only their placeholders
+        run: |
+          hopsworks-docs reset-helm-values
+          if ! git diff --exit-code --stat -- docs/setup_installation/common/helm_chart_values/; then
+            echo "::error::Generated Helm values content is committed in docs/setup_installation/common/helm_chart_values/ (files above)."
+            echo "Those pages are generated during the documentation build and must hold only their placeholder."
+            echo "To fix it, in your branch run:"
+            echo "  uv run --extra cli hopsworks-docs reset-helm-values"
+            echo "  git add docs/setup_installation/common/helm_chart_values/"
+            echo "  git commit -m 'Reset the Helm values pages to their placeholders'"
+            echo "The reset keeps any edit outside the generated block (a page title, its intro line, a new page)."
+            echo "To preview the values locally, run gen-helm-values, then the reset above before committing."
+            exit 1
+          fi
+
       - name: Install Hopsworks Python API
         run: uv pip install hopsworks-api/python
 
@@ -48,21 +67,26 @@ jobs:
           NEXUS_USER: ${{ secrets.NEXUS_USER }}
           NEXUS_PASSWORD: ${{ secrets.NEXUS_PASSWORD }}
           BASE_REF: ${{ github.base_ref }}
+        # --strict turns the generator's warnings (a subchart without a page, an
+        # upstream chart without a docs link, a missing common value or RonDB
+        # schema, a RonDB override the schema does not declare, two entries on
+        # a page with one anchor, a null override with values listed under it)
+        # into a failed PR check; the deploy workflow only warns.
         run: |
           if [ "$BASE_REF" = "main" ]; then
             # dev docs: latest chart from the (private) dev channel. Fork PRs do
             # not receive secrets, so skip (keep the page placeholder) when the
             # Nexus credentials are absent rather than failing the build.
             if [ -n "$NEXUS_USER" ]; then
-              hopsworks-docs gen-helm-values \
+              hopsworks-docs gen-helm-values --strict \
                 --repo-url https://nexus.hops.works/repository/hopsworks-helm-dev
             else
               echo "Nexus credentials not available (e.g. a fork PR); keeping the values reference placeholder."
             fi
           else
             # release docs: latest patch of the chart version (major.minor) for this docs version.
             # The released repo is public, so fetch anonymously (clear any creds).
-            NEXUS_USER= NEXUS_PASSWORD= hopsworks-docs gen-helm-values \
+            NEXUS_USER= NEXUS_PASSWORD= hopsworks-docs gen-helm-values --strict \
               --repo-url https://nexus.hops.works/repository/hopsworks-helm \
               --chart-version "${BASE_REF#branch-}"
           fi

README.md

@@ -92,18 +92,33 @@ After adding your new page in the docs folder, you also need to add it to this f
 
 ## Helm chart values reference
 
-The `setup_installation/common/helm_chart_values.md` page renders a placeholder locally; its `## Values` table is injected at build time by the `hopsworks-docs gen-helm-values` step and is **never committed**.
+The pages under `setup_installation/common/helm_chart_values/` render placeholders locally; their values tables are injected at build time by the `hopsworks-docs gen-helm-values` step and are **never committed**.
+The step reads the `## Values` table of the chart `README.md` plus that of every `charts/<name>/README.md`, with `<name>.` put before each subchart key; the first row of a key wins, so a chart whose root README embeds the subchart tables (`helm-docs -u`, every release up to 5.1) and one whose README does not give the same pages.
+It writes the rows of each top-level key into the stub page of that name (`kafka.md` gets the `kafka.*` rows), together with the subchart's deployment condition from the root `Chart.yaml` and links to the upstream charts it installs.
+A row's default is what Hopsworks deploys: the root `values.yaml` settings for the subchart are merged over the subchart's own defaults the way Helm merges them; a subchart `values.yaml` that PyYAML cannot parse keeps its own defaults, with a warning.
+`index.md` gets the common values (`_COMMON_VALUES` in `scripts/helm_values.py`), the overview table, plus the rows of any top-level key that has no stub page; to give a new subchart its own page, add a stub with the generation markers and a `nav:` entry.
+For RonDB the page also renders the `values.schema.json` of the pinned RonDB chart, with Hopsworks' overrides (the wrapper subchart's `values.yaml`, then the root `values.yaml`) merged over its defaults the way Helm merges them; an overridden entry also names the RonDB chart's own default.
+A values row the generator cannot parse always fails the step.
+The PR check runs it with `--strict`, which also fails on a top-level key without a page, an upstream chart repository without a docs link in `_UPSTREAM_DOCS`, a common value the chart no longer has, a missing RonDB schema, a RonDB override of a key the schema does not declare (usually a value RonDB renamed, which the override then stops setting), two entries on a page with the same anchor (usually a helm-docs comment on a key under `rondb.rondb`, which the RonDB schema already lists), or a Hopsworks override that sets a key to null while values under it are listed (Helm removes them, but their entries would still show the chart defaults); the deploy workflow only warns.
 CI fetches the chart from Nexus rather than the (private) `hopsworks-helm` git repo: release builds (`branch-x.y`) read the public `hopsworks-helm` repo anonymously and pick the latest patch of the chart version matching the docs version, while the development build (`main`) reads the private `hopsworks-helm-dev` repo and uses its newest published chart.
-The development path requires the read-only `NEXUS_USER` and `NEXUS_PASSWORD` repository secrets; without them the `main` build's generation step fails.
-Older chart versions that predate the chart's `## Values` section keep the page placeholder (the build warns rather than failing).
+The development path requires the read-only `NEXUS_USER` and `NEXUS_PASSWORD` repository secrets; without them the `main` build skips the generation step and keeps the placeholders.
+Older chart versions that predate the chart's `## Values` section keep the page placeholders (the build warns rather than failing).
 
-To preview the table locally against a chart checkout:
+To preview the pages locally against a chart checkout:
 
 ```bash
+helm dependency build <path-to-hopsworks-helm>/charts/rondb # only needed for the RonDB chart values
 uv run --extra cli hopsworks-docs gen-helm-values --chart <path-to-hopsworks-helm>
 ```
 
-This rewrites the page in place, so restore it (`git checkout docs/setup_installation/common/helm_chart_values.md`) before committing.
+This rewrites the pages in place, so put the placeholders back before committing:
+
+```bash
+uv run --extra cli hopsworks-docs reset-helm-values
+```
+
+The reset only touches the text between the generation markers, so edits to a page title or intro survive it.
+The PR check runs the same reset and fails if it changes a committed page, since a committed copy of the values would be published as-is by any build that leaves the page alone.
 
 ## Checking links
 

docs/css/custom.css

@@ -2468,3 +2468,87 @@ body.hops-zoom-lock {
   background: var(--md-default-bg-color);
   border: 1px solid var(--hops-border);
 }
+
+/* ---- Helm values reference: one entry per chart value ----------
+   The generated pages under setup_installation/common/helm_chart_values/
+   wrap their definition lists in .hops-values. Keys run to 90 characters,
+   so they cannot sit in a table (a table chip never wraps, see above);
+   each key is a hairline-separated row with its type, default and
+   description under it at table type size. */
+.md-typeset .hops-values dl {
+  margin: 0;
+}
+.md-typeset .hops-values dt {
+  margin: 0;
+  padding-top: 0.5rem;
+  border-top: 1px solid var(--hops-border);
+  font-size: 0.72rem;
+}
+.md-typeset .hops-values dd {
+  margin: 0.15rem 0 0.5rem 0;
+  font-size: 0.68rem;
+  line-height: 1.45;
+}
+.md-typeset .hops-values dd p {
+  margin: 0;
+}
+/* Deprecated values stay listed (an old override still has to validate) but
+   recede: a muted label after the key and a muted description. The muting is
+   the theme's light foreground, not opacity, which took the text below 4.5:1. */
+.md-typeset .hops-values-deprecated {
+  margin-left: 0.4em;
+  padding: 0.05em 0.4em;
+  border: 1px solid var(--hops-border-strong);
+  border-radius: 0.25rem;
+  color: var(--md-default-fg-color--light);
+  font-size: 0.58rem;
+  letter-spacing: 0.04em;
+  text-transform: uppercase;
+  vertical-align: middle;
+}
+.md-typeset .hops-values dt:has(.hops-values-deprecated) + dd {
+  color: var(--md-default-fg-color--light);
+}
+/* Filter box (js/values-filter.js), pinned under the header while the long
+   pages scroll. */
+.md-typeset .hops-values-filterbar {
+  position: sticky;
+  top: 2.4rem;
+  z-index: 1;
+  display: flex;
+  align-items: center;
+  gap: 0.6rem;
+  margin: 1rem 0 0.5rem;
+  padding: 0.4rem 0;
+  background: var(--md-default-bg-color);
+}
+.md-typeset .hops-values-filter {
+  flex: 1;
+  min-width: 0;
+  padding: 0.35rem 0.7rem;
+  border: 1px solid var(--hops-border-strong);
+  border-radius: 1rem;
+  background: var(--hops-surface);
+  color: var(--md-default-fg-color);
+  font: inherit;
+  font-size: 0.72rem;
+}
+.md-typeset .hops-values-filter:focus {
+  outline: 2px solid var(--hops-accent);
+  outline-offset: 1px;
+}
+.md-typeset .hops-values-count {
+  color: var(--md-default-fg-color--light);
+  font-size: 0.64rem;
+  white-space: nowrap;
+}
+/* A linked key or section lands below the header and the pinned filter bar,
+   not under it. */
+.md-content:has(.hops-values-filterbar) :is(h2, h3, dt)[id] {
+  scroll-margin-top: 5.6rem;
+}
+/* The filter hides a section's "Defaults as YAML" block with the hidden
+   attribute, which Material's `details { display: flow-root }` outranks. */
+.md-typeset details[hidden] {
+  display: none;
+}

docs/index.md

@@ -234,7 +234,7 @@ Independent [feature, training and inference pipelines](concepts/fti.md), connec
 
 | Task | Start here |
 | --- | --- |
-| :material-rocket-launch-outline: Deploy | [AWS](setup_installation/aws/getting_started.md), [Azure](setup_installation/azure/getting_started.md), [GCP](setup_installation/gcp/getting_started.md), [on-prem](setup_installation/on_prem/contact_hopsworks.md), [Helm values](setup_installation/common/helm_chart_values.md) |
+| :material-rocket-launch-outline: Deploy | [AWS](setup_installation/aws/getting_started.md), [Azure](setup_installation/azure/getting_started.md), [GCP](setup_installation/gcp/getting_started.md), [on-prem](setup_installation/on_prem/contact_hopsworks.md), [Helm values][helm-chart-values-reference] |
 | :material-monitor-dashboard: Operate | [Administration](setup_installation/admin/index.md), [monitoring](setup_installation/admin/monitoring/grafana.md), [alerts](setup_installation/admin/alert.md), [HA and DR](setup_installation/admin/ha-dr/intro.md), [service operations](setup_installation/admin/operationLogs.md) |
 | :material-wrench-outline: Troubleshoot | [Model serving](user_guides/mlops/serving/troubleshooting.md), [Python deployments](user_guides/projects/python-deployment/troubleshooting.md), [online ingestion](user_guides/fs/feature_group/online_ingestion_observability.md), [Jupyter session capacity](user_guides/projects/jupyter/session_capacity_warnings.md) |
 | :material-arrow-up-circle-outline: Upgrade | [3.x to 4.0 migration](user_guides/migration/40_migration.md), [Airflow 3 upgrade](user_guides/projects/airflow/airflow3_upgrade.md), [Airflow 3 operator notes](setup_installation/admin/airflow3.md) |
@@ -255,7 +255,7 @@ Independent [feature, training and inference pipelines](concepts/fti.md), connec
 :material-tune:{ .hops-colophon-ico } Configure and query
 { .hops-colophon-cap }
 
-- [Helm chart values](setup_installation/common/helm_chart_values.md)
+- [Helm chart values][helm-chart-values-reference]
 - [Cluster configuration](setup_installation/admin/variables.md)
 - [Query engine (Trino)](user_guides/projects/trino/query_engine.md)
 - [Vector similarity search](user_guides/fs/vector_similarity_search.md)

docs/js/values-filter.js

@@ -0,0 +1,58 @@
+// Filter box for the generated Helm values pages: narrows every .hops-values
+// list to the entries whose key or description contains the query, and hides
+// the sections left empty. Without this script every entry stays visible.
+document.addEventListener("DOMContentLoaded", function () {
+  var lists = document.querySelectorAll(".md-typeset .hops-values");
+  var terms = document.querySelectorAll(".md-typeset .hops-values dt");
+  if (terms.length < 25) return;
+
+  var bar = document.createElement("div");
+  bar.className = "hops-values-filterbar";
+  var input = document.createElement("input");
+  input.type = "search";
+  input.className = "hops-values-filter";
+  input.placeholder = "Filter " + terms.length + " values by key or description";
+  input.setAttribute("aria-label", "Filter values");
+  var count = document.createElement("span");
+  count.className = "hops-values-count";
+  count.setAttribute("aria-live", "polite");
+  bar.append(input, count);
+
+  // A list's section is the heading and the "Defaults as YAML" block before it.
+  function sectionOf(list) {
+    var parts = [];
+    var el = list.previousElementSibling;
+    while (el && el.tagName === "DETAILS") {
+      parts.push(el);
+      el = el.previousElementSibling;
+    }
+    if (el && /^H[23]$/.test(el.tagName)) parts.push(el);
+    return parts;
+  }
+
+  var first = lists[0];
+  var firstSection = sectionOf(first);
+  var before = firstSection.length ? firstSection[firstSection.length - 1] : first;
+  before.parentNode.insertBefore(bar, before);
+
+  input.addEventListener("input", function () {
+    var query = input.value.trim().toLowerCase();
+    var shown = 0;
+    terms.forEach(function (dt) {
+      var dd = dt.nextElementSibling;
+      var text = (dt.textContent + " " + (dd ? dd.textContent : "")).toLowerCase();
+      var match = !query || text.indexOf(query) !== -1;
+      dt.hidden = !match;
+      if (dd) dd.hidden = !match;
+      if (match) shown++;
+    });
+    lists.forEach(function (list) {
+      var empty = list.querySelector("dt:not([hidden])") === null;
+      list.hidden = empty;
+      sectionOf(list).forEach(function (el) {
+        el.hidden = empty;
+      });
+    });
+    count.textContent = query ? shown + " of " + terms.length + " values" : "";
+  });
+});

docs/setup_installation/common/helm_chart_values/airflow.md

@@ -0,0 +1,9 @@
+# Airflow values { #helm-values-airflow }
+
+Values under `airflow` configure Apache Airflow, which schedules and orchestrates Hopsworks jobs.
+
+<!-- BEGIN GENERATED VALUES -->
+
+_The values table is generated from the Hopsworks Helm chart during the documentation build._
+
+<!-- END GENERATED VALUES -->

docs/setup_installation/common/helm_chart_values/arrowflight.md

@@ -0,0 +1,9 @@
+# Arrow Flight values { #helm-values-arrowflight }
+
+Values under `arrowflight` configure the Arrow Flight server, which serves fast reads of feature groups and training datasets.
+
+<!-- BEGIN GENERATED VALUES -->
+
+_The values table is generated from the Hopsworks Helm chart during the documentation build._
+
+<!-- END GENERATED VALUES -->

docs/setup_installation/common/helm_chart_values/certs-operator.md

@@ -0,0 +1,9 @@
+# Certs operator values { #helm-values-certs-operator }
+
+Values under `certs-operator` configure the operator that issues the TLS certificates of the Hopsworks services and removes them on uninstall.
+
+<!-- BEGIN GENERATED VALUES -->
+
+_The values table is generated from the Hopsworks Helm chart during the documentation build._
+
+<!-- END GENERATED VALUES -->

docs/setup_installation/common/helm_chart_values/consul.md

@@ -0,0 +1,9 @@
+# Consul values { #helm-values-consul }
+
+Values under `consul` configure Consul, which provides service discovery and DNS between the Hopsworks services.
+
+<!-- BEGIN GENERATED VALUES -->
+
+_The values table is generated from the Hopsworks Helm chart during the documentation build._
+
+<!-- END GENERATED VALUES -->

docs/setup_installation/common/helm_chart_values/docker-registry.md

@@ -0,0 +1,9 @@
+# Docker registry values { #helm-values-docker-registry }
+
+Values under `docker-registry` configure the in-cluster Docker registry that stores the images Hopsworks builds, such as project Python environments.
+
+<!-- BEGIN GENERATED VALUES -->
+
+_The values table is generated from the Hopsworks Helm chart during the documentation build._
+
+<!-- END GENERATED VALUES -->

docs/setup_installation/common/helm_chart_values/global.md

@@ -0,0 +1,9 @@
+# Global values { #helm-values-global }
+
+Values under `global` are shared by every subchart: image registry and pull secrets, storage class, scheduling, cloud provider and which optional services are enabled.
+
+<!-- BEGIN GENERATED VALUES -->
+
+_The values table is generated from the Hopsworks Helm chart during the documentation build._
+
+<!-- END GENERATED VALUES -->

docs/setup_installation/common/helm_chart_values/grafana.md

@@ -0,0 +1,9 @@
+# Grafana values { #helm-values-grafana }
+
+Values under `grafana` configure Grafana and the Hopsworks monitoring dashboards.
+
+<!-- BEGIN GENERATED VALUES -->
+
+_The values table is generated from the Hopsworks Helm chart during the documentation build._
+
+<!-- END GENERATED VALUES -->

docs/setup_installation/common/helm_chart_values/hive.md

@@ -0,0 +1,9 @@
+# Hive values { #helm-values-hive }
+
+Values under `hive` configure the Hive metastore, which holds the table metadata of the offline feature store.
+
+<!-- BEGIN GENERATED VALUES -->
+
+_The values table is generated from the Hopsworks Helm chart during the documentation build._
+
+<!-- END GENERATED VALUES -->

docs/setup_installation/common/helm_chart_values/hopsfs.md

@@ -0,0 +1,9 @@
+# HopsFS values { #helm-values-hopsfs }
+
+Values under `hopsfs` configure HopsFS, the distributed file system behind datasets and the offline feature store.
+
+<!-- BEGIN GENERATED VALUES -->
+
+_The values table is generated from the Hopsworks Helm chart during the documentation build._
+
+<!-- END GENERATED VALUES -->