From 2cf3ba47fec109c64afd726f33f9fafa5c92ae13 Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Wed, 23 Sep 2026 06:04:11 +0200 Subject: [PATCH 01/42] [HWORKS-3275] ML software factory https://hopsworks.atlassian.net/browse/HWORKS-3275 The /hops command becomes three software factories for Claude Code (hopsworks-api): /hops ml builds a verified ML system recorded in system.yaml, /hops dashboard a Superset dashboard, /hops app a Python app, with two sub-agents for training and inference. The backend pins Hugging Face imports to one Hub commit (hopsworks-ee). The client installation page lists the files hops setup writes. It now names the three factories the /hops command offers and the two agents it ships, drops hops-fti, and says what happens to a file an earlier version wrote and this one no longer ships. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/user_guides/client_installation/index.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/user_guides/client_installation/index.md b/docs/user_guides/client_installation/index.md index 867f329ecb..ba3ed176af 100644 --- a/docs/user_guides/client_installation/index.md +++ b/docs/user_guides/client_installation/index.md @@ -66,13 +66,15 @@ It then writes the following files into the current directory: | --- | --- | | `AGENTS.md` | Instructions for the agent: the project you are connected to, where the `hopsworks` library is installed on this machine, and how to use the `hops` CLI and the skills. | | `.claude/skills/hops/SKILL.md` | A reference for the `hops` CLI. | -| `.claude/commands/hops.md` | The `/hops` slash command for Claude Code. | -| `.claude/agents/hops-fti.md` | A Claude Code sub-agent that reviews a project against the feature, training and inference pipeline pattern. | +| `.claude/commands/hops.md` | The `/hops` slash command for Claude Code: `/hops ml` builds an ML system, `/hops dashboard` a Superset dashboard, `/hops app` a Python app. | +| `.claude/agents/hops-train-agent.md` | The Claude Code sub-agent `/hops ml` runs to train a model until it meets its target. | +| `.claude/agents/hops-infer-agent.md` | The Claude Code sub-agent `/hops ml` runs to build inference until it meets its SLA. | | `.claude/settings.local.json` | Allows `Bash(hops *)`, so Claude Code can run the CLI without asking before each command. | `AGENTS.md` is read by Claude Code, Codex, GitHub Copilot and OpenCode. The files under `.claude/` are read by Claude Code only. Running `hops setup` again in a directory that already has these files updates the files you have not edited and leaves the ones you have edited unchanged. +A file an earlier version wrote and this one no longer ships, such as `.claude/agents/hops-fti.md`, is removed when you have not edited it and reported otherwise. Pass `--no-scaffold` to authenticate without writing any files. ### Add the Hopsworks skills From 91cd8760a1e539f621695e99b48c2ed091d341ca Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Thu, 24 Sep 2026 00:10:11 +0200 Subject: [PATCH 02/42] [HWORKS-3275] ML software factory https://hopsworks.atlassian.net/browse/HWORKS-3275 Split /hops so its menus and questions run on a fast model and everything that builds runs on a stronger one. A command's model is fixed per command and a sub-agent cannot ask the user anything, so /hops (model: haiku) is now the menu for exploring data, status, and the questions for a dashboard or an app, and it hands each build to hops-dashboard-builder or hops-app-builder, pinned to opus because a sub-agent that inherits from a Haiku command runs on Haiku. The ML factory moves to /hops-ml on the session model, with the training and inference agents inheriting it; the state its first requirements question needs is injected into the prompt, so that question comes without tool round trips. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/user_guides/client_installation/index.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/docs/user_guides/client_installation/index.md b/docs/user_guides/client_installation/index.md index ba3ed176af..e9cd605686 100644 --- a/docs/user_guides/client_installation/index.md +++ b/docs/user_guides/client_installation/index.md @@ -66,9 +66,12 @@ It then writes the following files into the current directory: | --- | --- | | `AGENTS.md` | Instructions for the agent: the project you are connected to, where the `hopsworks` library is installed on this machine, and how to use the `hops` CLI and the skills. | | `.claude/skills/hops/SKILL.md` | A reference for the `hops` CLI. | -| `.claude/commands/hops.md` | The `/hops` slash command for Claude Code: `/hops ml` builds an ML system, `/hops dashboard` a Superset dashboard, `/hops app` a Python app. | -| `.claude/agents/hops-train-agent.md` | The Claude Code sub-agent `/hops ml` runs to train a model until it meets its target. | -| `.claude/agents/hops-infer-agent.md` | The Claude Code sub-agent `/hops ml` runs to build inference until it meets its SLA. | +| `.claude/commands/hops.md` | The `/hops` slash command for Claude Code: a fast menu to explore data, build or edit a Superset dashboard (`/hops dashboard`) or a Python app (`/hops app`), and show status. It runs on Haiku; the building is done by the agents below. | +| `.claude/commands/hops-ml.md` | The `/hops-ml` slash command: builds an ML system from its requirements to a pull request, on your session's model. | +| `.claude/agents/hops-dashboard-builder.md` | The Claude Code sub-agent `/hops` runs to build, edit or delete a dashboard. | +| `.claude/agents/hops-app-builder.md` | The Claude Code sub-agent `/hops` runs to build, edit or delete an app and fix it until it serves. | +| `.claude/agents/hops-train-agent.md` | The Claude Code sub-agent `/hops-ml` runs to train a model until it meets its target. | +| `.claude/agents/hops-infer-agent.md` | The Claude Code sub-agent `/hops-ml` runs to build inference until it meets its SLA. | | `.claude/settings.local.json` | Allows `Bash(hops *)`, so Claude Code can run the CLI without asking before each command. | `AGENTS.md` is read by Claude Code, Codex, GitHub Copilot and OpenCode. From 6f4c24a9aeda3ca5e2440a24daaaff9d59ae02ab Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Thu, 24 Sep 2026 05:55:47 +0200 Subject: [PATCH 03/42] [HWORKS-3275] ML software factory https://hopsworks.atlassian.net/browse/HWORKS-3275 Document /hops-ml as the requirements interview on Haiku and the new /hops-build as the builder on the session model in the file table of the client installation guide. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/user_guides/client_installation/index.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/user_guides/client_installation/index.md b/docs/user_guides/client_installation/index.md index e9cd605686..db3f16a906 100644 --- a/docs/user_guides/client_installation/index.md +++ b/docs/user_guides/client_installation/index.md @@ -67,11 +67,12 @@ It then writes the following files into the current directory: | `AGENTS.md` | Instructions for the agent: the project you are connected to, where the `hopsworks` library is installed on this machine, and how to use the `hops` CLI and the skills. | | `.claude/skills/hops/SKILL.md` | A reference for the `hops` CLI. | | `.claude/commands/hops.md` | The `/hops` slash command for Claude Code: a fast menu to explore data, build or edit a Superset dashboard (`/hops dashboard`) or a Python app (`/hops app`), and show status. It runs on Haiku; the building is done by the agents below. | -| `.claude/commands/hops-ml.md` | The `/hops-ml` slash command: builds an ML system from its requirements to a pull request, on your session's model. | +| `.claude/commands/hops-ml.md` | The `/hops-ml` slash command: a fast interview on Haiku for a new ML system (what to predict, batch, real-time or agentic, how often, which data, how predictions are used), recorded in `system.yaml` as you answer. | +| `.claude/commands/hops-build.md` | The `/hops-build` slash command: completes the specification the interview recorded and builds the ML system to a pull request, on your session's model. | | `.claude/agents/hops-dashboard-builder.md` | The Claude Code sub-agent `/hops` runs to build, edit or delete a dashboard. | | `.claude/agents/hops-app-builder.md` | The Claude Code sub-agent `/hops` runs to build, edit or delete an app and fix it until it serves. | -| `.claude/agents/hops-train-agent.md` | The Claude Code sub-agent `/hops-ml` runs to train a model until it meets its target. | -| `.claude/agents/hops-infer-agent.md` | The Claude Code sub-agent `/hops-ml` runs to build inference until it meets its SLA. | +| `.claude/agents/hops-train-agent.md` | The Claude Code sub-agent `/hops-build` runs to train a model until it meets its target. | +| `.claude/agents/hops-infer-agent.md` | The Claude Code sub-agent `/hops-build` runs to build inference until it meets its SLA. | | `.claude/settings.local.json` | Allows `Bash(hops *)`, so Claude Code can run the CLI without asking before each command. | `AGENTS.md` is read by Claude Code, Codex, GitHub Copilot and OpenCode. From fc0376e648d53dfcc2a65ffb4d75b4cca723eed9 Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Thu, 24 Sep 2026 08:54:57 +0200 Subject: [PATCH 04/42] [HWORKS-3275] ML software factory https://hopsworks.atlassian.net/browse/HWORKS-3275 Document `hops build` in the client installation guide: the interview and its example systems, how a new data source's secret is passed, the launch of /hops-build in a new tmux window, and the ML systems progress panel in the Hopsworks UI. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/user_guides/client_installation/index.md | 22 ++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-) diff --git a/docs/user_guides/client_installation/index.md b/docs/user_guides/client_installation/index.md index db3f16a906..88af6a7b13 100644 --- a/docs/user_guides/client_installation/index.md +++ b/docs/user_guides/client_installation/index.md @@ -67,7 +67,7 @@ It then writes the following files into the current directory: | `AGENTS.md` | Instructions for the agent: the project you are connected to, where the `hopsworks` library is installed on this machine, and how to use the `hops` CLI and the skills. | | `.claude/skills/hops/SKILL.md` | A reference for the `hops` CLI. | | `.claude/commands/hops.md` | The `/hops` slash command for Claude Code: a fast menu to explore data, build or edit a Superset dashboard (`/hops dashboard`) or a Python app (`/hops app`), and show status. It runs on Haiku; the building is done by the agents below. | -| `.claude/commands/hops-ml.md` | The `/hops-ml` slash command: a fast interview on Haiku for a new ML system (what to predict, batch, real-time or agentic, how often, which data, how predictions are used), recorded in `system.yaml` as you answer. | +| `.claude/commands/hops-ml.md` | The `/hops-ml` slash command: the `hops build` interview inside Claude Code, on Haiku, recorded in `system.yaml` as you answer. | | `.claude/commands/hops-build.md` | The `/hops-build` slash command: completes the specification the interview recorded and builds the ML system to a pull request, on your session's model. | | `.claude/agents/hops-dashboard-builder.md` | The Claude Code sub-agent `/hops` runs to build, edit or delete a dashboard. | | `.claude/agents/hops-app-builder.md` | The Claude Code sub-agent `/hops` runs to build, edit or delete an app and fix it until it serves. | @@ -109,6 +109,26 @@ hops skills list hops skills show hops-fg ``` +### Build an ML system + +```bash +hops build +``` + +`hops build` asks what the ML system should predict, or offers an example system (Churn next month, Personalized recommendations, Customer Service Agent) that runs on synthetic data and includes an app. +It then asks the questions that follow from the system type: how often predictions are made for a batch system, the latency and throughput for a real-time one, the LLM for an agentic one, the data to learn from, how the predictions are used, and where the code goes. +One Claude Code call on Haiku reads your description and recommends the system type and a name; the other questions are plain prompts. +Each answer is written to `/system.yaml` in the current directory as you give it. + +A new data source is created with `hops datasource create`, and its password or key is read without echo and passed to it in an environment variable, so it never appears on the command line or in `system.yaml`. + +When the interview is done, `hops build` starts Claude Code with `/hops-build `, which completes the specification and builds the feature, training and inference pipelines. +Inside tmux, as in the Hopsworks terminal, it opens a new tmux window named after the system, so several systems can be built at once. +Pass `--no-launch` to record the interview only, and `hops build ` to resume a system. + +In the Hopsworks terminal, a system in your home directory is registered under `~/.hops/builds/`, and the Hopsworks UI shows an **ML systems** button beside **Terminal** while one is being built. +It opens a panel with each system's phases, what is done and what is left, which you can minimize or close. + ## Hopsworks Java Library If you want to interact with the Hopsworks Feature Store from environments such as Spark, Flink or Beam, you can use the Hopsworks Feature Store (Hopsworks) Java library. From 78d505c21a271857b762f4b921661d30270fdd1c Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Thu, 24 Sep 2026 09:15:09 +0200 Subject: [PATCH 05/42] [HWORKS-3275] ML software factory https://hopsworks.atlassian.net/browse/HWORKS-3275 Describe the new first question of `hops build`: a new ML system or an example ML system (churn, personalized recommendations, help desk agent). Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01RfHMsBbYDQZqoD58XVqeYE --- docs/user_guides/client_installation/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user_guides/client_installation/index.md b/docs/user_guides/client_installation/index.md index 88af6a7b13..d4747d90e3 100644 --- a/docs/user_guides/client_installation/index.md +++ b/docs/user_guides/client_installation/index.md @@ -115,7 +115,7 @@ hops skills show hops-fg hops build ``` -`hops build` asks what the ML system should predict, or offers an example system (Churn next month, Personalized recommendations, Customer Service Agent) that runs on synthetic data and includes an app. +`hops build` first asks what you want to build: a new ML system, which it asks you to describe, or an example ML system (churn, batch; personalized recommendations, real-time; a help desk agent, agentic) that runs on synthetic data and includes an app. It then asks the questions that follow from the system type: how often predictions are made for a batch system, the latency and throughput for a real-time one, the LLM for an agentic one, the data to learn from, how the predictions are used, and where the code goes. One Claude Code call on Haiku reads your description and recommends the system type and a name; the other questions are plain prompts. Each answer is written to `/system.yaml` in the current directory as you give it. From f62bf3d230fd73419ec2744c9f6c1dae11b20679 Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Mon, 28 Sep 2026 10:43:30 +0200 Subject: [PATCH 06/42] [HWORKS-3275] ML software factory https://hopsworks.atlassian.net/browse/HWORKS-3275 Describe the project's ML systems registry in the client installation guide: what hops build registers, the ML systems panel every member of the project sees with its access icons, and hops mlsystem. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01RfHMsBbYDQZqoD58XVqeYE --- docs/user_guides/client_installation/index.md | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/docs/user_guides/client_installation/index.md b/docs/user_guides/client_installation/index.md index d4747d90e3..e168432791 100644 --- a/docs/user_guides/client_installation/index.md +++ b/docs/user_guides/client_installation/index.md @@ -126,8 +126,17 @@ When the interview is done, `hops build` starts Claude Code with `/hops-build ` to resume a system. -In the Hopsworks terminal, a system in your home directory is registered under `~/.hops/builds/`, and the Hopsworks UI shows an **ML systems** button beside **Terminal** while one is being built. -It opens a panel with each system's phases, what is done and what is left, which you can minimize or close. +`hops build` registers each system with the project, by the HopsFS directory of its code, or by its GitHub repository when you build from an external client. +While a project has registered systems, the Hopsworks UI shows an **ML systems** button beside **Terminal**, for every member of the project. +It opens a panel you can minimize or close, with a list of the project's systems: an open folder marks the ones whose code you can open, a lock the ones you cannot, and a link the ones in a GitHub repository. +Selecting a system opens its directory in the file browser and shows its phases, what is done and what is left. +A system whose directory is deleted disappears from the list. + +```bash +hops mlsystem list # the project's systems and whether you can open their code +hops mlsystem register [--name N] # register or refresh one by hand +hops mlsystem remove # remove it from the list; its code is kept +``` ## Hopsworks Java Library From 36c128373dbb8f22bde88e156d1769e03b9bd14d Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Mon, 28 Sep 2026 11:19:25 +0200 Subject: [PATCH 07/42] [HWORKS-3275] ML software factory https://hopsworks.atlassian.net/browse/HWORKS-3275 Document hops build --example and the Create example ML system entry in the Hopsworks UI's ML systems panel, which replaces the Wizard and shows on a cluster with the terminal even before a project has a system. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01RfHMsBbYDQZqoD58XVqeYE --- docs/user_guides/client_installation/index.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/user_guides/client_installation/index.md b/docs/user_guides/client_installation/index.md index e168432791..ff03158345 100644 --- a/docs/user_guides/client_installation/index.md +++ b/docs/user_guides/client_installation/index.md @@ -125,12 +125,14 @@ A new data source is created with `hops datasource create`, and its password or When the interview is done, `hops build` starts Claude Code with `/hops-build `, which completes the specification and builds the feature, training and inference pipelines. Inside tmux, as in the Hopsworks terminal, it opens a new tmux window named after the system, so several systems can be built at once. Pass `--no-launch` to record the interview only, and `hops build ` to resume a system. +`hops build --example ` (`churn-example`, `recs-example` or `helpdesk-example`) builds an example without the menu, and resumes it if it already exists. `hops build` registers each system with the project, by the HopsFS directory of its code, or by its GitHub repository when you build from an external client. -While a project has registered systems, the Hopsworks UI shows an **ML systems** button beside **Terminal**, for every member of the project. +The Hopsworks UI shows an **ML systems** button beside **Terminal**, for every member of the project, when the project has registered systems or the cluster has the terminal. It opens a panel you can minimize or close, with a list of the project's systems: an open folder marks the ones whose code you can open, a lock the ones you cannot, and a link the ones in a GitHub repository. Selecting a system opens its directory in the file browser and shows its phases, what is done and what is left. A system whose directory is deleted disappears from the list. +**Create example ML system**, the last entry in the list, asks which example to build, starts the terminal and runs `hops build --example ` in it. ```bash hops mlsystem list # the project's systems and whether you can open their code From 8405282c8f1aba37142bebdbc976f898becba3e8 Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Mon, 28 Sep 2026 21:07:22 +0200 Subject: [PATCH 08/42] [HWORKS-3275] ML software factory https://hopsworks.atlassian.net/browse/HWORKS-3275 Describe the Brewer button, the ML systems panel's new name, and the architecture page it opens: the pipelines and the data flow between them, redrawn as system.yaml changes, with changed boxes marked until they are clicked, and each box's part of the specification viewable and editable. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01RfHMsBbYDQZqoD58XVqeYE --- docs/user_guides/client_installation/index.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/user_guides/client_installation/index.md b/docs/user_guides/client_installation/index.md index ff03158345..8dad8a33f6 100644 --- a/docs/user_guides/client_installation/index.md +++ b/docs/user_guides/client_installation/index.md @@ -128,9 +128,11 @@ Pass `--no-launch` to record the interview only, and `hops build ` to resu `hops build --example ` (`churn-example`, `recs-example` or `helpdesk-example`) builds an example without the menu, and resumes it if it already exists. `hops build` registers each system with the project, by the HopsFS directory of its code, or by its GitHub repository when you build from an external client. -The Hopsworks UI shows an **ML systems** button beside **Terminal**, for every member of the project, when the project has registered systems or the cluster has the terminal. +The Hopsworks UI shows a **Brewer** button (the software factory) beside **Terminal**, for every member of the project, when the project has registered systems or the cluster has the terminal. It opens a panel you can minimize or close, with a list of the project's systems: an open folder marks the ones whose code you can open, a lock the ones you cannot, and a link the ones in a GitHub repository. Selecting a system opens its directory in the file browser and shows its phases, what is done and what is left. +**show architecture** opens the system's architecture in the main view: its data sources, feature, training and inference pipelines and app, with the data flowing between them, redrawn as `system.yaml` changes. +A box whose part of the specification changed since you last looked is marked until you click it; clicking a box shows that part of `system.yaml`, which you can edit and save, and boxes can be dragged. A system whose directory is deleted disappears from the list. **Create example ML system**, the last entry in the list, asks which example to build, starts the terminal and runs `hops build --example ` in it. From 71626c22ee3cf231a1a437f410eb0f75586fe588 Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Mon, 28 Sep 2026 23:15:50 +0200 Subject: [PATCH 09/42] [HWORKS-3275] ML software factory https://hopsworks.atlassian.net/browse/HWORKS-3275 The hops build section says /hops-build adds a section to the root AGENTS.md of the repository: the system is built from system.yaml, a change to it is checked against the pipelines and assets, and a changed component's downstream is found with the hops lineage commands. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01RfHMsBbYDQZqoD58XVqeYE --- docs/user_guides/client_installation/index.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/user_guides/client_installation/index.md b/docs/user_guides/client_installation/index.md index 8dad8a33f6..3dc4b83cde 100644 --- a/docs/user_guides/client_installation/index.md +++ b/docs/user_guides/client_installation/index.md @@ -123,6 +123,7 @@ Each answer is written to `/system.yaml` in the current directory as you g A new data source is created with `hops datasource create`, and its password or key is read without echo and passed to it in an environment variable, so it never appears on the command line or in `system.yaml`. When the interview is done, `hops build` starts Claude Code with `/hops-build `, which completes the specification and builds the feature, training and inference pipelines. +It also adds a section to the `AGENTS.md` at the root of the repository, saying the system is built from `system.yaml`: a coding agent there checks what a change to `system.yaml` means for the pipelines and the assets they create, and finds what a changed component affects downstream with `hops fg lineage`, `hops fv lineage`, `hops td lineage`, `hops model lineage` and `hops deployment lineage`. Inside tmux, as in the Hopsworks terminal, it opens a new tmux window named after the system, so several systems can be built at once. Pass `--no-launch` to record the interview only, and `hops build ` to resume a system. `hops build --example ` (`churn-example`, `recs-example` or `helpdesk-example`) builds an example without the menu, and resumes it if it already exists. From b8af6dd3668bf4179c188511367f373d5c60578a Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Mon, 28 Sep 2026 23:40:21 +0200 Subject: [PATCH 10/42] [HWORKS-3275] ML software factory https://hopsworks.atlassian.net/browse/HWORKS-3275 AGENTS.md is in each ML system's directory, with a CLAUDE.md that imports it, and hops build and Brewer start Claude Code there. Selecting a system in Brewer brings its Terminal tab to the front, or opens one with Claude Code in the system's directory. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01RfHMsBbYDQZqoD58XVqeYE --- docs/user_guides/client_installation/index.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/user_guides/client_installation/index.md b/docs/user_guides/client_installation/index.md index 3dc4b83cde..3ceea470e9 100644 --- a/docs/user_guides/client_installation/index.md +++ b/docs/user_guides/client_installation/index.md @@ -123,7 +123,7 @@ Each answer is written to `/system.yaml` in the current directory as you g A new data source is created with `hops datasource create`, and its password or key is read without echo and passed to it in an environment variable, so it never appears on the command line or in `system.yaml`. When the interview is done, `hops build` starts Claude Code with `/hops-build `, which completes the specification and builds the feature, training and inference pipelines. -It also adds a section to the `AGENTS.md` at the root of the repository, saying the system is built from `system.yaml`: a coding agent there checks what a change to `system.yaml` means for the pipelines and the assets they create, and finds what a changed component affects downstream with `hops fg lineage`, `hops fv lineage`, `hops td lineage`, `hops model lineage` and `hops deployment lineage`. +Each system's directory holds an `AGENTS.md`, with a `CLAUDE.md` that imports it, saying the system is built from `system.yaml`; `hops build` and Brewer start Claude Code in that directory, so it reads them, checks what a change to `system.yaml` means for the pipelines and the assets they create, and finds what a changed component affects downstream with `hops fg lineage`, `hops fv lineage`, `hops td lineage`, `hops model lineage` and `hops deployment lineage`. Inside tmux, as in the Hopsworks terminal, it opens a new tmux window named after the system, so several systems can be built at once. Pass `--no-launch` to record the interview only, and `hops build ` to resume a system. `hops build --example ` (`churn-example`, `recs-example` or `helpdesk-example`) builds an example without the menu, and resumes it if it already exists. @@ -131,7 +131,7 @@ Pass `--no-launch` to record the interview only, and `hops build ` to resu `hops build` registers each system with the project, by the HopsFS directory of its code, or by its GitHub repository when you build from an external client. The Hopsworks UI shows a **Brewer** button (the software factory) beside **Terminal**, for every member of the project, when the project has registered systems or the cluster has the terminal. It opens a panel you can minimize or close, with a list of the project's systems: an open folder marks the ones whose code you can open, a lock the ones you cannot, and a link the ones in a GitHub repository. -Selecting a system opens its directory in the file browser and shows its phases, what is done and what is left. +Selecting a system opens its directory in the file browser and shows its phases, what is done and what is left, and brings its Terminal tab to the front, or opens one with Claude Code started in the system's directory when the Terminal is open. **show architecture** opens the system's architecture in the main view: its data sources, feature, training and inference pipelines and app, with the data flowing between them, redrawn as `system.yaml` changes. A box whose part of the specification changed since you last looked is marked until you click it; clicking a box shows that part of `system.yaml`, which you can edit and save, and boxes can be dragged. A system whose directory is deleted disappears from the list. From 2d1fa4c1fc769490e0c063731bb15f60b18c9090 Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Tue, 29 Sep 2026 06:15:56 +0200 Subject: [PATCH 11/42] [HWORKS-3275] ML software factory https://hopsworks.atlassian.net/browse/HWORKS-3275 A system's directory holds only AGENTS.md for coding agents; the CLAUDE.md that imported it is gone. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01RfHMsBbYDQZqoD58XVqeYE --- docs/user_guides/client_installation/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user_guides/client_installation/index.md b/docs/user_guides/client_installation/index.md index 3ceea470e9..bd847619d6 100644 --- a/docs/user_guides/client_installation/index.md +++ b/docs/user_guides/client_installation/index.md @@ -123,7 +123,7 @@ Each answer is written to `/system.yaml` in the current directory as you g A new data source is created with `hops datasource create`, and its password or key is read without echo and passed to it in an environment variable, so it never appears on the command line or in `system.yaml`. When the interview is done, `hops build` starts Claude Code with `/hops-build `, which completes the specification and builds the feature, training and inference pipelines. -Each system's directory holds an `AGENTS.md`, with a `CLAUDE.md` that imports it, saying the system is built from `system.yaml`; `hops build` and Brewer start Claude Code in that directory, so it reads them, checks what a change to `system.yaml` means for the pipelines and the assets they create, and finds what a changed component affects downstream with `hops fg lineage`, `hops fv lineage`, `hops td lineage`, `hops model lineage` and `hops deployment lineage`. +Each system's directory holds an `AGENTS.md` saying the system is built from `system.yaml`; `hops build` and Brewer start Claude Code in that directory, so it reads it, checks what a change to `system.yaml` means for the pipelines and the assets they create, and finds what a changed component affects downstream with `hops fg lineage`, `hops fv lineage`, `hops td lineage`, `hops model lineage` and `hops deployment lineage`. Inside tmux, as in the Hopsworks terminal, it opens a new tmux window named after the system, so several systems can be built at once. Pass `--no-launch` to record the interview only, and `hops build ` to resume a system. `hops build --example ` (`churn-example`, `recs-example` or `helpdesk-example`) builds an example without the menu, and resumes it if it already exists. From 56a4b694c9de3233ad5b125b421f37095eb356d9 Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Tue, 29 Sep 2026 06:53:53 +0200 Subject: [PATCH 12/42] [HWORKS-3275] ML software factory https://hopsworks.atlassian.net/browse/HWORKS-3275 Brewer's links are system.yaml and architecture, and Delete removes a system's entry, optionally with the assets it created and its GitHub repository; hops mlsystem delete does the same from a shell. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01RfHMsBbYDQZqoD58XVqeYE --- docs/user_guides/client_installation/index.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/user_guides/client_installation/index.md b/docs/user_guides/client_installation/index.md index bd847619d6..7db24b5a1c 100644 --- a/docs/user_guides/client_installation/index.md +++ b/docs/user_guides/client_installation/index.md @@ -132,15 +132,17 @@ Pass `--no-launch` to record the interview only, and `hops build ` to resu The Hopsworks UI shows a **Brewer** button (the software factory) beside **Terminal**, for every member of the project, when the project has registered systems or the cluster has the terminal. It opens a panel you can minimize or close, with a list of the project's systems: an open folder marks the ones whose code you can open, a lock the ones you cannot, and a link the ones in a GitHub repository. Selecting a system opens its directory in the file browser and shows its phases, what is done and what is left, and brings its Terminal tab to the front, or opens one with Claude Code started in the system's directory when the Terminal is open. -**show architecture** opens the system's architecture in the main view: its data sources, feature, training and inference pipelines and app, with the data flowing between them, redrawn as `system.yaml` changes. +**architecture** opens the system's architecture in the main view: its data sources, feature, training and inference pipelines and app, with the data flowing between them, redrawn as `system.yaml` changes. A box whose part of the specification changed since you last looked is marked until you click it; clicking a box shows that part of `system.yaml`, which you can edit and save, and boxes can be dragged. A system whose directory is deleted disappears from the list. +**Delete** asks what to delete: the system's entry in the list only, that and every asset the system created (its app, deployments, jobs, models, feature view and training data, the feature groups it writes, the data sources it created and its cloned environments; feature groups it only reads are kept), or those and its GitHub repository, which is deleted only when the build created it for this system alone. The assets are deleted in the terminal, downstream first, and the entry last, so a delete that fails part way leaves the system in the list to be deleted again. The code directory is kept. **Create example ML system**, the last entry in the list, asks which example to build, starts the terminal and runs `hops build --example ` in it. ```bash hops mlsystem list # the project's systems and whether you can open their code hops mlsystem register [--name N] # register or refresh one by hand hops mlsystem remove # remove it from the list; its code is kept +hops mlsystem delete --assets [--repo] # also delete what it created, and its repository ``` ## Hopsworks Java Library From 59c3a161f2e0a4c9130c24dea72c4101c3d206e4 Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Tue, 29 Sep 2026 12:35:46 +0200 Subject: [PATCH 13/42] [HWORKS-3275] ML software factory https://hopsworks.atlassian.net/browse/HWORKS-3275 The help desk example is described: documents uploaded to Resources/helpdesk-docs, embedded into a vector-indexed feature group, a LangGraph agent over them and the customer's events, and the LLM endpoint and key kept as account environment variables. The agent deployment example serves the paths a prediction request arrives on, /predict and /v1/models/:predict, where it served only /query. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01RfHMsBbYDQZqoD58XVqeYE --- docs/user_guides/agents/deployments/index.md | 18 ++++++++++++++++++ docs/user_guides/client_installation/index.md | 2 ++ 2 files changed, 20 insertions(+) diff --git a/docs/user_guides/agents/deployments/index.md b/docs/user_guides/agents/deployments/index.md index 7c7a7e91b4..6bf6c09130 100644 --- a/docs/user_guides/agents/deployments/index.md +++ b/docs/user_guides/agents/deployments/index.md @@ -119,6 +119,7 @@ The entrypoint links to the file in the repository at that commit. The file below shows a simple agent program that uses LlamaIndex, FastAPI, and OpenTelemetry. +The deployment runs the file with `python`, so the file is the server: it listens on port 8080, answers the readiness probe on `/`, and serves `/predict` and `/v1/models/:predict`, the two paths a prediction request arrives on. Set `ANTHROPIC_API_KEY` in the deployment environment. Hopsworks injects the `OTEL_EXPORTER_OTLP_*` environment variables for the deployment, so the @@ -218,6 +219,23 @@ FastAPIInstrumentor.instrument_app( ) +@agent_app.get("/") +def ready(): + return {"status": "ok"} + + +# Where the Hopsworks inference endpoint forwards a request. +@agent_app.post("/predict") +def predict(payload: dict): + return predictor.predict(payload.get("instances", [payload])[0]) + + +# KServe's path, which deployment.predict() uses inside the cluster. +@agent_app.post("/v1/models/{target}") +def kserve_predict(target: str, payload: dict): + return predictor.predict(payload.get("instances", [payload])[0]) + + @agent_app.post("/query") def query(payload: dict): return predictor.predict(payload) diff --git a/docs/user_guides/client_installation/index.md b/docs/user_guides/client_installation/index.md index 7db24b5a1c..0e5ff7ba03 100644 --- a/docs/user_guides/client_installation/index.md +++ b/docs/user_guides/client_installation/index.md @@ -116,6 +116,8 @@ hops build ``` `hops build` first asks what you want to build: a new ML system, which it asks you to describe, or an example ML system (churn, batch; personalized recommendations, real-time; a help desk agent, agentic) that runs on synthetic data and includes an app. +The help desk agent answers from documents you upload to `Resources/helpdesk-docs` (PDF, text, Markdown, Word or OpenDocument), which a job cuts into passages and embeds with a sentence-transformers model downloaded into the Model Registry, and from the customer's recent events; it is a LangGraph agent deployment with a JavaScript chat app. +For it, `hops build` asks for an OpenAI-compatible LLM endpoint, model and API key (read without echo) and saves them as your account environment variables `LLM_URL`, `LLM_MODEL` and `LLM_API_KEY`, which the agent reads. It then asks the questions that follow from the system type: how often predictions are made for a batch system, the latency and throughput for a real-time one, the LLM for an agentic one, the data to learn from, how the predictions are used, and where the code goes. One Claude Code call on Haiku reads your description and recommends the system type and a name; the other questions are plain prompts. Each answer is written to `/system.yaml` in the current directory as you give it. From 722c931d64a721f1ada6a82a6bfd50bee8601f3b Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Wed, 30 Sep 2026 09:55:55 +0200 Subject: [PATCH 14/42] [HWORKS-3275] ML software factory https://hopsworks.atlassian.net/browse/HWORKS-3275 Brewer: a repository the build creates is named hops-, and deleting a system's assets also deletes its code directory, which the page said was kept. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01RfHMsBbYDQZqoD58XVqeYE --- docs/user_guides/client_installation/index.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/user_guides/client_installation/index.md b/docs/user_guides/client_installation/index.md index 0e5ff7ba03..a60af24202 100644 --- a/docs/user_guides/client_installation/index.md +++ b/docs/user_guides/client_installation/index.md @@ -131,13 +131,14 @@ Pass `--no-launch` to record the interview only, and `hops build ` to resu `hops build --example ` (`churn-example`, `recs-example` or `helpdesk-example`) builds an example without the menu, and resumes it if it already exists. `hops build` registers each system with the project, by the HopsFS directory of its code, or by its GitHub repository when you build from an external client. +A GitHub repository the build creates, an example's included, is named `hops-`, or `hops--` when you already have one of that name. The Hopsworks UI shows a **Brewer** button (the software factory) beside **Terminal**, for every member of the project, when the project has registered systems or the cluster has the terminal. It opens a panel you can minimize or close, with a list of the project's systems: an open folder marks the ones whose code you can open, a lock the ones you cannot, and a link the ones in a GitHub repository. Selecting a system opens its directory in the file browser and shows its phases, what is done and what is left, and brings its Terminal tab to the front, or opens one with Claude Code started in the system's directory when the Terminal is open. **architecture** opens the system's architecture in the main view: its data sources, feature, training and inference pipelines and app, with the data flowing between them, redrawn as `system.yaml` changes. A box whose part of the specification changed since you last looked is marked until you click it; clicking a box shows that part of `system.yaml`, which you can edit and save, and boxes can be dragged. A system whose directory is deleted disappears from the list. -**Delete** asks what to delete: the system's entry in the list only, that and every asset the system created (its app, deployments, jobs, models, feature view and training data, the feature groups it writes, the data sources it created and its cloned environments; feature groups it only reads are kept), or those and its GitHub repository, which is deleted only when the build created it for this system alone. The assets are deleted in the terminal, downstream first, and the entry last, so a delete that fails part way leaves the system in the list to be deleted again. The code directory is kept. +**Delete** asks what to delete: the system's entry in the list only, that and every asset the system created (its app, deployments, jobs, models, feature view and training data, the feature groups it writes, the data sources it created and its cloned environments; feature groups it only reads are kept), or those and its GitHub repository, which is deleted only when the build created it for this system alone. The assets are deleted in the terminal, downstream first, and the entry last, so a delete that fails part way leaves the system in the list to be deleted again. Deleting the assets also deletes the code directory; deleting the entry only keeps it. **Create example ML system**, the last entry in the list, asks which example to build, starts the terminal and runs `hops build --example ` in it. ```bash From 22910fdc311be5e53deb078ddc7592f06125e45c Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Wed, 30 Sep 2026 11:11:31 +0200 Subject: [PATCH 15/42] [HWORKS-3275] ML software factory https://hopsworks.atlassian.net/browse/HWORKS-3275 Brewer: the UI's ML systems are the Factory page under AI/ML, with the requirements form, the system page, Login to GitHub, and Back from the architecture and status pages. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01RfHMsBbYDQZqoD58XVqeYE --- docs/user_guides/client_installation/index.md | 20 +++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-) diff --git a/docs/user_guides/client_installation/index.md b/docs/user_guides/client_installation/index.md index a60af24202..aa117b39d7 100644 --- a/docs/user_guides/client_installation/index.md +++ b/docs/user_guides/client_installation/index.md @@ -125,21 +125,29 @@ Each answer is written to `/system.yaml` in the current directory as you g A new data source is created with `hops datasource create`, and its password or key is read without echo and passed to it in an environment variable, so it never appears on the command line or in `system.yaml`. When the interview is done, `hops build` starts Claude Code with `/hops-build `, which completes the specification and builds the feature, training and inference pipelines. -Each system's directory holds an `AGENTS.md` saying the system is built from `system.yaml`; `hops build` and Brewer start Claude Code in that directory, so it reads it, checks what a change to `system.yaml` means for the pipelines and the assets they create, and finds what a changed component affects downstream with `hops fg lineage`, `hops fv lineage`, `hops td lineage`, `hops model lineage` and `hops deployment lineage`. +Each system's directory holds an `AGENTS.md` saying the system is built from `system.yaml`; `hops build` and the **Factory** page start Claude Code in that directory, so it reads it, checks what a change to `system.yaml` means for the pipelines and the assets they create, and finds what a changed component affects downstream with `hops fg lineage`, `hops fv lineage`, `hops td lineage`, `hops model lineage` and `hops deployment lineage`. Inside tmux, as in the Hopsworks terminal, it opens a new tmux window named after the system, so several systems can be built at once. Pass `--no-launch` to record the interview only, and `hops build ` to resume a system. `hops build --example ` (`churn-example`, `recs-example` or `helpdesk-example`) builds an example without the menu, and resumes it if it already exists. `hops build` registers each system with the project, by the HopsFS directory of its code, or by its GitHub repository when you build from an external client. A GitHub repository the build creates, an example's included, is named `hops-`, or `hops--` when you already have one of that name. -The Hopsworks UI shows a **Brewer** button (the software factory) beside **Terminal**, for every member of the project, when the project has registered systems or the cluster has the terminal. -It opens a panel you can minimize or close, with a list of the project's systems: an open folder marks the ones whose code you can open, a lock the ones you cannot, and a link the ones in a GitHub repository. -Selecting a system opens its directory in the file browser and shows its phases, what is done and what is left, and brings its Terminal tab to the front, or opens one with Claude Code started in the system's directory when the Terminal is open. -**architecture** opens the system's architecture in the main view: its data sources, feature, training and inference pipelines and app, with the data flowing between them, redrawn as `system.yaml` changes. +**Factory**, under AI/ML in the project menu, lists the project's systems for every member, when the project has registered systems or the cluster has the terminal: each with its type, status, phases done, owner and last update, and an open folder for the ones whose code you can open, a lock for the ones you cannot, and a link for the ones in a GitHub repository. +**Login to GitHub** runs `github-login` in a Terminal tab; the page shows whether the terminal's GitHub CLI is logged in, which the build needs to create the repository. +**New** opens the main requirements of a new system: its name, which is also its directory's and, as `hops-`, its GitHub repository's (lowercase letters, digits and hyphens), what it should predict, its type (batch, real-time or agentic) with the cadence or the latency and throughput, its data (feature groups in the project, or synthetic data described in a sentence), and how its predictions are used. +For an agentic system the LLM's endpoint, model and key are saved as your account environment variables, `LLM_URL`, `LLM_MODEL` and `LLM_API_KEY`. +**New**, then **Example**, opens one of the example systems with its requirements filled in. +**Create** runs `hops build --answers` in a Terminal tab named after the system, which asks only what the page left out and starts Claude Code on `/hops-build `; the page then locks the requirements and opens the system once it is registered. + +A system's page shows its phases, what is done and what is left, what it has made, and its requirements, locked. +**Open in Terminal** brings the system's Terminal tab to the front, or opens one with Claude Code started in its directory. +**system.yaml** opens the specification to read or edit. +**Architecture** opens the system's architecture: its data sources, feature, training and inference pipelines and app, with the data flowing between them, redrawn as `system.yaml` changes. A box whose part of the specification changed since you last looked is marked until you click it; clicking a box shows that part of `system.yaml`, which you can edit and save, and boxes can be dragged. +**Status**, once every phase is done, checks the system's jobs over the last day and its deployments and app, and shows the report. +**Back** on the architecture and status pages, like the browser's back button, returns to the system's page. A system whose directory is deleted disappears from the list. **Delete** asks what to delete: the system's entry in the list only, that and every asset the system created (its app, deployments, jobs, models, feature view and training data, the feature groups it writes, the data sources it created and its cloned environments; feature groups it only reads are kept), or those and its GitHub repository, which is deleted only when the build created it for this system alone. The assets are deleted in the terminal, downstream first, and the entry last, so a delete that fails part way leaves the system in the list to be deleted again. Deleting the assets also deletes the code directory; deleting the entry only keeps it. -**Create example ML system**, the last entry in the list, asks which example to build, starts the terminal and runs `hops build --example ` in it. ```bash hops mlsystem list # the project's systems and whether you can open their code From c073aca27c0b8cd9acf9e881208ff3e5fe8d0e90 Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Wed, 30 Sep 2026 15:54:47 +0200 Subject: [PATCH 16/42] [HWORKS-3275] ML software factory https://hopsworks.atlassian.net/browse/HWORKS-3275 Terminal: Start the Terminal documents the start screen: Resources, Spark Cluster and the LLM Provider, where the key is kept and which coding agents use it. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01RfHMsBbYDQZqoD58XVqeYE --- .../jupyter/session_capacity_warnings.md | 4 +-- docs/user_guides/projects/terminal/start.md | 33 +++++++++++++++++++ mkdocs.yml | 1 + 3 files changed, 36 insertions(+), 2 deletions(-) create mode 100644 docs/user_guides/projects/terminal/start.md diff --git a/docs/user_guides/projects/jupyter/session_capacity_warnings.md b/docs/user_guides/projects/jupyter/session_capacity_warnings.md index 36e52c5fd8..54f7498492 100644 --- a/docs/user_guides/projects/jupyter/session_capacity_warnings.md +++ b/docs/user_guides/projects/jupyter/session_capacity_warnings.md @@ -12,7 +12,7 @@ The badges sit next to the action button on three pages: ![WebSocket warnings on the Jupyter page](../../../assets/images/guides/jupyter/websocket-warnings-jupyter.png) -- **Terminal**: next to the `Start Terminal` button inside the terminal panel. +- **Terminal**: next to the `Start` button inside the terminal panel. ![WebSocket warnings on the Terminal panel](../../../assets/images/guides/jupyter/websocket-warnings-terminal.png) @@ -38,7 +38,7 @@ On a single-instance deployment the two badges always agree, since the only inst ## What happens when a badge turns red -- `Run Jupyter`, `Start Terminal`, and per-row `Start App` buttons are disabled while the **cluster** badge is red (no instance has capacity to serve a new session). +- `Run Jupyter`, the terminal's `Start`, and per-row `Start App` buttons are disabled while the **cluster** badge is red (no instance has capacity to serve a new session). While only the instance badge is red the buttons stay enabled because refreshing or signing back in may land you on a different instance pod that still has capacity. - An already-running Jupyter server keeps working. Opening a new notebook tab inside a running Jupyter server may still fail if the pod that hosts it is at its per-session cap: the new tab's kernel WebSocket upgrade is closed with a `1013 TRY_AGAIN_LATER` close rather than attaching. diff --git a/docs/user_guides/projects/terminal/start.md b/docs/user_guides/projects/terminal/start.md new file mode 100644 index 0000000000..f80b7c9faf --- /dev/null +++ b/docs/user_guides/projects/terminal/start.md @@ -0,0 +1,33 @@ +--- +description: Start the Hopsworks terminal with its resources, an optional Spark cluster, and the LLM provider its coding agents use +--- + +# Start the Terminal + +The terminal panel opens on a **Start** button, with keyboard focus on it, and a line under it saying what it starts, such as `Python, 1 CPU, 2048 MB`. +Its settings are kept in your browser, so the next visit starts the same terminal without entering them again. + +## Resources and Spark Cluster + +**Settings**, then **Resources**, sets the terminal's CPU cores and memory, and its GPUs when the cluster has GPU nodes. +With more than 0 GPUs the GPU terminal starts; otherwise the Python terminal. +While a terminal runs, **Apply & Restart** restarts it with the new resources. + +**Settings**, then **Spark Cluster**, starts a Spark cluster with the terminal when **Start a Spark cluster with the terminal** is checked; it is unchecked by default. +The terminal pod is then the Spark driver, with the driver cores and memory set there, and runs the given number of executors, with Spark Connect optional. +A Spark terminal has no GPUs. + +## LLM Provider + +**LLM Provider**, to the right of **Start**, shows the provider the terminal's coding agents use; **Subscription**, the default, leaves `claude`, `codex`, `copilot` and `opencode` on their own logins. +Clicking it shows the providers: Kimi, DeepMind, Berget, GLM, Anthropic, OpenAI, Meta and Grok. +For a provider, pick a model or type a model id, enter its API key and press **Save key**, then **Test**, which checks the key against the provider's model list without spending tokens and says whether the model is listed. + +The key is saved as a private secret in your Hopsworks account, `llm__api_key`, not in the browser; the provider and model are kept in the browser. +When the terminal starts, the key is set as the provider's own variable (for example `KIMI_API_KEY`), and: + +- `codex` and `opencode` use the provider and model: every provider is written into `~/.codex/config.toml` as `model_providers.hops-`, so switching needs only that provider's key. +- `claude` uses them for providers with an Anthropic-compatible API (Kimi, GLM and Anthropic), through `ANTHROPIC_BASE_URL`, `ANTHROPIC_AUTH_TOKEN` and `ANTHROPIC_MODEL`; for the others it keeps its own login. +- `copilot` keeps its GitHub login. + +A provider with no saved key disables **Start** until you save one or pick **Subscription**. diff --git a/mkdocs.yml b/mkdocs.yml index b8e126d025..20b06137dc 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -175,6 +175,7 @@ nav: - Session Capacity Warnings: user_guides/projects/jupyter/session_capacity_warnings.md - Apps: user_guides/projects/apps/index.md - Terminal: + - Start the Terminal: user_guides/projects/terminal/start.md - Teleport Sessions: user_guides/projects/terminal/teleport.md - Jobs: - Run Python Job: user_guides/projects/jobs/python_job.md From e78d6219e0863ce4dd7d29ad49cde6b4d7b5bef0 Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Wed, 30 Sep 2026 16:12:09 +0200 Subject: [PATCH 17/42] [HWORKS-3275] ML software factory https://hopsworks.atlassian.net/browse/HWORKS-3275 Terminal: the start screen has no summary line under Start. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01RfHMsBbYDQZqoD58XVqeYE --- docs/user_guides/projects/terminal/start.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user_guides/projects/terminal/start.md b/docs/user_guides/projects/terminal/start.md index f80b7c9faf..f1ce770e2e 100644 --- a/docs/user_guides/projects/terminal/start.md +++ b/docs/user_guides/projects/terminal/start.md @@ -4,7 +4,7 @@ description: Start the Hopsworks terminal with its resources, an optional Spark # Start the Terminal -The terminal panel opens on a **Start** button, with keyboard focus on it, and a line under it saying what it starts, such as `Python, 1 CPU, 2048 MB`. +The terminal panel opens on a **Start** button, with keyboard focus on it. Its settings are kept in your browser, so the next visit starts the same terminal without entering them again. ## Resources and Spark Cluster From b6633c67ea0811ea60bcdd28b4f7b1b144ac0413 Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Wed, 30 Sep 2026 16:22:38 +0200 Subject: [PATCH 18/42] [HWORKS-3275] ML software factory https://hopsworks.atlassian.net/browse/HWORKS-3275 Terminal: which Claude Code variables a provider sets, and the warning for a settings.json env block that overrides them. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01RfHMsBbYDQZqoD58XVqeYE --- docs/user_guides/projects/terminal/start.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/user_guides/projects/terminal/start.md b/docs/user_guides/projects/terminal/start.md index f1ce770e2e..0ce494768e 100644 --- a/docs/user_guides/projects/terminal/start.md +++ b/docs/user_guides/projects/terminal/start.md @@ -27,7 +27,8 @@ The key is saved as a private secret in your Hopsworks account, `llm__ When the terminal starts, the key is set as the provider's own variable (for example `KIMI_API_KEY`), and: - `codex` and `opencode` use the provider and model: every provider is written into `~/.codex/config.toml` as `model_providers.hops-`, so switching needs only that provider's key. -- `claude` uses them for providers with an Anthropic-compatible API (Kimi, GLM and Anthropic), through `ANTHROPIC_BASE_URL`, `ANTHROPIC_AUTH_TOKEN` and `ANTHROPIC_MODEL`; for the others it keeps its own login. +- `claude` uses them for providers with an Anthropic-compatible API (Kimi, GLM and Anthropic), through `ANTHROPIC_BASE_URL`, `ANTHROPIC_AUTH_TOKEN`, `ANTHROPIC_MODEL` and the model variables each provider's Claude Code guide sets (for Kimi, `kimi-k3[1m]` with its 1M-token context and `kimi-k2.7-code` for the background tier); for the others it keeps its own login. + An `env` block in `~/.claude/settings.json` overrides these, so the terminal warns at start when it sets any of them. - `copilot` keeps its GitHub login. A provider with no saved key disables **Start** until you save one or pick **Subscription**. From c45572324d6301b8fac44337f9f723b452f0ee6e Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Wed, 30 Sep 2026 16:32:35 +0200 Subject: [PATCH 19/42] [HWORKS-3275] ML software factory https://hopsworks.atlassian.net/browse/HWORKS-3275 Terminal: Meta runs Claude Code too, codex gets each provider's model settings, and the provider picker sits under Start as cards. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01RfHMsBbYDQZqoD58XVqeYE --- docs/user_guides/projects/terminal/start.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/user_guides/projects/terminal/start.md b/docs/user_guides/projects/terminal/start.md index 0ce494768e..fdd6c44cbb 100644 --- a/docs/user_guides/projects/terminal/start.md +++ b/docs/user_guides/projects/terminal/start.md @@ -19,15 +19,15 @@ A Spark terminal has no GPUs. ## LLM Provider -**LLM Provider**, to the right of **Start**, shows the provider the terminal's coding agents use; **Subscription**, the default, leaves `claude`, `codex`, `copilot` and `opencode` on their own logins. -Clicking it shows the providers: Kimi, DeepMind, Berget, GLM, Anthropic, OpenAI, Meta and Grok. +**LLM Provider**, under **Start**, shows the provider the terminal's coding agents use; **Subscription**, the default, leaves `claude`, `codex`, `copilot` and `opencode` on their own logins. +Clicking it shows the providers as cards, each saying whether its key is saved: Kimi, DeepMind, Berget, GLM, Anthropic, OpenAI, Meta and Grok. For a provider, pick a model or type a model id, enter its API key and press **Save key**, then **Test**, which checks the key against the provider's model list without spending tokens and says whether the model is listed. The key is saved as a private secret in your Hopsworks account, `llm__api_key`, not in the browser; the provider and model are kept in the browser. When the terminal starts, the key is set as the provider's own variable (for example `KIMI_API_KEY`), and: -- `codex` and `opencode` use the provider and model: every provider is written into `~/.codex/config.toml` as `model_providers.hops-`, so switching needs only that provider's key. -- `claude` uses them for providers with an Anthropic-compatible API (Kimi, GLM and Anthropic), through `ANTHROPIC_BASE_URL`, `ANTHROPIC_AUTH_TOKEN`, `ANTHROPIC_MODEL` and the model variables each provider's Claude Code guide sets (for Kimi, `kimi-k3[1m]` with its 1M-token context and `kimi-k2.7-code` for the background tier); for the others it keeps its own login. +- `codex` and `opencode` use the provider and model: every provider is written into `~/.codex/config.toml` as `model_providers.hops-`, so switching needs only that provider's key, and codex gets the model settings the provider's guide gives, such as Meta's and Kimi's 1M-token context window. +- `claude` uses them for providers with an Anthropic-compatible API (Kimi, GLM, Anthropic and Meta), through `ANTHROPIC_BASE_URL`, `ANTHROPIC_AUTH_TOKEN`, `ANTHROPIC_MODEL` and the model variables each provider's Claude Code guide sets (for Kimi, `kimi-k3[1m]` with its 1M-token context and `kimi-k2.7-code` for the background tier); for the others it keeps its own login. An `env` block in `~/.claude/settings.json` overrides these, so the terminal warns at start when it sets any of them. - `copilot` keeps its GitHub login. From 54933c20449235da3d55f384742943015625aae9 Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Wed, 30 Sep 2026 17:07:41 +0200 Subject: [PATCH 20/42] [HWORKS-3275] ML software factory https://hopsworks.atlassian.net/browse/HWORKS-3275 Terminal: OpenRouter is an LLM provider, Claude Code included. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01RfHMsBbYDQZqoD58XVqeYE --- docs/user_guides/projects/terminal/start.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/user_guides/projects/terminal/start.md b/docs/user_guides/projects/terminal/start.md index fdd6c44cbb..ca261b783b 100644 --- a/docs/user_guides/projects/terminal/start.md +++ b/docs/user_guides/projects/terminal/start.md @@ -20,14 +20,14 @@ A Spark terminal has no GPUs. ## LLM Provider **LLM Provider**, under **Start**, shows the provider the terminal's coding agents use; **Subscription**, the default, leaves `claude`, `codex`, `copilot` and `opencode` on their own logins. -Clicking it shows the providers as cards, each saying whether its key is saved: Kimi, DeepMind, Berget, GLM, Anthropic, OpenAI, Meta and Grok. +Clicking it shows the providers as cards, each saying whether its key is saved: Kimi, DeepMind, Berget, GLM, Anthropic, OpenAI, Meta, OpenRouter and Grok. For a provider, pick a model or type a model id, enter its API key and press **Save key**, then **Test**, which checks the key against the provider's model list without spending tokens and says whether the model is listed. The key is saved as a private secret in your Hopsworks account, `llm__api_key`, not in the browser; the provider and model are kept in the browser. When the terminal starts, the key is set as the provider's own variable (for example `KIMI_API_KEY`), and: - `codex` and `opencode` use the provider and model: every provider is written into `~/.codex/config.toml` as `model_providers.hops-`, so switching needs only that provider's key, and codex gets the model settings the provider's guide gives, such as Meta's and Kimi's 1M-token context window. -- `claude` uses them for providers with an Anthropic-compatible API (Kimi, GLM, Anthropic and Meta), through `ANTHROPIC_BASE_URL`, `ANTHROPIC_AUTH_TOKEN`, `ANTHROPIC_MODEL` and the model variables each provider's Claude Code guide sets (for Kimi, `kimi-k3[1m]` with its 1M-token context and `kimi-k2.7-code` for the background tier); for the others it keeps its own login. +- `claude` uses them for providers with an Anthropic-compatible API (Kimi, GLM, Anthropic, Meta and OpenRouter), through `ANTHROPIC_BASE_URL`, `ANTHROPIC_AUTH_TOKEN`, `ANTHROPIC_MODEL` and the model variables each provider's Claude Code guide sets (for Kimi, `kimi-k3[1m]` with its 1M-token context and `kimi-k2.7-code` for the background tier); for the others it keeps its own login. An `env` block in `~/.claude/settings.json` overrides these, so the terminal warns at start when it sets any of them. - `copilot` keeps its GitHub login. From e6e80e24724b004f42f0d97692e3bee7cbc0db3a Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Wed, 30 Sep 2026 17:28:34 +0200 Subject: [PATCH 21/42] [HWORKS-3275] ML software factory https://hopsworks.atlassian.net/browse/HWORKS-3275 Terminal: Grok is no longer an LLM provider. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01RfHMsBbYDQZqoD58XVqeYE --- docs/user_guides/projects/terminal/start.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user_guides/projects/terminal/start.md b/docs/user_guides/projects/terminal/start.md index ca261b783b..170f4f9d28 100644 --- a/docs/user_guides/projects/terminal/start.md +++ b/docs/user_guides/projects/terminal/start.md @@ -20,7 +20,7 @@ A Spark terminal has no GPUs. ## LLM Provider **LLM Provider**, under **Start**, shows the provider the terminal's coding agents use; **Subscription**, the default, leaves `claude`, `codex`, `copilot` and `opencode` on their own logins. -Clicking it shows the providers as cards, each saying whether its key is saved: Kimi, DeepMind, Berget, GLM, Anthropic, OpenAI, Meta, OpenRouter and Grok. +Clicking it shows the providers as cards, each saying whether its key is saved: Kimi, DeepMind, Berget, GLM, Anthropic, OpenAI, Meta and OpenRouter. For a provider, pick a model or type a model id, enter its API key and press **Save key**, then **Test**, which checks the key against the provider's model list without spending tokens and says whether the model is listed. The key is saved as a private secret in your Hopsworks account, `llm__api_key`, not in the browser; the provider and model are kept in the browser. From bab2ef282d9944d83388f365c95df4872518c80a Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Sun, 4 Oct 2026 12:16:16 +0200 Subject: [PATCH 22/42] [HWORKS-3275] ML software factory https://hopsworks.atlassian.net/browse/HWORKS-3275 Factory: it sits in the project menu below Catalog and holds two factories. The ML system factory's examples are under New ML System. A new section describes the medallion layer factory: the archived medallion_table tag, Tag as bronze tables in the dltHub ingestion review, New Medallion Layer and its silver tasks, the engine, the incremental job over [HOPS_START_TIME, HOPS_END_TIME), and hops medallion silver and delete. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01RfHMsBbYDQZqoD58XVqeYE --- docs/user_guides/client_installation/index.md | 29 +++++++++++++++++-- 1 file changed, 26 insertions(+), 3 deletions(-) diff --git a/docs/user_guides/client_installation/index.md b/docs/user_guides/client_installation/index.md index aa117b39d7..311688cf80 100644 --- a/docs/user_guides/client_installation/index.md +++ b/docs/user_guides/client_installation/index.md @@ -132,11 +132,12 @@ Pass `--no-launch` to record the interview only, and `hops build ` to resu `hops build` registers each system with the project, by the HopsFS directory of its code, or by its GitHub repository when you build from an external client. A GitHub repository the build creates, an example's included, is named `hops-`, or `hops--` when you already have one of that name. -**Factory**, under AI/ML in the project menu, lists the project's systems for every member, when the project has registered systems or the cluster has the terminal: each with its type, status, phases done, owner and last update, and an open folder for the ones whose code you can open, a lock for the ones you cannot, and a link for the ones in a GitHub repository. +**Factory**, in the project menu below Catalog, holds the ML system factory and the medallion layer factory. +Its ML system factory lists the project's systems for every member, when the project has registered systems or the cluster has the terminal: each with its type, status, phases done, owner and last update, and an open folder for the ones whose code you can open, a lock for the ones you cannot, and a link for the ones in a GitHub repository. **Login to GitHub** runs `github-login` in a Terminal tab; the page shows whether the terminal's GitHub CLI is logged in, which the build needs to create the repository. -**New** opens the main requirements of a new system: its name, which is also its directory's and, as `hops-`, its GitHub repository's (lowercase letters, digits and hyphens), what it should predict, its type (batch, real-time or agentic) with the cadence or the latency and throughput, its data (feature groups in the project, or synthetic data described in a sentence), and how its predictions are used. +**New ML System** opens the main requirements of a new system: its name, which is also its directory's and, as `hops-`, its GitHub repository's (lowercase letters, digits and hyphens), what it should predict, its type (batch, real-time or agentic) with the cadence or the latency and throughput, its data (feature groups in the project, or synthetic data described in a sentence), and how its predictions are used. For an agentic system the LLM's endpoint, model and key are saved as your account environment variables, `LLM_URL`, `LLM_MODEL` and `LLM_API_KEY`. -**New**, then **Example**, opens one of the example systems with its requirements filled in. +The examples are listed under **New ML System**; each opens with its requirements filled in. **Create** runs `hops build --answers` in a Terminal tab named after the system, which asks only what the page left out and starts Claude Code on `/hops-build `; the page then locks the requirements and opens the system once it is registered. A system's page shows its phases, what is done and what is left, what it has made, and its requirements, locked. @@ -156,6 +157,28 @@ hops mlsystem remove # remove it from the list; its code is hops mlsystem delete --assets [--repo] # also delete what it created, and its repository ``` +### Build a silver medallion layer + +A medallion layer organizes tables as bronze (raw data as it arrived), silver (cleansed and conformed) and gold (consumption-ready). +Hopsworks installs an archived schematized tag, `medallion_table`, whose `layer` is `bronze`, `silver` or `gold` and whose `lifecycle` is `dev`, `staging` or `prod`; every change of a table's value is kept in the tag history. +When you ingest data with a dltHub data source, **Tag as bronze tables** in the review, off by default, tags every feature group it creates as bronze. + +**New Medallion Layer**, in the Factory, builds a silver layer from the project's bronze feature groups. +A project without feature groups has nothing to build from: ingest raw data as bronze tables first. +The page asks for the bronze tables to build from (only those tagged bronze are shown while any are), the silver tasks (deduplicate, cast types, standardize values, handle nulls, validate with a rejects table, protect personal data, conform entities, surrogate keys, referential checks), additional tasks in your own words, the engine, the refresh cadence and the lifecycle. +The engine is dbt on Trino unless an additional task needs code SQL does not express well, when PySpark is suggested. +**Create** runs `hops medallion silver --answers` in a Terminal tab named after the layer, which records it in `/layer.yaml` and starts Claude Code on `/hops-silver `. + +The build profiles the bronze tables, designs the silver tables, writes and tests the code, backfills the whole bronze history once, schedules the job and tags the silver feature groups `layer: silver`, and verifies one window. +Silver tables are feature groups, materialized, never views. +Each scheduled run processes only the bronze rows that arrived in its window, `[HOPS_START_TIME, HOPS_END_TIME)`, which Hopsworks sets for every scheduled run, so the silver tables are refreshed incrementally; running a window again changes nothing. +A layer's page shows its phases, the silver tables and job it made, the bronze tables it reads, and its tasks; **Delete** removes it from the Factory, or also deletes its job, silver tables and directory, never the bronze tables. + +```bash +hops medallion silver --answers answers.json # record a silver layer and build it with Claude Code +hops medallion delete --assets # also delete its job, silver tables and directory +``` + ## Hopsworks Java Library If you want to interact with the Hopsworks Feature Store from environments such as Spark, Flink or Beam, you can use the Hopsworks Feature Store (Hopsworks) Java library. From a948d993fad9a5fcc842e4ff92dde05964e807b8 Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Sun, 4 Oct 2026 13:21:45 +0200 Subject: [PATCH 23/42] [HWORKS-3275] ML software factory https://hopsworks.atlassian.net/browse/HWORKS-3275 Silver tables are in third normal form, and a medallion layer's specification is its system.yaml, whose changes the layer's page offers to apply: a new lifecycle retags, a new refresh reschedules, and a changed task, engine or source writes new table versions. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01RfHMsBbYDQZqoD58XVqeYE --- docs/user_guides/client_installation/index.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/user_guides/client_installation/index.md b/docs/user_guides/client_installation/index.md index 311688cf80..efdf39ceee 100644 --- a/docs/user_guides/client_installation/index.md +++ b/docs/user_guides/client_installation/index.md @@ -167,11 +167,12 @@ When you ingest data with a dltHub data source, **Tag as bronze tables** in the A project without feature groups has nothing to build from: ingest raw data as bronze tables first. The page asks for the bronze tables to build from (only those tagged bronze are shown while any are), the silver tasks (deduplicate, cast types, standardize values, handle nulls, validate with a rejects table, protect personal data, conform entities, surrogate keys, referential checks), additional tasks in your own words, the engine, the refresh cadence and the lifecycle. The engine is dbt on Trino unless an additional task needs code SQL does not express well, when PySpark is suggested. -**Create** runs `hops medallion silver --answers` in a Terminal tab named after the layer, which records it in `/layer.yaml` and starts Claude Code on `/hops-silver `. +**Create** runs `hops medallion silver --answers` in a Terminal tab named after the layer, which records its specification in `/system.yaml` and starts Claude Code on `/hops-silver `. The build profiles the bronze tables, designs the silver tables, writes and tests the code, backfills the whole bronze history once, schedules the job and tags the silver feature groups `layer: silver`, and verifies one window. -Silver tables are feature groups, materialized, never views. +Silver tables are feature groups, materialized, never views, and in third normal form: one table per entity or event, every column depending on its table's key alone, lookups in tables of their own, and no aggregates, which belong in gold. Each scheduled run processes only the bronze rows that arrived in its window, `[HOPS_START_TIME, HOPS_END_TIME)`, which Hopsworks sets for every scheduled run, so the silver tables are refreshed incrementally; running a window again changes nothing. +`system.yaml` drives the layer's lifecycle, as an ML system's does: when it changes after the build (its bronze tables, tasks, engine, refresh or lifecycle), the layer's page says what changed and **Apply changes** runs `/hops-silver apply`, which retags for a new lifecycle, reschedules for a new refresh, and for a changed task, engine or source writes a new version of each silver table whose content changes, backfilled from the whole bronze history, and switches the job to it; earlier versions are kept. A layer's page shows its phases, the silver tables and job it made, the bronze tables it reads, and its tasks; **Delete** removes it from the Factory, or also deletes its job, silver tables and directory, never the bronze tables. ```bash From c9c57992f4772a136b587ee482982db531956fd2 Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Sun, 4 Oct 2026 16:07:30 +0200 Subject: [PATCH 24/42] [HWORKS-3275] ML software factory https://hopsworks.atlassian.net/browse/HWORKS-3275 The silver layer section describes the settings (history, deletes, schema changes, late-data lookback, the rejected-rows limit with an alert, freshness), lineage to the bronze tables, partitioning decided from the bronze files, the schedule's catch-up, and Status, Backfill, hops medallion status and hops medallion backfill. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01RfHMsBbYDQZqoD58XVqeYE --- docs/user_guides/client_installation/index.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/user_guides/client_installation/index.md b/docs/user_guides/client_installation/index.md index efdf39ceee..f4b4a1e749 100644 --- a/docs/user_guides/client_installation/index.md +++ b/docs/user_guides/client_installation/index.md @@ -167,16 +167,21 @@ When you ingest data with a dltHub data source, **Tag as bronze tables** in the A project without feature groups has nothing to build from: ingest raw data as bronze tables first. The page asks for the bronze tables to build from (only those tagged bronze are shown while any are), the silver tasks (deduplicate, cast types, standardize values, handle nulls, validate with a rejects table, protect personal data, conform entities, surrogate keys, referential checks), additional tasks in your own words, the engine, the refresh cadence and the lifecycle. The engine is dbt on Trino unless an additional task needs code SQL does not express well, when PySpark is suggested. +It also asks how the tables behave, with defaults: history (`latest`, one row per key, or `full`, every version by time), deletes in bronze (`ignore` or `propagate`), bronze schema changes (`fail`, or `evolve` by adding new columns), a late-data lookback re-read before each window (none, a day or a week), the share of rejected rows above which a run fails, with an alert when one fails, and a freshness target. **Create** runs `hops medallion silver --answers` in a Terminal tab named after the layer, which records its specification in `/system.yaml` and starts Claude Code on `/hops-silver `. The build profiles the bronze tables, designs the silver tables, writes and tests the code, backfills the whole bronze history once, schedules the job and tags the silver feature groups `layer: silver`, and verifies one window. Silver tables are feature groups, materialized, never views, and in third normal form: one table per entity or event, every column depending on its table's key alone, lookups in tables of their own, and no aggregates, which belong in gold. Each scheduled run processes only the bronze rows that arrived in its window, `[HOPS_START_TIME, HOPS_END_TIME)`, which Hopsworks sets for every scheduled run, so the silver tables are refreshed incrementally; running a window again changes nothing. `system.yaml` drives the layer's lifecycle, as an ML system's does: when it changes after the build (its bronze tables, tasks, engine, refresh or lifecycle), the layer's page says what changed and **Apply changes** runs `/hops-silver apply`, which retags for a new lifecycle, reschedules for a new refresh, and for a changed task, engine or source writes a new version of each silver table whose content changes, backfilled from the whole bronze history, and switches the job to it; earlier versions are kept. +Every silver table records its bronze tables as its parents, so the lineage shows them; its partitioning (none, or by hour, day or week) is decided from the volume and time span of the bronze table's files; and the job is scheduled with catch-up, so windows missed while the scheduler was down are replayed. +**Status** on a built layer's page reports the job's runs and each table's rows, last write against the freshness target, rejected share against the limit, and file layout, and **Backfill** reprocesses every bronze row into the silver tables. A layer's page shows its phases, the silver tables and job it made, the bronze tables it reads, and its tasks; **Delete** removes it from the Factory, or also deletes its job, silver tables and directory, never the bronze tables. ```bash hops medallion silver --answers answers.json # record a silver layer and build it with Claude Code +hops medallion status # write the layer's health report, status/report.html +hops medallion backfill # reprocess every bronze row into the silver tables hops medallion delete --assets # also delete its job, silver tables and directory ``` From fc4590772265970ce373848102b2ad14bf813860 Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Sun, 4 Oct 2026 18:07:38 +0200 Subject: [PATCH 25/42] [HWORKS-3275] ML software factory https://hopsworks.atlassian.net/browse/HWORKS-3275 Delete removes a silver layer, a gold layer or both; bronze is never deleted. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01RfHMsBbYDQZqoD58XVqeYE --- docs/user_guides/client_installation/index.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/user_guides/client_installation/index.md b/docs/user_guides/client_installation/index.md index f4b4a1e749..0615601d99 100644 --- a/docs/user_guides/client_installation/index.md +++ b/docs/user_guides/client_installation/index.md @@ -176,13 +176,13 @@ Each scheduled run processes only the bronze rows that arrived in its window, `[ `system.yaml` drives the layer's lifecycle, as an ML system's does: when it changes after the build (its bronze tables, tasks, engine, refresh or lifecycle), the layer's page says what changed and **Apply changes** runs `/hops-silver apply`, which retags for a new lifecycle, reschedules for a new refresh, and for a changed task, engine or source writes a new version of each silver table whose content changes, backfilled from the whole bronze history, and switches the job to it; earlier versions are kept. Every silver table records its bronze tables as its parents, so the lineage shows them; its partitioning (none, or by hour, day or week) is decided from the volume and time span of the bronze table's files; and the job is scheduled with catch-up, so windows missed while the scheduler was down are replayed. **Status** on a built layer's page reports the job's runs and each table's rows, last write against the freshness target, rejected share against the limit, and file layout, and **Backfill** reprocesses every bronze row into the silver tables. -A layer's page shows its phases, the silver tables and job it made, the bronze tables it reads, and its tasks; **Delete** removes it from the Factory, or also deletes its job, silver tables and directory, never the bronze tables. +A layer's page shows its phases, the silver tables and job it made, the bronze tables it reads, and its tasks; **Delete** removes it from the Factory, or deletes its silver layer, its gold layer, or both, with their jobs and tables; bronze tables are the source of truth and are never deleted. ```bash hops medallion silver --answers answers.json # record a silver layer and build it with Claude Code hops medallion status # write the layer's health report, status/report.html hops medallion backfill # reprocess every bronze row into the silver tables -hops medallion delete --assets # also delete its job, silver tables and directory +hops medallion delete --assets [--layer silver|gold] # delete the silver or gold layer, or both; never bronze ``` ## Hopsworks Java Library From 44946cbcfdea348f84f7e3c101c47ceac9638d1d Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Sun, 4 Oct 2026 18:57:58 +0200 Subject: [PATCH 26/42] [HWORKS-3275] ML software factory https://hopsworks.atlassian.net/browse/HWORKS-3275 Each bronze table of a silver layer has its own refresh and the layer one job per refresh; a layer's page shows its requirements, and the Status page has a box for asking Claude Code about the report. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01RfHMsBbYDQZqoD58XVqeYE --- docs/user_guides/client_installation/index.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/user_guides/client_installation/index.md b/docs/user_guides/client_installation/index.md index 0615601d99..014fe44a8b 100644 --- a/docs/user_guides/client_installation/index.md +++ b/docs/user_guides/client_installation/index.md @@ -167,6 +167,7 @@ When you ingest data with a dltHub data source, **Tag as bronze tables** in the A project without feature groups has nothing to build from: ingest raw data as bronze tables first. The page asks for the bronze tables to build from (only those tagged bronze are shown while any are), the silver tasks (deduplicate, cast types, standardize values, handle nulls, validate with a rejects table, protect personal data, conform entities, surrogate keys, referential checks), additional tasks in your own words, the engine, the refresh cadence and the lifecycle. The engine is dbt on Trino unless an additional task needs code SQL does not express well, when PySpark is suggested. +Each bronze table has its own refresh, hourly, daily or weekly; a silver table refreshes as often as its most frequently updated source, and the layer gets one job per refresh, each with its own schedule and freshness target. It also asks how the tables behave, with defaults: history (`latest`, one row per key, or `full`, every version by time), deletes in bronze (`ignore` or `propagate`), bronze schema changes (`fail`, or `evolve` by adding new columns), a late-data lookback re-read before each window (none, a day or a week), the share of rejected rows above which a run fails, with an alert when one fails, and a freshness target. **Create** runs `hops medallion silver --answers` in a Terminal tab named after the layer, which records its specification in `/system.yaml` and starts Claude Code on `/hops-silver `. @@ -175,7 +176,8 @@ Silver tables are feature groups, materialized, never views, and in third normal Each scheduled run processes only the bronze rows that arrived in its window, `[HOPS_START_TIME, HOPS_END_TIME)`, which Hopsworks sets for every scheduled run, so the silver tables are refreshed incrementally; running a window again changes nothing. `system.yaml` drives the layer's lifecycle, as an ML system's does: when it changes after the build (its bronze tables, tasks, engine, refresh or lifecycle), the layer's page says what changed and **Apply changes** runs `/hops-silver apply`, which retags for a new lifecycle, reschedules for a new refresh, and for a changed task, engine or source writes a new version of each silver table whose content changes, backfilled from the whole bronze history, and switches the job to it; earlier versions are kept. Every silver table records its bronze tables as its parents, so the lineage shows them; its partitioning (none, or by hour, day or week) is decided from the volume and time span of the bronze table's files; and the job is scheduled with catch-up, so windows missed while the scheduler was down are replayed. -**Status** on a built layer's page reports the job's runs and each table's rows, last write against the freshness target, rejected share against the limit, and file layout, and **Backfill** reprocesses every bronze row into the silver tables. +A layer's page shows its requirements as they were filled in. +**Status** on a built layer's page reports the job's runs and each table's rows, last write against the freshness target, rejected share against the limit, and file layout, with a box for asking Claude Code anything about the report, filled in with a request to fix the problems it found; **Backfill** reprocesses every bronze row into the silver tables, running each job. A layer's page shows its phases, the silver tables and job it made, the bronze tables it reads, and its tasks; **Delete** removes it from the Factory, or deletes its silver layer, its gold layer, or both, with their jobs and tables; bronze tables are the source of truth and are never deleted. ```bash From ab3d0345296b9ec9fc842123be2b3d6d2bfc2cb4 Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Sun, 4 Oct 2026 23:34:57 +0200 Subject: [PATCH 27/42] [HWORKS-3275] ML software factory https://hopsworks.atlassian.net/browse/HWORKS-3275 Document gold medallion layers: a Kimball star or snowflake model built from silver tables as data marts, each with its own requirements (business purpose, reusable tables, grain, metrics, change handling, verification, quality and access), tables and jobs at its own refresh, added, edited and deleted one by one. Silver layers gain per-job delete, Add tables, and links to their GitHub repository and dbt code, and the hops medallion command list covers gold, marts, job-delete and add-tables. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01RfHMsBbYDQZqoD58XVqeYE --- docs/user_guides/client_installation/index.md | 37 ++++++++++++++++--- 1 file changed, 32 insertions(+), 5 deletions(-) diff --git a/docs/user_guides/client_installation/index.md b/docs/user_guides/client_installation/index.md index 014fe44a8b..beb371a546 100644 --- a/docs/user_guides/client_installation/index.md +++ b/docs/user_guides/client_installation/index.md @@ -178,13 +178,40 @@ Each scheduled run processes only the bronze rows that arrived in its window, `[ Every silver table records its bronze tables as its parents, so the lineage shows them; its partitioning (none, or by hour, day or week) is decided from the volume and time span of the bronze table's files; and the job is scheduled with catch-up, so windows missed while the scheduler was down are replayed. A layer's page shows its requirements as they were filled in. **Status** on a built layer's page reports the job's runs and each table's rows, last write against the freshness target, rejected share against the limit, and file layout, with a box for asking Claude Code anything about the report, filled in with a request to fix the problems it found; **Backfill** reprocesses every bronze row into the silver tables, running each job. -A layer's page shows its phases, the silver tables and job it made, the bronze tables it reads, and its tasks; **Delete** removes it from the Factory, or deletes its silver layer, its gold layer, or both, with their jobs and tables; bronze tables are the source of truth and are never deleted. +A layer's page shows its phases, the silver tables and jobs it made, the bronze tables it reads, and its tasks, with links to the layer's GitHub repository (`hops-`, which the build creates and pushes every commit to) and to its dbt code in the file browser. +Every job has a delete icon that asks whether to also delete the feature groups only that job writes; **Add tables** adds bronze tables, each with its refresh, and describes the silver tables wanted from them, which Claude Code designs and builds with the rest. +**Delete** removes the layer from the Factory, or deletes it with its jobs, tables and directory; the tables a layer reads are never deleted. + +### Build a gold medallion layer of data marts + +**New Medallion Layer** with **gold** builds a gold layer from silver tables: a Kimball dimensional model, a star or snowflake schema, for the queries the layer will serve. +The page asks for those queries, the model, the silver tables to read, the standards every data mart follows (naming, modeling, documentation and quality, proposed and editable), and the first data mart. + +A gold layer is a set of data marts, each added, changed and deleted on its own, with its own fact and dimension tables and its own jobs, `--`, at its own refresh. +A data mart's requirements are: + +- **Business purpose**: who the analysts are, the decisions and reports it supports, and who approves its business definitions. +- **Existing tables**: the gold tables it can reuse, or new ones built from silver. +- **Row grain**: what one row represents, what identifies it, and whether rows are individual events, periodic snapshots or aggregates. +- **Metrics**: each metric's exact formula, exclusions, filters, currency and unit. +- **Freshness and changes**: the refresh and freshness target, how late arrivals, updates and deletes are processed, and whether corrections restate published results. +- **Verification**: example questions with the answers expected, in plain English; the totals that must reconcile, and with what; and how refreshes and reruns are proven correct. The build runs every check after the backfill and again after a refresh, records the results, and does not mark the mart built until each passes. +- **Quality and access**: the invariants to test, what happens when a check fails (fail the run, quarantine the failing rows, or warn), who may read which rows and columns, and the projects it is shared with. + +**Create** runs `hops medallion gold --answers` and starts Claude Code on `/hops-gold `, which builds each mart: its requirements, the design of its facts and dimensions (a dimension used by several marts is built once and shared), the dbt models with their tests, the backfill with the verification, the schedule and tags (`layer: gold`, the silver tables as parents), and a verified refresh. +The layer's page shows each mart with its phases, tables, jobs and verification results: **Edit** changes its requirements and applies the change, **Add data mart** adds one, and deleting a mart deletes its jobs and, if asked, its tables that no other mart lists. ```bash -hops medallion silver --answers answers.json # record a silver layer and build it with Claude Code -hops medallion status # write the layer's health report, status/report.html -hops medallion backfill # reprocess every bronze row into the silver tables -hops medallion delete --assets [--layer silver|gold] # delete the silver or gold layer, or both; never bronze +hops medallion silver --answers answers.json # record a silver layer and build it with Claude Code +hops medallion gold --answers answers.json # record a gold layer and its first data mart +hops medallion mart-add --answers mart.json # add a data mart to a gold layer +hops medallion mart-update --answers mart.json # change a data mart's requirements +hops medallion mart-delete [--tables] # delete a data mart's jobs, and its own tables +hops medallion job-delete [--tables] # delete one job, and the tables only it writes +hops medallion add-tables --answers new.json # add bronze tables to a silver layer +hops medallion status # write the layer's health report, status/report.html +hops medallion backfill # recompute the layer's tables from the whole history +hops medallion delete --assets # delete the layer, its jobs and tables; never what it reads ``` ## Hopsworks Java Library From a86221c635c4a269a4dcb55d926ca92b9f42d09d Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Mon, 5 Oct 2026 12:31:22 +0200 Subject: [PATCH 28/42] [HWORKS-3275] ML software factory https://hopsworks.atlassian.net/browse/HWORKS-3275 A medallion's silver and gold layers share one GitHub repository, hops-, each layer a directory in it; a gold layer joins the repository of the silver layer it reads, and the build reuses the repository or creates it. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01RfHMsBbYDQZqoD58XVqeYE --- docs/user_guides/client_installation/index.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/user_guides/client_installation/index.md b/docs/user_guides/client_installation/index.md index 5124199c20..22057c0eb5 100644 --- a/docs/user_guides/client_installation/index.md +++ b/docs/user_guides/client_installation/index.md @@ -188,7 +188,8 @@ Each scheduled run processes only the bronze rows that arrived in its window, `[ Every silver table records its bronze tables as its parents, so the lineage shows them; its partitioning (none, or by hour, day or week) is decided from the volume and time span of the bronze table's files; and the job is scheduled with catch-up, so windows missed while the scheduler was down are replayed. A layer's page shows its requirements as they were filled in. **Status** on a built layer's page reports the job's runs and each table's rows, last write against the freshness target, rejected share against the limit, and file layout, with a box for asking Claude Code anything about the report, filled in with a request to fix the problems it found; **Backfill** reprocesses every bronze row into the silver tables, running each job. -A layer's page shows its phases, the silver tables and jobs it made, the bronze tables it reads, and its tasks, with links to the layer's GitHub repository (`hops-`, which the build creates and pushes every commit to) and to its dbt code in the file browser. +A layer's page shows its phases, the silver tables and jobs it made, the bronze tables it reads, and its tasks, with links to the layer's GitHub repository and to its dbt code in the file browser. +A medallion's silver and gold layers share one GitHub repository, `hops-`, where the prefix is the layer's name without `-silver` or `-gold`: each layer is a directory in it, a gold layer joins the repository of the silver layer it reads, and the build uses the repository when it exists, creates it when it does not, and pushes every commit to it. Every job has a delete icon that asks whether to also delete the feature groups only that job writes; **Add tables** adds bronze tables, each with its refresh, and describes the silver tables wanted from them, which Claude Code designs and builds with the rest. **Delete** removes the layer from the Factory, or deletes it with its jobs, tables and directory; the tables a layer reads are never deleted. From 9d6913e89ef8125625be4744ad9955f820f688e5 Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Mon, 5 Oct 2026 22:34:58 +0200 Subject: [PATCH 29/42] [HWORKS-3275] ML software factory https://hopsworks.atlassian.net/browse/HWORKS-3275 A batch or real-time ML system built by the factory can now log its predictions' features and be monitored. The Factory's new system form ends with a Monitoring panel: a Feature logging checkbox, on by default, and a text box for what to monitor and alert on in the user's words. The answers reach system.yaml as requirements.monitoring {feature_logging, watch}. The client installation guide describes the panel. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01RfHMsBbYDQZqoD58XVqeYE --- docs/user_guides/client_installation/index.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/user_guides/client_installation/index.md b/docs/user_guides/client_installation/index.md index 22057c0eb5..844a798343 100644 --- a/docs/user_guides/client_installation/index.md +++ b/docs/user_guides/client_installation/index.md @@ -147,6 +147,8 @@ Its ML system factory lists the project's systems for every member, when the pro **Login to GitHub** runs `github-login` in a Terminal tab; the page shows whether the terminal's GitHub CLI is logged in, which the build needs to create the repository. **New ML System** opens the main requirements of a new system: its name, which is also its directory's and, as `hops-`, its GitHub repository's (lowercase letters, digits and hyphens), what it should predict, its type (batch, real-time or agentic) with the cadence or the latency and throughput, its data (feature groups in the project, or synthetic data described in a sentence), and how its predictions are used. For an agentic system the LLM's endpoint, model and key are saved as your account environment variables, `LLM_URL`, `LLM_MODEL` and `LLM_API_KEY`. +For a batch or real-time system, **Monitoring** sets whether every prediction logs the features it used (on by default) and, in your own words, what to monitor and alert on, such as drift in a feature against the training data or a failed job. +The build turns them into feature logging on the feature view, feature monitoring checks and alerts, and sends a failure alert for every job the system owns to the project's alert receiver. The examples are listed under **New ML System**; each opens with its requirements filled in. **Create** runs `hops build --answers` in a Terminal tab named after the system, which asks only what the page left out and starts Claude Code on `/hops-build `; the page then locks the requirements and opens the system once it is registered. From 2b2db1d3eb30a51a199029dc8c6f9851c4370aaa Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Tue, 6 Oct 2026 13:40:59 +0200 Subject: [PATCH 30/42] [HWORKS-3275] ML software factory https://hopsworks.atlassian.net/browse/HWORKS-3275 hops mlsystem -h did not list a create command, since a system is created with hops build, and hops medallion had the same gap with its silver and gold commands. The client installation guide lists both. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01RfHMsBbYDQZqoD58XVqeYE --- docs/user_guides/client_installation/index.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/user_guides/client_installation/index.md b/docs/user_guides/client_installation/index.md index 844a798343..b568d9f76d 100644 --- a/docs/user_guides/client_installation/index.md +++ b/docs/user_guides/client_installation/index.md @@ -163,6 +163,7 @@ A system whose directory is deleted disappears from the list. **Delete** asks what to delete: the system's entry in the list only, that and every asset the system created (its app, deployments, jobs, models, feature view and training data, the feature groups it writes, the data sources it created and its cloned environments; feature groups it only reads are kept), or those and its GitHub repository, which is deleted only when the build created it for this system alone. The assets are deleted in the terminal, downstream first, and the entry last, so a delete that fails part way leaves the system in the list to be deleted again. Deleting the assets also deletes the code directory; deleting the entry only keeps it. ```bash +hops mlsystem create [] # the same as hops build: interview, then build with Claude Code hops mlsystem list # the project's systems and whether you can open their code hops mlsystem register [--name N] # register or refresh one by hand hops mlsystem remove # remove it from the list; its code is kept @@ -217,6 +218,7 @@ The layer's page shows each mart with its phases, tables, jobs and verification ```bash hops medallion silver --answers answers.json # record a silver layer and build it with Claude Code hops medallion gold --answers answers.json # record a gold layer and its first data mart +hops medallion create silver|gold --answers answers.json # the same two commands hops medallion mart-add --answers mart.json # add a data mart to a gold layer hops medallion mart-update --answers mart.json # change a data mart's requirements hops medallion mart-delete [--tables] # delete a data mart's jobs, and its own tables From f1119844b35982bd921a76807c66a6765b9ac400 Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Tue, 6 Oct 2026 13:44:20 +0200 Subject: [PATCH 31/42] [HWORKS-3275] ML software factory https://hopsworks.atlassian.net/browse/HWORKS-3275 hops build is removed: an ML system is created with hops mlsystem create, which is the same interview and build, so the two names were one command too many. References to hops build now name hops mlsystem create; the /hops-build slash command keeps its name. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01RfHMsBbYDQZqoD58XVqeYE --- docs/user_guides/client_installation/index.md | 22 +++++++++---------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/docs/user_guides/client_installation/index.md b/docs/user_guides/client_installation/index.md index b568d9f76d..46c250ed2b 100644 --- a/docs/user_guides/client_installation/index.md +++ b/docs/user_guides/client_installation/index.md @@ -77,7 +77,7 @@ It then writes the following files into the current directory: | `AGENTS.md` | Instructions for the agent: the project you are connected to, where the `hopsworks` library is installed on this machine, and how to use the `hops` CLI and the skills. | | `.claude/skills/hops/SKILL.md` | A reference for the `hops` CLI. | | `.claude/commands/hops.md` | The `/hops` slash command for Claude Code: a fast menu to explore data, build or edit a Superset dashboard (`/hops dashboard`) or a Python app (`/hops app`), and show status. It runs on Haiku; the building is done by the agents below. | -| `.claude/commands/hops-ml.md` | The `/hops-ml` slash command: the `hops build` interview inside Claude Code, on Haiku, recorded in `system.yaml` as you answer. | +| `.claude/commands/hops-ml.md` | The `/hops-ml` slash command: the `hops mlsystem create` interview inside Claude Code, on Haiku, recorded in `system.yaml` as you answer. | | `.claude/commands/hops-build.md` | The `/hops-build` slash command: completes the specification the interview recorded and builds the ML system to a pull request, on your session's model. | | `.claude/agents/hops-dashboard-builder.md` | The Claude Code sub-agent `/hops` runs to build, edit or delete a dashboard. | | `.claude/agents/hops-app-builder.md` | The Claude Code sub-agent `/hops` runs to build, edit or delete an app and fix it until it serves. | @@ -122,25 +122,25 @@ hops skills show hops-fg ### Build an ML system ```bash -hops build +hops mlsystem create ``` -`hops build` first asks what you want to build: a new ML system, which it asks you to describe, or an example ML system (churn, batch; personalized recommendations, real-time; a help desk agent, agentic) that runs on synthetic data and includes an app. +`hops mlsystem create` first asks what you want to build: a new ML system, which it asks you to describe, or an example ML system (churn, batch; personalized recommendations, real-time; a help desk agent, agentic) that runs on synthetic data and includes an app. The help desk agent answers from documents you upload to `Resources/helpdesk-docs` (PDF, text, Markdown, Word or OpenDocument), which a job cuts into passages and embeds with a sentence-transformers model downloaded into the Model Registry, and from the customer's recent events; it is a LangGraph agent deployment with a JavaScript chat app. -For it, `hops build` asks for an OpenAI-compatible LLM endpoint, model and API key (read without echo) and saves them as your account environment variables `LLM_URL`, `LLM_MODEL` and `LLM_API_KEY`, which the agent reads. +For it, `hops mlsystem create` asks for an OpenAI-compatible LLM endpoint, model and API key (read without echo) and saves them as your account environment variables `LLM_URL`, `LLM_MODEL` and `LLM_API_KEY`, which the agent reads. It then asks the questions that follow from the system type: how often predictions are made for a batch system, the latency and throughput for a real-time one, the LLM for an agentic one, the data to learn from, how the predictions are used, and where the code goes. One Claude Code call on Haiku reads your description and recommends the system type and a name; the other questions are plain prompts. Each answer is written to `/system.yaml` in the current directory as you give it. A new data source is created with `hops datasource create`, and its password or key is read without echo and passed to it in an environment variable, so it never appears on the command line or in `system.yaml`. -When the interview is done, `hops build` starts Claude Code with `/hops-build `, which completes the specification and builds the feature, training and inference pipelines. -Each system's directory holds an `AGENTS.md` saying the system is built from `system.yaml`; `hops build` and the **Factory** page start Claude Code in that directory, so it reads it, checks what a change to `system.yaml` means for the pipelines and the assets they create, and finds what a changed component affects downstream with `hops fg lineage`, `hops fv lineage`, `hops td lineage`, `hops model lineage` and `hops deployment lineage`. +When the interview is done, `hops mlsystem create` starts Claude Code with `/hops-build `, which completes the specification and builds the feature, training and inference pipelines. +Each system's directory holds an `AGENTS.md` saying the system is built from `system.yaml`; `hops mlsystem create` and the **Factory** page start Claude Code in that directory, so it reads it, checks what a change to `system.yaml` means for the pipelines and the assets they create, and finds what a changed component affects downstream with `hops fg lineage`, `hops fv lineage`, `hops td lineage`, `hops model lineage` and `hops deployment lineage`. Inside tmux, as in the Hopsworks terminal, it opens a new tmux window named after the system, so several systems can be built at once. -Pass `--no-launch` to record the interview only, and `hops build ` to resume a system. -`hops build --example ` (`churn-example`, `recs-example` or `helpdesk-example`) builds an example without the menu, and resumes it if it already exists. +Pass `--no-launch` to record the interview only, and `hops mlsystem create ` to resume a system. +`hops mlsystem create --example ` (`churn-example`, `recs-example` or `helpdesk-example`) builds an example without the menu, and resumes it if it already exists. -`hops build` registers each system with the project, by the HopsFS directory of its code, or by its GitHub repository when you build from an external client. +`hops mlsystem create` registers each system with the project, by the HopsFS directory of its code, or by its GitHub repository when you build from an external client. A GitHub repository the build creates, an example's included, is named `hops-`, or `hops--` when you already have one of that name. **Factory**, in the project menu below Catalog, holds the ML system factory and the medallion layer factory. Its ML system factory lists the project's systems for every member, when the project has registered systems or the cluster has the terminal: each with its type, status, phases done, owner and last update, and an open folder for the ones whose code you can open, a lock for the ones you cannot, and a link for the ones in a GitHub repository. @@ -150,7 +150,7 @@ For an agentic system the LLM's endpoint, model and key are saved as your accoun For a batch or real-time system, **Monitoring** sets whether every prediction logs the features it used (on by default) and, in your own words, what to monitor and alert on, such as drift in a feature against the training data or a failed job. The build turns them into feature logging on the feature view, feature monitoring checks and alerts, and sends a failure alert for every job the system owns to the project's alert receiver. The examples are listed under **New ML System**; each opens with its requirements filled in. -**Create** runs `hops build --answers` in a Terminal tab named after the system, which asks only what the page left out and starts Claude Code on `/hops-build `; the page then locks the requirements and opens the system once it is registered. +**Create** runs `hops mlsystem create --answers` in a Terminal tab named after the system, which asks only what the page left out and starts Claude Code on `/hops-build `; the page then locks the requirements and opens the system once it is registered. A system's page shows its phases, what is done and what is left, what it has made, and its requirements, locked. **Open in Terminal** brings the system's Terminal tab to the front, or opens one with Claude Code started in its directory. @@ -163,7 +163,7 @@ A system whose directory is deleted disappears from the list. **Delete** asks what to delete: the system's entry in the list only, that and every asset the system created (its app, deployments, jobs, models, feature view and training data, the feature groups it writes, the data sources it created and its cloned environments; feature groups it only reads are kept), or those and its GitHub repository, which is deleted only when the build created it for this system alone. The assets are deleted in the terminal, downstream first, and the entry last, so a delete that fails part way leaves the system in the list to be deleted again. Deleting the assets also deletes the code directory; deleting the entry only keeps it. ```bash -hops mlsystem create [] # the same as hops build: interview, then build with Claude Code +hops mlsystem create [] # interview for a new system, then build it with Claude Code hops mlsystem list # the project's systems and whether you can open their code hops mlsystem register [--name N] # register or refresh one by hand hops mlsystem remove # remove it from the list; its code is kept From b52cf0d51e7758cf7a0689b267cd211083eb662a Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Tue, 6 Oct 2026 14:11:54 +0200 Subject: [PATCH 32/42] [HWORKS-3275] ML software factory https://hopsworks.atlassian.net/browse/HWORKS-3275 The CLI has one top-level command for the software factories, hops factory. hops factory list lists the two built-in factories, mlsystem and medallion, with the commands each takes, and their command groups moved under it: hops factory mlsystem create|list|register|remove| delete|status and hops factory medallion silver|gold|create|...; the top-level hops mlsystem and hops medallion are gone. Every reference names the new commands. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01RfHMsBbYDQZqoD58XVqeYE --- docs/user_guides/client_installation/index.md | 57 ++++++++++--------- 1 file changed, 29 insertions(+), 28 deletions(-) diff --git a/docs/user_guides/client_installation/index.md b/docs/user_guides/client_installation/index.md index 46c250ed2b..80d37ae743 100644 --- a/docs/user_guides/client_installation/index.md +++ b/docs/user_guides/client_installation/index.md @@ -77,7 +77,7 @@ It then writes the following files into the current directory: | `AGENTS.md` | Instructions for the agent: the project you are connected to, where the `hopsworks` library is installed on this machine, and how to use the `hops` CLI and the skills. | | `.claude/skills/hops/SKILL.md` | A reference for the `hops` CLI. | | `.claude/commands/hops.md` | The `/hops` slash command for Claude Code: a fast menu to explore data, build or edit a Superset dashboard (`/hops dashboard`) or a Python app (`/hops app`), and show status. It runs on Haiku; the building is done by the agents below. | -| `.claude/commands/hops-ml.md` | The `/hops-ml` slash command: the `hops mlsystem create` interview inside Claude Code, on Haiku, recorded in `system.yaml` as you answer. | +| `.claude/commands/hops-ml.md` | The `/hops-ml` slash command: the `hops factory mlsystem create` interview inside Claude Code, on Haiku, recorded in `system.yaml` as you answer. | | `.claude/commands/hops-build.md` | The `/hops-build` slash command: completes the specification the interview recorded and builds the ML system to a pull request, on your session's model. | | `.claude/agents/hops-dashboard-builder.md` | The Claude Code sub-agent `/hops` runs to build, edit or delete a dashboard. | | `.claude/agents/hops-app-builder.md` | The Claude Code sub-agent `/hops` runs to build, edit or delete an app and fix it until it serves. | @@ -122,25 +122,25 @@ hops skills show hops-fg ### Build an ML system ```bash -hops mlsystem create +hops factory mlsystem create ``` -`hops mlsystem create` first asks what you want to build: a new ML system, which it asks you to describe, or an example ML system (churn, batch; personalized recommendations, real-time; a help desk agent, agentic) that runs on synthetic data and includes an app. +`hops factory mlsystem create` first asks what you want to build: a new ML system, which it asks you to describe, or an example ML system (churn, batch; personalized recommendations, real-time; a help desk agent, agentic) that runs on synthetic data and includes an app. The help desk agent answers from documents you upload to `Resources/helpdesk-docs` (PDF, text, Markdown, Word or OpenDocument), which a job cuts into passages and embeds with a sentence-transformers model downloaded into the Model Registry, and from the customer's recent events; it is a LangGraph agent deployment with a JavaScript chat app. -For it, `hops mlsystem create` asks for an OpenAI-compatible LLM endpoint, model and API key (read without echo) and saves them as your account environment variables `LLM_URL`, `LLM_MODEL` and `LLM_API_KEY`, which the agent reads. +For it, `hops factory mlsystem create` asks for an OpenAI-compatible LLM endpoint, model and API key (read without echo) and saves them as your account environment variables `LLM_URL`, `LLM_MODEL` and `LLM_API_KEY`, which the agent reads. It then asks the questions that follow from the system type: how often predictions are made for a batch system, the latency and throughput for a real-time one, the LLM for an agentic one, the data to learn from, how the predictions are used, and where the code goes. One Claude Code call on Haiku reads your description and recommends the system type and a name; the other questions are plain prompts. Each answer is written to `/system.yaml` in the current directory as you give it. A new data source is created with `hops datasource create`, and its password or key is read without echo and passed to it in an environment variable, so it never appears on the command line or in `system.yaml`. -When the interview is done, `hops mlsystem create` starts Claude Code with `/hops-build `, which completes the specification and builds the feature, training and inference pipelines. -Each system's directory holds an `AGENTS.md` saying the system is built from `system.yaml`; `hops mlsystem create` and the **Factory** page start Claude Code in that directory, so it reads it, checks what a change to `system.yaml` means for the pipelines and the assets they create, and finds what a changed component affects downstream with `hops fg lineage`, `hops fv lineage`, `hops td lineage`, `hops model lineage` and `hops deployment lineage`. +When the interview is done, `hops factory mlsystem create` starts Claude Code with `/hops-build `, which completes the specification and builds the feature, training and inference pipelines. +Each system's directory holds an `AGENTS.md` saying the system is built from `system.yaml`; `hops factory mlsystem create` and the **Factory** page start Claude Code in that directory, so it reads it, checks what a change to `system.yaml` means for the pipelines and the assets they create, and finds what a changed component affects downstream with `hops fg lineage`, `hops fv lineage`, `hops td lineage`, `hops model lineage` and `hops deployment lineage`. Inside tmux, as in the Hopsworks terminal, it opens a new tmux window named after the system, so several systems can be built at once. -Pass `--no-launch` to record the interview only, and `hops mlsystem create ` to resume a system. -`hops mlsystem create --example ` (`churn-example`, `recs-example` or `helpdesk-example`) builds an example without the menu, and resumes it if it already exists. +Pass `--no-launch` to record the interview only, and `hops factory mlsystem create ` to resume a system. +`hops factory mlsystem create --example ` (`churn-example`, `recs-example` or `helpdesk-example`) builds an example without the menu, and resumes it if it already exists. -`hops mlsystem create` registers each system with the project, by the HopsFS directory of its code, or by its GitHub repository when you build from an external client. +`hops factory mlsystem create` registers each system with the project, by the HopsFS directory of its code, or by its GitHub repository when you build from an external client. A GitHub repository the build creates, an example's included, is named `hops-`, or `hops--` when you already have one of that name. **Factory**, in the project menu below Catalog, holds the ML system factory and the medallion layer factory. Its ML system factory lists the project's systems for every member, when the project has registered systems or the cluster has the terminal: each with its type, status, phases done, owner and last update, and an open folder for the ones whose code you can open, a lock for the ones you cannot, and a link for the ones in a GitHub repository. @@ -150,7 +150,7 @@ For an agentic system the LLM's endpoint, model and key are saved as your accoun For a batch or real-time system, **Monitoring** sets whether every prediction logs the features it used (on by default) and, in your own words, what to monitor and alert on, such as drift in a feature against the training data or a failed job. The build turns them into feature logging on the feature view, feature monitoring checks and alerts, and sends a failure alert for every job the system owns to the project's alert receiver. The examples are listed under **New ML System**; each opens with its requirements filled in. -**Create** runs `hops mlsystem create --answers` in a Terminal tab named after the system, which asks only what the page left out and starts Claude Code on `/hops-build `; the page then locks the requirements and opens the system once it is registered. +**Create** runs `hops factory mlsystem create --answers` in a Terminal tab named after the system, which asks only what the page left out and starts Claude Code on `/hops-build `; the page then locks the requirements and opens the system once it is registered. A system's page shows its phases, what is done and what is left, what it has made, and its requirements, locked. **Open in Terminal** brings the system's Terminal tab to the front, or opens one with Claude Code started in its directory. @@ -163,11 +163,12 @@ A system whose directory is deleted disappears from the list. **Delete** asks what to delete: the system's entry in the list only, that and every asset the system created (its app, deployments, jobs, models, feature view and training data, the feature groups it writes, the data sources it created and its cloned environments; feature groups it only reads are kept), or those and its GitHub repository, which is deleted only when the build created it for this system alone. The assets are deleted in the terminal, downstream first, and the entry last, so a delete that fails part way leaves the system in the list to be deleted again. Deleting the assets also deletes the code directory; deleting the entry only keeps it. ```bash -hops mlsystem create [] # interview for a new system, then build it with Claude Code -hops mlsystem list # the project's systems and whether you can open their code -hops mlsystem register [--name N] # register or refresh one by hand -hops mlsystem remove # remove it from the list; its code is kept -hops mlsystem delete --assets [--repo] # also delete what it created, and its repository +hops factory list # the factories: mlsystem and medallion +hops factory mlsystem create [] # interview for a new system, then build it with Claude Code +hops factory mlsystem list # the project's systems and whether you can open their code +hops factory mlsystem register [--name N] # register or refresh one by hand +hops factory mlsystem remove # remove it from the list; its code is kept +hops factory mlsystem delete --assets [--repo] # also delete what it created, and its repository ``` ### Build a silver medallion layer @@ -182,7 +183,7 @@ The page asks for the bronze tables to build from (only those tagged bronze are The engine is dbt on Trino unless an additional task needs code SQL does not express well, when PySpark is suggested. Each bronze table has its own refresh, hourly, daily or weekly; a silver table refreshes as often as its most frequently updated source, and the layer gets one job per refresh, each with its own schedule and freshness target. It also asks how the tables behave, with defaults: history (`latest`, one row per key, or `full`, every version by time), deletes in bronze (`ignore` or `propagate`), bronze schema changes (`fail`, or `evolve` by adding new columns), a late-data lookback re-read before each window (none, a day or a week), the share of rejected rows above which a run fails, with an alert when one fails, and a freshness target. -**Create** runs `hops medallion silver --answers` in a Terminal tab named after the layer, which records its specification in `/system.yaml` and starts Claude Code on `/hops-silver `. +**Create** runs `hops factory medallion silver --answers` in a Terminal tab named after the layer, which records its specification in `/system.yaml` and starts Claude Code on `/hops-silver `. The build profiles the bronze tables, designs the silver tables, writes and tests the code, backfills the whole bronze history once, schedules the job and tags the silver feature groups `layer: silver`, and verifies one window. Silver tables are feature groups, materialized, never views, and in third normal form: one table per entity or event, every column depending on its table's key alone, lookups in tables of their own, and no aggregates, which belong in gold. @@ -212,21 +213,21 @@ A data mart's requirements are: - **Verification**: example questions with the answers expected, in plain English; the totals that must reconcile, and with what; and how refreshes and reruns are proven correct. The build runs every check after the backfill and again after a refresh, records the results, and does not mark the mart built until each passes. - **Quality and access**: the invariants to test, what happens when a check fails (fail the run, quarantine the failing rows, or warn), who may read which rows and columns, and the projects it is shared with. -**Create** runs `hops medallion gold --answers` and starts Claude Code on `/hops-gold `, which builds each mart: its requirements, the design of its facts and dimensions (a dimension used by several marts is built once and shared), the dbt models with their tests, the backfill with the verification, the schedule and tags (`layer: gold`, the silver tables as parents), and a verified refresh. +**Create** runs `hops factory medallion gold --answers` and starts Claude Code on `/hops-gold `, which builds each mart: its requirements, the design of its facts and dimensions (a dimension used by several marts is built once and shared), the dbt models with their tests, the backfill with the verification, the schedule and tags (`layer: gold`, the silver tables as parents), and a verified refresh. The layer's page shows each mart with its phases, tables, jobs and verification results: **Edit** changes its requirements and applies the change, **Add data mart** adds one, and deleting a mart deletes its jobs and, if asked, its tables that no other mart lists. ```bash -hops medallion silver --answers answers.json # record a silver layer and build it with Claude Code -hops medallion gold --answers answers.json # record a gold layer and its first data mart -hops medallion create silver|gold --answers answers.json # the same two commands -hops medallion mart-add --answers mart.json # add a data mart to a gold layer -hops medallion mart-update --answers mart.json # change a data mart's requirements -hops medallion mart-delete [--tables] # delete a data mart's jobs, and its own tables -hops medallion job-delete [--tables] # delete one job, and the tables only it writes -hops medallion add-tables --answers new.json # add bronze tables to a silver layer -hops medallion status # write the layer's health report, status/report.html -hops medallion backfill # recompute the layer's tables from the whole history -hops medallion delete --assets # delete the layer, its jobs and tables; never what it reads +hops factory medallion silver --answers answers.json # record a silver layer and build it with Claude Code +hops factory medallion gold --answers answers.json # record a gold layer and its first data mart +hops factory medallion create silver|gold --answers answers.json # the same two commands +hops factory medallion mart-add --answers mart.json # add a data mart to a gold layer +hops factory medallion mart-update --answers mart.json # change a data mart's requirements +hops factory medallion mart-delete [--tables] # delete a data mart's jobs, and its own tables +hops factory medallion job-delete [--tables] # delete one job, and the tables only it writes +hops factory medallion add-tables --answers new.json # add bronze tables to a silver layer +hops factory medallion status # write the layer's health report, status/report.html +hops factory medallion backfill # recompute the layer's tables from the whole history +hops factory medallion delete --assets # delete the layer, its jobs and tables; never what it reads ``` ## Hopsworks Java Library From afe85aaff4df1b9a9a196a2710193b55b74a0159 Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Tue, 6 Oct 2026 15:23:41 +0200 Subject: [PATCH 33/42] [HWORKS-3275] ML software factory https://hopsworks.atlassian.net/browse/HWORKS-3275 The client installation guide describes creating, cloning, importing, exporting, editing and deleting a project's own factories, with an example definition and the hops factory commands. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01RfHMsBbYDQZqoD58XVqeYE --- docs/user_guides/client_installation/index.md | 43 +++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/docs/user_guides/client_installation/index.md b/docs/user_guides/client_installation/index.md index 80d37ae743..82a9e19499 100644 --- a/docs/user_guides/client_installation/index.md +++ b/docs/user_guides/client_installation/index.md @@ -230,6 +230,49 @@ hops factory medallion backfill # recompute the layer' hops factory medallion delete --assets # delete the layer, its jobs and tables; never what it reads ``` +### Create your own factory + +A factory is a YAML definition: the questions of its creation form, the phases of its build, and the instructions Claude Code follows to build what the answers describe. +The **Factory** page shows a section for each of the project's factories, with **New** opening the form it generates. +The two built-in factories, `mlsystem` and `medallion`, are read-only; clone one to change it. + +**Factory > Manage factories** lists every factory with its version and how many systems it built. +A data owner can create a factory, clone any factory, import a YAML file, export one, enable or disable a project factory, and delete one that has no systems left. +The editor changes the questions, phases and build instructions as a form or as YAML, previews the form beside it, and saves each change as a new version; a system keeps the version it was built with. +Importing shows the factory's build instructions in full first: Claude Code follows them in your Terminal, with your credentials. + +A clone of a built-in keeps the built-in's form and build; its own questions are recorded in `requirements.extra` and its instructions in `factory.instructions` of each system's `system.yaml`, which the built-in build follows too. + +```yaml +apiVersion: hopsworks.ai/factory/v1 +kind: Factory +name: churn-review +title: Churn review +form: + sections: + - id: basics + title: Basics + fields: + - {id: name, type: slug, label: Name, required: true} + - {id: question, type: textarea, label: "What should it answer?", required: true} +phases: + - {key: build, label: Build, minutes: 20} +build: + skills: [hops-superset] + instructions: Build a dashboard that answers requirements.question. +``` + +```bash +hops factory validate churn-review.yaml # check a definition without a cluster +hops factory import churn-review.yaml # review it, then add it to the project +hops factory clone mlsystem fraud-ml # start from a built-in +hops factory export churn-review # write churn-review.factory.yaml +hops factory churn-review create --answers answers.json +hops factory delete churn-review # refused while it has systems +``` + +The `hops-factory` skill lists every field type and rule. + ## Hopsworks Java Library If you want to interact with the Hopsworks Feature Store from environments such as Spark or Beam, you can use the Hopsworks Feature Store (Hopsworks) Java library. From 3563e9f9f6fda0b7bc03359d06324af4634aaa0c Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Tue, 6 Oct 2026 22:49:51 +0200 Subject: [PATCH 34/42] [HWORKS-3275] ML software factory https://hopsworks.atlassian.net/browse/HWORKS-3275 The built-in factories are now five plain YAML definitions with no conditional logic: ml-batch, ml-realtime and ml-agent replace the one ML system factory, whose form hid most questions behind the system type, and medallion-silver and medallion-gold replace the medallion factory. A form has no conditions any more: every question of a section is shown, and a section with defaults is collapsed to a summary of its answers with Edit; the batch and real-time factories have their own Monitoring section. The format gains answer keys, labelled options, single feature group pickers, lists of entries, account variables kept out of system.yaml, and presets, which carry the ML examples. The guide describes the three ML factories, the silver and gold factories, collapsed sections and the format's new question types. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01RfHMsBbYDQZqoD58XVqeYE --- docs/user_guides/client_installation/index.md | 24 +++++++++++-------- 1 file changed, 14 insertions(+), 10 deletions(-) diff --git a/docs/user_guides/client_installation/index.md b/docs/user_guides/client_installation/index.md index 82a9e19499..7be2e56861 100644 --- a/docs/user_guides/client_installation/index.md +++ b/docs/user_guides/client_installation/index.md @@ -145,12 +145,14 @@ A GitHub repository the build creates, an example's included, is named `hops-`, its GitHub repository's (lowercase letters, digits and hyphens), what it should predict, its type (batch, real-time or agentic) with the cadence or the latency and throughput, its data (feature groups in the project, or synthetic data described in a sentence), and how its predictions are used. +**New ML System** lists three factories, **Batch ML system**, **Real-time ML system** and **Agentic system**, and their examples. +Each opens its form: the system's name, which is also its directory's and, as `hops-`, its GitHub repository's (lowercase letters, digits and hyphens), what it should predict, its targets (a batch system's cadence and run time, a real-time or agentic system's latency and throughput), its data (feature groups in the project, synthetic data described in a sentence, and files), and how its predictions are used. +Sections with defaults show a one-line summary of their answers, with **Edit** to change them. For an agentic system the LLM's endpoint, model and key are saved as your account environment variables, `LLM_URL`, `LLM_MODEL` and `LLM_API_KEY`. -For a batch or real-time system, **Monitoring** sets whether every prediction logs the features it used (on by default) and, in your own words, what to monitor and alert on, such as drift in a feature against the training data or a failed job. +For a batch or real-time system, **Monitoring** (collapsed) sets whether every prediction logs the features it used (on by default) and, in your own words, what to monitor and alert on, such as drift in a feature against the training data or a failed job. The build turns them into feature logging on the feature view, feature monitoring checks and alerts, and sends a failure alert for every job the system owns to the project's alert receiver. -The examples are listed under **New ML System**; each opens with its requirements filled in. -**Create** runs `hops factory mlsystem create --answers` in a Terminal tab named after the system, which asks only what the page left out and starts Claude Code on `/hops-build `; the page then locks the requirements and opens the system once it is registered. +The examples are listed under **New ML System**; each opens its factory's form filled in. +**Create** runs `hops factory ml-batch create --answers` (or `ml-realtime`, `ml-agent`) in a Terminal tab named after the system, which records it and starts Claude Code on `/hops-build `; the page opens the system once it is registered. A system's page shows its phases, what is done and what is left, what it has made, and its requirements, locked. **Open in Terminal** brings the system's Terminal tab to the front, or opens one with Claude Code started in its directory. @@ -177,13 +179,13 @@ A medallion layer organizes tables as bronze (raw data as it arrived), silver (c Hopsworks installs an archived schematized tag, `medallion_table`, whose `layer` is `bronze`, `silver` or `gold` and whose `lifecycle` is `dev`, `staging` or `prod`; every change of a table's value is kept in the tag history. When you ingest data with a dltHub data source, **Tag as bronze tables** in the review, off by default, tags every feature group it creates as bronze. -**New Medallion Layer**, in the Factory, builds a silver layer from the project's bronze feature groups. +**New Medallion Layer**, in the Factory, lists two factories, **Silver layer** and **Gold layer**; the silver one builds a silver layer from the project's bronze feature groups. A project without feature groups has nothing to build from: ingest raw data as bronze tables first. The page asks for the bronze tables to build from (only those tagged bronze are shown while any are), the silver tasks (deduplicate, cast types, standardize values, handle nulls, validate with a rejects table, protect personal data, conform entities, surrogate keys, referential checks), additional tasks in your own words, the engine, the refresh cadence and the lifecycle. The engine is dbt on Trino unless an additional task needs code SQL does not express well, when PySpark is suggested. Each bronze table has its own refresh, hourly, daily or weekly; a silver table refreshes as often as its most frequently updated source, and the layer gets one job per refresh, each with its own schedule and freshness target. It also asks how the tables behave, with defaults: history (`latest`, one row per key, or `full`, every version by time), deletes in bronze (`ignore` or `propagate`), bronze schema changes (`fail`, or `evolve` by adding new columns), a late-data lookback re-read before each window (none, a day or a week), the share of rejected rows above which a run fails, with an alert when one fails, and a freshness target. -**Create** runs `hops factory medallion silver --answers` in a Terminal tab named after the layer, which records its specification in `/system.yaml` and starts Claude Code on `/hops-silver `. +**Create** runs `hops factory medallion-silver create --answers` in a Terminal tab named after the layer, which records its specification in `/system.yaml` and starts Claude Code on `/hops-silver `. The build profiles the bronze tables, designs the silver tables, writes and tests the code, backfills the whole bronze history once, schedules the job and tags the silver feature groups `layer: silver`, and verifies one window. Silver tables are feature groups, materialized, never views, and in third normal form: one table per entity or event, every column depending on its table's key alone, lookups in tables of their own, and no aggregates, which belong in gold. @@ -199,7 +201,7 @@ Every job has a delete icon that asks whether to also delete the feature groups ### Build a gold medallion layer of data marts -**New Medallion Layer** with **gold** builds a gold layer from silver tables: a Kimball dimensional model, a star or snowflake schema, for the queries the layer will serve. +**New Medallion Layer** with **Gold layer** builds a gold layer from silver tables: a Kimball dimensional model, a star or snowflake schema, for the queries the layer will serve. The page asks for those queries, the model, the silver tables to read, the standards every data mart follows (naming, modeling, documentation and quality, proposed and editable), and the first data mart. A gold layer is a set of data marts, each added, changed and deleted on its own, with its own fact and dimension tables and its own jobs, `--`, at its own refresh. @@ -213,7 +215,7 @@ A data mart's requirements are: - **Verification**: example questions with the answers expected, in plain English; the totals that must reconcile, and with what; and how refreshes and reruns are proven correct. The build runs every check after the backfill and again after a refresh, records the results, and does not mark the mart built until each passes. - **Quality and access**: the invariants to test, what happens when a check fails (fail the run, quarantine the failing rows, or warn), who may read which rows and columns, and the projects it is shared with. -**Create** runs `hops factory medallion gold --answers` and starts Claude Code on `/hops-gold `, which builds each mart: its requirements, the design of its facts and dimensions (a dimension used by several marts is built once and shared), the dbt models with their tests, the backfill with the verification, the schedule and tags (`layer: gold`, the silver tables as parents), and a verified refresh. +**Create** runs `hops factory medallion-gold create --answers` and starts Claude Code on `/hops-gold `, which builds each mart: its requirements, the design of its facts and dimensions (a dimension used by several marts is built once and shared), the dbt models with their tests, the backfill with the verification, the schedule and tags (`layer: gold`, the silver tables as parents), and a verified refresh. The layer's page shows each mart with its phases, tables, jobs and verification results: **Edit** changes its requirements and applies the change, **Add data mart** adds one, and deleting a mart deletes its jobs and, if asked, its tables that no other mart lists. ```bash @@ -234,14 +236,16 @@ hops factory medallion delete --assets # delete the layer, it A factory is a YAML definition: the questions of its creation form, the phases of its build, and the instructions Claude Code follows to build what the answers describe. The **Factory** page shows a section for each of the project's factories, with **New** opening the form it generates. -The two built-in factories, `mlsystem` and `medallion`, are read-only; clone one to change it. +The five built-in factories, `ml-batch`, `ml-realtime`, `ml-agent`, `medallion-silver` and `medallion-gold`, are read-only; clone one to change it. +A form has no conditions: every question of a section is shown, and a section with defaults can be collapsed to a summary of its answers with **Edit**. **Factory > Manage factories** lists every factory with its version and how many systems it built. A data owner can create a factory, clone any factory, import a YAML file, export one, enable or disable a project factory, and delete one that has no systems left. The editor changes the questions, phases and build instructions as a form or as YAML, previews the form beside it, and saves each change as a new version; a system keeps the version it was built with. Importing shows the factory's build instructions in full first: Claude Code follows them in your Terminal, with your credentials. -A clone of a built-in keeps the built-in's form and build; its own questions are recorded in `requirements.extra` and its instructions in `factory.instructions` of each system's `system.yaml`, which the built-in build follows too. +A clone of a built-in keeps the built-in's questions and build; answers the built-in build does not read are recorded in `requirements.extra` and the clone's instructions in `factory.instructions` of each system's `system.yaml`, which the built-in build follows too. +A definition's questions can be text, numbers, checkboxes, one or some of a list of options, one or several feature groups, a list of entries each with its own questions, and account variables, which are saved in your account and never in `system.yaml`; presets are named sets of starting answers, listed under the factory's **New** button. ```yaml apiVersion: hopsworks.ai/factory/v1 From 7a2ed78b7298ef9d1a65732078afde716cbe68eb Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Tue, 6 Oct 2026 23:20:46 +0200 Subject: [PATCH 35/42] [HWORKS-3275] ML software factory https://hopsworks.atlassian.net/browse/HWORKS-3275 hops factory run builds a system with any factory and replaces the hops factory mlsystem and hops factory medallion groups and the per-factory hops factory groups. A new system's answers come from --answers, as the Factory page writes them, or each question of the factory's definition is asked in the terminal, with --preset to start from one of its examples. A system already recorded is resumed from its system.yaml, which holds the factory, its version and every answer, so it needs no answers file. The systems the factories built are listed, reported on, registered and deleted with hops factory system, which also holds the medallion layer commands (data marts, jobs, added tables, backfills, dir); status and delete handle ML systems and layers alike. hopsworks-api: run, the terminal questions (an account variable is asked only when the account lacks it, then saved there) and resume in factory.py; the built-in builds become functions taking the answers, and the Haiku interview and its menu are deleted, since the factory's form now asks the questions. Skills, templates and tests follow. hopsworks-front: the typed commands use run and system. hopsworks-ee: the built-in definitions' header comments. logicalclocks.github.io: the CLI sections. loadtest: a docstring. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01RfHMsBbYDQZqoD58XVqeYE --- docs/user_guides/client_installation/index.md | 68 +++++++++---------- 1 file changed, 34 insertions(+), 34 deletions(-) diff --git a/docs/user_guides/client_installation/index.md b/docs/user_guides/client_installation/index.md index 7be2e56861..544b113d87 100644 --- a/docs/user_guides/client_installation/index.md +++ b/docs/user_guides/client_installation/index.md @@ -77,7 +77,7 @@ It then writes the following files into the current directory: | `AGENTS.md` | Instructions for the agent: the project you are connected to, where the `hopsworks` library is installed on this machine, and how to use the `hops` CLI and the skills. | | `.claude/skills/hops/SKILL.md` | A reference for the `hops` CLI. | | `.claude/commands/hops.md` | The `/hops` slash command for Claude Code: a fast menu to explore data, build or edit a Superset dashboard (`/hops dashboard`) or a Python app (`/hops app`), and show status. It runs on Haiku; the building is done by the agents below. | -| `.claude/commands/hops-ml.md` | The `/hops-ml` slash command: the `hops factory mlsystem create` interview inside Claude Code, on Haiku, recorded in `system.yaml` as you answer. | +| `.claude/commands/hops-ml.md` | The `/hops-ml` slash command: an ML system interview inside Claude Code, on Haiku, recorded in `system.yaml` as you answer. | | `.claude/commands/hops-build.md` | The `/hops-build` slash command: completes the specification the interview recorded and builds the ML system to a pull request, on your session's model. | | `.claude/agents/hops-dashboard-builder.md` | The Claude Code sub-agent `/hops` runs to build, edit or delete a dashboard. | | `.claude/agents/hops-app-builder.md` | The Claude Code sub-agent `/hops` runs to build, edit or delete an app and fix it until it serves. | @@ -122,25 +122,24 @@ hops skills show hops-fg ### Build an ML system ```bash -hops factory mlsystem create +hops factory run ml-batch # or ml-realtime, ml-agent ``` -`hops factory mlsystem create` first asks what you want to build: a new ML system, which it asks you to describe, or an example ML system (churn, batch; personalized recommendations, real-time; a help desk agent, agentic) that runs on synthetic data and includes an app. +`hops factory run ` asks the factory's questions in the terminal, section by section, with the factory's defaults: for `ml-batch` the system's name, what it should predict, its cadence and run time, its data (feature groups in the project, synthetic data described in a sentence, and files), how its predictions are used, and its monitoring. +`--answers answers.json` takes the answers from a file instead, as the **Factory** page writes it, and `--preset ` starts from one of the factory's examples (`churn-example`, `recs-example`, `gis-example`, `helpdesk-example` or `run-example`, as `hops factory get ` lists them under `presets`). The help desk agent answers from documents you upload to `Resources/helpdesk-docs` (PDF, text, Markdown, Word or OpenDocument), which a job cuts into passages and embeds with a sentence-transformers model downloaded into the Model Registry, and from the customer's recent events; it is a LangGraph agent deployment with a JavaScript chat app. -For it, `hops factory mlsystem create` asks for an OpenAI-compatible LLM endpoint, model and API key (read without echo) and saves them as your account environment variables `LLM_URL`, `LLM_MODEL` and `LLM_API_KEY`, which the agent reads. -It then asks the questions that follow from the system type: how often predictions are made for a batch system, the latency and throughput for a real-time one, the LLM for an agentic one, the data to learn from, how the predictions are used, and where the code goes. -One Claude Code call on Haiku reads your description and recommends the system type and a name; the other questions are plain prompts. -Each answer is written to `/system.yaml` in the current directory as you give it. +For an agentic system, `hops factory run ml-agent` asks for an OpenAI-compatible LLM endpoint, model and API key (read without echo), unless your account already has them, and saves them as your account environment variables `LLM_URL`, `LLM_MODEL` and `LLM_API_KEY`, which the agent reads. +The answers are written to `/system.yaml` in the current directory, and what they leave out, such as where the code goes, is asked next. A new data source is created with `hops datasource create`, and its password or key is read without echo and passed to it in an environment variable, so it never appears on the command line or in `system.yaml`. -When the interview is done, `hops factory mlsystem create` starts Claude Code with `/hops-build `, which completes the specification and builds the feature, training and inference pipelines. -Each system's directory holds an `AGENTS.md` saying the system is built from `system.yaml`; `hops factory mlsystem create` and the **Factory** page start Claude Code in that directory, so it reads it, checks what a change to `system.yaml` means for the pipelines and the assets they create, and finds what a changed component affects downstream with `hops fg lineage`, `hops fv lineage`, `hops td lineage`, `hops model lineage` and `hops deployment lineage`. +`hops factory run` then starts Claude Code with `/hops-build `, which completes the specification and builds the feature, training and inference pipelines. +Each system's directory holds an `AGENTS.md` saying the system is built from `system.yaml`; `hops factory run` and the **Factory** page start Claude Code in that directory, so it reads it, checks what a change to `system.yaml` means for the pipelines and the assets they create, and finds what a changed component affects downstream with `hops fg lineage`, `hops fv lineage`, `hops td lineage`, `hops model lineage` and `hops deployment lineage`. Inside tmux, as in the Hopsworks terminal, it opens a new tmux window named after the system, so several systems can be built at once. -Pass `--no-launch` to record the interview only, and `hops factory mlsystem create ` to resume a system. -`hops factory mlsystem create --example ` (`churn-example`, `recs-example` or `helpdesk-example`) builds an example without the menu, and resumes it if it already exists. +Pass `--no-launch` to record the system only. +`hops factory run `, or `hops factory run ` inside the system's directory, resumes a system: its `system.yaml` records the factory, the factory version and every answer, so it needs no answers file. -`hops factory mlsystem create` registers each system with the project, by the HopsFS directory of its code, or by its GitHub repository when you build from an external client. +`hops factory run` registers each system with the project, by the HopsFS directory of its code, or by its GitHub repository when you build from an external client. A GitHub repository the build creates, an example's included, is named `hops-`, or `hops--` when you already have one of that name. **Factory**, in the project menu below Catalog, holds the ML system factory and the medallion layer factory. Its ML system factory lists the project's systems for every member, when the project has registered systems or the cluster has the terminal: each with its type, status, phases done, owner and last update, and an open folder for the ones whose code you can open, a lock for the ones you cannot, and a link for the ones in a GitHub repository. @@ -152,7 +151,7 @@ For an agentic system the LLM's endpoint, model and key are saved as your accoun For a batch or real-time system, **Monitoring** (collapsed) sets whether every prediction logs the features it used (on by default) and, in your own words, what to monitor and alert on, such as drift in a feature against the training data or a failed job. The build turns them into feature logging on the feature view, feature monitoring checks and alerts, and sends a failure alert for every job the system owns to the project's alert receiver. The examples are listed under **New ML System**; each opens its factory's form filled in. -**Create** runs `hops factory ml-batch create --answers` (or `ml-realtime`, `ml-agent`) in a Terminal tab named after the system, which records it and starts Claude Code on `/hops-build `; the page opens the system once it is registered. +**Create** runs `hops factory run ml-batch --answers` (or `ml-realtime`, `ml-agent`) in a Terminal tab named after the system, which records it and starts Claude Code on `/hops-build `; the page opens the system once it is registered. A system's page shows its phases, what is done and what is left, what it has made, and its requirements, locked. **Open in Terminal** brings the system's Terminal tab to the front, or opens one with Claude Code started in its directory. @@ -165,12 +164,14 @@ A system whose directory is deleted disappears from the list. **Delete** asks what to delete: the system's entry in the list only, that and every asset the system created (its app, deployments, jobs, models, feature view and training data, the feature groups it writes, the data sources it created and its cloned environments; feature groups it only reads are kept), or those and its GitHub repository, which is deleted only when the build created it for this system alone. The assets are deleted in the terminal, downstream first, and the entry last, so a delete that fails part way leaves the system in the list to be deleted again. Deleting the assets also deletes the code directory; deleting the entry only keeps it. ```bash -hops factory list # the factories: mlsystem and medallion -hops factory mlsystem create [] # interview for a new system, then build it with Claude Code -hops factory mlsystem list # the project's systems and whether you can open their code -hops factory mlsystem register [--name N] # register or refresh one by hand -hops factory mlsystem remove # remove it from the list; its code is kept -hops factory mlsystem delete --assets [--repo] # also delete what it created, and its repository +hops factory list # the factories and how many systems each built +hops factory run [--answers F] [--preset P] # a new system, then build it with Claude Code +hops factory run # resume a system from its system.yaml +hops factory system list [--factory ] # the project's systems and whether you can open their code +hops factory system register [--name N] # register or refresh one by hand +hops factory system status # write its health report, status/report.html +hops factory system remove # remove it from the list; its code is kept +hops factory system delete --assets [--repo] # also delete what it created, and its repository ``` ### Build a silver medallion layer @@ -185,7 +186,7 @@ The page asks for the bronze tables to build from (only those tagged bronze are The engine is dbt on Trino unless an additional task needs code SQL does not express well, when PySpark is suggested. Each bronze table has its own refresh, hourly, daily or weekly; a silver table refreshes as often as its most frequently updated source, and the layer gets one job per refresh, each with its own schedule and freshness target. It also asks how the tables behave, with defaults: history (`latest`, one row per key, or `full`, every version by time), deletes in bronze (`ignore` or `propagate`), bronze schema changes (`fail`, or `evolve` by adding new columns), a late-data lookback re-read before each window (none, a day or a week), the share of rejected rows above which a run fails, with an alert when one fails, and a freshness target. -**Create** runs `hops factory medallion-silver create --answers` in a Terminal tab named after the layer, which records its specification in `/system.yaml` and starts Claude Code on `/hops-silver `. +**Create** runs `hops factory run medallion-silver --answers` in a Terminal tab named after the layer, which records its specification in `/system.yaml` and starts Claude Code on `/hops-silver `. The build profiles the bronze tables, designs the silver tables, writes and tests the code, backfills the whole bronze history once, schedules the job and tags the silver feature groups `layer: silver`, and verifies one window. Silver tables are feature groups, materialized, never views, and in third normal form: one table per entity or event, every column depending on its table's key alone, lookups in tables of their own, and no aggregates, which belong in gold. @@ -215,21 +216,20 @@ A data mart's requirements are: - **Verification**: example questions with the answers expected, in plain English; the totals that must reconcile, and with what; and how refreshes and reruns are proven correct. The build runs every check after the backfill and again after a refresh, records the results, and does not mark the mart built until each passes. - **Quality and access**: the invariants to test, what happens when a check fails (fail the run, quarantine the failing rows, or warn), who may read which rows and columns, and the projects it is shared with. -**Create** runs `hops factory medallion-gold create --answers` and starts Claude Code on `/hops-gold `, which builds each mart: its requirements, the design of its facts and dimensions (a dimension used by several marts is built once and shared), the dbt models with their tests, the backfill with the verification, the schedule and tags (`layer: gold`, the silver tables as parents), and a verified refresh. +**Create** runs `hops factory run medallion-gold --answers` and starts Claude Code on `/hops-gold `, which builds each mart: its requirements, the design of its facts and dimensions (a dimension used by several marts is built once and shared), the dbt models with their tests, the backfill with the verification, the schedule and tags (`layer: gold`, the silver tables as parents), and a verified refresh. The layer's page shows each mart with its phases, tables, jobs and verification results: **Edit** changes its requirements and applies the change, **Add data mart** adds one, and deleting a mart deletes its jobs and, if asked, its tables that no other mart lists. ```bash -hops factory medallion silver --answers answers.json # record a silver layer and build it with Claude Code -hops factory medallion gold --answers answers.json # record a gold layer and its first data mart -hops factory medallion create silver|gold --answers answers.json # the same two commands -hops factory medallion mart-add --answers mart.json # add a data mart to a gold layer -hops factory medallion mart-update --answers mart.json # change a data mart's requirements -hops factory medallion mart-delete [--tables] # delete a data mart's jobs, and its own tables -hops factory medallion job-delete [--tables] # delete one job, and the tables only it writes -hops factory medallion add-tables --answers new.json # add bronze tables to a silver layer -hops factory medallion status # write the layer's health report, status/report.html -hops factory medallion backfill # recompute the layer's tables from the whole history -hops factory medallion delete --assets # delete the layer, its jobs and tables; never what it reads +hops factory run medallion-silver [--answers answers.json] # record a silver layer and build it with Claude Code +hops factory run medallion-gold [--answers answers.json] # record a gold layer and its first data mart +hops factory system mart-add --answers mart.json # add a data mart to a gold layer +hops factory system mart-update --answers mart.json # change a data mart's requirements +hops factory system mart-delete [--tables] # delete a data mart's jobs, and its own tables +hops factory system job-delete [--tables] # delete one job, and the tables only it writes +hops factory system add-tables --answers new.json # add bronze tables to a silver layer +hops factory system status # write the layer's health report, status/report.html +hops factory system backfill # recompute the layer's tables from the whole history +hops factory system delete --assets # delete the layer, its jobs and tables; never what it reads ``` ### Create your own factory @@ -269,9 +269,9 @@ build: ```bash hops factory validate churn-review.yaml # check a definition without a cluster hops factory import churn-review.yaml # review it, then add it to the project -hops factory clone mlsystem fraud-ml # start from a built-in +hops factory clone ml-batch fraud-ml # start from a built-in hops factory export churn-review # write churn-review.factory.yaml -hops factory churn-review create --answers answers.json +hops factory run churn-review # answer its questions, then build with Claude Code hops factory delete churn-review # refused while it has systems ``` From 38ac6742d7de72a10fe22aeacf3b7f2528b6cf49 Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Wed, 7 Oct 2026 00:35:18 +0200 Subject: [PATCH 36/42] [HWORKS-3275] ML software factory https://hopsworks.atlassian.net/browse/HWORKS-3275 A built system is changed the same way whatever its factory: a factory definition declares its changes (changes: id, label, description, a form, and the instructions the build follows). hops factory run --change takes the answers from --answers or asks them, checks them against the form and system.yaml, records the request in the system's system.yaml (changes, status: pending) and resumes the build, which carries out pending requests first. A change's form may pick one of the system's entries (field type entry: the list at a dotted path of system.yaml, optionally prefilling the other fields). This replaces the medallion-only verbs of hops factory system (mart-add, mart-update, mart-delete, job-delete, add-tables, backfill): the gold built-in declares add-mart, edit-mart, delete-mart and delete-job, the silver one add-tables and delete-job, and a backfill is hops job run over the whole history. hops factory system keeps list, register, status, remove, delete and dir, plus delete-assets, the one way a build deletes jobs and feature groups, which refuses a table the system reads or one of a lower medallion layer. The gold factory is titled Gold Data Mart, and a layer's name is its directory name: one Name field instead of Name and Directory. hopsworks-api: changes and entry fields in factory_spec, run --change, system delete-assets and dir; the layer verbs and their helpers are deleted; the build templates carry out pending change requests; the hops-factory and hops-medallion skills document changes. hopsworks-ee: the validator accepts changes and entry fields; the medallion built-ins declare their changes. hopsworks-front: a system's page has a Change menu rendered from its factory's definition, a change dialog built on FactoryForm (entry fields, fill), and the pending requests; the layer page's Add data mart, Edit, Add tables and delete icons open the same changes; the hand-written mart and add-tables dialogs (MartFields) are deleted. logicalclocks.github.io: changes, and the layer commands. loadtest: the gold built-in's changes. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01RfHMsBbYDQZqoD58XVqeYE --- docs/user_guides/client_installation/index.md | 20 +++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/docs/user_guides/client_installation/index.md b/docs/user_guides/client_installation/index.md index 544b113d87..b850f7cffd 100644 --- a/docs/user_guides/client_installation/index.md +++ b/docs/user_guides/client_installation/index.md @@ -180,7 +180,7 @@ A medallion layer organizes tables as bronze (raw data as it arrived), silver (c Hopsworks installs an archived schematized tag, `medallion_table`, whose `layer` is `bronze`, `silver` or `gold` and whose `lifecycle` is `dev`, `staging` or `prod`; every change of a table's value is kept in the tag history. When you ingest data with a dltHub data source, **Tag as bronze tables** in the review, off by default, tags every feature group it creates as bronze. -**New Medallion Layer**, in the Factory, lists two factories, **Silver layer** and **Gold layer**; the silver one builds a silver layer from the project's bronze feature groups. +**New Medallion Layer**, in the Factory, lists two factories, **Silver layer** and **Gold Data Mart**; the silver one builds a silver layer from the project's bronze feature groups. A project without feature groups has nothing to build from: ingest raw data as bronze tables first. The page asks for the bronze tables to build from (only those tagged bronze are shown while any are), the silver tasks (deduplicate, cast types, standardize values, handle nulls, validate with a rejects table, protect personal data, conform entities, surrogate keys, referential checks), additional tasks in your own words, the engine, the refresh cadence and the lifecycle. The engine is dbt on Trino unless an additional task needs code SQL does not express well, when PySpark is suggested. @@ -202,7 +202,7 @@ Every job has a delete icon that asks whether to also delete the feature groups ### Build a gold medallion layer of data marts -**New Medallion Layer** with **Gold layer** builds a gold layer from silver tables: a Kimball dimensional model, a star or snowflake schema, for the queries the layer will serve. +**New Medallion Layer** with **Gold Data Mart** builds a gold layer from silver tables: a Kimball dimensional model, a star or snowflake schema, for the queries the layer will serve. The page asks for those queries, the model, the silver tables to read, the standards every data mart follows (naming, modeling, documentation and quality, proposed and editable), and the first data mart. A gold layer is a set of data marts, each added, changed and deleted on its own, with its own fact and dimension tables and its own jobs, `--`, at its own refresh. @@ -222,16 +222,20 @@ The layer's page shows each mart with its phases, tables, jobs and verification ```bash hops factory run medallion-silver [--answers answers.json] # record a silver layer and build it with Claude Code hops factory run medallion-gold [--answers answers.json] # record a gold layer and its first data mart -hops factory system mart-add --answers mart.json # add a data mart to a gold layer -hops factory system mart-update --answers mart.json # change a data mart's requirements -hops factory system mart-delete [--tables] # delete a data mart's jobs, and its own tables -hops factory system job-delete [--tables] # delete one job, and the tables only it writes -hops factory system add-tables --answers new.json # add bronze tables to a silver layer +hops factory run medallion-gold --change add-mart [--answers mart.json] # add a data mart to a gold layer +hops factory run medallion-gold --change edit-mart [--answers mart.json] # change a data mart's requirements +hops factory run medallion-gold --change delete-mart # delete a data mart's jobs, and if asked its own tables +hops factory run medallion-gold --change delete-job # delete one job, and if asked the tables only it writes +hops factory run medallion-silver --change add-tables [--answers new.json] # add bronze tables to a silver layer hops factory system status # write the layer's health report, status/report.html -hops factory system backfill # recompute the layer's tables from the whole history +hops job run --start-time 1970-01-01 --end-time --wait # backfill: recompute a job's tables from the whole history hops factory system delete --assets # delete the layer, its jobs and tables; never what it reads ``` +A change to a built system, a layer's or any other, is one of its factory's `changes`: **Change** on the system's page lists them, and the layer pages' **Add data mart**, **Edit**, **Add tables** and delete icons open them. +Each opens the change's form; saving records the request in the system's `system.yaml` (`changes`, `status: pending`) with `hops factory run --change `, and resumes the build in a Terminal tab, which carries out the pending requests first. +A build deletes jobs and feature groups only with `hops factory system delete-assets`, which refuses a table the system reads or one of a lower medallion layer. + ### Create your own factory A factory is a YAML definition: the questions of its creation form, the phases of its build, and the instructions Claude Code follows to build what the answers describe. From a35a0376cd68134345fcf27ef089dbf4620da558 Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Wed, 7 Oct 2026 08:01:00 +0200 Subject: [PATCH 37/42] [HWORKS-3275] ML software factory https://hopsworks.atlassian.net/browse/HWORKS-3275 The gold medallion factory is titled Data Mart, so New Medallion Layer lists it as Data Mart and its page reads New Data Mart. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01RfHMsBbYDQZqoD58XVqeYE --- docs/user_guides/client_installation/index.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/user_guides/client_installation/index.md b/docs/user_guides/client_installation/index.md index b850f7cffd..2272934a1d 100644 --- a/docs/user_guides/client_installation/index.md +++ b/docs/user_guides/client_installation/index.md @@ -180,7 +180,7 @@ A medallion layer organizes tables as bronze (raw data as it arrived), silver (c Hopsworks installs an archived schematized tag, `medallion_table`, whose `layer` is `bronze`, `silver` or `gold` and whose `lifecycle` is `dev`, `staging` or `prod`; every change of a table's value is kept in the tag history. When you ingest data with a dltHub data source, **Tag as bronze tables** in the review, off by default, tags every feature group it creates as bronze. -**New Medallion Layer**, in the Factory, lists two factories, **Silver layer** and **Gold Data Mart**; the silver one builds a silver layer from the project's bronze feature groups. +**New Medallion Layer**, in the Factory, lists two factories, **Silver layer** and **Data Mart**; the silver one builds a silver layer from the project's bronze feature groups. A project without feature groups has nothing to build from: ingest raw data as bronze tables first. The page asks for the bronze tables to build from (only those tagged bronze are shown while any are), the silver tasks (deduplicate, cast types, standardize values, handle nulls, validate with a rejects table, protect personal data, conform entities, surrogate keys, referential checks), additional tasks in your own words, the engine, the refresh cadence and the lifecycle. The engine is dbt on Trino unless an additional task needs code SQL does not express well, when PySpark is suggested. @@ -202,7 +202,7 @@ Every job has a delete icon that asks whether to also delete the feature groups ### Build a gold medallion layer of data marts -**New Medallion Layer** with **Gold Data Mart** builds a gold layer from silver tables: a Kimball dimensional model, a star or snowflake schema, for the queries the layer will serve. +**New Medallion Layer** with **Data Mart** builds a gold layer from silver tables: a Kimball dimensional model, a star or snowflake schema, for the queries the layer will serve. The page asks for those queries, the model, the silver tables to read, the standards every data mart follows (naming, modeling, documentation and quality, proposed and editable), and the first data mart. A gold layer is a set of data marts, each added, changed and deleted on its own, with its own fact and dimension tables and its own jobs, `--`, at its own refresh. From 959688b9abae82ca604b8d862195a43d8a4a0924 Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Wed, 7 Oct 2026 08:59:44 +0200 Subject: [PATCH 38/42] [HWORKS-3275] ML software factory https://hopsworks.atlassian.net/browse/HWORKS-3275 Mistral is an LLM provider for the terminal's coding agents, with Mistral Large 4 (mistral-large-4) as its model, through its OpenAI-compatible chat completions API at https://api.mistral.ai/v1 and the key in MISTRAL_API_KEY. Mistral has no Anthropic-compatible API, so codex and opencode use it and Claude Code keeps its own login, as with DeepMind and Berget. hopsworks-ee: the MISTRAL provider. hopsworks-front: its logo, from @lobehub/icons-static-svg 1.95.1 as the other color logos. logicalclocks.github.io: the provider list. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01RfHMsBbYDQZqoD58XVqeYE --- docs/user_guides/projects/terminal/start.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user_guides/projects/terminal/start.md b/docs/user_guides/projects/terminal/start.md index 170f4f9d28..6b5e8219cf 100644 --- a/docs/user_guides/projects/terminal/start.md +++ b/docs/user_guides/projects/terminal/start.md @@ -20,7 +20,7 @@ A Spark terminal has no GPUs. ## LLM Provider **LLM Provider**, under **Start**, shows the provider the terminal's coding agents use; **Subscription**, the default, leaves `claude`, `codex`, `copilot` and `opencode` on their own logins. -Clicking it shows the providers as cards, each saying whether its key is saved: Kimi, DeepMind, Berget, GLM, Anthropic, OpenAI, Meta and OpenRouter. +Clicking it shows the providers as cards, each saying whether its key is saved: Kimi, DeepMind, Berget, Mistral, GLM, Anthropic, OpenAI, Meta and OpenRouter. For a provider, pick a model or type a model id, enter its API key and press **Save key**, then **Test**, which checks the key against the provider's model list without spending tokens and says whether the model is listed. The key is saved as a private secret in your Hopsworks account, `llm__api_key`, not in the browser; the provider and model are kept in the browser. From 3b21e0353c3161d1371b0df189579cd7cc4ec365 Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Wed, 7 Oct 2026 09:29:16 +0200 Subject: [PATCH 39/42] [HWORKS-3275] ML software factory https://hopsworks.atlassian.net/browse/HWORKS-3275 A data mart can ask for dashboards. The Data Mart factory's mart form (Create, Add data mart and Edit data mart) has a Dashboards question, answered in plain text: for each dashboard, who reads it, the questions it answers, and the charts and filters wanted. It is recorded as requirements.dashboards, and a mart has a seventh phase, dashboards, which /hops-gold runs once the mart is verified: it builds each dashboard in Superset over the mart's gold tables, as a script in marts//dashboards/ that creates it or updates it in place, checks each chart against the mart's verified numbers, and records it as dashboards: [{name, id, url, script, charts}]. With the question empty the phase is done at once. Editing the text reruns the phase, updating dashboards in place. hopsworks-api: the requirement and phase, the dashboards phase of /hops-gold, and gold-marts.md. hopsworks-ee: the Data Mart definition's Dashboards section and phase. hopsworks-front: the phase, and the mart's dashboards, asked for and then linked, on the layer page. logicalclocks.github.io: the requirement and the phase. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01RfHMsBbYDQZqoD58XVqeYE --- docs/user_guides/client_installation/index.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/user_guides/client_installation/index.md b/docs/user_guides/client_installation/index.md index 2272934a1d..da88154465 100644 --- a/docs/user_guides/client_installation/index.md +++ b/docs/user_guides/client_installation/index.md @@ -215,9 +215,10 @@ A data mart's requirements are: - **Freshness and changes**: the refresh and freshness target, how late arrivals, updates and deletes are processed, and whether corrections restate published results. - **Verification**: example questions with the answers expected, in plain English; the totals that must reconcile, and with what; and how refreshes and reruns are proven correct. The build runs every check after the backfill and again after a refresh, records the results, and does not mark the mart built until each passes. - **Quality and access**: the invariants to test, what happens when a check fails (fail the run, quarantine the failing rows, or warn), who may read which rows and columns, and the projects it is shared with. +- **Dashboards**: in plain text, the dashboards to build from the mart: for each, who reads it, the questions it answers, and the charts and filters wanted. Optional. -**Create** runs `hops factory run medallion-gold --answers` and starts Claude Code on `/hops-gold `, which builds each mart: its requirements, the design of its facts and dimensions (a dimension used by several marts is built once and shared), the dbt models with their tests, the backfill with the verification, the schedule and tags (`layer: gold`, the silver tables as parents), and a verified refresh. -The layer's page shows each mart with its phases, tables, jobs and verification results: **Edit** changes its requirements and applies the change, **Add data mart** adds one, and deleting a mart deletes its jobs and, if asked, its tables that no other mart lists. +**Create** runs `hops factory run medallion-gold --answers` and starts Claude Code on `/hops-gold `, which builds each mart: its requirements, the design of its facts and dimensions (a dimension used by several marts is built once and shared), the dbt models with their tests, the backfill with the verification, the schedule and tags (`layer: gold`, the silver tables as parents), a verified refresh, and last the Superset dashboards the mart asks for, each over its gold tables and checked against the mart's verified numbers. +The layer's page shows each mart with its phases, tables, jobs, verification results and links to its dashboards: **Edit** changes its requirements and applies the change, **Add data mart** adds one, and deleting a mart deletes its jobs and, if asked, its tables that no other mart lists. ```bash hops factory run medallion-silver [--answers answers.json] # record a silver layer and build it with Claude Code From 3dde6ca2eced3ca1c2c81eb8e266aa9f3c826de8 Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Wed, 7 Oct 2026 15:38:16 +0200 Subject: [PATCH 40/42] [HWORKS-3275] ML software factory https://hopsworks.atlassian.net/browse/HWORKS-3275 New Medallion Layer gets an Examples submenu, and its first example is a bronze layer of generated web shop data: clicks, customers, products and orders, as offline Delta feature groups tagged bronze, so silver and gold layers can be built on them without a source system. A backfill writes the 30 days up to the last midnight (1,000,000 clicks, 10,000 customers, 1,000 products, 20,000 orders), an hourly job writes 10,000 clicks an hour and a daily job writes each day's new and changed customers, products and orders. About 0.001% of clicks arrive twice, under a new ingest_id, for a silver layer to remove. The example is built by a new built-in factory, medallion-bronze, which is listed only through its examples, and /hops-bronze. The Factory guide describes the example and lists medallion-bronze among the built-in factories. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 --- docs/user_guides/client_installation/index.md | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/docs/user_guides/client_installation/index.md b/docs/user_guides/client_installation/index.md index da88154465..2e8d539ad1 100644 --- a/docs/user_guides/client_installation/index.md +++ b/docs/user_guides/client_installation/index.md @@ -181,7 +181,13 @@ Hopsworks installs an archived schematized tag, `medallion_table`, whose `layer` When you ingest data with a dltHub data source, **Tag as bronze tables** in the review, off by default, tags every feature group it creates as bronze. **New Medallion Layer**, in the Factory, lists two factories, **Silver layer** and **Data Mart**; the silver one builds a silver layer from the project's bronze feature groups. -A project without feature groups has nothing to build from: ingest raw data as bronze tables first. +A project without feature groups has nothing to build from: ingest raw data as bronze tables first, or build the example bronze layer. + +**Examples** in the same menu lists example layers. **Synthetic clickstream (bronze)** builds a bronze layer of generated web shop data with `hops factory run medallion-bronze --preset clickstream-example`, which copies the generator into `clickstream-bronze/` and starts Claude Code on `/hops-bronze clickstream-bronze`. +It writes four offline Delta feature groups tagged `layer: bronze`: `clickstream_customers`, `clickstream_products`, `clickstream_orders` and `clickstream_clicks`. +A backfill job writes the 30 days up to the last midnight: 10,000 customers, 1,000 products, 20,000 orders and 1,000,000 clicks. +An hourly job writes 10,000 clicks an hour, and a daily job writes the day's new customers, products and orders and its changes: profile updates, price changes, discontinued products and order status changes. +The data is raw on purpose, for a silver layer to cleanse: about 0.001% of clicks arrive twice under a new `ingest_id`, and an order's lines are a JSON array in its `items` column. The page asks for the bronze tables to build from (only those tagged bronze are shown while any are), the silver tasks (deduplicate, cast types, standardize values, handle nulls, validate with a rejects table, protect personal data, conform entities, surrogate keys, referential checks), additional tasks in your own words, the engine, the refresh cadence and the lifecycle. The engine is dbt on Trino unless an additional task needs code SQL does not express well, when PySpark is suggested. Each bronze table has its own refresh, hourly, daily or weekly; a silver table refreshes as often as its most frequently updated source, and the layer gets one job per refresh, each with its own schedule and freshness target. @@ -221,6 +227,7 @@ A data mart's requirements are: The layer's page shows each mart with its phases, tables, jobs, verification results and links to its dashboards: **Edit** changes its requirements and applies the change, **Add data mart** adds one, and deleting a mart deletes its jobs and, if asked, its tables that no other mart lists. ```bash +hops factory run medallion-bronze --preset clickstream-example # build the example bronze layer of generated data hops factory run medallion-silver [--answers answers.json] # record a silver layer and build it with Claude Code hops factory run medallion-gold [--answers answers.json] # record a gold layer and its first data mart hops factory run medallion-gold --change add-mart [--answers mart.json] # add a data mart to a gold layer @@ -241,7 +248,7 @@ A build deletes jobs and feature groups only with `hops factory system delete-as A factory is a YAML definition: the questions of its creation form, the phases of its build, and the instructions Claude Code follows to build what the answers describe. The **Factory** page shows a section for each of the project's factories, with **New** opening the form it generates. -The five built-in factories, `ml-batch`, `ml-realtime`, `ml-agent`, `medallion-silver` and `medallion-gold`, are read-only; clone one to change it. +The six built-in factories, `ml-batch`, `ml-realtime`, `ml-agent`, `medallion-bronze`, `medallion-silver` and `medallion-gold`, are read-only; clone one to change it. A form has no conditions: every question of a section is shown, and a section with defaults can be collapsed to a summary of its answers with **Edit**. **Factory > Manage factories** lists every factory with its version and how many systems it built. From d5531e5b453ed84aa1e742f8d75df5e40394d232 Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Wed, 7 Oct 2026 19:13:40 +0200 Subject: [PATCH 41/42] [HWORKS-3275] ML software factory https://hopsworks.atlassian.net/browse/HWORKS-3275 The New Data Mart form asks only for the layer, its questions, the first mart's refresh and the silver tables; the mart is named after the layer, and its requirements are folded away and drafted by the build when left blank. On a cluster with Platform Intelligence, Suggest beside the silver tables selects the tables the answers call for and drafts the folded questions still blank, through a new POST factories/suggest-sources endpoint that keeps only candidate tables and answers to the questions asked. A medallion layer's table picker no longer offers the Show logging feature groups switch. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 --- docs/user_guides/client_installation/index.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/user_guides/client_installation/index.md b/docs/user_guides/client_installation/index.md index 2e8d539ad1..8d0069bc8a 100644 --- a/docs/user_guides/client_installation/index.md +++ b/docs/user_guides/client_installation/index.md @@ -209,7 +209,10 @@ Every job has a delete icon that asks whether to also delete the feature groups ### Build a gold medallion layer of data marts **New Medallion Layer** with **Data Mart** builds a gold layer from silver tables: a Kimball dimensional model, a star or snowflake schema, for the queries the layer will serve. -The page asks for those queries, the model, the silver tables to read, the standards every data mart follows (naming, modeling, documentation and quality, proposed and editable), and the first data mart. +The page asks for those queries, the model, the first data mart's refresh and freshness target, and the silver tables to read. +On a cluster with Platform Intelligence, **Suggest** selects the silver tables the answers so far call for, and drafts answers to the folded questions below that are still blank; change any of them as you like. +The standards every data mart follows (naming, modeling, documentation and quality, proposed and editable) and the first data mart's requirements below are folded away: anything left blank is drafted by the build from the layer's questions and the silver tables, for you to confirm, and recorded in `system.yaml`, where you can edit it later. +The first data mart is named after the layer. A gold layer is a set of data marts, each added, changed and deleted on its own, with its own fact and dimension tables and its own jobs, `--`, at its own refresh. A data mart's requirements are: From d9d1fbfe980e56479576faaec0f68cea33c48ddb Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Wed, 7 Oct 2026 21:30:18 +0200 Subject: [PATCH 42/42] [HWORKS-3275] ML software factory https://hopsworks.atlassian.net/browse/HWORKS-3275 The medallion layer factories are renamed analytics everywhere: the built-in factories analytics-bronze, analytics-silver and analytics-gold, the analytics_table tag, the hops-analytics skill, the analytics_factory_enabled variable, and the Factory's New Analytics menu, now a primary button like New ML System. Bronze, silver and gold keep their names. None of this has shipped, so nothing migrates; a cluster that ran the branch renames its factory rows, definitions, tag and variable once. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5.5 --- docs/user_guides/client_installation/index.md | 42 +++++++++---------- 1 file changed, 21 insertions(+), 21 deletions(-) diff --git a/docs/user_guides/client_installation/index.md b/docs/user_guides/client_installation/index.md index 8d0069bc8a..f8f80543ef 100644 --- a/docs/user_guides/client_installation/index.md +++ b/docs/user_guides/client_installation/index.md @@ -141,7 +141,7 @@ Pass `--no-launch` to record the system only. `hops factory run` registers each system with the project, by the HopsFS directory of its code, or by its GitHub repository when you build from an external client. A GitHub repository the build creates, an example's included, is named `hops-`, or `hops--` when you already have one of that name. -**Factory**, in the project menu below Catalog, holds the ML system factory and the medallion layer factory. +**Factory**, in the project menu below Catalog, holds the ML system factory and the analytics layer factory. Its ML system factory lists the project's systems for every member, when the project has registered systems or the cluster has the terminal: each with its type, status, phases done, owner and last update, and an open folder for the ones whose code you can open, a lock for the ones you cannot, and a link for the ones in a GitHub repository. **Login to GitHub** runs `github-login` in a Terminal tab; the page shows whether the terminal's GitHub CLI is logged in, which the build needs to create the repository. **New ML System** lists three factories, **Batch ML system**, **Real-time ML system** and **Agentic system**, and their examples. @@ -174,16 +174,16 @@ hops factory system remove # remove it from the list; i hops factory system delete --assets [--repo] # also delete what it created, and its repository ``` -### Build a silver medallion layer +### Build a silver analytics layer -A medallion layer organizes tables as bronze (raw data as it arrived), silver (cleansed and conformed) and gold (consumption-ready). -Hopsworks installs an archived schematized tag, `medallion_table`, whose `layer` is `bronze`, `silver` or `gold` and whose `lifecycle` is `dev`, `staging` or `prod`; every change of a table's value is kept in the tag history. +An analytics layer organizes tables as bronze (raw data as it arrived), silver (cleansed and conformed) and gold (consumption-ready). +Hopsworks installs an archived schematized tag, `analytics_table`, whose `layer` is `bronze`, `silver` or `gold` and whose `lifecycle` is `dev`, `staging` or `prod`; every change of a table's value is kept in the tag history. When you ingest data with a dltHub data source, **Tag as bronze tables** in the review, off by default, tags every feature group it creates as bronze. -**New Medallion Layer**, in the Factory, lists two factories, **Silver layer** and **Data Mart**; the silver one builds a silver layer from the project's bronze feature groups. +**New Analytics**, in the Factory, lists two factories, **Silver layer** and **Data Mart**; the silver one builds a silver layer from the project's bronze feature groups. A project without feature groups has nothing to build from: ingest raw data as bronze tables first, or build the example bronze layer. -**Examples** in the same menu lists example layers. **Synthetic clickstream (bronze)** builds a bronze layer of generated web shop data with `hops factory run medallion-bronze --preset clickstream-example`, which copies the generator into `clickstream-bronze/` and starts Claude Code on `/hops-bronze clickstream-bronze`. +**Examples** in the same menu lists example layers. **Synthetic clickstream (bronze)** builds a bronze layer of generated web shop data with `hops factory run analytics-bronze --preset clickstream-example`, which copies the generator into `clickstream-bronze/` and starts Claude Code on `/hops-bronze clickstream-bronze`. It writes four offline Delta feature groups tagged `layer: bronze`: `clickstream_customers`, `clickstream_products`, `clickstream_orders` and `clickstream_clicks`. A backfill job writes the 30 days up to the last midnight: 10,000 customers, 1,000 products, 20,000 orders and 1,000,000 clicks. An hourly job writes 10,000 clicks an hour, and a daily job writes the day's new customers, products and orders and its changes: profile updates, price changes, discontinued products and order status changes. @@ -192,7 +192,7 @@ The page asks for the bronze tables to build from (only those tagged bronze are The engine is dbt on Trino unless an additional task needs code SQL does not express well, when PySpark is suggested. Each bronze table has its own refresh, hourly, daily or weekly; a silver table refreshes as often as its most frequently updated source, and the layer gets one job per refresh, each with its own schedule and freshness target. It also asks how the tables behave, with defaults: history (`latest`, one row per key, or `full`, every version by time), deletes in bronze (`ignore` or `propagate`), bronze schema changes (`fail`, or `evolve` by adding new columns), a late-data lookback re-read before each window (none, a day or a week), the share of rejected rows above which a run fails, with an alert when one fails, and a freshness target. -**Create** runs `hops factory run medallion-silver --answers` in a Terminal tab named after the layer, which records its specification in `/system.yaml` and starts Claude Code on `/hops-silver `. +**Create** runs `hops factory run analytics-silver --answers` in a Terminal tab named after the layer, which records its specification in `/system.yaml` and starts Claude Code on `/hops-silver `. The build profiles the bronze tables, designs the silver tables, writes and tests the code, backfills the whole bronze history once, schedules the job and tags the silver feature groups `layer: silver`, and verifies one window. Silver tables are feature groups, materialized, never views, and in third normal form: one table per entity or event, every column depending on its table's key alone, lookups in tables of their own, and no aggregates, which belong in gold. @@ -202,13 +202,13 @@ Every silver table records its bronze tables as its parents, so the lineage show A layer's page shows its requirements as they were filled in. **Status** on a built layer's page reports the job's runs and each table's rows, last write against the freshness target, rejected share against the limit, and file layout, with a box for asking Claude Code anything about the report, filled in with a request to fix the problems it found; **Backfill** reprocesses every bronze row into the silver tables, running each job. A layer's page shows its phases, the silver tables and jobs it made, the bronze tables it reads, and its tasks, with links to the layer's GitHub repository and to its dbt code in the file browser. -A medallion's silver and gold layers share one GitHub repository, `hops-`, where the prefix is the layer's name without `-silver` or `-gold`: each layer is a directory in it, a gold layer joins the repository of the silver layer it reads, and the build uses the repository when it exists, creates it when it does not, and pushes every commit to it. +An analytics pipeline's silver and gold layers share one GitHub repository, `hops-`, where the prefix is the layer's name without `-silver` or `-gold`: each layer is a directory in it, a gold layer joins the repository of the silver layer it reads, and the build uses the repository when it exists, creates it when it does not, and pushes every commit to it. Every job has a delete icon that asks whether to also delete the feature groups only that job writes; **Add tables** adds bronze tables, each with its refresh, and describes the silver tables wanted from them, which Claude Code designs and builds with the rest. **Delete** removes the layer from the Factory, or deletes it with its jobs, tables and directory; the tables a layer reads are never deleted. -### Build a gold medallion layer of data marts +### Build a gold analytics layer of data marts -**New Medallion Layer** with **Data Mart** builds a gold layer from silver tables: a Kimball dimensional model, a star or snowflake schema, for the queries the layer will serve. +**New Analytics** with **Data Mart** builds a gold layer from silver tables: a Kimball dimensional model, a star or snowflake schema, for the queries the layer will serve. The page asks for those queries, the model, the first data mart's refresh and freshness target, and the silver tables to read. On a cluster with Platform Intelligence, **Suggest** selects the silver tables the answers so far call for, and drafts answers to the folded questions below that are still blank; change any of them as you like. The standards every data mart follows (naming, modeling, documentation and quality, proposed and editable) and the first data mart's requirements below are folded away: anything left blank is drafted by the build from the layer's questions and the silver tables, for you to confirm, and recorded in `system.yaml`, where you can edit it later. @@ -226,18 +226,18 @@ A data mart's requirements are: - **Quality and access**: the invariants to test, what happens when a check fails (fail the run, quarantine the failing rows, or warn), who may read which rows and columns, and the projects it is shared with. - **Dashboards**: in plain text, the dashboards to build from the mart: for each, who reads it, the questions it answers, and the charts and filters wanted. Optional. -**Create** runs `hops factory run medallion-gold --answers` and starts Claude Code on `/hops-gold `, which builds each mart: its requirements, the design of its facts and dimensions (a dimension used by several marts is built once and shared), the dbt models with their tests, the backfill with the verification, the schedule and tags (`layer: gold`, the silver tables as parents), a verified refresh, and last the Superset dashboards the mart asks for, each over its gold tables and checked against the mart's verified numbers. +**Create** runs `hops factory run analytics-gold --answers` and starts Claude Code on `/hops-gold `, which builds each mart: its requirements, the design of its facts and dimensions (a dimension used by several marts is built once and shared), the dbt models with their tests, the backfill with the verification, the schedule and tags (`layer: gold`, the silver tables as parents), a verified refresh, and last the Superset dashboards the mart asks for, each over its gold tables and checked against the mart's verified numbers. The layer's page shows each mart with its phases, tables, jobs, verification results and links to its dashboards: **Edit** changes its requirements and applies the change, **Add data mart** adds one, and deleting a mart deletes its jobs and, if asked, its tables that no other mart lists. ```bash -hops factory run medallion-bronze --preset clickstream-example # build the example bronze layer of generated data -hops factory run medallion-silver [--answers answers.json] # record a silver layer and build it with Claude Code -hops factory run medallion-gold [--answers answers.json] # record a gold layer and its first data mart -hops factory run medallion-gold --change add-mart [--answers mart.json] # add a data mart to a gold layer -hops factory run medallion-gold --change edit-mart [--answers mart.json] # change a data mart's requirements -hops factory run medallion-gold --change delete-mart # delete a data mart's jobs, and if asked its own tables -hops factory run medallion-gold --change delete-job # delete one job, and if asked the tables only it writes -hops factory run medallion-silver --change add-tables [--answers new.json] # add bronze tables to a silver layer +hops factory run analytics-bronze --preset clickstream-example # build the example bronze layer of generated data +hops factory run analytics-silver [--answers answers.json] # record a silver layer and build it with Claude Code +hops factory run analytics-gold [--answers answers.json] # record a gold layer and its first data mart +hops factory run analytics-gold --change add-mart [--answers mart.json] # add a data mart to a gold layer +hops factory run analytics-gold --change edit-mart [--answers mart.json] # change a data mart's requirements +hops factory run analytics-gold --change delete-mart # delete a data mart's jobs, and if asked its own tables +hops factory run analytics-gold --change delete-job # delete one job, and if asked the tables only it writes +hops factory run analytics-silver --change add-tables [--answers new.json] # add bronze tables to a silver layer hops factory system status # write the layer's health report, status/report.html hops job run --start-time 1970-01-01 --end-time --wait # backfill: recompute a job's tables from the whole history hops factory system delete --assets # delete the layer, its jobs and tables; never what it reads @@ -245,13 +245,13 @@ hops factory system delete --assets # delete the layer, its jobs A change to a built system, a layer's or any other, is one of its factory's `changes`: **Change** on the system's page lists them, and the layer pages' **Add data mart**, **Edit**, **Add tables** and delete icons open them. Each opens the change's form; saving records the request in the system's `system.yaml` (`changes`, `status: pending`) with `hops factory run --change `, and resumes the build in a Terminal tab, which carries out the pending requests first. -A build deletes jobs and feature groups only with `hops factory system delete-assets`, which refuses a table the system reads or one of a lower medallion layer. +A build deletes jobs and feature groups only with `hops factory system delete-assets`, which refuses a table the system reads or one of a lower analytics layer. ### Create your own factory A factory is a YAML definition: the questions of its creation form, the phases of its build, and the instructions Claude Code follows to build what the answers describe. The **Factory** page shows a section for each of the project's factories, with **New** opening the form it generates. -The six built-in factories, `ml-batch`, `ml-realtime`, `ml-agent`, `medallion-bronze`, `medallion-silver` and `medallion-gold`, are read-only; clone one to change it. +The six built-in factories, `ml-batch`, `ml-realtime`, `ml-agent`, `analytics-bronze`, `analytics-silver` and `analytics-gold`, are read-only; clone one to change it. A form has no conditions: every question of a section is shown, and a section with defaults can be collapsed to a summary of its answers with **Edit**. **Factory > Manage factories** lists every factory with its version and how many systems it built.