Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/component-paths.json
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
"lint_job": "lint-control-plane",
"paths": [
"components/control-plane/**",
"OPENSHELL_VERSION",
"components/api-server/go.mod",
"components/api-server/go.sum",
"components/api-server/pkg/api/grpc/**",
Expand Down
11 changes: 11 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -239,6 +239,17 @@ check-dependency-age: test-dependency-age-policy
sync-openshell-version:
PYTHONDONTWRITEBYTECODE=1 python3 scripts/sync_openshell_version.py --stamp

.PHONY: vendor-openshell-chart
vendor-openshell-chart:
@. ./OPENSHELL_VERSION && \
echo "Vendoring OpenShell chart $$OPENSHELL_TAG from $$OPENSHELL_CHART_REPO..." && \
git clone --depth 1 --branch "$$OPENSHELL_TAG" "$$OPENSHELL_CHART_REPO" /tmp/openshell-chart-vendor && \
rm -rf charts/openshell && \
cp -r /tmp/openshell-chart-vendor/deploy/helm/openshell charts/openshell && \
rm -rf /tmp/openshell-chart-vendor && \
grep -rl $$'\xe2\x80\x94' charts/openshell/ | xargs -r sed -i 's/\xe2\x80\x94/-/g' && \
echo "Vendored charts/openshell/ at $$OPENSHELL_TAG"

.PHONY: test-openshell-version-policy
test-openshell-version-policy:
PYTHONDONTWRITEBYTECODE=1 python3 -m unittest scripts/test_sync_openshell_version.py
Expand Down
4 changes: 2 additions & 2 deletions OPENSHELL_VERSION
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
# OpenShell upstream coordinates. Gateway, supervisor, and CLI image tags
# and the chart ref use OPENSHELL_TAG. The console image is pinned by digest
# (it uses its own tagging scheme). Update this file when bumping versions.
OPENSHELL_TAG=v0.0.116-rhaiv.6
OPENSHELL_TAG=v0.1.2-rhaiv.0

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Major] First minor-version bump (0.0.x -> 0.1.x): the default gateway/supervisor images move to v0.1.2-rhaiv.0, and the linked upstream changes include Agent Sandbox v1.0.3 plus breaking proto/policy changes. Because no Go code changed here, a green build does not validate runtime interop. The concrete risk: gateway 0.1.2 may require a different Agent Sandbox API version than the v1beta1 documented in specs/platform/global-architecture.spec.md (~L894, "v1beta1 for gateway 0.0.109"); if the installed CRD version no longer matches, sandbox RPCs surface as gRPC Unimplemented at runtime. The lessons-log entry in this PR defers this as unverified. Please confirm the required Agent Sandbox API version for 0.1.2 (ideally via the OpenShift e2e gate) and update that spec line rather than deferring it. Confidence: Medium.

OPENSHELL_CHART_REPO=https://github.com/opendatahub-io/openshell.git
OPENSHELL_GATEWAY_IMAGE=quay.io/opendatahub/odh-openshell-gateway
OPENSHELL_SUPERVISOR_IMAGE=quay.io/opendatahub/odh-openshell-supervisor
OPENSHELL_CLI_IMAGE=quay.io/opendatahub/odh-openshell-cli
OPENSHELL_CONSOLE_IMAGE=quay.io/gkrumbach07/openshell-dashboard
OPENSHELL_CONSOLE_DIGEST=sha256:c69c1f34c574556684710a7d2d2a3654f164b855efe0d273a098782447068fc5
OPENSHELL_CONSOLE_DIGEST=sha256:1d36331138c37aa75285869a21d2aa35ebdab5b1f6980f028b28df405ec5a927

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Minor] The v0.1.2 console bump to sha-71335e5 (digest sha256:1d363311...) is correctly mirrored in components/control-plane/internal/gateway/config.go L30, but the pinned-image documentation in specs/platform/e2e-console-browser-testing.spec.md still points at the previous build: L650 shows @sha256:c69c1f34... tag sha-978bcb5, and L766 says "OpenShell console (sha-978bcb5)". Update those two references so the docs match the code, per the "image references must match across the stack" convention.

Separately (pre-existing, not introduced here): the console default still resolves to a personal registry quay.io/gkrumbach07/openshell-dashboard. A platform-owned mirror would be preferable for a production default image; the HYPERSHELL_CONSOLE_IMAGE override mitigates but does not change the default.

23 changes: 23 additions & 0 deletions charts/openshell/.helmignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# Patterns to ignore when building packages.
.DS_Store
.git/
.gitignore
.bzr/
.bzrignore
.hg/
.hgignore
.svn/
*.swp
*.bak
*.tmp
*.orig
*~
.project
.idea/
*.tmproj
.vscode/

# Ignore development files
README.md.gotmpl
skaffold.yaml
ci/
13 changes: 13 additions & 0 deletions charts/openshell/Chart.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

apiVersion: v2
# Chart name determines the OCI image name: ghcr.io/nvidia/openshell/helm-chart:<version>
name: helm-chart
description: runtime environment for autonomous agents
type: application
# version and appVersion are patched to the release semver by CI before helm package.
# appVersion doubles as the default image tag (image.tag defaults to appVersion when
# empty), so a released chart automatically pulls matching gateway, sandbox, and supervisor images.
version: 0.0.0
appVersion: "0.0.0"
486 changes: 486 additions & 0 deletions charts/openshell/README.md

Large diffs are not rendered by default.

310 changes: 310 additions & 0 deletions charts/openshell/README.md.gotmpl
Original file line number Diff line number Diff line change
@@ -0,0 +1,310 @@
# OpenShell Helm Chart

<!--
This file is generated by helm-docs.
Edit README.md.gotmpl and values.yaml, then run `mise run helm:docs`.
-->

> **Experimental** - the Kubernetes deployment path is under active development. Expect rough edges and breaking changes.

This chart deploys the OpenShell gateway into a Kubernetes cluster. It is published as an OCI artifact to GHCR at `oci://ghcr.io/nvidia/openshell/helm-chart`.

By default, this chart also creates the namespace-scoped resources needed by
sandboxes. For a shared-gateway deployment, install it with
`workspaceResources.enabled=false`, then install the
`deploy/helm/openshell-workspace` chart in every pre-provisioned workspace
namespace. The gateway and workspace releases can then be upgraded and removed
independently. Use Kubernetes `operator` workspace mode when one gateway serves
multiple pre-provisioned workspace namespaces.

## Cluster-scoped vs namespaced objects

Most objects in this chart are namespaced and land in the release namespace.
Only two are cluster-scoped:

| Object | Default name |
| --- | --- |
| `ClusterRole` | `<fullname>-node-reader-<release namespace>` |
| `ClusterRoleBinding` | `<fullname>-node-reader-<release namespace>` |

By default the release creates both, so an install by a cluster-admin is
unchanged. On clusters where cluster-scoped RBAC is owned by a different team,
split the install in two.

A cluster-admin applies the cluster-scoped objects once per gateway
ServiceAccount, rendered from the same values the release uses:

```shell
helm template openshell oci://ghcr.io/nvidia/openshell/helm-chart --version <version> \
--namespace openshell -f my-values.yaml \
--set rbac.create=true \
--set rbac.clusterScoped.create=true \
--set agentSandbox.preflight.enabled=false \
--show-only templates/clusterrole.yaml \
--show-only templates/clusterrolebinding.yaml | kubectl apply -f -
```

A namespace-admin then installs and upgrades the release with cluster-scoped
objects omitted, using [`ci/values-namespace-admin.yaml`](ci/values-namespace-admin.yaml)
or the equivalent `--set`:

```shell
helm upgrade --install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version <version> \
--namespace openshell -f my-values.yaml \
--set rbac.clusterScoped.create=false
```

The gateway ServiceAccount name and namespace do not change, so the
pre-created `ClusterRoleBinding` keeps matching the release. This works with
`serviceAccount.create=false` too: the `ClusterRoleBinding` subject follows
`serviceAccount.name`, so render the admin step with the same values.

### Which flag the installer needs

In the default `shared` workspace mode the release also creates a namespaced
sandbox `Role` granting Agent Sandbox (`agents.x-k8s.io`) permissions.
Kubernetes forbids granting permissions you do not hold, and the built-in
`admin` ClusterRole does not cover that CRD, so an installer holding only
`admin` cannot create it. `rbac.clusterScoped.create=false` alone is then not
enough and the install fails with `attempting to grant RBAC permissions not
currently held`.

| Workspace mode | Installer holds | Use |
| --- | --- | --- |
| `shared` | built-in `admin` only | `rbac.create=false`, cluster-admin pre-creates all gateway RBAC |
| `shared` | `admin` plus the sandbox permissions in the namespace | `rbac.clusterScoped.create=false` |
| `managed`, `operator` | built-in `admin` only | `rbac.clusterScoped.create=false` |

`managed` and `operator` render no namespaced sandbox `Role`, so no extra grant
is needed there.

With `rbac.create=false` the cluster-admin applies the namespaced RBAC too,
adding it to the same render:

```shell
helm template openshell oci://ghcr.io/nvidia/openshell/helm-chart --version <version> \
--namespace openshell -f my-values.yaml \
--set rbac.create=true \
--set rbac.clusterScoped.create=true \
--set agentSandbox.preflight.enabled=false \
--show-only templates/clusterrole.yaml \
--show-only templates/clusterrolebinding.yaml \
--show-only templates/role.yaml \
--show-only templates/rolebinding.yaml \
--show-only templates/peer-role.yaml | kubectl apply -f -
```

To grant the installer the sandbox permissions instead, bind it to a Role
carrying the same rules as the chart's `openshell-sandbox` Role.

The certgen hook and credential driver RBAC keep their own flags
(`pkiInitJob.enabled` and
`server.credentialDrivers.kubernetesSecrets.rbac.create`).

### Migrating an existing release

Helm deletes objects that leave a release manifest, so setting
`rbac.clusterScoped.create=false` on a release that already owns the
`ClusterRole` and `ClusterRoleBinding` deletes them. The gateway then loses
TokenReview until a cluster-admin re-applies them. Hand ownership over first, as
cluster-admin, so nothing is deleted:

```shell
kubectl annotate clusterrole "openshell-node-reader-<namespace>" \
helm.sh/resource-policy=keep --overwrite
kubectl annotate clusterrolebinding "openshell-node-reader-<namespace>" \
helm.sh/resource-policy=keep --overwrite
```

The objects then survive the upgrade that sets the flag, and the cluster-admin
owns them from that point on. Fresh installs need no such step.

`rbac.clusterScoped.create` is independent of
`server.drivers.kubernetes.workspaceMode`. Managed and operator modes change
what the `ClusterRole` contains, but they never force the namespaced release to
apply it. Re-run the cluster-admin step after changing values that affect the
`ClusterRole` rules.


## Prerequisites

> **Required:** Your cluster CNI MUST enforce Kubernetes `NetworkPolicy` for
> ingress and egress in every sandbox namespace. OpenShell creates the policies,
> but Kubernetes accepts them even if no CNI enforces them. Without enforcement,
> sandbox workloads may connect directly and bypass supervisor network policy.
> Verify CNI support before installing OpenShell.

The Kubernetes Agent Sandbox CRDs and controller must be installed on the cluster before deploying OpenShell. Install them with:

```shell
kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/latest/download/sandbox.yaml
```

The chart does not install this cluster-scoped dependency. By default, it
fails before creating gateway resources when the cluster serves neither
supported Sandbox API (`agents.x-k8s.io/v1beta1` or
`agents.x-k8s.io/v1alpha1`). Disable the check with
`agentSandbox.preflight.enabled=false` for offline `helm template` rendering,
where Helm cannot discover cluster APIs.

## Install on Kubernetes

```shell
helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version <version>
```

## Install on OpenShift

See the full [OpenShift install guide](https://docs.nvidia.com/openshell/latest/kubernetes/openshift) for details. Quick start:

```shell
# Precreate the openshell namespace
oc create ns openshell

# Deploy openshell with overrides to allow SCC assignment of fsGroup and runAsUser for the gateway
helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version <version> -n openshell \
--set server.disableTls=true \
--set podSecurityContext.fsGroup=null \
--set securityContext.runAsUser=null
```

On OpenShift 4.22+, end-to-end TLS is supported via `BackendTLSPolicy`. See the
[OpenShift install guide](https://docs.nvidia.com/openshell/latest/kubernetes/openshift#end-to-end-tls-openshift-422) for details.

## Available versions

| Tag | Source | Notes |
| --- | --- | --- |
| `<semver>` (e.g. `0.6.0`) | Tagged GitHub release | Tracks the matching gateway, sandbox, and supervisor image versions. Recommended for production. |
| `<semver>-pre.N` (e.g. `0.1.0-pre.3`) | A specific prerelease candidate | Immutable candidate pin that tracks images with the same exact version. |
| `0.0.0-dev` | Latest commit on `main` | Floating tag, overwritten on every push. `appVersion` is `dev`, so images resolve to the `:dev` tag. |
| `0.0.0-dev.<commit-sha>` | A specific commit on `main` | Per-commit pin. Chart version and `appVersion` both use the full 40-character commit SHA, which matches the image tag pushed by CI. |

Prerelease and `dev` tags are intended for testing changes ahead of a release. Production deployments should pin to a stable tagged release.

## Configuration

See [`values.yaml`](values.yaml) for source defaults. Selected overlays:

- [`ci/values-gateway.yaml`](ci/values-gateway.yaml) - gateway-only configuration
- [`ci/values-cert-manager.yaml`](ci/values-cert-manager.yaml) - cert-manager integration
- [`ci/values-keycloak.yaml`](ci/values-keycloak.yaml) - Keycloak OIDC integration
- [`ci/values-high-availability.yaml`](ci/values-high-availability.yaml) - CI overlay for multi-replica external PostgreSQL testing
- [`ci/values-spire.yaml`](ci/values-spire.yaml) - SPIFFE/SPIRE provider token grants
- [`ci/values-spire-stack.yaml`](ci/values-spire-stack.yaml) - SPIRE hardened chart values for local development

### Database backend

By default, OpenShell uses SQLite and runs the gateway as a StatefulSet so the
database is backed by a per-pod PVC:

```yaml
server:
dbUrl: "sqlite:/var/openshell/openshell.db"
```

#### External PostgreSQL

Use external PostgreSQL when the gateway should connect to a database managed
outside this chart. The OpenShell chart does not deploy a database; install
PostgreSQL separately using the chart, operator, or managed service that fits
your environment, then pass the connection URI through a Secret.

Create a Secret containing the PostgreSQL connection URI if one does not
already exist:

```bash
kubectl create secret generic my-pg-credentials -n openshell \
--from-literal=uri="postgresql://user:pass@host:5432/dbname"
```

Then install the chart pointing at that Secret:

```bash
helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version <version> \
-n openshell \
--set workload.kind=deployment \
--set server.externalDbSecret=my-pg-credentials
```

Use `workload.kind=deployment` for external database-backed multi-replica
gateways. `workload.kind=statefulset` is still available for single-replica
SQLite installs and for operators who explicitly need StatefulSet identity or
storage semantics.

### Credential storage

By default, the chart uses the gateway's encrypted database credential storage.
The gateway writes encrypted provider credential envelopes to the OpenShell
database. The chart creates a retained Kubernetes Secret with the shared
key-encryption key and injects that key into every gateway pod, so the same
default works for single-replica and external database-backed HA deployments.

Use `kubernetes-secrets` or `vault` instead when credentials should live in a
cluster or external secret backend. Enabling one external credential driver
disables the default credential-storage key-encryption key Secret and env injection.

#### OpenShift

Append these flags to any of the PostgreSQL commands above for OpenShift:

```
--set server.disableTls=true \
--set podSecurityContext.fsGroup=null \
--set securityContext.runAsUser=null
```

### High availability

Set `replicaCount` above `1` only with `server.externalDbSecret`; the default
SQLite database is per pod and cannot coordinate multiple gateway replicas.
The chart creates a headless peer Service for gateway-to-gateway relay traffic.
StatefulSet pods use stable pod DNS names through that headless Service.
Deployment pods advertise their pod IP with `OPENSHELL_PEER_ENDPOINT`, because
Kubernetes does not assign stable per-pod DNS names to Deployment replicas.

Gateway peer traffic uses Kubernetes ServiceAccount identity. Each gateway pod
mounts a projected, pod-bound ServiceAccount token with audience
`openshell-gateway-peer`; receiving replicas validate that token with the
Kubernetes TokenReview API, verify the live pod UID and Helm selector labels,
and authorize only the internal `PeerRelay` RPC. The chart does not create or
accept a shared gateway peer Secret.

With gateway TLS enabled, peer calls use the chart CA and client TLS Secret for
server verification and mTLS. The client verifies the stable gateway Service
DNS name while connecting directly to the owning pod. Custom TLS Secrets must
include that Service DNS name in the server certificate and provide the CA and
client credentials configured by `server.tls`.

## Secret bootstrap

By default, a pre-install/pre-upgrade hook Job runs `openshell-gateway generate-certs`
to create the gateway's server/client mTLS Secrets and sandbox JWT signing Secret.
The Job uses the gateway image itself, so air-gapped environments only need to
mirror that one image (no separate openssl/alpine sidecar).

When `certManager.enabled=true`, cert-manager owns the TLS Secrets and the chart
runs the same hook in JWT-only mode because cert-manager does not create the
sandbox JWT signing Secret. This precedence applies even if
`pkiInitJob.enabled` remains true. Set `pkiInitJob.enabled=false` only when an
external non-cert-manager TLS source manages TLS and you pre-create the sandbox
JWT signing Secret.

## SPIFFE/SPIRE provider token grants

Set `server.providerTokenGrants.spiffe.enabled=true` to let the gateway and
sandbox supervisors use SPIFFE JWT-SVIDs for dynamic provider token grants. The
chart keeps supervisor-to-gateway authentication on gateway-minted sandbox JWTs,
mounts the SPIFFE CSI socket into the gateway pod, exports
`OPENSHELL_GATEWAY_SPIFFE_WORKLOAD_API_SOCKET`, and passes the socket path to
the Kubernetes driver so sandbox pods can mount the same socket.

For local development, uncomment the SPIRE Helm releases in `skaffold.yaml` and
add `ci/values-spire.yaml` to the OpenShell release values files.

The gateway verifies supervisor JWT-SVIDs with JWT bundles fetched from the
SPIFFE Workload API, so this path does not require access to the SPIRE OIDC
discovery endpoint or its TLS CA.

{{ template "chart.valuesSection" . }}
{{ template "helm-docs.versionFooter" . }}
11 changes: 11 additions & 0 deletions charts/openshell/ci/values-cert-manager.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# Merge after values.yaml when cert-manager CRDs are installed, e.g.:
# helm install ... -f values.yaml -f ci/values-cert-manager.yaml
# Or add this file to skaffold manifests.helm.releases[].valuesFiles.
server:
disableTls: false

certManager:
enabled: true
Loading
Loading