Skiff

Skiff allows HPC users to run their own private Kubernetes "mini Cloud" on a typical HPC cluster and issue commands using Kubernetes-native tools without requiring root privileges.

A Slurm job starts a K3s cluster of the nodes it is given, each node inside a rootless Apptainer container (a "bubble"). Pods run as unprivileged processes on those nodes, with the cluster network, Services, DNS, volumes and the job's NVIDIA GPUs Kubernetes users expect. When the job ends, the cluster is gone. Skiff can also run a cluster without Slurm (see below).

Get started

1. Download the latest release, skiff-<version>.tar.gz, and unpack it in your home directory on the cluster (it must be shared by the compute nodes):

   tar -xzf skiff-<version>.tar.gz -C ~
   cd ~/skiff-<version>/scripts

2. Fetch what a cluster needs, on the login node (or any node with Internet access):

   bash skiff.sh download

   It gets the Skiff image of this release from Docker Hub (about 200 MB), takes the programs Skiff needs on a host out of it (Apptainer, slirp4netns, socat and kubectl, so nothing has to be installed), and puts the images of the cluster's system pods in its image cache. Everything goes to ~/.skiff. If the compute nodes are of another architecture than the login node (x86_64 and arm64), it fetches for both: Slurm says which the nodes are, and --arch adds one it does not (bash skiff.sh download --arch arm64).

3. Check that a compute node has what Skiff needs:

   srun -N1 bash skiff.sh check

   It lists everything that is missing, each with what to do about it (some things only your administrator can provide, and it says what to ask for). Fix what it reports and run it again until the node is "ready for Skiff". If it warns that the node cannot reach the image registries, the cluster still starts (step 2 fetched its images), but the images of your own pods have to be fetched the same way, on the login node, each named exactly as your pods name it: bash skiff.sh download docker.io/library/nginx:1.29.

4. Submit the cluster as a job, from the same folder; --nodes is its size:

   sbatch --nodes=2 skiff.slurm

5. Use it. When the job's output (skiff-cluster-<job ID>.out) says how to connect, point kubectl at it:

   export KUBECONFIG=${HOME}/.skiff/kubeconfig
   export PATH=${HOME}/.skiff/tools/$(uname -m)/bin:${PATH}
   kubectl get nodes

   For a first try, a web server, a Service in front of it, and a client that reaches it by name (the server takes a few seconds to listen once its pods are Ready, hence the retries):

   kubectl create deployment hello --image=nginxinc/nginx-unprivileged:1.29 --replicas=2
   kubectl expose deployment hello --port=80 --target-port=8080
   kubectl run client --image=busybox:1.37 --restart=Never -- sleep 3600
   kubectl wait --for=condition=Ready pod -l app=hello --timeout=5m
   kubectl wait --for=condition=Ready pod/client --timeout=5m
   kubectl exec client -- sh -c 'for i in $(seq 10); do wget -qO- -T 2 http://hello && break; sleep 2; done'
   kubectl get pods -o wide
   kubectl delete deployment,service hello && kubectl delete pod client

6. Stop it by ending the job, which stops the pods before the cluster goes:

   scancel --batch --signal=TERM <job ID>

   To remove Skiff, delete ~/.skiff (it also holds the data of your PersistentVolumeClaims) and the folder of the release.

Problems

If a node lacks something Skiff needs, the check in step 3 says what it is, and so does the job's output. If something does not work, open an issue on GitHub, with the output of skiff.sh check on a compute node, the job's output, ~/.skiff/job-results/<job ID>.cause if there is one, the operating system of the nodes, the version of Slurm, and what your administrator changed for Skiff.

Learn more

The Skiff Guide has the rest:

• Using Skiff: requirements, running without Slurm, configuration, the directories Skiff uses, storage, containers, GPUs, images, what to do when a job ends unexpectedly, and running from the repository.
• Developing Skiff: the architecture, building, testing it locally with Vagrant or on a Slurm cluster, and releases.

The components have their own documents: Skifflet, Plaid and the tests.

Without Slurm

Skiff also runs where there is no Slurm, on machines you log in to: it then starts the cluster itself, on one machine or on several that share your home and reach one another with ssh keys. It is just as rootless: everything runs as you, an ordinary user, with nothing installed and no root privileges, as in a Slurm job. After steps 1 and 2, check each machine, then start the cluster in the background instead of submitting a job, and stop it when you are done:

bash skiff.sh check
bash skiff.sh start                  # this machine; or: bash skiff.sh start host1 host2 (the first runs the controller)
bash skiff.sh status
bash skiff.sh stop

start says the job and its output file (skiff-cluster-<job ID>.out in the current folder); use the cluster as in step 5. The cluster assumes the machines are yours alone; the guide has the details.

Acknowledgements

Skiff builds upon the legacy of previous projects that have explored the integration of Kubernetes in HPC from different viewpoints: KNoC, HPK, and hpktainer. Check them out if you are interested in the history that led to the current design.

The developers also acknowledge the use of generative AI tools (Gemini by Google, Sol and Astra by OpenAI, and most recently Claude Sonnet and Opus by Anthropic) for assistance during the development of this software.