Create ray-vllm.mdx

docs.json

@@ -261,6 +261,7 @@
                 "pages": [
                   "instant-clusters/pytorch",
                   "instant-clusters/axolotl",
+                  "instant-clusters/ray-vllm",
                   "instant-clusters/slurm"
                 ]
               }

instant-clusters/ray-vllm.mdx

@@ -0,0 +1,234 @@
+---
+title: "Deploy an Instant Cluster with Ray and vLLM"
+sidebarTitle: "Ray + vLLM"
+description: "Run distributed inference across multiple nodes using Ray and vLLM on an Instant Cluster."
+tag: BETA
+---
+
+This tutorial shows how to use Instant Clusters with Ray to run distributed inference on large language models. By combining Ray's cluster management with vLLM's tensor and pipeline parallelism, you can serve models that exceed the memory of a single node — for example, a 70B parameter model across multiple 8×H100 pods.
+
+Ray handles the cluster topology; vLLM uses it to split the model across GPUs both within each node (tensor parallelism) and across nodes (pipeline parallelism).
+
+<Note>
+  Distributed inference with Ray and vLLM on Instant Clusters is currently in beta. Join our [Discord](https://discord.gg/runpod) to provide feedback and get support.
+</Note>
+
+---
+
+## Requirements
+
+- A Runpod account with sufficient credits for a multi-node cluster
+- Basic familiarity with large language model inference and distributed GPU setups
+
+---
+
+## Step 1: Deploy an Instant Cluster
+
+1. Open the [Instant Clusters page](https://console.runpod.io/instant-clusters).
+2. Click **Create Cluster**.
+3. Name your cluster and configure it. For this walkthrough, set **Pod Count** to **2** and select **8× H100 SXM GPUs** per pod. Use the **Runpod PyTorch** template as your base image.
+
+   <Note>
+   Increase `/dev/shm` when configuring your pod. The default (64 MB) is too small for large tensor-parallel workloads. Set it to at least 8 GB. In the pod configuration, add the environment variable `MALLOC_ARENA_MAX=1` and set `--shm-size` to `8g` in your Docker run options.
+   </Note>
+
+4. Click **Deploy Cluster**. You are redirected to the Instant Clusters page.
+
+---
+
+## Step 2: Start the Ray head on pod-0
+
+The first pod (`CLUSTERNAME-pod-0`) runs the Ray head node. All other pods connect to it as workers.
+
+1. Click your cluster to expand the pod list.
+2. Click **CLUSTERNAME-pod-0**, then click **Connect → Web Terminal**.
+3. In the terminal, clone the reference scripts:
+
+   ```bash
+   {/* [STO-463] Replace with canonical repo URL once scripts are published. */}
+   git clone https://github.com/runpod/ray-vllm-cluster.git
+   ```
+
+4. Run the head startup script:
+
+   ```bash
+   bash ray-vllm-cluster/head.sh
+   ```
+
+   The script sets the correct NIC address and starts Ray:
+
+   ```bash
+   # head.sh (excerpt — see full script in the repo)
+   export RAY_NODE_IP_ADDRESS=$(hostname -I | awk '{print $1}')
+   export VLLM_HOST_IP=$RAY_NODE_IP_ADDRESS
+
+   ray start --head \
+     --node-ip-address=$RAY_NODE_IP_ADDRESS \
+     --port=6379 \
+     --num-gpus=$NUM_TRAINERS
+   ```
+
+   <Note>
+   `RAY_NODE_IP_ADDRESS` and `VLLM_HOST_IP` must be set to the pod's internal network IP — not `0.0.0.0`. Setting them prevents Ray and vLLM from binding to the wrong interface on multi-NIC pods.
+   </Note>
+
+---
+
+## Step 3: Join the worker pods to the cluster
+
+Repeat this for each remaining pod in the cluster (`pod-1`, `pod-2`, …).
+
+1. In the Instant Clusters page, click the next pod and open its **Web Terminal**.
+2. Clone the same scripts:
+
+   ```bash
+   {/* [STO-463] Replace with canonical repo URL once scripts are published. */}
+   git clone https://github.com/runpod/ray-vllm-cluster.git
+   ```
+
+3. Run the worker startup script:
+
+   ```bash
+   bash ray-vllm-cluster/worker.sh
+   ```
+
+   The script waits for the head to be reachable, then joins:
+
+   ```bash
+   # worker.sh (excerpt — see full script in the repo)
+   export RAY_NODE_IP_ADDRESS=$(hostname -I | awk '{print $1}')
+   export VLLM_HOST_IP=$RAY_NODE_IP_ADDRESS
+
+   # Wait for Ray head
+   until bash -c ">/dev/tcp/$MASTER_ADDR/6379" 2>/dev/null; do
+     echo "Waiting for Ray head at $MASTER_ADDR:6379..."
+     sleep 2
+   done
+
+   ray start \
+     --address=$MASTER_ADDR:6379 \
+     --node-ip-address=$RAY_NODE_IP_ADDRESS \
+     --num-gpus=$NUM_TRAINERS
+   ```
+
+   `$MASTER_ADDR` is injected automatically by Runpod into all pods in the cluster — it resolves to `pod-0`.
+
+---
+
+## Step 4: Verify the cluster
+
+Run this on `pod-0` to confirm all nodes have joined:
+
+```bash
+ray status
+```
+
+Expected output for a two-pod cluster with 8 GPUs each:
+
+```
+======== Autoscaler status: ... ========
+Node status
+---------------------------------------------------------------
+Healthy:
+ 2 node(s) with resources: {"GPU": 8.0, ...}
+```
+
+You can also open the Ray dashboard. The console shows the dashboard port in the **Connect** dialog for `pod-0`.
+
+---
+
+## Step 5: Launch distributed inference with vLLM
+
+Run this on `pod-0` only. vLLM uses the Ray cluster that is already running.
+
+```bash
+bash ray-vllm-cluster/serve.sh
+```
+
+The script launches vLLM with tensor parallelism across GPUs within each node and pipeline parallelism across nodes:
+
+```bash
+# serve.sh (excerpt — see full script in the repo)
+python -m vllm.entrypoints.openai.api_server \
+  --model meta-llama/Meta-Llama-3-70B \
+  --tensor-parallel-size $NUM_TRAINERS \   # GPUs per node (e.g., 8)
+  --pipeline-parallel-size $NUM_NODES \    # number of nodes (e.g., 2)
+  --host 0.0.0.0 \
+  --port 8000
+```
+
+<Tip>
+`--tensor-parallel-size` should equal the number of GPUs per node (`$NUM_TRAINERS`). `--pipeline-parallel-size` should equal the number of nodes (`$NUM_NODES`). Both are injected as environment variables by Runpod.
+</Tip>
+
+vLLM connects to the running Ray cluster automatically. It may take several minutes to load model weights across all nodes.
+
+---
+
+## Step 6: Test the endpoint
+
+Once vLLM reports that it is ready, validate from `pod-0`:
+
+```bash
+# Check the server is healthy
+curl http://localhost:8000/health
+
+# Confirm the model is loaded
+curl http://localhost:8000/v1/models
+```
+
+Then send a test request:
+
+```bash
+curl http://localhost:8000/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -d '{
+    "model": "meta-llama/Meta-Llama-3-70B",
+    "messages": [{"role": "user", "content": "Hello!"}]
+  }'
+```
+
+---
+
+## Step 7: Clean up
+
+When you are done, return to the [Instant Clusters page](https://console.runpod.io/instant-clusters) and delete your cluster. Leaving it running continues to incur charges.
+
+---
+
+## Environment variables reference
+
+Runpod injects these environment variables into every pod in the cluster. The startup scripts rely on them.
+
+| Variable | Description |
+|---|---|
+| `MASTER_ADDR` | Hostname of `pod-0`, the Ray head node |
+| `MASTER_PORT` | Port for inter-node communication (default: `29500`) |
+| `NUM_NODES` | Total number of pods in the cluster |
+| `NUM_TRAINERS` | Number of GPUs per pod |
+| `NODE_RANK` | Index of this pod (`0` for head, `1+` for workers) |
+
+---
+
+## Common issues
+
+**Ray workers don't join**
+Confirm `$MASTER_ADDR` resolves from each worker pod. Run `ping $MASTER_ADDR` in a worker terminal. If it fails, the cluster network may still be initializing — wait 30 seconds and try again.
+
+**vLLM OOM during model load**
+Check that `/dev/shm` is large enough (at least 8 GB for 70B models). Also verify that `--tensor-parallel-size` matches the number of GPUs per node — a mismatch causes uneven shard sizes.
+
+**`VLLM_HOST_IP` binding error**
+This error occurs when vLLM tries to bind to `0.0.0.0` on a pod with multiple network interfaces. Make sure `VLLM_HOST_IP` is set to the internal IP (`hostname -I | awk '{print $1}'`) before starting the server.
+
+**Stale Ray cluster after restart**
+If you restart a pod, Ray does not automatically rejoin the cluster. Rerun `head.sh` on `pod-0` first, then `worker.sh` on all other pods.
+
+---
+
+## Next steps
+
+- Adapt the serve script to load your own model from a [GlobalStore](/storage/globalstore) or [Network Volume](/storage/network-volumes) mount.
+- Scale up by increasing the pod count and adjusting `--pipeline-parallel-size` accordingly.
+- Try [Axolotl on an Instant Cluster](/instant-clusters/axolotl) for distributed fine-tuning.
+- Review the [Instant Cluster configuration reference](/instant-clusters/configuration) for full details on environment variables and networking.