Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 48 additions & 0 deletions mkdocs/docs/concepts/tasks.md
Original file line number Diff line number Diff line change
Expand Up @@ -179,6 +179,54 @@ Jobs on each node communicate using their private IP addresses. Use `DSTACK_MAST
For convenience, `~/.ssh/config` is preconfigured with these options, so a simple `ssh <node_ip>` is enough.
For a list of nodes IPs check the `DSTACK_NODES_IPS` environment variable.

### Node groups

A task can define multiple node groups. Each group has its own `nodes` count,
`resources`, `commands`, and `ports`.

<div editor-title=".dstack.yml">

```yaml
type: task
name: ray-cluster

python: 3.12

groups:
- name: head
nodes: 1
commands:
- pip uninstall -y ray && pip install -U "ray[default]"
- ray start --head --port=6379 --block
resources:
cpu: 2
memory: 4GB..
ports:
- 8265

- name: workers
nodes: 2
commands:
- pip uninstall -y ray && pip install -U "ray[default]"
- ray start --address=${{ groups[0].nodes[0].IP_ADDRESS }}:6379 --block
resources:
gpu: H100:8
```

</div>

Commands in any group can reference the internal IP address of any node in the run via
`${{ groups[i].nodes[j].IP_ADDRESS }}`, where `i` is the index of the group in `groups` and `j` is
the index of the node within that group.

Node `groups[0].nodes[0]` is the run's master node — it is what `DSTACK_MASTER_NODE_IP` resolves

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO, this can use a blockquote to highlight this?

to, and `startup_order` and `stop_criteria` apply to it across all groups.

> Currently, only `resources`, `commands`, and `ports` can be configured per node group. [`groups`](../reference/dstack.yml/task.md#groups) and top-level `nodes` are mutually exclusive.
> Group `resources` are not inherited from the task's top-level `resources`. A group that
> omits `resources` gets the default values.
> Support for other properties is coming soon.

### Resources

When you specify a resource value like `cpu` or `memory`,
Expand Down
Loading