Describe the bug
InteractiveLauncher.launch consults the local GPU count before it honors an
explicit --nproc-per-node, so the flag is overridden by the device probe in
two ways.
nemo_automodel/components/launcher/interactive.py:
nproc_per_node: int | None = launcher_config
...
num_devices = determine_local_world_size(nproc_per_node="gpu")
assert num_devices > 0, "Expected num-devices to be > 0"
if nproc_per_node == 1 or num_devices == 1:
logger.info("Launching job locally on a single device")
return self._run_recipe_in_process(recipe_target, config)
else:
effective_nproc = nproc_per_node if nproc_per_node is not None else num_devices
1. --nproc-per-node N is silently downgraded to a single process on a
one-GPU host.
num_devices == 1 short-circuits to the in-process path regardless of what the
user asked for, so on a single-GPU box:
automodel config.yaml --nproc-per-node 4
runs one process, not four, and logs "Launching job locally on a single
device" without mentioning that the requested value was discarded. Running
several ranks on one device is a normal way to exercise FSDP/DDP or
pipeline-parallel code paths on a dev box, and torchrun --nproc-per-node 4
supports it directly.
2. --nproc-per-node 1 fails on a host with no visible CUDA device.
determine_local_world_size(nproc_per_node="gpu") raises when
torch.cuda.is_available() is false, and it runs before the
nproc_per_node == 1 branch — the one path that needs no device count at all,
because it runs the recipe in-process:
$ automodel config.yaml --nproc-per-node 1
...
File ".../torch/distributed/run.py", line 745, in determine_local_world_size
raise ValueError("Cuda is not available.") from e
ValueError: Cuda is not available.
This also fires when CUDA_VISIBLE_DEVICES="" is set. The error surfaces from
inside torch.distributed.run, so it does not say which AutoModel flag or
requirement is involved.
Steps/Code to reproduce bug
Symptom 2 reproduces on any host without a visible CUDA device:
automodel examples/llm_finetune/llama3_2/llama3_2_1b_squad.yaml --nproc-per-node 1
Symptom 1 reproduces on a single-GPU host with --nproc-per-node 4, or in a
unit test by stubbing determine_local_world_size to return 1: the launcher
takes the in-process branch and torch.distributed.run.run is never called.
Expected behavior
An explicit --nproc-per-node N is honored: 1 runs in-process, N > 1
launches torchrun with N workers. The GPU count is only probed when the flag
is not supplied, since that is the only case where it is needed to pick a
default.
--help describes the flag as "Number of workers per node for local/interactive
jobs", with no mention of it being clamped to the visible device count.
Environment overview
main at 3ddef9b. Symptom 2 observed on a CPU-only host; symptom 1 is
reachable in a CPU unit test by stubbing the probe.
Additional context
To be clear about scope: fixing symptom 2 does not make CPU training work —
the recipe still requires CUDA further down. It makes the documented flag
behave as documented and moves the failure to where the framework can report
its own requirement, instead of failing in a device probe that the requested
single-process path never needed.
Happy to send a PR: probe only when nproc_per_node is None, plus CPU unit
tests for both symptoms.
Describe the bug
InteractiveLauncher.launchconsults the local GPU count before it honors anexplicit
--nproc-per-node, so the flag is overridden by the device probe intwo ways.
nemo_automodel/components/launcher/interactive.py:1.
--nproc-per-node Nis silently downgraded to a single process on aone-GPU host.
num_devices == 1short-circuits to the in-process path regardless of what theuser asked for, so on a single-GPU box:
runs one process, not four, and logs "Launching job locally on a single
device" without mentioning that the requested value was discarded. Running
several ranks on one device is a normal way to exercise FSDP/DDP or
pipeline-parallel code paths on a dev box, and
torchrun --nproc-per-node 4supports it directly.
2.
--nproc-per-node 1fails on a host with no visible CUDA device.determine_local_world_size(nproc_per_node="gpu")raises whentorch.cuda.is_available()is false, and it runs before thenproc_per_node == 1branch — the one path that needs no device count at all,because it runs the recipe in-process:
This also fires when
CUDA_VISIBLE_DEVICES=""is set. The error surfaces frominside
torch.distributed.run, so it does not say which AutoModel flag orrequirement is involved.
Steps/Code to reproduce bug
Symptom 2 reproduces on any host without a visible CUDA device:
Symptom 1 reproduces on a single-GPU host with
--nproc-per-node 4, or in aunit test by stubbing
determine_local_world_sizeto return 1: the launchertakes the in-process branch and
torch.distributed.run.runis never called.Expected behavior
An explicit
--nproc-per-node Nis honored:1runs in-process,N > 1launches torchrun with
Nworkers. The GPU count is only probed when the flagis not supplied, since that is the only case where it is needed to pick a
default.
--helpdescribes the flag as "Number of workers per node for local/interactivejobs", with no mention of it being clamped to the visible device count.
Environment overview
mainat 3ddef9b. Symptom 2 observed on a CPU-only host; symptom 1 isreachable in a CPU unit test by stubbing the probe.
Additional context
To be clear about scope: fixing symptom 2 does not make CPU training work —
the recipe still requires CUDA further down. It makes the documented flag
behave as documented and moves the failure to where the framework can report
its own requirement, instead of failing in a device probe that the requested
single-process path never needed.
Happy to send a PR: probe only when
nproc_per_node is None, plus CPU unittests for both symptoms.