Conversation
Require IteratorDone before treating a worker as complete, and report its exit code through UnexpectedWorkerExitError when it exits without completion. Recheck the queue after a timeout before declaring failure. Port the crash and completion-race coverage to the shared subprocess tests, preserving compatibility with Python 3.10.
|
Currently a problem for dfine node. |
| """Raised when not even the smallest unit of work fits in memory.""" | ||
|
|
||
|
|
||
| class UnexpectedWorkerExitError(RuntimeError): |
There was a problem hiding this comment.
A RuntimeError is what _perform_state treats as retryable: it resets the state to TrainModelDownloaded and starts _train again. The non-finite loss from the description fails the same way every time, so the node would keep retraining and never reach ReadyForCleanup. A worker killed by the OOM killer should be retried though. Maybe CriticalError when the worker exited on its own, and a retry only when a signal killed it?
| raise UnexpectedWorkerExitError( | ||
| f'{process.name} exited with code {process.exitcode} without sending IteratorDone' | ||
| ) from e |
There was a problem hiding this comment.
_iterator_wrapper catches Exception, but sys.exit(1) raises SystemExit. So the worker's reason never reaches the queue and only the exit code is left. process.name is this same literal for detection too, so a failed training and a failed detection produce the same message in the loop. Could _iterator_wrapper catch SystemExit as well and pass the reason along? This error would then be left for the cases where there is nothing to report anyway: os._exit, SIGKILL, the OOM killer. The module docstring would need the new case too.
2 participants
Motivation
For example, non-finite training loss calls
sys.exit(1)inside the worker while the main node stays alive. The worker exits without sendingIteratorDone, but an empty queue could previously be treated as successful completion. Failed training or incomplete detection could therefore advance through the success pipeline.Implementation
IteratorDonefor successful completion; otherwise report the worker's exit code.Validation: 236 library unit tests passed on macOS (Python 3.13.12), including all 12 subprocess tests; Ruff and
git diff --checkpassed. Live Learning Loop integration and CUDA execution were not tested.