Introduce logger on dask workers by sbrandstaeter · Pull Request #170 · queens-py/queens

sbrandstaeter · 2025-06-07T12:23:53Z

Description and Context:
What and Why?

Summary

This PR introduces logging support for code running inside Dask workers, specifically within the driver.run and dataprocessor.get_data_from_file methods, as discussed in issue #165. This should enable debugging these methods even on an HPC cluster.

Motivation

Currently, there's no logging when these methods are executed on a Dask worker, making it difficult to debug or monitor behavior during distributed execution.

What's Proposed

A dedicated logger (self.logger_on_dask_worker) is initialized inside the above methods if it doesn't already exist.
This allows logging within Dask workers to be captured reliably.

Logging Setup

Logs are written both to file and to the standard stream.
One log file per worker is used. If a worker restarts and gets the same ID (e.g. due to restart_worker=True), it reuses the same log file.
Log files are stored under <experiment_dir>/dask_workers_logs.
The stream output might also be captured by job schedulers like SLURM, depending on the environment.

Notes

This PR is meant to establish a solid initial foundation for Dask-side logging. While the current solution may not cover all edge cases or logging best practices, it's a step in the right direction and open for future improvements. Feedback on design and setup is welcome.

Related Issues and Pull Requests

Closes Logging for dask cluster and workers, i.e., drivers, is not working #165
Related to

Interested Parties

gilrrei

@sbrandstaeter I like this feature and I think we should merge this :)

leahaeusel

Since Sebastian asked us to have a look at this PR in a past developer meeting, I have added my point of view on the open conversations. Hopefully, this will help to get this merged soon 🤞

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

bennoschoenstein · 2026-10-06T15:15:19Z

I rebased this PR onto the current main and reworked the design.

What you get

Every job writes its own log file: <experiment_dir>/<job_id>/worker.log. It contains all messages from the driver and the data processors of this job, for example which file was read or why a job failed. Before, these messages were lost on the dask workers, or ended up mixed across jobs in the SLURM output.

How it works

Driver.run() is now final (cannot be overridden). It sets up the logging for one job, calls the new abstract _run(), logs the exception if the job fails, and closes the log file afterwards.
Drivers and data processors log via self.logger_on_worker. These loggers are children of one parent logger (queens.worker), so the log file of a job is attached only to the parent and all of them write into it.
On a dask worker QUEENS' normal logging is not set up. In this case the messages are also printed to the stream, so they still show up in the SLURM output.

Options (all drivers)

worker_log_level (default logging.INFO, names like "DEBUG" also work): which messages go into the log. If QUEENS runs with debug=True, the worker log shows debug messages too, regardless of this setting.
write_worker_log_files (default True): set to False to switch the log file off. The Function driver has it off by default, because it usually runs many cheap jobs.

What changes for users

Custom drivers must implement _run() instead of run() (the tutorial is updated). Everything else is backwards compatible.
The FAQ on directories now documents worker.log.

Copilot

Copilot review overview

🟡 Changes recommended

The implementation contradicts the documented per-worker layout and introduces an incompatible custom-driver contract.

Review effort: Balanced
Findings: 1 · 1 · 1

Open (3)

Replacing run with _run breaks existing Driver subclasses · New
Worker logs use per-job paths instead of one file per worker · New
Tests do not cover Dask stream fallback or handler cleanup · New

leahaeusel

Thank you very much @bennoschoenstein for taking care of getting this important feature ready to merge! I really like the new job-specific log files and think they will be very helpful for users in the future.

Co-authored-by: Sebastian Brandstaeter <sebastian.brandstaeter@unibw.de>

Replace worker_log_level and write_worker_log_files by one argument: worker_log_level=None switches the job log files off, any level switches them on. The Function driver writes no log files by default. Remove the stream handler on dask workers, so that the SLURM output is not touched by QUEENS anymore. Define the format of log files once.

Run 8 jobs on 4 workers with the Pool and Local schedulers and check that the messages of the driver and the data processor of a job end up in the log file of that job. The assumption of one job per process at a time is stated in the docstring of setup_logger_on_worker.

Keep run as the method that custom drivers implement, as before. The log file of a job is set up in __call__, which the schedulers call. A direct call of run behaves as on main. This avoids the breaking change of the abstract _run method.

bennoschoenstein · 2026-10-09T08:24:52Z

Update after the review (replaces my comment from Oct 6 where it differs):

What you get

Every job writes its own log file <experiment_dir>/<job_id>/worker.log with the messages of the driver and the data processors of this job, e.g., which file was read or why the job failed.

How it works

Driver.__call__, which the schedulers call, sets up the log file of the job, calls run, logs the exception if the job fails, and closes the log file afterwards. run stays the method that custom drivers implement, so nothing breaks.
Drivers and data processors log via self.logger_on_worker. These loggers are children of one parent logger (queens.worker), so the log file is attached only to the parent and all of them write into it.
The SLURM output of the dask workers is not touched.

Option (all drivers)

worker_log_level (default logging.INFO, names like "DEBUG" also work): level of the job log file. None switches the log files off, which is the default of the Function driver. If QUEENS runs locally with debug=True, the log file shows debug messages too.

Rebased onto the current main.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot

🟡 Changes recommended

Logger levels currently leak after cleanup and can make log files more verbose than configured.

1 open finding

Reset parent worker logger level to NOTSET · New

1 resolved since last review

Replacing run with _run breaks existing Driver subclasses

🧠 Review effort: Balanced

Reset the level of the worker logger after a job, so that it does not leak into logging outside of jobs. Open the log file in write mode, so that a rerun of a job starts a new file. Keep the level of the log file if the process logs on a lower level, e.g., in debug mode.

leahaeusel

The changes look good to me 🚀 Nice catch with the write mode to overwrite old log files on reruns!

sbrandstaeter

@bennoschoenstein Many thanks for taking over and pushing this over the finish line. Looks good to me but of course can't approve.

bennoschoenstein · 2026-10-12T11:09:52Z

@rjoussen could you please take a final look at this PR? 👯

bennoschoenstein · 2026-10-12T11:11:55Z

My approve is just to test, if recent pushers can approve or not

leahaeusel · 2026-10-12T11:43:46Z

Good idea to test this! I guess the test failed 😅 I don't really know why, though, because we have this setting activated:

sbrandstaeter · 2026-10-12T12:08:35Z

Good idea to test this! I guess the test failed 😅 I don't really know why, though, because we have this setting activated:

Yeah that is what we wanted to test. No idea why it does not work...

rjoussen · 2026-10-12T12:20:35Z

@bennoschoenstein I will look into it today or tomorrow.

rjoussen

I just have one last note.

rjoussen

thanks for your efforts!

sbrandstaeter force-pushed the introduce-logger-on-dask-worker branch from 143eff2 to 01c9f02 Compare June 7, 2025 22:06

gilrrei reviewed Jun 11, 2025

View reviewed changes

Comment thread queens/drivers/_driver.py Outdated

gilrrei reviewed Aug 23, 2025

View reviewed changes

Comment thread queens/utils/config_directories.py Outdated

sbrandstaeter force-pushed the introduce-logger-on-dask-worker branch from 01c9f02 to de5bd40 Compare August 23, 2025 10:56

sbrandstaeter commented Aug 23, 2025

View reviewed changes

Comment thread src/queens/drivers/_driver.py Outdated

sbrandstaeter commented Aug 23, 2025

View reviewed changes

Comment thread src/queens/drivers/_driver.py Outdated

sbrandstaeter commented Aug 23, 2025

View reviewed changes

Comment thread src/queens/drivers/function.py Outdated

gilrrei reviewed Aug 24, 2025

View reviewed changes

Comment thread src/queens/drivers/function.py Outdated

Comment thread src/queens/data_processors/_data_processor.py Outdated

sbrandstaeter force-pushed the introduce-logger-on-dask-worker branch from e687545 to 6cf1c78 Compare October 8, 2025 10:58

leahaeusel reviewed Oct 17, 2025

View reviewed changes

Comment thread src/queens/drivers/_driver.py Outdated

Comment thread src/queens/drivers/_driver.py Outdated

Comment thread src/queens/drivers/function.py Outdated

Comment thread src/queens/data_processors/_data_processor.py Outdated

leahaeusel mentioned this pull request Oct 27, 2025

Meeting on 27.10.2025 queens-py/queens-meetings#20

Closed

7 of 8 tasks

sbrandstaeter mentioned this pull request Nov 6, 2025

Clean up _manage_paths in the driver object #238

Open

sbrandstaeter force-pushed the introduce-logger-on-dask-worker branch from 6cf1c78 to 86ac5d0 Compare November 6, 2025 15:13

leahaeusel mentioned this pull request Nov 7, 2025

Remove empty experiment dirs #242

Merged

sbrandstaeter mentioned this pull request Mar 27, 2026

Switch driver metadata files to pickle #296

Open

sbrandstaeter marked this pull request as draft April 27, 2026 12:03

bennoschoenstein self-assigned this Oct 5, 2026

bennoschoenstein force-pushed the introduce-logger-on-dask-worker branch from acc4d16 to f16f8ec Compare October 6, 2026 15:05

bennoschoenstein requested review from leahaeusel and a balanced review from Copilot October 6, 2026 15:06

Copilot AI reviewed Oct 6, 2026

bennoschoenstein requested a review from rjoussen October 6, 2026 15:16

leahaeusel requested a balanced review from Copilot October 7, 2026 14:44

Copilot started reviewing on behalf of leahaeusel October 7, 2026 14:45

View session

Copilot AI reviewed Oct 7, 2026

View reviewed changes

Comment thread src/queens/drivers/_driver.py Outdated

Comment thread src/queens/drivers/_driver.py Outdated

Comment thread src/queens/utils/logger_settings.py Outdated

leahaeusel reviewed Oct 8, 2026

View reviewed changes

Comment thread src/queens/drivers/_driver.py Outdated

Comment thread src/queens/utils/logger_settings.py Outdated

Comment thread src/queens/data_processors/_data_processor.py

Comment thread src/queens/drivers/_driver.py Outdated

sbrandstaeter marked this pull request as ready for review October 8, 2026 08:24

bennoschoenstein and others added 2 commits October 9, 2026 10:16

feat: introduce logger on workers with one log file per job

f0ca000

Co-authored-by: Sebastian Brandstaeter <sebastian.brandstaeter@unibw.de>

bennoschoenstein added 3 commits October 9, 2026 10:16

test: adapt to the new signature of NumpyFile

40388e8

bennoschoenstein force-pushed the introduce-logger-on-dask-worker branch from f16f8ec to 40388e8 Compare October 9, 2026 08:22

bennoschoenstein requested review from leahaeusel and a balanced review from Copilot October 9, 2026 08:25

Copilot AI reviewed Oct 9, 2026

leahaeusel requested a balanced review from Copilot October 9, 2026 10:41

Copilot started reviewing on behalf of leahaeusel October 9, 2026 10:42

View session

Copilot AI reviewed Oct 9, 2026

View reviewed changes

Comment thread src/queens/utils/logger_settings.py

leahaeusel previously approved these changes Oct 9, 2026

View reviewed changes

sbrandstaeter commented Oct 9, 2026

View reviewed changes

bennoschoenstein previously approved these changes Oct 12, 2026

View reviewed changes

rjoussen reviewed Oct 12, 2026

View reviewed changes

Comment thread src/queens/drivers/jobscript.py Outdated

feat: log the result of a job on level INFO

5a2ea25

bennoschoenstein dismissed stale reviews from leahaeusel and themself via 5a2ea25 October 13, 2026 07:06

leahaeusel approved these changes Oct 13, 2026

View reviewed changes

rjoussen approved these changes Oct 13, 2026

View reviewed changes

bennoschoenstein merged commit e8400f1 into queens-py:main Oct 13, 2026
18 checks passed

Conversation

sbrandstaeter commented Jun 7, 2025

Description and Context: What and Why?

Summary

Motivation

What's Proposed

Logging Setup

Notes

Related Issues and Pull Requests

Interested Parties

Uh oh!

Uh oh!

gilrrei left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

leahaeusel left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

bennoschoenstein commented Oct 6, 2026

What you get

How it works

Options (all drivers)

What changes for users

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

leahaeusel left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bennoschoenstein commented Oct 9, 2026

What you get

How it works

Option (all drivers)

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

leahaeusel left a comment

Choose a reason for hiding this comment

Uh oh!

sbrandstaeter left a comment

Choose a reason for hiding this comment

Uh oh!

bennoschoenstein commented Oct 12, 2026

Uh oh!

bennoschoenstein commented Oct 12, 2026

Uh oh!

leahaeusel commented Oct 12, 2026

Uh oh!

sbrandstaeter commented Oct 12, 2026

Uh oh!

rjoussen commented Oct 12, 2026

Description and Context:
What and Why?