Skip to content

Fix standalone server not catching SIGINT/SIGTERM - #3799

Merged
vigoo merged 8 commits into
golemcloud:mainfrom
baltasarblanco:fix/standalone-server-sigint
Sep 30, 2026
Merged

vigoo merged 8 commits into
golemcloud:mainfrom
baltasarblanco:fix/standalone-server-sigint

Conversation

@baltasarblanco

Copy link
Copy Markdown
Contributor

Fixes #1603

Problem

golem server run never installs a signal handler — the run path just awaits the services' JoinSet until completion. Running natively that is invisible: with no handler installed, the kernel applies SIGINT's default disposition and the process dies. But when the standalone server runs as PID 1 in a container (the Docker repro in the issue), the kernel only delivers a signal to a PID namespace's init process if init has installed a handler for it — default dispositions are never applied (pid_namespaces(7)). So SIGINT (Ctrl+C, kill -INT 1) and SIGTERM (docker stop) are silently discarded, which matches every observation in the issue thread.

Fix

Wrap the join_next() loop in ServerSubcommand::Run in a tokio::select! against a shutdown signal (SIGINT + SIGTERM via tokio::signal::unix on Unix, ctrl_c() elsewhere). On signal: log, join_set.shutdown().await, and return. Aborting the tasks drops the router task, which owns the worker executor's RunDetails; its Drop cancels the graph-wide shutdown token and stops the epoch thread — the same mechanism already documented in cli/golem/src/router.rs ("Dropping its RunDetails cancels the shutdown token that drives background services").

One file changed, no new dependencies (tokio already has the signal feature enabled workspace-wide).

Verification

Reproduced the PID 1 condition without Docker, since the bug's trigger is being the init of a PID namespace: sudo unshare --fork --pid --mount-proc ./target/debug/golem server run -vv.

Before (main):

  • Normal process: kill -INT <pid> kills it instantly (kernel default action, no shutdown of any kind).
  • As PID 1: the same kill -INT is silently ignored; the server keeps running and can only be stopped with SIGKILL.

After (this branch):

  • Both cases shut down cleanly on SIGINT and SIGTERM, logging Received shutdown signal, stopping Golem server, with exit code 0.
  • The shutdown token propagates to the background services; depending on timing you can see the scheduler log Shutdown requested, stopping scheduler background loop before the process exits.

cargo build -p golem and cargo make fix pass.

Note: on unpatched versions, docker run --init works around the issue, since tini becomes PID 1 and forwards signals.

@baltasarblanco
baltasarblanco requested a review from a team August 31, 2026 16:32
@netlify

netlify Bot commented Aug 31, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for golemcloud canceled.

Name Link
🔨 Latest commit 78d4a0d
🔍 Latest deploy log https://app.netlify.com/projects/golemcloud/deploys/6abcf5908c9b480008aa9b26

@github-actions

github-actions Bot commented Aug 31, 2026 •

Copy link
Copy Markdown

✅ All contributors have signed the CLA.
Posted by the CLA Assistant Lite bot.

@github-actions

Copy link
Copy Markdown

Hi @baltasarblanco, thanks for your interest in contributing!

This project requires that pull request authors are vouched, and you are not in the list of vouched users.

This PR will be closed automatically. See https://github.com/golemcloud/golem/blob/main/CONTRIBUTING.md for more details.

@github-actions github-actions Bot closed this Aug 31, 2026
@baltasarblanco

Copy link
Copy Markdown
Contributor Author

I have read the CLA Document and I hereby sign the CLA

github-actions Bot added a commit that referenced this pull request Aug 31, 2026
@vigoo vigoo reopened this Sep 1, 2026
Comment thread cli/golem/src/command_handler.rs
@vigoo
vigoo merged commit b1bce37 into golemcloud:main Sep 30, 2026
66 checks passed
@github-actions github-actions Bot locked and limited conversation to collaborators Sep 30, 2026
@baltasarblanco
baltasarblanco deleted the fix/standalone-server-sigint branch September 30, 2026 14:55
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Golem standalone server doesn't catch SIGINT

2 participants