feat: SLS-497 Add explicit prestart hooks to the runpod-python SDK - #570
Open
jasonwang-runpod wants to merge 4 commits into
Open
feat: SLS-497 Add explicit prestart hooks to the runpod-python SDK#570jasonwang-runpod wants to merge 4 commits into
jasonwang-runpod wants to merge 4 commits into
Conversation
A handler that dies during a model load or a CUDA fault usually explains itself on stdout/stderr, but only the exception reached the platform, so the useful part was lost. Tee both streams into a per-context ring buffer and attach the tail to the error the worker reports. Every reported field is clipped so a huge message or log cannot push the job-done body past its limit.
Startup work placed before runpod.serverless.start() ran outside the SDK's view: a model load that hung or crashed left requests sitting in IN_QUEUE until the TTL expired, with the reason buried in worker logs. Accept an optional initializer and init_timeout. The worker now runs the initializer as a fourth concurrent task, keeps taking requests, and holds handler execution until initialization finishes. If initialization fails, the worker reports the reason and its captured logs against the request it is holding through the existing job-done route, fails any request a long-poll returns afterwards, and exits so the platform respawns it under existing backoff.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
There was a problem hiding this comment.
Pull request overview
Adds supervised prestart hooks across supported Serverless worker modes, including startup-failure reporting and bounded stdout/stderr capture.
Changes:
- Adds ordered sync/async prestart hooks with optional phase timeout.
- Gates handlers during initialization and drains failed queue workers.
- Captures bounded logs for startup and handler failures.
Reviewed changes
Copilot reviewed 14 out of 14 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
runpod/serverless/__init__.py |
Exposes hooks and validates runtime modes. |
runpod/serverless/worker.py |
Installs output capture for workers. |
runpod/serverless/modules/rp_prestart.py |
Implements hook registration and execution. |
runpod/serverless/modules/rp_capture.py |
Implements bounded contextual output capture. |
runpod/serverless/modules/rp_scale.py |
Integrates queue gating, failure delivery, and shutdown. |
runpod/serverless/modules/rp_local.py |
Runs hooks before local handlers. |
runpod/serverless/modules/rp_fastapi.py |
Runs hooks through FastAPI lifespan. |
runpod/serverless/modules/rp_job.py |
Attaches captured logs to handler failures. |
docs/serverless/worker.md |
Documents prestart configuration and modes. |
tests/test_serverless/test_prestart.py |
Tests the public hook contract and mode guards. |
tests/test_serverless/test_prestart_lifecycle.py |
Tests queue lifecycle and process termination. |
tests/test_serverless/test_capture.py |
Tests capture and output bounds. |
tests/test_serverless/test_modules/test_local.py |
Tests local prestart behavior. |
tests/test_serverless/test_init.py |
Verifies the new public export. |
馃挕 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
jasonwang-runpod
marked this pull request as ready for review
August 19, 2026 22:16
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Workers often load models or start inference engines before calling
runpod.serverless.start(). The SDK cannot observe that work, so prestart failures leave requests inIN_QUEUEwhile the useful error remains only in worker logs.Solution
Add explicit prestart hooks that the SDK supervises before handler execution.
@runpod.serverless.register_prestart_hook. One optional timeout covers the complete prestart phase.prestart_failed, then the worker drains and exits.Depends on: https://github.com/runpod/ai-api/pull/1023. AI API should deploy first so it classifies
prestart_failedbefore this SDK emits it.Testing
uv run pytest --no-cov -q: 700 passed.PrestartTimeoutunder a five-second bound.Supersedes #567.