From 306f18b763cd2eca444f0aa49e7ebfb88542c83d Mon Sep 17 00:00:00 2001 From: lgunreddi Date: Thu, 9 Jul 2026 08:14:21 -0400 Subject: [PATCH 01/17] Create batch-jobs.mdx --- serverless/batch-jobs.mdx | 247 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 247 insertions(+) create mode 100644 serverless/batch-jobs.mdx diff --git a/serverless/batch-jobs.mdx b/serverless/batch-jobs.mdx new file mode 100644 index 000000000..e74b3b47f --- /dev/null +++ b/serverless/batch-jobs.mdx @@ -0,0 +1,247 @@ +--- +title: "Batch jobs" +description: "Submit large collections of inference requests as a single named batch, processed asynchronously within a 24-hour SLA." +--- + +Use batch jobs to run large volumes of inference requests against a serverless endpoint without waiting for each result in real time. Batch jobs run asynchronously on dedicated workers that are separate from your endpoint's standard `/run` traffic, so submitting a batch never delays your interactive requests. + +## When to use batch vs /run + +| | Batch | `/run` | +|---|---|---| +| **Use case** | Bulk, offline workloads | Interactive, real-time inference | +| **Latency** | Completed within 24h SLA | Seconds to minutes | +| **Traffic isolation** | Dedicated batch workers | Standard serverless workers | +| **Result delivery** | Poll or subscribe to notifications | Synchronous or async poll | + +Choose batch when your workload can tolerate multi-hour latency — for example, nightly dataset processing, pre-computing embeddings, or running evaluations. + +## Batch lifecycle + +A batch moves through the following states: + +``` +OPEN → FINALIZED → RUNNING → COMPLETED + → FAILED + → CANCELLED +``` + +- **OPEN** — The batch is a draft. You can add, update, or remove individual requests. Batch workers have not started any work. +- **FINALIZED** — The batch is locked. No further requests can be added or removed. The batch is now eligible for execution and will be picked up by batch workers. +- **RUNNING** — At least one request in the batch is being processed by a worker. +- **COMPLETED** — All requests have reached a terminal state (completed or failed). +- **FAILED** — The batch itself failed before or during execution (distinct from individual request failures within an otherwise-completed batch). +- **CANCELLED** — You cancelled the batch. See [Cancellation](#cancellation) for details. + +You must call `/finalize` before the batch begins processing. An OPEN batch will not be executed. + +## API walkthrough + +### 1. Create a batch + +```bash +POST /v2/{endpoint_id}/batch +Authorization: Bearer {api_key} +Content-Type: application/json +``` + +You can create an empty batch and add requests later, or include an initial list of requests in the same call. Each request in the `requests` array uses the same shape as a standard `/run` call — a JSON object with an `input` field. + +```json +{ + "name": "nightly-embeddings-2026-07-09", + "requests": [ + { "input": { "text": "The quick brown fox" } }, + { "input": { "text": "Jumped over the lazy dog" } } + ] +} +``` + +**Response:** + +```json +{ + "id": "batch_01j9abc123", + "status": "OPEN", + "name": "nightly-embeddings-2026-07-09", + "endpointId": "abc123xyz", + "itemCount": 2, + "createdAt": "2026-07-09T08:00:00Z" +} +``` + +### 2. Add more requests + +While the batch is OPEN, append additional requests: + +```bash +POST /v2/{endpoint_id}/batch/{batch_id}/requests +Authorization: Bearer {api_key} +Content-Type: application/json +``` + +```json +{ + "requests": [ + { "input": { "text": "More text to embed" } } + ] +} +``` + +You can call this endpoint multiple times to build up large batches incrementally. + +### 3. Finalize the batch + +Once you've added all requests, finalize the batch to make it eligible for execution: + +```bash +POST /v2/{endpoint_id}/batch/{batch_id}/finalize +Authorization: Bearer {api_key} +``` + +After finalization, the batch status transitions to `FINALIZED` and requests are locked. You can no longer add or remove individual requests. + +### 4. Poll batch status + +Check overall progress by fetching the batch summary: + +```bash +GET /v2/{endpoint_id}/batch/{batch_id} +Authorization: Bearer {api_key} +``` + +**Response:** + +```json +{ + "id": "batch_01j9abc123", + "status": "RUNNING", + "name": "nightly-embeddings-2026-07-09", + "itemCount": 1000, + "queuedCount": 742, + "inProgressCount": 8, + "completedCount": 244, + "failedCount": 6, + "progress": 0.25, + "createdAt": "2026-07-09T08:00:00Z", + "finalizedAt": "2026-07-09T08:01:00Z" +} +``` + +Poll this endpoint at whatever interval suits your workflow. When `status` is `COMPLETED`, `FAILED`, or `CANCELLED`, the batch has reached a terminal state. + +### 5. Retrieve results + +Fetch paginated results for all child requests in the batch: + +```bash +GET /v2/{endpoint_id}/batch/{batch_id}/requests +Authorization: Bearer {api_key} +``` + +**Response:** + +```json +{ + "requests": [ + { + "id": "req_abc001", + "status": "COMPLETED", + "output": { "embedding": [0.12, 0.34, ...] }, + "startedAt": "2026-07-09T09:15:00Z", + "completedAt": "2026-07-09T09:15:02Z" + }, + { + "id": "req_abc002", + "status": "FAILED", + "error": "Handler raised an exception: timeout exceeded", + "startedAt": "2026-07-09T09:15:01Z", + "completedAt": "2026-07-09T09:15:10Z" + } + ], + "nextCursor": "cursor_xyz" +} +``` + +The results are paginated. Pass `nextCursor` as a query parameter to retrieve the next page. + +## Full API reference + +| Method | Path | Description | +|--------|------|-------------| +| `POST` | `/v2/{endpoint_id}/batch` | Create a new batch, optionally with initial requests | +| `POST` | `/v2/{endpoint_id}/batch/{id}/requests` | Append requests to an OPEN batch | +| `POST` | `/v2/{endpoint_id}/batch/{id}/finalize` | Lock the batch and make it eligible for execution | +| `PUT` | `/v2/{endpoint_id}/batch/{id}` | Update batch attributes (e.g. display name) | +| `DELETE` | `/v2/{endpoint_id}/batch/{id}/requests/{requestId}` | Remove a single request from an OPEN batch | +| `GET` | `/v2/{endpoint_id}/batch` | List all batches for an endpoint, newest first | +| `GET` | `/v2/{endpoint_id}/batch/{id}` | Batch summary with counts and progress | +| `POST` | `/v2/{endpoint_id}/batch/{id}/cancel` | Cancel a batch | +| `GET` | `/v2/{endpoint_id}/batch/{id}/requests` | Paginated child request list | + +For full request and response schemas, see the [API reference](/api-reference/endpoint/batch). + +## Monitoring batches in the console + +Open your endpoint in the Runpod console and select the **Batch** tab to see all batches. Each row shows the batch name, status, and progress counts. + +Click a batch to open the detail view, which shows: + +- Top-level status and progress +- Per-request rows with status, timestamps, and error messages for failed requests +- Links to the full request detail view for each child request + +The child request list is sorted by failures first, then in-progress, then queued, then completed. + +## Notifications + +When a batch reaches a terminal state (COMPLETED, FAILED, or CANCELLED), Runpod sends: + +- **Console Inbox notification** — includes batch ID, endpoint name, terminal status, and item counts (completed / failed / total) +- **Webhook event** — if your account has a webhook subscription configured for batch events + +Notifications are sent once per terminal state transition and are not fired for intermediate progress. + +## Cancellation + +To cancel a batch: + +```bash +POST /v2/{endpoint_id}/batch/{batch_id}/cancel +Authorization: Bearer {api_key} +``` + +Cancellation behavior: + +- **Queued requests** are cancelled immediately and are not billed. +- **In-progress requests** are allowed to finish and are billed normally. + +The batch status transitions to `CANCELLED` once all in-progress work has drained. + +## Limits + +| Limit | Value | +|-------|-------| +| Queued items per endpoint | 50,000 | +| Open (draft) batches per user | 100 | + +If you need to exceed these limits, [contact support](https://www.runpod.io/contact). + +## Billing + +Batch jobs are billed at the same rate as standard serverless requests on your endpoint. There is no batch discount at launch. Billing is based on the compute time used by each child request, regardless of whether the batch was later cancelled (in-progress requests that completed before cancellation are billed normally). + +## Error handling + +**Individual request failures** — A failed child request does not fail the entire batch. The batch continues processing remaining requests and reaches COMPLETED status. Inspect failed requests via the console or the `GET .../requests` endpoint; each failed request includes an error message from the handler. + +**Batch-level failure** — If the batch itself fails (status `FAILED`), it indicates a systemic problem rather than individual handler errors. Contact support if you see this state and cannot explain it from request-level errors. + +**Redis durability** — Batch jobs use the same Redis-backed queue as standard serverless requests. In the event of a Redis failure, queued batch requests may be lost. This is an MVP limitation that applies equally to `/run` traffic. + +## Known limitations + +- Batch jobs inherit the GPU type configured on your endpoint. You cannot specify a different GPU per batch or per request. +- There is no per-request scheduling or ordering. Requests within a batch are processed in an unspecified order. +- Cost estimation before finalization is not available at launch. +- Batch workers are scheduled based on global queue urgency and off-peak capacity. Start time within the 24h SLA is not guaranteed. From 913f824d645fdb814ef8ac43275bf73ae2cb046c Mon Sep 17 00:00:00 2001 From: lgunreddi Date: Thu, 9 Jul 2026 08:17:22 -0400 Subject: [PATCH 02/17] Update docs.json --- docs.json | 1 + 1 file changed, 1 insertion(+) diff --git a/docs.json b/docs.json index 474ced9ad..f5cbf4ecc 100644 --- a/docs.json +++ b/docs.json @@ -143,6 +143,7 @@ { "group": "Advanced workflows", "pages": [ + "serverless/advanced-workflows/batch-jobs" ] }, { From d4f14b84776bda0c87e8e7b9fd986e5e81ac73bc Mon Sep 17 00:00:00 2001 From: lgunreddi Date: Thu, 9 Jul 2026 08:17:47 -0400 Subject: [PATCH 03/17] Rename serverless/batch-jobs.mdx to serverless/advanced-workflows/batch-jobs.mdx --- serverless/{ => advanced-workflows}/batch-jobs.mdx | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename serverless/{ => advanced-workflows}/batch-jobs.mdx (100%) diff --git a/serverless/batch-jobs.mdx b/serverless/advanced-workflows/batch-jobs.mdx similarity index 100% rename from serverless/batch-jobs.mdx rename to serverless/advanced-workflows/batch-jobs.mdx From bb37a2010d15b57e5a35c1bfcf78d496056b5880 Mon Sep 17 00:00:00 2001 From: lgunreddi Date: Wed, 15 Jul 2026 12:07:00 -0400 Subject: [PATCH 04/17] Update batch-jobs.mdx --- serverless/advanced-workflows/batch-jobs.mdx | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/serverless/advanced-workflows/batch-jobs.mdx b/serverless/advanced-workflows/batch-jobs.mdx index e74b3b47f..10d062067 100644 --- a/serverless/advanced-workflows/batch-jobs.mdx +++ b/serverless/advanced-workflows/batch-jobs.mdx @@ -83,12 +83,14 @@ Content-Type: application/json ```json { "requests": [ - { "input": { "text": "More text to embed" } } + { "input": { "text": "More text to embed" } }, + { "input": { "text": "Another piece of text" } }, + { "input": { "text": "And another one" } } ] } ``` - -You can call this endpoint multiple times to build up large batches incrementally. +Each call accepts up to 100 requests in a single array. You can call this +endpoint multiple times to build up large batches incrementally. ### 3. Finalize the batch @@ -229,7 +231,7 @@ If you need to exceed these limits, [contact support](https://www.runpod.io/cont ## Billing -Batch jobs are billed at the same rate as standard serverless requests on your endpoint. There is no batch discount at launch. Billing is based on the compute time used by each child request, regardless of whether the batch was later cancelled (in-progress requests that completed before cancellation are billed normally). +Batch jobs are billed at the same rate as standard serverless requests on your endpoint. For enterprise customers, flex worker discounts apply to batch jobs. Billing is based on the compute time used by each child request, regardless of whether the batch was later cancelled (in-progress requests that completed before cancellation are billed normally). ## Error handling From 0406cbf2033429cb2e3177c4a1aa1890b587410d Mon Sep 17 00:00:00 2001 From: lgunreddi Date: Wed, 15 Jul 2026 12:19:18 -0400 Subject: [PATCH 05/17] Update docs.json --- docs.json | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs.json b/docs.json index 880926327..bdbd824ec 100644 --- a/docs.json +++ b/docs.json @@ -151,7 +151,6 @@ "pages": [ "serverless/advanced-workflows/batch-jobs" ] - "pages": [] }, { "group": "Load balancing", @@ -1305,4 +1304,4 @@ "destination": "/serverless/endpoints/send-requests" } ] -} \ No newline at end of file +} From e7d975ad6a71dd5eaf049543140ebbc2368bebff Mon Sep 17 00:00:00 2001 From: "promptless[bot]" Date: Wed, 19 Aug 2026 14:15:43 +0000 Subject: [PATCH 06/17] Apply PR #699 review feedback to batch-jobs.mdx - Rename batch state OPEN -> DRAFT throughout (diagram, lifecycle, API table) - Fix create request to a bare array of job inputs; response is {id, status: DRAFT} - Fix results pagination to offset/limit/total/hasMore (was nextCursor) - Update limits table (10 active batches/endpoint, 5,000 requests/batch, 50,000 queued/endpoint) + enterprise note - Replace per-call request count with 10 MiB body-size limit --- serverless/advanced-workflows/batch-jobs.mdx | 54 +++++++++----------- 1 file changed, 25 insertions(+), 29 deletions(-) diff --git a/serverless/advanced-workflows/batch-jobs.mdx b/serverless/advanced-workflows/batch-jobs.mdx index 10d062067..eabe2391b 100644 --- a/serverless/advanced-workflows/batch-jobs.mdx +++ b/serverless/advanced-workflows/batch-jobs.mdx @@ -21,19 +21,19 @@ Choose batch when your workload can tolerate multi-hour latency — for example, A batch moves through the following states: ``` -OPEN → FINALIZED → RUNNING → COMPLETED - → FAILED - → CANCELLED +DRAFT → FINALIZED → RUNNING → COMPLETED + → FAILED + → CANCELLED ``` -- **OPEN** — The batch is a draft. You can add, update, or remove individual requests. Batch workers have not started any work. +- **DRAFT** — The batch is a draft. You can add, update, or remove individual requests. Batch workers have not started any work. - **FINALIZED** — The batch is locked. No further requests can be added or removed. The batch is now eligible for execution and will be picked up by batch workers. - **RUNNING** — At least one request in the batch is being processed by a worker. - **COMPLETED** — All requests have reached a terminal state (completed or failed). - **FAILED** — The batch itself failed before or during execution (distinct from individual request failures within an otherwise-completed batch). - **CANCELLED** — You cancelled the batch. See [Cancellation](#cancellation) for details. -You must call `/finalize` before the batch begins processing. An OPEN batch will not be executed. +You must call `/finalize` before the batch begins processing. A DRAFT batch will not be executed. ## API walkthrough @@ -45,16 +45,13 @@ Authorization: Bearer {api_key} Content-Type: application/json ``` -You can create an empty batch and add requests later, or include an initial list of requests in the same call. Each request in the `requests` array uses the same shape as a standard `/run` call — a JSON object with an `input` field. +The request body is a top-level JSON array. Send an empty array `[]` to create a batch and add requests later, or send a populated array to include an initial list of requests. Each element uses the same shape as a standard `/run` call — a JSON object with an `input` field. ```json -{ - "name": "nightly-embeddings-2026-07-09", - "requests": [ - { "input": { "text": "The quick brown fox" } }, - { "input": { "text": "Jumped over the lazy dog" } } - ] -} +[ + { "input": { "text": "The quick brown fox" } }, + { "input": { "text": "Jumped over the lazy dog" } } +] ``` **Response:** @@ -62,17 +59,13 @@ You can create an empty batch and add requests later, or include an initial list ```json { "id": "batch_01j9abc123", - "status": "OPEN", - "name": "nightly-embeddings-2026-07-09", - "endpointId": "abc123xyz", - "itemCount": 2, - "createdAt": "2026-07-09T08:00:00Z" + "status": "DRAFT" } ``` ### 2. Add more requests -While the batch is OPEN, append additional requests: +While the batch is DRAFT, append additional requests: ```bash POST /v2/{endpoint_id}/batch/{batch_id}/requests @@ -89,8 +82,8 @@ Content-Type: application/json ] } ``` -Each call accepts up to 100 requests in a single array. You can call this -endpoint multiple times to build up large batches incrementally. + +Request body size is limited to 10 MiB per call. You can call this endpoint multiple times to build up large batches incrementally. ### 3. Finalize the batch @@ -118,7 +111,6 @@ Authorization: Bearer {api_key} { "id": "batch_01j9abc123", "status": "RUNNING", - "name": "nightly-embeddings-2026-07-09", "itemCount": 1000, "queuedCount": 742, "inProgressCount": 8, @@ -161,21 +153,24 @@ Authorization: Bearer {api_key} "completedAt": "2026-07-09T09:15:10Z" } ], - "nextCursor": "cursor_xyz" + "total": 1000, + "offset": 0, + "limit": 50, + "hasMore": true } ``` -The results are paginated. Pass `nextCursor` as a query parameter to retrieve the next page. +The results are paginated. Pass the `offset` and `limit` query parameters to page through results. The `hasMore` field indicates whether more pages remain. ## Full API reference | Method | Path | Description | |--------|------|-------------| | `POST` | `/v2/{endpoint_id}/batch` | Create a new batch, optionally with initial requests | -| `POST` | `/v2/{endpoint_id}/batch/{id}/requests` | Append requests to an OPEN batch | +| `POST` | `/v2/{endpoint_id}/batch/{id}/requests` | Append requests to a DRAFT batch | | `POST` | `/v2/{endpoint_id}/batch/{id}/finalize` | Lock the batch and make it eligible for execution | | `PUT` | `/v2/{endpoint_id}/batch/{id}` | Update batch attributes (e.g. display name) | -| `DELETE` | `/v2/{endpoint_id}/batch/{id}/requests/{requestId}` | Remove a single request from an OPEN batch | +| `DELETE` | `/v2/{endpoint_id}/batch/{id}/requests/{requestId}` | Remove a single request from a DRAFT batch | | `GET` | `/v2/{endpoint_id}/batch` | List all batches for an endpoint, newest first | | `GET` | `/v2/{endpoint_id}/batch/{id}` | Batch summary with counts and progress | | `POST` | `/v2/{endpoint_id}/batch/{id}/cancel` | Cancel a batch | @@ -224,10 +219,11 @@ The batch status transitions to `CANCELLED` once all in-progress work has draine | Limit | Value | |-------|-------| -| Queued items per endpoint | 50,000 | -| Open (draft) batches per user | 100 | +| Active batches per endpoint | 10 | +| Requests per batch | 5,000 | +| Queued requests per endpoint | 50,000 | -If you need to exceed these limits, [contact support](https://www.runpod.io/contact). +Limits are configurable for enterprise accounts. Contact sales for custom limits. ## Billing From 6aba18343d5f97de78968f0b00cdc6d24d077664 Mon Sep 17 00:00:00 2001 From: lgunreddi Date: Wed, 19 Aug 2026 10:19:03 -0400 Subject: [PATCH 07/17] Update docs.json --- docs.json | 1 - 1 file changed, 1 deletion(-) diff --git a/docs.json b/docs.json index d14d6988e..34e744a36 100644 --- a/docs.json +++ b/docs.json @@ -1908,7 +1908,6 @@ "source": "/pods/clusters/overview", "destination": "/instant-clusters" } - ] ], "description": "Developer documentation for building, deploying, and scaling AI applications on Runpod.", "metadata": { From 8723234856a3cd344dd69498c113d6474a1f39db Mon Sep 17 00:00:00 2001 From: lgunreddi Date: Wed, 19 Aug 2026 11:51:33 -0400 Subject: [PATCH 08/17] Update batch-jobs.mdx --- serverless/advanced-workflows/batch-jobs.mdx | 1 + 1 file changed, 1 insertion(+) diff --git a/serverless/advanced-workflows/batch-jobs.mdx b/serverless/advanced-workflows/batch-jobs.mdx index eabe2391b..c690eed47 100644 --- a/serverless/advanced-workflows/batch-jobs.mdx +++ b/serverless/advanced-workflows/batch-jobs.mdx @@ -1,6 +1,7 @@ --- title: "Batch jobs" description: "Submit large collections of inference requests as a single named batch, processed asynchronously within a 24-hour SLA." +tag: BETA --- Use batch jobs to run large volumes of inference requests against a serverless endpoint without waiting for each result in real time. Batch jobs run asynchronously on dedicated workers that are separate from your endpoint's standard `/run` traffic, so submitting a batch never delays your interactive requests. From e110955073fd04f5246cab9ca3c31eb15ae402f8 Mon Sep 17 00:00:00 2001 From: lgunreddi Date: Wed, 19 Aug 2026 12:00:36 -0400 Subject: [PATCH 09/17] Update release-notes.mdx --- release-notes.mdx | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/release-notes.mdx b/release-notes.mdx index 568b7ad70..f23a07217 100644 --- a/release-notes.mdx +++ b/release-notes.mdx @@ -9,6 +9,10 @@ rss: true +**August 19, 2026** + +

New Release [Batch Jobs (Beta)](/serverless/advanced-workflows/batch-jobs)

Submit large sets of inference requests to a Serverless endpoint as a single managed unit. Create a batch, finalize it to start processing, and poll for status and progress using per-request counts. See [Batch Jobs](/serverless/advanced-workflows/batch-jobs) to get started. + **August 18, 2026**

New Release [REST API v2](/api-reference-v2/overview)

REST API v2 is now generally available. v2 moves to a new base URL (`https://api.runpod.io/v2`), reorganizes resource paths, standardizes request and response shapes, and adds new capabilities including catalog endpoints, pod log streaming, and Serverless observability. See the [migration guide](/api-reference-v2/migrate-from-v1) to move your existing integrations. From f004a64ca671916c88157874afbf33ff6f6e612d Mon Sep 17 00:00:00 2001 From: lgunreddi Date: Wed, 19 Aug 2026 12:07:14 -0400 Subject: [PATCH 10/17] Update batch-jobs.mdx --- serverless/advanced-workflows/batch-jobs.mdx | 2 ++ 1 file changed, 2 insertions(+) diff --git a/serverless/advanced-workflows/batch-jobs.mdx b/serverless/advanced-workflows/batch-jobs.mdx index c690eed47..3c5156595 100644 --- a/serverless/advanced-workflows/batch-jobs.mdx +++ b/serverless/advanced-workflows/batch-jobs.mdx @@ -6,6 +6,8 @@ tag: BETA Use batch jobs to run large volumes of inference requests against a serverless endpoint without waiting for each result in real time. Batch jobs run asynchronously on dedicated workers that are separate from your endpoint's standard `/run` traffic, so submitting a batch never delays your interactive requests. +Batch Jobs is currently in beta. Join our [Discord](https://discord.gg/runpod) to provide feedback and get support. + ## When to use batch vs /run | | Batch | `/run` | From de47cba93952e7fe842285360f61a3276c1ada17 Mon Sep 17 00:00:00 2001 From: lgunreddi Date: Wed, 19 Aug 2026 12:11:06 -0400 Subject: [PATCH 11/17] Update release-notes.mdx --- release-notes.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/release-notes.mdx b/release-notes.mdx index f23a07217..b4b096151 100644 --- a/release-notes.mdx +++ b/release-notes.mdx @@ -11,7 +11,7 @@ rss: true **August 19, 2026** -

New Release [Batch Jobs (Beta)](/serverless/advanced-workflows/batch-jobs)

Submit large sets of inference requests to a Serverless endpoint as a single managed unit. Create a batch, finalize it to start processing, and poll for status and progress using per-request counts. See [Batch Jobs](/serverless/advanced-workflows/batch-jobs) to get started. +

New Release [Batch Jobs (BETA)](/serverless/advanced-workflows/batch-jobs)

Submit large sets of inference requests to a Serverless endpoint as a single managed unit. Create a batch, finalize it to start processing, and poll for status and progress using per-request counts. See [Batch Jobs](/serverless/advanced-workflows/batch-jobs) to get started. **August 18, 2026** From c745be620b230683960ab796efd2b5916f0660c2 Mon Sep 17 00:00:00 2001 From: "promptless[bot]" Date: Wed, 19 Aug 2026 16:11:54 +0000 Subject: [PATCH 12/17] Apply PR #699 review feedback: correct batch states and poll response shape Batch states are DRAFT, FINALIZED, FAILED, CANCELLED (not RUNNING/COMPLETED); an actively-processing batch stays FINALIZED and progress is inferred from the requestTotal/requestInProgress/requestCompleted/requestFailed counts. Reshapes the poll response example and updates polling, notifications, and error-handling prose accordingly. --- serverless/advanced-workflows/batch-jobs.mdx | 35 +++++++++----------- 1 file changed, 15 insertions(+), 20 deletions(-) diff --git a/serverless/advanced-workflows/batch-jobs.mdx b/serverless/advanced-workflows/batch-jobs.mdx index 3c5156595..ad99d13a1 100644 --- a/serverless/advanced-workflows/batch-jobs.mdx +++ b/serverless/advanced-workflows/batch-jobs.mdx @@ -24,16 +24,13 @@ Choose batch when your workload can tolerate multi-hour latency — for example, A batch moves through the following states: ``` -DRAFT → FINALIZED → RUNNING → COMPLETED - → FAILED - → CANCELLED +DRAFT → FINALIZED → FAILED + → CANCELLED ``` - **DRAFT** — The batch is a draft. You can add, update, or remove individual requests. Batch workers have not started any work. -- **FINALIZED** — The batch is locked. No further requests can be added or removed. The batch is now eligible for execution and will be picked up by batch workers. -- **RUNNING** — At least one request in the batch is being processed by a worker. -- **COMPLETED** — All requests have reached a terminal state (completed or failed). -- **FAILED** — The batch itself failed before or during execution (distinct from individual request failures within an otherwise-completed batch). +- **FINALIZED** — The batch is locked; no further requests can be added or removed. Batch workers process the requests while the batch stays in this state, and there is no separate `RUNNING` or `COMPLETED` batch status. Track progress through the `requestTotal`, `requestInProgress`, `requestCompleted`, and `requestFailed` counts — all requests have finished when `requestCompleted + requestFailed` equals `requestTotal`. +- **FAILED** — The batch itself failed before or during execution (distinct from individual request failures in a batch whose other requests finished successfully). - **CANCELLED** — You cancelled the batch. See [Cancellation](#cancellation) for details. You must call `/finalize` before the batch begins processing. A DRAFT batch will not be executed. @@ -113,19 +110,17 @@ Authorization: Bearer {api_key} ```json { "id": "batch_01j9abc123", - "status": "RUNNING", - "itemCount": 1000, - "queuedCount": 742, - "inProgressCount": 8, - "completedCount": 244, - "failedCount": 6, - "progress": 0.25, - "createdAt": "2026-07-09T08:00:00Z", - "finalizedAt": "2026-07-09T08:01:00Z" + "endpointId": "abc123xyz", + "status": "FINALIZED", + "requestTotal": 1000, + "requestInProgress": 8, + "requestCompleted": 244, + "requestFailed": 6, + "createdAt": 1783584000000 } ``` -Poll this endpoint at whatever interval suits your workflow. When `status` is `COMPLETED`, `FAILED`, or `CANCELLED`, the batch has reached a terminal state. +Poll this endpoint at whatever interval suits your workflow. A batch that is still processing reports `status: FINALIZED`; there is no `RUNNING` or `COMPLETED` status. All requests have finished when `requestCompleted + requestFailed` equals `requestTotal`. The batch reaches a terminal state only when `status` is `FAILED` or `CANCELLED`. The `createdAt` field is a Unix epoch timestamp in milliseconds. ### 5. Retrieve results @@ -175,7 +170,7 @@ The results are paginated. Pass the `offset` and `limit` query parameters to pag | `PUT` | `/v2/{endpoint_id}/batch/{id}` | Update batch attributes (e.g. display name) | | `DELETE` | `/v2/{endpoint_id}/batch/{id}/requests/{requestId}` | Remove a single request from a DRAFT batch | | `GET` | `/v2/{endpoint_id}/batch` | List all batches for an endpoint, newest first | -| `GET` | `/v2/{endpoint_id}/batch/{id}` | Batch summary with counts and progress | +| `GET` | `/v2/{endpoint_id}/batch/{id}` | Batch summary with request counts | | `POST` | `/v2/{endpoint_id}/batch/{id}/cancel` | Cancel a batch | | `GET` | `/v2/{endpoint_id}/batch/{id}/requests` | Paginated child request list | @@ -195,7 +190,7 @@ The child request list is sorted by failures first, then in-progress, then queue ## Notifications -When a batch reaches a terminal state (COMPLETED, FAILED, or CANCELLED), Runpod sends: +When a batch reaches a terminal state (`FAILED` or `CANCELLED`), Runpod sends: - **Console Inbox notification** — includes batch ID, endpoint name, terminal status, and item counts (completed / failed / total) - **Webhook event** — if your account has a webhook subscription configured for batch events @@ -234,7 +229,7 @@ Batch jobs are billed at the same rate as standard serverless requests on your e ## Error handling -**Individual request failures** — A failed child request does not fail the entire batch. The batch continues processing remaining requests and reaches COMPLETED status. Inspect failed requests via the console or the `GET .../requests` endpoint; each failed request includes an error message from the handler. +**Individual request failures** — A failed child request does not fail the entire batch. The batch stays `FINALIZED` and continues processing the remaining requests; overall completion is inferred from the request counts (all requests are done when `requestCompleted + requestFailed` equals `requestTotal`). Inspect failed requests via the console or the `GET .../requests` endpoint; each failed request includes an error message from the handler. **Batch-level failure** — If the batch itself fails (status `FAILED`), it indicates a systemic problem rather than individual handler errors. Contact support if you see this state and cannot explain it from request-level errors. From f48c5950d7d65f403fd29656904383c479d01313 Mon Sep 17 00:00:00 2001 From: "promptless[bot]" Date: Wed, 19 Aug 2026 23:46:40 +0000 Subject: [PATCH 13/17] Remove 24-hour SLA references from batch jobs docs --- serverless/advanced-workflows/batch-jobs.mdx | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/serverless/advanced-workflows/batch-jobs.mdx b/serverless/advanced-workflows/batch-jobs.mdx index ad99d13a1..917b81e62 100644 --- a/serverless/advanced-workflows/batch-jobs.mdx +++ b/serverless/advanced-workflows/batch-jobs.mdx @@ -1,6 +1,6 @@ --- title: "Batch jobs" -description: "Submit large collections of inference requests as a single named batch, processed asynchronously within a 24-hour SLA." +description: "Submit large collections of inference requests as a single named batch, processed asynchronously." tag: BETA --- @@ -13,7 +13,7 @@ Use batch jobs to run large volumes of inference requests against a serverless e | | Batch | `/run` | |---|---|---| | **Use case** | Bulk, offline workloads | Interactive, real-time inference | -| **Latency** | Completed within 24h SLA | Seconds to minutes | +| **Latency** | Multi-hour | Seconds to minutes | | **Traffic isolation** | Dedicated batch workers | Standard serverless workers | | **Result delivery** | Poll or subscribe to notifications | Synchronous or async poll | @@ -240,4 +240,4 @@ Batch jobs are billed at the same rate as standard serverless requests on your e - Batch jobs inherit the GPU type configured on your endpoint. You cannot specify a different GPU per batch or per request. - There is no per-request scheduling or ordering. Requests within a batch are processed in an unspecified order. - Cost estimation before finalization is not available at launch. -- Batch workers are scheduled based on global queue urgency and off-peak capacity. Start time within the 24h SLA is not guaranteed. +- Batch workers are scheduled based on global queue urgency and off-peak capacity. Start time is not guaranteed. From 8722e77f0d1d497189bc0943efa86f2297d9f9f5 Mon Sep 17 00:00:00 2001 From: "promptless[bot]" Date: Thu, 20 Aug 2026 14:34:36 +0000 Subject: [PATCH 14/17] Restore 24-hour SLA references in batch jobs docs Reverts commit f48c5950 at PR author @lavanya-gunreddi's request (PR #699 comment 5357223965). Restores the 24-hour SLA wording in the frontmatter description, the batch-vs-/run comparison table's Latency cell, and the known-limitations scheduling bullet. --- serverless/advanced-workflows/batch-jobs.mdx | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/serverless/advanced-workflows/batch-jobs.mdx b/serverless/advanced-workflows/batch-jobs.mdx index 917b81e62..ad99d13a1 100644 --- a/serverless/advanced-workflows/batch-jobs.mdx +++ b/serverless/advanced-workflows/batch-jobs.mdx @@ -1,6 +1,6 @@ --- title: "Batch jobs" -description: "Submit large collections of inference requests as a single named batch, processed asynchronously." +description: "Submit large collections of inference requests as a single named batch, processed asynchronously within a 24-hour SLA." tag: BETA --- @@ -13,7 +13,7 @@ Use batch jobs to run large volumes of inference requests against a serverless e | | Batch | `/run` | |---|---|---| | **Use case** | Bulk, offline workloads | Interactive, real-time inference | -| **Latency** | Multi-hour | Seconds to minutes | +| **Latency** | Completed within 24h SLA | Seconds to minutes | | **Traffic isolation** | Dedicated batch workers | Standard serverless workers | | **Result delivery** | Poll or subscribe to notifications | Synchronous or async poll | @@ -240,4 +240,4 @@ Batch jobs are billed at the same rate as standard serverless requests on your e - Batch jobs inherit the GPU type configured on your endpoint. You cannot specify a different GPU per batch or per request. - There is no per-request scheduling or ordering. Requests within a batch are processed in an unspecified order. - Cost estimation before finalization is not available at launch. -- Batch workers are scheduled based on global queue urgency and off-peak capacity. Start time is not guaranteed. +- Batch workers are scheduled based on global queue urgency and off-peak capacity. Start time within the 24h SLA is not guaranteed. From 92aec72d8b406a55b0e638c650c89e133952303c Mon Sep 17 00:00:00 2001 From: lgunreddi Date: Thu, 20 Aug 2026 11:37:48 -0400 Subject: [PATCH 15/17] Update batch-jobs.mdx --- serverless/advanced-workflows/batch-jobs.mdx | 1 + 1 file changed, 1 insertion(+) diff --git a/serverless/advanced-workflows/batch-jobs.mdx b/serverless/advanced-workflows/batch-jobs.mdx index ad99d13a1..34ecf4806 100644 --- a/serverless/advanced-workflows/batch-jobs.mdx +++ b/serverless/advanced-workflows/batch-jobs.mdx @@ -220,6 +220,7 @@ The batch status transitions to `CANCELLED` once all in-progress work has draine | Active batches per endpoint | 10 | | Requests per batch | 5,000 | | Queued requests per endpoint | 50,000 | +| Daily jobs | 1,000,000 max | Limits are configurable for enterprise accounts. Contact sales for custom limits. From 873d70c410e5a87931c7e7adc5f4104a545c047f Mon Sep 17 00:00:00 2001 From: lgunreddi Date: Thu, 20 Aug 2026 11:55:21 -0400 Subject: [PATCH 16/17] Update batch-jobs.mdx --- serverless/advanced-workflows/batch-jobs.mdx | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/serverless/advanced-workflows/batch-jobs.mdx b/serverless/advanced-workflows/batch-jobs.mdx index 34ecf4806..26317f160 100644 --- a/serverless/advanced-workflows/batch-jobs.mdx +++ b/serverless/advanced-workflows/batch-jobs.mdx @@ -220,9 +220,8 @@ The batch status transitions to `CANCELLED` once all in-progress work has draine | Active batches per endpoint | 10 | | Requests per batch | 5,000 | | Queued requests per endpoint | 50,000 | -| Daily jobs | 1,000,000 max | -Limits are configurable for enterprise accounts. Contact sales for custom limits. + The maximum queued requests per endpoint can go up to 1,000,000 daily jobs. Limits are configurable for enterprise accounts. Contact sales for custom limits. ## Billing From a345ee4dd78d42b7e9fd95bd0981a8e67e1bb8c4 Mon Sep 17 00:00:00 2001 From: "promptless[bot]" Date: Thu, 20 Aug 2026 15:55:49 +0000 Subject: [PATCH 17/17] Remove 24-hour SLA references from batch jobs docs --- serverless/advanced-workflows/batch-jobs.mdx | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/serverless/advanced-workflows/batch-jobs.mdx b/serverless/advanced-workflows/batch-jobs.mdx index 26317f160..f13354c85 100644 --- a/serverless/advanced-workflows/batch-jobs.mdx +++ b/serverless/advanced-workflows/batch-jobs.mdx @@ -1,6 +1,6 @@ --- title: "Batch jobs" -description: "Submit large collections of inference requests as a single named batch, processed asynchronously within a 24-hour SLA." +description: "Submit large collections of inference requests as a single named batch, processed asynchronously." tag: BETA --- @@ -13,7 +13,7 @@ Use batch jobs to run large volumes of inference requests against a serverless e | | Batch | `/run` | |---|---|---| | **Use case** | Bulk, offline workloads | Interactive, real-time inference | -| **Latency** | Completed within 24h SLA | Seconds to minutes | +| **Latency** | Multi-hour | Seconds to minutes | | **Traffic isolation** | Dedicated batch workers | Standard serverless workers | | **Result delivery** | Poll or subscribe to notifications | Synchronous or async poll | @@ -240,4 +240,4 @@ Batch jobs are billed at the same rate as standard serverless requests on your e - Batch jobs inherit the GPU type configured on your endpoint. You cannot specify a different GPU per batch or per request. - There is no per-request scheduling or ordering. Requests within a batch are processed in an unspecified order. - Cost estimation before finalization is not available at launch. -- Batch workers are scheduled based on global queue urgency and off-peak capacity. Start time within the 24h SLA is not guaranteed. +- Runpod schedules batch workers based on global queue urgency and off-peak capacity, so start times aren't guaranteed.