Share caption policy and retries across native vLLM and API workers

README.md

@@ -178,6 +178,13 @@ prompts, sampling, and output fields. Existing `vllm:` configurations are also
 accepted by API workers for backward compatibility; endpoint-local `model`
 values take precedence over the broadcast model name.
 
+Direct vLLM and API workers share JSON validation, output transformations,
+refusal detection, and semantic retries. Configure those under
+`inference.output_processing`; use `response_format` and `retry_response_format`
+for constrained decoding on either backend. See the
+[structured JSON recipe](recipes/STRUCTURED_JSON.md) for examples and native
+vLLM tuning details.
+
 ---
 
 ## dataset formats

examples/worker.openai-compatible.yaml

@@ -4,6 +4,16 @@ worker:
   token: "replace-with-captionflow-worker-token"
   name: "my-api-pool"
 
+  # Caption policy normally belongs in orchestrator.inference.output_processing
+  # and works identically with direct vLLM. Optional local overrides:
+  # output_processing:
+  #   refusal_markers: []
+  #   validate_json_output: true
+  #   repair_invalid_json_escapes: true
+  #   canonicalize_json_output: true
+  #   normalize_yxyx_bboxes: true
+  #   deduplicate_json_elements: true
+
   # Provider API keys stay in this process. They are never sent to CaptionFlow.
   openai_compatible:
     # Keep enough local items ready to fill all discovered request slots.
@@ -13,24 +23,9 @@ worker:
     image_format: "jpeg"
     image_quality: 90
 
-    # Optional local override for refusal detection. An empty list disables
-    # text-based refusal detection. This takes precedence over the shared
-    # orchestrator inference.refusal_markers setting.
-    # refusal_markers:
-    #   - "i cannot describe"
-    #   - "unable to provide a caption"
-
-    # Structured-output validation is optional. When enabled, invalid JSON is
-    # treated like an empty/refused result and can use inference.retry_prompt.
-    # The remaining transforms require validate_json_output: true.
-    # validate_json_output: true
-    # repair_invalid_json_escapes: true
-    # canonicalize_json_output: true
-    # normalize_yxyx_bboxes: true
-    # deduplicate_json_elements: true
-
-    # Provider-specific fields applied only to fallback requests. This is
-    # useful when the retry needs a stricter or smaller response schema.
+    # Provider-specific fields applied only to fallback requests. Prefer shared
+    # inference.retry_response_format for constraints that also apply to direct
+    # vLLM workers.
     # retry_extra_body:
     #   response_format:
     #     type: "json_object"

recipes/STRUCTURED_JSON.md

@@ -1,104 +1,140 @@
 # Structured JSON captions
 
-CaptionFlow's OpenAI-compatible worker can request provider-constrained JSON,
-reject malformed responses, and retry failed items with a smaller schema or
-different sampling settings. This works for Ideogram-style captions and other
-JSON caption formats without coupling CaptionFlow to one schema.
+Direct vLLM and OpenAI-compatible workers use the same caption pipeline:
+prompt formatting, output validation and transformations, semantic retries,
+and stage success/failure accounting. Choose either backend without losing
+JSON captioning or recovery behavior.
 
-## Configure the worker
+## Shared inference configuration
 
-Provider-specific request fields remain local to the worker. Put the primary
-structured-output constraint in an endpoint's `extra_body` and enable local
-validation on the worker:
-
-```yaml
-worker:
-  server: "ws://orchestrator.example:8765"
-  token: "replace-with-captionflow-worker-token"
-
-  openai_compatible:
-    validate_json_output: true
-    canonicalize_json_output: true
-
-    # Optional repairs for common model failures. Bounding boxes are expected
-    # in [ymin, xmin, ymax, xmax] order.
-    repair_invalid_json_escapes: true
-    normalize_yxyx_bboxes: true
-    deduplicate_json_elements: true
-
-    endpoints:
-      - name: "local-vllm"
-        base_url: "http://inference.example:8000/v1"
-        api_key_env: "VLLM_API_KEY"
-        model: "vision-model"
-        initial_concurrency: 16
-        max_concurrency: 16
-        extra_body:
-          response_format:
-            type: "json_schema"
-            json_schema:
-              name: "caption"
-              strict: true
-              schema:
-                type: "object"
-                additionalProperties: false
-                required: ["description"]
-                properties:
-                  description: {type: "string"}
-```
-
-`validate_json_output` checks strict JSON syntax, including rejecting
-non-standard `NaN` and `Infinity` values. It does not itself implement JSON
-Schema validation; use the endpoint's `response_format` when the provider
-supports constrained decoding. The optional transforms require
-`validate_json_output: true`.
-
-`repair_invalid_json_escapes` only escapes backslashes that cannot begin a
-valid JSON escape. `normalize_yxyx_bboxes` corrects inverted coordinate pairs;
-it does not infer boxes, clamp their range, or assess localization quality.
-`deduplicate_json_elements` removes exact duplicate objects from arrays named
-`elements`.
-
-## Configure recovery
-
-The orchestrator owns the shared prompt and standard sampling settings. A
-response that is empty, refused, rejected by a provider safety filter, or
-invalid JSON can receive one semantic retry:
+Put caption policy and decoding constraints in the orchestrator's `inference`
+configuration (the legacy `vllm` spelling also works):
 
 ```yaml
 orchestrator:
   inference:
+    model: "vision-model"
     inference_prompts:
       - >-
-        Describe the image as one JSON object matching the requested schema.
-        Return JSON only.
+        Describe the visible image as one JSON object matching the requested
+        schema. Use only visible evidence, without speculation. Return JSON only.
     sampling:
       temperature: 0.3
       max_tokens: 2048
+
+    output_processing:
+      validate_json_output: true
+      canonicalize_json_output: true
+      # Optional repairs; boxes must use [ymin, xmin, ymax, xmax] order.
+      repair_invalid_json_escapes: true
+      normalize_yxyx_bboxes: true
+      deduplicate_json_elements: true
+      # Optional: override refusal substrings, or [] to disable detection.
+      refusal_markers: ["i cannot describe", "unable to provide a caption"]
+
+    response_format:
+      type: "json_schema"
+      json_schema:
+        name: "caption"
+        strict: true
+        schema:
+          type: "object"
+          additionalProperties: false
+          required: ["description"]
+          properties:
+            description: {type: "string"}
+
     retry_prompt: >-
       Try again. Return exactly one valid JSON object with a concise,
       visually grounded description and no surrounding prose.
     retry_sampling:
       temperature: 0.1
       max_tokens: 1024
+    retry_response_format:
+      type: "json_object"
 ```
 
-`retry_sampling` accepts the standard OpenAI sampling fields supported by the
-worker. It can be configured globally under `inference` or per stage. Changes
-received from the orchestrator apply without restarting the worker.
-
-Some providers need a different constrained-decoding body for the retry. Set
-that locally so provider details and credentials never pass through the
-orchestrator:
+An empty caption, configured refusal match, invalid JSON, or provider content
+rejection can receive one semantic retry. Unrelated HTTP/transport failures
+use the endpoint's request retry policy, not the semantic prompt. Only items
+with an accepted output proceed to the next stage; failed items are not
+reported as successful empty captions. With multiple prompts, one accepted
+output is sufficient, and that item is not retried.
+
+`output_processing`, `response_format`, `retry_response_format`, `retry_prompt`,
+`retry_sampling`, and `retry_without_image` can also be set per stage. Stage
+output policy merges over inference defaults. Set `retry_without_image: true`
+only when deliberately retrying from text/metadata; it omits the image on both
+backends. Prompt, output-policy, and retry changes apply on config reload
+without reloading native model weights.
+
+`validate_json_output` checks JSON syntax and rejects non-finite numbers. It
+does **not** validate against JSON Schema. `response_format` requests constrained
+decoding from the backend; support for specific schemas depends on the model,
+provider, and vLLM version. Supported envelope types are `text`, `json_object`,
+and `json_schema`; a retry inherits the primary constraint unless overridden.
+Use `retry_response_format: {type: text}` to request unconstrained retry decoding.
+
+The optional transforms require `validate_json_output: true`:
+
+- `repair_invalid_json_escapes` escapes only invalid backslashes, preserving
+  existing valid escape pairs. It cannot repair truncated JSON.
+- `canonicalize_json_output` emits compact JSON with literal Unicode.
+- `normalize_yxyx_bboxes` sorts inverted coordinate pairs. It does not infer
+  boxes, clamp their range, or assess localization quality.
+- `deduplicate_json_elements` removes exact duplicates from arrays named
+  `elements`, including nested arrays of that name.
+
+## Direct vLLM worker
+
+Use a normal GPU worker config and launch with `caption-flow worker --vllm`.
+The worker translates `response_format` into vLLM's native structured-output
+parameters, including the older guided-decoding API. It retains native tensor
+parallelism, memory/cache configuration, token checks, and image resize recovery.
+
+Native `sampling` and `retry_sampling` are passed through to `SamplingParams`,
+including vLLM-specific controls such as `top_k`, `min_p`, `seed`, and
+`repetition_penalty`. Retry settings overlay primary settings without mutating
+the cached primary parameters. Use fields supported by your installed vLLM.
+Do not also configure native `structured_outputs`/`guided_decoding` when using
+the shared `response_format` envelope.
+
+## OpenAI-compatible worker
+
+Launch with `caption-flow worker --openai-compatible`. Endpoint credentials,
+request concurrency, image encoding, and provider-specific request extensions
+remain local:
 
 ```yaml
 worker:
+  server: "ws://orchestrator.example:8765"
+  token: "replace-with-captionflow-worker-token"
   openai_compatible:
-    retry_extra_body:
-      response_format:
-        type: "json_object"
+    endpoints:
+      - name: "local-vllm"
+        base_url: "http://inference.example:8000/v1"
+        api_key_env: "VLLM_API_KEY"
+        model: "vision-model"
+        initial_concurrency: 16
+        max_concurrency: 16
 ```
 
-Retry request fields override the endpoint's primary `extra_body`, except for
-`model` and `messages`, which CaptionFlow reserves. Non-standard sampling such
-as vLLM's `repetition_penalty` can also be placed in `retry_extra_body`.
+This adapter forwards shared `response_format` and standard OpenAI sampling
+fields to chat completions. Non-standard provider fields belong in endpoint
+`extra_body`; retry-only extensions belong in `openai_compatible.retry_extra_body`.
+Shared response formats override endpoint defaults; local `retry_extra_body`
+overrides the shared retry body. Neither extension can replace `model` or
+`messages`.
+
+## Compatibility and local overrides
+
+Either worker can override shared policy using `worker.output_processing`.
+Precedence is inference defaults, stage policy, then local worker policy.
+The legacy `inference.refusal_markers` is still accepted, with
+`inference.output_processing.refusal_markers` taking precedence.
+
+Existing JSON/refusal flags under `worker.openai_compatible` remain accepted
+as compatibility aliases for local output policy. New configs should use
+shared inference policy; explicit `worker.output_processing` overrides those
+legacy aliases. Existing endpoint `extra_body.response_format` and
+`retry_extra_body.response_format` continue working for API workers.

src/caption_flow/models.py

@@ -179,12 +179,13 @@ class ProcessingStage:
     dtype: Optional[str] = None
     gpu_memory_utilization: Optional[float] = None
 
-    # Optional semantic fallback for empty/refused caption responses. API
-    # workers can omit the image on the fallback request when an upstream
-    # provider rejects the image before inference.
+    # Caption policy and recovery apply to every inference backend.
     retry_prompt: Optional[str] = None
     retry_without_image: bool = False
     retry_sampling: Optional[Dict[str, Any]] = None
+    output_processing: Optional[Dict[str, Any]] = None
+    response_format: Optional[Dict[str, Any]] = None
+    retry_response_format: Optional[Dict[str, Any]] = None
 
 
 @dataclass

src/caption_flow/utils/image_processor.py

@@ -40,13 +40,18 @@ def prepare_for_inference(item: ProcessingItem) -> Image.Image:
 
         if item.image is not None:
             image = item.image
-            item.metadata["image_width"], item.metadata["image_height"] = image.size
-            item.metadata["image_format"] = image.format or "unknown"
+            for name, value in (
+                ("image_width", image.width),
+                ("image_height", image.height),
+                ("image_format", image.format or "unknown"),
+            ):
+                if item.metadata.get(name) is None:
+                    item.metadata[name] = value
             # item.image = None
             return image
 
-        item.image = None
         image = ImageProcessor.decode_image_data(item.image_data)
+        item.image = image
         item.image_data = b""
         item.metadata["image_format"] = image.format or "unknown"
         item.metadata["image_width"], item.metadata["image_height"] = image.size