Skip to content

fix(openai-chat): read ogg, opus, flac, aac, aiff, webm and mpeg input_audio as their true media types - #16

Merged
MaximeRivest merged 1 commit into
lm15-dev:mainfrom
jonasherfort:fix/ingest-more-audio-formats
Sep 29, 2026
Merged

MaximeRivest merged 1 commit into
lm15-dev:mainfrom
jonasherfort:fix/ingest-more-audio-formats

Conversation

@jonasherfort

Copy link
Copy Markdown
Contributor

request_from_openai_chat accepts an input_audio block only when its format is wav or mp3. Any other format raises ValueError: messages[1].content[1].input_audio.format must be one of ['mp3', 'wav'] at ingest, before a provider is chosen. DSPy renders dspy.Audio as input_audio with the file's MIME subtype as the format, so an ogg/opus phone recording sent to vertex:gemini-3.8-flash fails even though Gemini takes audio/ogg natively. An .mp3 file fails too, because DSPy labels it mpeg. wav | mp3 is the list for OpenAI's own server. Google's Chat Completions server accepts every audio MIME type, and MAP-12 rule 4 already says ingest is not where a wire gap is hidden.

Each format now reads as its true media type: wav as audio/wav, mp3 and mpeg as audio/mpeg, and ogg, opus, flac, aac, aiff and webm as audio/<format>. No format is relabelled as another type. An unknown format is still malformed. Nothing changes at send time. Gemini sends the part as inlineData with its media type. The Chat Completions and Anthropic builders still refuse audio before the wire (MAP-10). m4a/mp4 is left out because no receipt shows which spelling Gemini accepts.

The contract change comes first: lm15-dev/lm15-contract#1 amends MAP-12 rule 4, updates the verdict row and adds the case openai_chat.ingest_input_audio_ogg. CONTRACT_PIN moves to that commit. New tests in tests/test_openai_chat_ingest.py cover the media type for each format, the ingest-to-Gemini inlineData path, the Chat Completions refusal of the same request and an unknown format. They failed before the fix (8 of 26 selected) and pass now. The full suite passes (3589 passed, 8 skipped), as do conformance/run_all.py --strict, the pinned harness (1829 of 1829, ingest 217 of 217; the new case fails on the old code) and tools/typecheck.py (0 new).

…t_audio as their true media types

request_from_openai_chat refused every input_audio format but wav and mp3,
before any provider was chosen, so a DSPy ogg/opus recording bound for Gemini
failed at ingest although Gemini takes audio/ogg natively. Each format now
reads as its true media type (MAP-12 rule 4 as amended in lm15-contract
changes/2026-09-29-input-audio-formats.md); an unknown format is still
malformed, and the Chat Completions and Anthropic builders still refuse audio
at send (MAP-10). Contract pin 307925a.

Signed-off-by: Jonas Dreyøe Herfort <jdreyoe@gmail.com>
@MaximeRivest

Copy link
Copy Markdown
Member

Thank you, Jonas. Clear diagnosis (DSPy's mpeg label for .mp3 was a good catch), each format read as its true media type with nothing relabelled, and tests that cover both sides: the ogg clip reaching Gemini as inlineData and the Chat Completions builder still refusing it. CI is green on every OS and Python version, and the pinned contract harness passes. Merged; lm15-dev/lm15-contract#1 is on main with the pinned commit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants