Repository navigation
fix(openai-chat): read ogg, opus, flac, aac, aiff, webm and mpeg input_audio as their true media types - #2
Open
jonasherfort wants to merge 1 commit into
Conversation
…t_audio as their true media types request_from_openai_chat refused every input_audio format but wav and mp3, before any provider was chosen, so a DSPy ogg/opus recording bound for Gemini failed at ingest although Gemini takes audio/ogg natively. Each format now reads as its true media type; an unknown format is still malformed, and the Chat Completions and Anthropic builders still refuse audio at send (MAP-10). The same change as the lm15-dev/lm15-python fix, on the line DSPy vendors; CONTRACT_PIN stays at b721ce4, whose MAP-12 rule 4 still lists wav and mp3. Signed-off-by: Jonas Dreyøe Herfort <jdreyoe@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
request_from_openai_chataccepts aninput_audioblock only when itsformatiswavormp3. Any other format raisesValueError: messages[1].content[1].input_audio.format must be one of ['mp3', 'wav']at ingest, before a provider is chosen. DSPy rendersdspy.Audioasinput_audiowith the file's MIME subtype as the format, so an ogg/opus phone recording sent tovertex:gemini-3.8-flashfails even though Gemini takesaudio/oggnatively. An.mp3file fails too, because DSPy labels itmpeg.wav | mp3is the list for OpenAI's own server. Google's Chat Completions server accepts every audio MIME type, and MAP-12 rule 4 already says ingest is not where a wire gap is hidden.Each format now reads as its true media type:
wavasaudio/wav,mp3andmpegasaudio/mpeg, andogg,opus,flac,aac,aiffandwebmasaudio/<format>. No format is relabelled as another type. An unknown format is still malformed. Nothing changes at send time. Gemini sends the part asinlineDatawith its media type. The Chat Completions and Anthropic builders still refuse audio before the wire (MAP-10).m4a/mp4is left out because no receipt shows which spelling Gemini accepts.This is the same table fix as lm15-dev#16, on the 1.0.0rc3 line (55fce2e) that DSPy vendors, so DSPy can re-vendor it without taking 1.1.0. The contract amendment is lm15-dev/lm15-contract#1. New tests in
tests/test_openai_chat_ingest.pycover the media type for each format, the ingest-to-GeminiinlineDatapath, the Chat Completions refusal of the same request and an unknown format; they fail before the fix and pass now. The full suite passes (3333 passed).