Place decoded audio where the file's frame sizes put it

[bdeluca]

Symptom

Audio read out of order was wrong: frames popped, went silent, or carried samples from elsewhere, in plain playback as much as when scrubbing, because the cache fills from several readers and every reader seeks. On a webm the whole track sat 3 ms late against picture. Seven defects in the ffmpeg reader and one latent fault, each below with its cause, fix, commit and the test that shows it.

Issues

1. A frame spliced across a seek

• Cause: a per-frame buffer half filled before a seek was kept and finished with samples decoded from the new position.
• Fix: half-filled buffers are cleared on a seek (FFMpegDecoder::do_seek).
• Commit: 3, Assemble audio frames from decoded packets correctly across a seek.
• Evidence: DecoderInternals.SeekClearsHalfAssembledAudioBuffers; the pop measured at up to 2925 against a 2900 peak in the harness's shuffled runs is gone.

2. Region merge could not absorb a nested region (latent)

• Cause: the merge of filled regions only joined two when an endpoint of one fell inside the other. Abutting regions, which is all a sequential decode produces, always merged, so this never showed in sequential play. A region wholly inside an existing one did not merge, which needs a packet to re-deliver samples already written to the same buffer; only the old label-based placement could do that, after a seek. The buffer then never reported complete.
• Fix: regions merge when they overlap or touch (PartiallyFilledAudioBuf::copy_samples_from_other_buffer). With issue 6 fixed the case cannot arise; the merge is now correct regardless.
• Commit: 3.
• Evidence: PartialAudioBuf.ContainedRegionStillCompletesTheFrame, PartialAudioBuf.AdjacentRegionsCompleteTheFrame. No harness case depends on it.

3. Priming packets skipped

• Cause: sample arithmetic was unsigned; a packet with a negative timestamp, which is how matroska hands AAC priming over, wrapped and was skipped.
• Fix: the arithmetic is signed.
• Commit: 3.
• Evidence: AudioIntegrity.* on mkv_aac_48k, mkv_mp3_44k1, mkv_ac3_48k (first packets at negative timestamps).

4. Decoder state survived a seek

• Cause: the reader flushed the decoder on a seek, and a flush does not reset every decoder. AC-3, E-AC-3, FLAC and ALAC have no flush callback, so after a seek back to the head AC-3 overlap-added its first frame with the tail of the last one decoded; AAC keeps its noise-substitution state across a flush.
• Fix: a seek opens a fresh codec context (FFMpegStream::open_audio_decoder), which is what the head of the stream always had.
• Commit: 4, Decode audio after a seek the way ffmpeg's tools do.
• Evidence: AudioIntegrity.RandomAccessIsRepeatable on avi_ac3_48k and clicks_m4a_aac_48k, whose frame 0 read after a seek differed from the sequential decode before, every later frame matching.

5. Program streams failed on every seek

• Cause: a seek into an MPEG program stream lands on a partial packet, the mp2 and AC-3 decoders reject it, and the reader treated the error as fatal. The request failed and the next one landed on the same packet.
• Fix: a rejected packet is skipped, as ffmpeg's own tools skip it (FFMpegDecoder::decode_next_frame, decode_and_store_next_frame).
• Commit: 4.
• Evidence: AudioIntegrity.* on mpg_mp2_48k, mpg_mp2_44k1, mpg_ac3_48k, which threw on every frame before.

6. Random access disagreed with sequential decode in millisecond containers

• Cause: matroska and webm record only the millisecond a packet starts in, a tick of 44 or 48 samples. A run started by a seek was placed from that rounded label, so two routes to the same packet disagreed by up to a tick and frames from different runs never met. The webm's 3 ms was the same defect at the head: a muxer that lost the vorbis lead-in labels the first kept sample after the start.
• Fix: AudioPtsRescaler. How a frame's position is found is chosen per stream, in FFMpegStream::probe_audio_start, from what libavcodec states about it. A codec with no frames, whose samples come straight from the packet's bytes (av_get_exact_bits_per_sample), with a time_base unit of one sample or less: the pts is the position, which is right for a raw PCM demuxer that cuts packets anywhere after a seek. A codec the library gives a frame size for (av_get_audio_frame_duration2): frame k starts at start + k * frame_size, k being the pts in samples over the frame size, rounded, since a pts is never out by anything near half a frame. Anything else is issue 8. The decoder runs with AV_CODEC_FLAG2_SKIP_MANUAL so frames arrive whole and the reader drops the priming the side data names; the first frames, decoded once at open, fix where the audio begins. Replaces the lead-in probe, the counting rule and av_rescale_delta.
• Commit: 5, Place decoded audio by pts, by frame size, or from a region table.
• Evidence: audio_pts_rescaler_test, which takes pts sequences recorded from thirteen of the harness's streams and requires any frame, from its own pts alone, to land where a count from the start of the stream puts it; AudioIntegrity.* on every matroska, webm and avi AC-3 case, all failing before; AudioIntegrity.ClicksLandOnTheirSample on clicks_webm_vorbis_48k for the 3 ms.

7. Pre-roll dropped audio that was needed

• Cause: the pre-roll a seek allows for was measured from wherever the demuxer landed. A program stream's mp2 parser rejects up to nine packets after a seek, so the first frame accepted could be 90 ms after the target, and the frames needed were dropped as pre-roll or came back with the decoder's warm-up in their first samples.
• Fix: pre-roll is dropped only up to the position asked for; a seek that lands with less than half the warm-up to spare is retried from twice as far back, up to 16x, and the distance is kept for the file (FFMpegDecoder::decode_audio_frame).
• Commit: 5.
• Evidence: AudioIntegrity.EveryFrameIsStillAPureTone and RandomAccessAgreesWithSequential on mpg_mp2_44k1, the last case to pass.

8. A pts cannot place a packet whose size changes

• Cause: vorbis picks one of three frame sizes per packet from the sound, Opus and FLAC may change theirs, and libavcodec states no frame size for them. Where such a packet starts is the sum of the sizes before it, and only the muxer counted. Its count, the pts, is not reliable: in real webm and matroska files the pts around a change of vorbis block size are tens of milliseconds from the samples before them, and inside a matroska block only the block's time is stored, so the demuxer adds durations for the other packets and after a seek onto the block sizes the first one without the packet before it. The same eight packets of one film come out as 35299 35311 35313 ... read through and 35299 35301 35303 ... after a seek.
• Fix: a region table per stream, AudioPtsRescaler::use_region_table: each packet's file position (AVPacket.pos) and the samples it decodes to, sized by the library without decoding (av_get_audio_frame_duration2 on the packet size, else AVCodecParserContext::duration, else the packet duration where the time_base is one sample). A region starts at the sum of the sizes before it. A seek finds the region that holds the wanted sample and goes to its block's stored pts (FFMpegStream::audio_seek_pts). A packet is named by its pos and which of the packets there it is, and the decoder hands that to the frame (AV_CODEC_FLAG_COPY_OPAQUE). No pts places anything. The table is filled only as far into the file as has been needed: playing on adds each region from the packet being decoded, a seek beyond it reads forward from where it stops. A packet that cannot be sized or has no pos, a packet the table does not hold and a frame tied to no packet throw.
• Commit: 5.
• Evidence: audio_pts_rescaler_test holds that block of the film with both sets of pts, and checks the table both ways, packet to start and sample to region, on the thirteen recorded streams. AudioIntegrity.* on every vorbis, Opus, FLAC, E-AC-3 and ALAC case and PCM in matroska, which take this path. Outside the tree, real files decoded through the reader, 1200 frames in order and again with a seek before every frame, each buffer located in a plain ffmpeg decode: before commit 5 a seek placed 1193 of 1198 frames of one Wikimedia webm elsewhere than playback did, 740 of 740 of an 830 MB matroska film, and ogg vorbis broke 18 times in plain playback; after it, thirteen files each sit at one constant offset with no breaks. A seek 625 s into the film costs 64 ms once, from the page cache. Only vorbis was found in real files; the pts and frame size paths have been run on the harness's media only.

Testing

• Rebased on upstream develop (a18a736). The commits apply cleanly. The second commit is not part of the change: it is the media_reader fix that keeps Buffer::size_ in step with set_buf_data, which upstream lacks. The harness commit before it shows why: resampling audio corrupts the heap and the harness aborts in its first decoder test; the fix turns it green. It should go upstream on its own ahead of this.
• ffmpeg_audio_integrity_test generates its own media with the vendored ffmpeg into tone_media and leaves it there to be played and checked by hand; it allows no tolerance, and compares sequential, shuffled and reverse decodes sample for sample: 8 of 8 pass. Containers: wav, aiff, mov, mp4, m4a, mkv, webm, avi, mxf, MPEG-TS, MPEG-PS, ogg, opus, raw mp3. Codecs: PCM, AAC, ALAC, FLAC, mp2, mp3, AC-3, E-AC-3, vorbis, opus. Rates: 44.1, 48 and 96 kHz.
• The click-track test runs over every container and codec pair the reader supports, with a picture in the file, 30 pairs plus 44.1 kHz variants of the lossy millisecond-container ones. Every click must land on the sample the source put it on, following the audio stream's own clock where a container starts it late.
• audio_pts_rescaler_test: 8 of 8 pass. Needs no media.
• Each commit builds on its own with tests on, and the harness was run at each. Commit 1: it crashes in its first decoder test, the heap corruption commit 2 fixes. Commit 2: 1 of 8 tests pass, 95 media cases fail. Commit 3: 3 of 8, 49 cases. Commit 4: 3 of 8, 45 cases, avi_ac3_48k and the mp4 AC-3, E-AC-3 and mp3 click cases now passing. Commit 5: 8 of 8, none. The harness commit adds FFMpegStream::first_packet_pts(), which its click test needs.
• Linux only. Nothing listened to on real footage; the real files under issue 8 were measured, not auditioned. A clock test set for doing that by eye and ear, an image sequence with a flash on every second and a wav with a one-frame burst of noise on the same frame, encoded into 44 container and codec pairs, is generated by scripts/av_sync_clock.py (not in this PR).

Known broken

Listed by name and reason in kKnownBroken in the harness, run only by AudioIntegrity.DISABLED_KnownBroken. gtest reports the disabled test on every run; --gtest_also_run_disabled_tests runs it and it fails on all twelve. A case that starts passing there is fixed and moves out of the list.

• Containers that cannot carry the codec's encoder delay, so the audio sits later than the source by that delay and every player that trusts the file is late too: avi with AC-3 (256 samples) and mp3 (1105); MPEG-TS with AAC (1024), mp2 (481), mp3 (1105), AC-3 (256); MPEG-PS with mp2 and AC-3.
• Raw mp3: the demuxer reports the LAME delay as the stream's start time, so the file's own clock puts the audio 1105 samples late.
• Ogg vorbis: the demuxer's per-packet timestamps disagree with the sizes of the frames the decoder returns. With the region table its frames join and seeks agree with playback, but the stream's first pts is -128 and the reader starts the audio one short frame early. Audio only; vorbis in matroska and webm is fine.

Commits

1. Add an audio integrity harness for the ffmpeg reader
2. media_reader: keep Buffer::size_ in sync with set_buf_data
3. Assemble audio frames from decoded packets correctly across a seek
4. Decode audio after a seek the way ffmpeg's tools do
5. Place decoded audio by pts, by frame size, or from a region table