MediaKit

The media plane of the Physiology Workbench family: capture of OS-mediated sensors — camera, microphone, motion, other apps' audio — fan-out of the high-volume buffers to reducers, the reducers that turn them into low-volume plain-data streams (pose skeletons, breath and vocal features, recognised labels, beat events, heart rate from face video), and the outbound half: spoken guidance and cue tones back out through the same audio path.

A library beside DeviceCore (the family's device-I/O layer), not above it: DeviceCore owns the radios the family drives itself over CoreBluetooth; this repo owns the sensors the OS has already terminated and hands over as buffers.

What works today

Two capture → reducer paths, both validated on real hardware or real audio, and the outbound audio path they turned out to need:

• Microphone → word spotting. MicrophoneCapture (explicit arm/disarm, drop-oldest chunk stream, published input level) feeding WordSpotter (SpeechTranscriber, word-onset timestamps; gated macOS 26 / iOS 26), with the pure pieces under them testable at the platform floor.

• System audio → beat tracking (macOS 14.4+). ProcessTapCapture taps one process's decoded output via a Core Audio process tap — pre-system- volume, player-agnostic. The DSP cores (OnsetEnvelope, estimateTempo, beatTimesDP, BeatPhasePredictor — ported from Ellis 2007 and Stark, Davies & Plumbley 2009, from the papers, never from GPL code) feed the BeatTracker reducer: plain (time, period, confidence) beat events, each predicted ~300 ms before it lands. Bench record in SPOTIFY-BEAT.md; design, references and constants in BEAT-TRACKING.md.

• Spoken guidance out. SpokenGuidance speaks instructions and plays cue tones through whatever output the host has chosen, stamping each emission at its onset and reporting it as a GuidanceEvent so the caller can put it on the recording — an unlogged signal into a channel that is also stimulus is a confound. Speech goes to AVSpeechSynthesizer; tones are synthesised here and scheduled at a known sample position, because a paced cue is the stimulus and wants better than callback precision. gating(_:) optionally filters a spotted-label stream against what the instrument just said, so the machine cannot answer its own question.

Beside the library sits a second product, MediaBench — five bench tools under the noun audio (devices, out, in, loopback, selfhear) that hear the outbound half and measure the inbound one. It is the one thing here that takes a family module: DeviceCore's BenchKit, which supplies the tool protocol and the typed run record. The library itself stays Apple-frameworks- only.

Try the beat path against whatever is playing (macOS):

swift run tap-probe --click   # taps Spotify by default, clicks on predicted beats

Using the package

.package(url: "https://github.com/PhysiologyWorkbench/MediaKit", from: "0.0.1")

Platform floors macOS 14 / iOS 17, Swift 6 language mode, strict concurrency. The library builds on both platforms; capture sources that need more are availability-gated (ProcessTapCapture macOS 14.4, WordSpotter macOS 26 / iOS 26) and everything under them compiles and tests at the floor:

swift build && swift test   # no hardware, no OS 26 needed

A clone needs DeviceCore beside it. The second product, MediaBench, takes DeviceCore's BenchKit by path, and SwiftPM resolves that path against the directory name, so the sibling must be checked out as DeviceCore:

git clone https://github.com/PhysiologyWorkbench/DeviceCore
git clone https://github.com/PhysiologyWorkbench/MediaKit
cd MediaKit && swift build && swift test

Rules this library enforces

• Nothing captures as a side effect. Capture is armed explicitly, and a live input level is published while armed — a hot microphone is legible as movement, not as a label claiming it is on. Nothing sounds as a side effect either: the output is claimed when something is played, not when the instrument is constructed.
• Drop, never queue. Backpressure on buffer fan-out is drop-oldest; a reducer that cannot keep up loses frames, not currency.
• Plain data out. Runs of Doubles, (time, label, confidence) events and (time, duration, content) emissions; no app vocabulary, no framework types across the boundary.
• Reducers replay. Every reducer runs over a recorded sidecar exactly as over live capture.
• Port from papers, not from GPL code. Every own algorithm carries its citation; GPL reference implementations are never consulted.

Records

ARCHITECTURE.md — the design and the reducer survey. BEAT-TRACKING.md — the beat stack's study and lessons. LESSONS.md — dated lessons. Contributor and agent orientation in CLAUDE.md; contribution basics in CONTRIBUTING.md.

License

MIT — see LICENSE.