piano_leds

Detects played piano keys from a "piano-roll" style video (falling colored bars over a keyboard, e.g. a screen recording of a Synthesia-like app) and drives an LED strip above a physical keyboard to match, with a synthesized audio preview of detected notes.

Structure

• src/main.py — ties everything together; the frame loop and keybindings
• src/calibration.py — maps camera pixel coordinates to piano keys
• src/detector.py — key press / note detection from frames
• src/sections.py — splits a video into note-sections separated by silence, cached to disk next to the video (see #todos below)
• src/synth.py — real-time audio preview of detected notes
• src/leds.py — MIDI-to-LED-index mapping, plus a rpi_ws281x/GPIO strip driver (currently unused by main.py; see LED strip setup)
• src/serial_leds.py — the LED strip driver main.py actually uses: sends Adalight-protocol frames over serial to an Arduino
• arduino_src/ — Arduino firmware (Adalight-FastLED.ino) that receives those serial frames and drives the physical WS2812B strip, plus a standalone rainbow test script
• config.json — calibration data (camera pixel-to-key mapping); has no LED settings, see below

Setup

uv venv
source .venv/bin/activate.fish   # bash: drop the .fish
uv pip install -r requirements.txt

Usage

uv run src/main.py [video_path]

Defaults to test.mp4 if no path is given. There are no build, lint, or test commands/configs in this repo — verification is manual, by running the app against a video file.

If config.json has no calibration data yet, the app starts in calibrate mode; otherwise it starts in detect mode. You can switch between them at any time while the app is running.

Example video

Example video: https://www.youtube.com/watch?v=ny3BwGgLh90

• grab with ./yt-dlp https://www.youtube.com/watch?v=ny3BwGgLh90
• convert to mp4: ffmpeg -i "test.webm" -c:v libx264 -c:a aac test.mp4

Calibrate mode

• click 4 points, in order:

  1. left of the leftmost key

  2. right of the rightmost key

  3. the middle of the narrow part of C4, between B and C#

  4. the horizontal detection line, placed so it crosses both black and white keys — in playback, each note marker must sit perfectly on a key; example of a good line placement:

     

• c — clear calibration points
• ENTER — save calibration to config.json and reload the detector
• SPACE — pause/play the video

Detect mode (requires a saved calibration)

• SPACE — pause/resume detection and mute the synth
• t — toggle the physical LED strip on/off (opens/closes the serial connection; safe to press without hardware attached, it just prints an error and stays off)
• numpad 1 / 3 — step one frame-step backward/forward, pause, and preview the detected notes as a short sound (and on the LED strip, if enabled) for the landed frame
• a — analyze the video into note-sections separated by silence; result is cached next to the video file (<video>.sections.json), invalidated by video or calibration changes, so this only has to be re-run when either changes
• RIGHT / LEFT — jump to and play the next/previous analyzed section, then stop at its end (requires a first)
• UP — replay the current section, or the last numpad-scrub preview sound if mid-scrub

Any mode

• s — switch to calibrate mode
• d — switch to detect mode (requires a saved calibration)
• q / ESC — quit

LED strip setup

The physical strip is driven over serial using the Adalight protocol: src/serial_leds.py (AdaLightStrip) sends frames to an Arduino running the Adalight-FastLED sketch in arduino_src/, which forwards them to a WS2812B strip via FastLED. src/leds.py also contains an alternative LedStrip driver that talks to a strip directly via rpi_ws281x/GPIO (for driving the strip straight from a Raspberry Pi's own pins instead of through an Arduino) — main.py doesn't use it currently.

Values you're likely to need to change for your own hardware:

• src/main.py
  • NUM_LEDS — number of LEDs actually spanning the keyboard width. Must be no larger than the Arduino's Num_Leds below.
• src/serial_leds.py (AdaLightStrip.__init__ defaults)
  • port — serial device the Arduino is on (e.g. /dev/ttyUSB0, or a COM port on Windows)
  • baud — must match SerialSpeed in the Arduino sketch
• src/leds.py
  • LOW_MIDI_SECTION / HIGH_MIDI_SECTION — the MIDI note range mapped across the strip; narrow or widen this if the strip should cover a different portion of the keyboard than A0..C8
• arduino_src/Adalight-FastLED.ino/Adalight-FastLED.ino.ino (re-flash the Arduino after changing)
  • Num_Leds — physical LED count wired to the Arduino
  • Brightness — max brightness (0-255)
  • PIN_DATA — data pin the strip is wired to (PIN_CLOCK too, for 4-wire strip types)
  • LED_TYPE / COLOR_ORDER — must match the strip's actual chip and wiring
  • SerialSpeed — must match baud above

#todos

• the dynamic bass-boost volume changes in synth.py (bass_boost()) are too extreme: low notes get boosted up to 2.5x versus 1x for high notes, which is too aggressive — tone this down.