POLY-SIM: Multimodal Speaker Identification

This repository contains my POLY-SIM challenge solution for Multimedia Data Mining at JKU (2026). The task is closed-set identification of 58 celebrities from MAV-Celeb v3 German/English data using frozen FaceNet face embeddings (512-d) and ECAPA-TDNN voice embeddings (192-d).

Protocol	Test language	Modalities
P3	training language	face + audio
P4	training language	audio only
P5	other language	face + audio
P6	other language	audio only

The challenge score is mean top-1 accuracy over P3-P6. Chance is 1.72%.

Reproducible 10-seed ratio study

The final study contains 180 val-selected runs:

• three recipes: dual-branch linear-sw1, shared single-sw0, and shared single-sw1;
• six synthetic ratios: 0, 0.03, 0.06, 0.09, 0.12, and 0.15;
• ten seeds per recipe/ratio cell.

sw1 adds per-modality AAM losses; sw0 trains only the fused pair objective. The rows below use each recipe's ratio with the highest 10-seed mean avg-P. Values are mean +/- sample standard deviation across ten seeds.

Model	Ratio	P3	P4	P5	P6	avg-P
dual branch AAM, stream supervision (linear-sw1)	0.12	99.5 +/- 0.5	97.7 +/- 0.3	93.2 +/- 2.1	91.1 +/- 1.1	95.37 +/- 0.84
shared AAM, fused objective (single-sw0)	0.15	98.5 +/- 0.5	96.2 +/- 0.9	94.8 +/- 1.1	90.3 +/- 1.1	94.96 +/- 0.48
shared AAM, stream supervision (single-sw1)	0.15	98.7 +/- 0.5	94.8 +/- 1.9	90.5 +/- 2.0	84.0 +/- 2.7	92.02 +/- 0.90
raw ECAPA + FaceNet score fusion, no training	-	99.2	95.5	95.9	94.0	96.1

Important observations:

• The best trained mean is 95.37, still 0.73 points below raw frozen-feature fusion at 96.1.
• The dual branch is the strongest trained recipe and retains a useful face path: 93.9% German and 73.5% English face-only accuracy at r=0.12.
• The shared fused-only recipe scores 73.0%/57.0% face-only at r=0.15. Adding stream supervision improves face emphasis but costs audio-only accuracy.
• At r=0.15, 211 synthetic rows are requested from only 172 unique eligible clips, so 39 rows are repeats. Results above the 12.25% unique-pool cap are oversampled rather than additional unique evidence.
• Raw ECAPA NCM reaches 95.5% German and 94.0% English audio-only. The trained models do not exceed the frozen ECAPA English result.

The complete seed-level evidence is committed under training_results/ratio_sweep_10seed_v1:

• summary.csv: mean and standard deviation for every recipe/ratio cell;
• results.csv and results.json: all 180 seed-level evaluations;
• plan.json: commands, input hashes, synthetic allocation, and provenance;
• execution.json: training execution records.

Published checkpoints

Three representative checkpoints are stored with Git LFS under important_checkpoints. For each recipe, the ratio is selected by 10-seed mean avg-P; within that ratio, the retained seed has the highest validation metric (lowest seed breaks ties).

File	Recipe	Ratio	Seed
linear_sw1_best.pt	dual branch + stream supervision	0.12	1
single_sw0_best.pt	shared branch, fused objective	0.15	2
single_sw1_best.pt	shared branch + stream supervision	0.15	3

No training media, synthetic media, pretrained encoder weights, or model caches are stored in the repository.

Repository layout

FOP/                    model, training, evaluation, and sweep runner
  main.py               train one configuration
  run_ratio_sweep.py    reproducible 3-recipe x 6-ratio x 10-seed study
  eval_protocol.py      P3-P6 checkpoint evaluation
  utils/                data loading, training, evaluation, checkpoint helpers
  tests/                model-specific tests
augmentation/           synthetic ASR/MT/TTS/lip-sync pipeline
scripts/                 study, training, and augmentation entry points
data_tools/              feature-tracker construction
tests/                   shared tests
metadata/                portable splits, manifest, exclusions, and hashes
training_results/        committed final aggregate and seed-level results
important_checkpoints/   three representative Git LFS checkpoints
data/                    local MAV-Celeb media/features; never committed
data_synthetic/          local generated samples/features; never committed

Setup

conda env create -f environment.yml
conda activate polysim
# Install the PyTorch build appropriate for the local GPU.

Reproduction

# Tests
python -m pytest -q

# Full final matrix: 3 recipes x 6 ratios x 10 seeds = 180 runs
cd FOP
python run_ratio_sweep.py --workers 4 --device cuda --batch_size 32

# Re-evaluate existing checkpoints without retraining
python run_ratio_sweep.py --evaluate_only

# Evaluate one published checkpoint
python eval_protocol.py --ckpt ../important_checkpoints/linear_sw1_best.pt

# Reproduce the zero-training references
cd ..
python scripts/study/raw_ncm_check.py
python scripts/study/raw_fusion.py

The manifest references local dataset paths for reproducibility, but those media files are intentionally excluded. Place course data under data/ and generated features under data_synthetic/ when reproducing the experiments.