The experimental workflow consists of three stages:

1. Data preparation
2. Model training
3. Word-form and phoneme analysis

Repository structure

.
├── data_preparation/        # Prepare transcripts, text-audio alignments and annotations
├── training/                # Train the models using the patched Fairseq code
│   ├── speech_learning/     # Model architecture and training components
│   ├── fairseq_changes.patch
│   └── req.txt              # Training and analysis dependencies
├── analysis/                # Analyze word-form and phoneme development
└── README.md

Environment setup

The training and analysis stages use a shared Conda environment. The data preparation scripts can use this environment as well, but Montreal Forced Aligner (MFA) should be installed in a separate environment to avoid dependency conflicts. See the data preparation instructions for details.

Run the following commands from the repository root.

1. Create the Conda environment

conda create --name fairseq_env python=3.10 pip=24.0 -y
conda activate fairseq_env

2. Set up Fairseq

The experiments use Fairseq at a specific commit together with the patch and our model implementation included in this repository:

cd training

git clone https://github.com/facebookresearch/fairseq.git
cd fairseq
git checkout 47e279842ac8776e3964b0e45c320ad1d2ea6096

git apply -p2 ../fairseq_changes.patch
cp -R ../speech_learning examples/

pip install --editable .
python setup.py build_ext --inplace
pip install -r ../req.txt

After completing the setup, return to the repository root:

cd ../..

Reproduce the experiments

Run the stages in order. Each stage has its own README with the required input format, commands, and expected outputs.

1. Prepare the data

Follow data_preparation/README.md to prepare the audio, transcripts, forced alignments, and linguistic annotations.

2. Train the model

Follow training/README.md to configure and run model training.

3. Run the analysis

Follow analysis/README.md to analyze word-form and phoneme development and compare the model results with CDI data.

Acknowledgments and attribution

Fairseq

Parts of this repository are based on or modify files from Fairseq, originally developed by Facebook AI Research / Meta AI and released under the MIT License. Modified Fairseq files and patches retain the original copyright notices and license terms.

If you use this code, please also cite Fairseq:

Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. “fairseq: A Fast, Extensible Toolkit for Sequence Modeling.” Proceedings of NAACL-HLT 2019: Demonstrations, 2019.

@inproceedings{ott2019fairseq,
  title     = {fairseq: A Fast, Extensible Toolkit for Sequence Modeling},
  author    = {Ott, Myle and Edunov, Sergey and Baevski, Alexei and Fan, Angela
               and Gross, Sam and Ng, Nathan and Grangier, David and Auli, Michael},
  booktitle = {Proceedings of NAACL-HLT 2019: Demonstrations},
  year      = {2019}
}

DinoSR

Parts of this repository are based on or modify code from DinoSR, developed by Alexander H. Liu, Heng-Jui Chang, Michael Auli, Wei-Ning Hsu, and James R. Glass.

The modified DinoSR-derived code included in this repository is redistributed with permission from the author and with clear attribution to the original DinoSR implementation. The original DinoSR repository does not currently include an explicit software license.

If you use this code, please also cite DinoSR:

Alexander H. Liu, Heng-Jui Chang, Michael Auli, Wei-Ning Hsu, and James R. Glass. “DinoSR: Self-Distillation and Online Clustering for Self-supervised Speech Representation Learning.” Advances in Neural Information Processing Systems 36 (NeurIPS 2023), 2023.

@inproceedings{liu2023dinosr,
  title     = {DinoSR: Self-Distillation and Online Clustering for Self-supervised Speech Representation Learning},
  author    = {Liu, Alexander H. and Chang, Heng-Jui and Auli, Michael and Hsu, Wei-Ning and Glass, James R.},
  booktitle = {Advances in Neural Information Processing Systems},
  volume    = {36},
  year      = {2023}
}

Text-Free Prosody-Aware Generative Spoken Language Modeling

The unit language-model component in this repository is based on Fairseq's TransformerUnitLanguageModel, introduced as part of the Multi-Stream Transformer Language Model (MS-TLM) in Text-Free Prosody-Aware Generative Spoken Language Modeling by Eugene Kharitonov, Ann Lee, Adam Polyak, Yossi Adi, Jade Copet, Kushal Lakhotia, Tu Anh Nguyen, Morgane Riviere, Abdelrahman Mohamed, Emmanuel Dupoux, and Wei-Ning Hsu. We adapt this architecture to model discovered speech units and their durations.

If you use this component, please also cite the original work:

Eugene Kharitonov, Ann Lee, Adam Polyak, Yossi Adi, Jade Copet, Kushal Lakhotia, Tu Anh Nguyen, Morgane Riviere, Abdelrahman Mohamed, Emmanuel Dupoux, and Wei-Ning Hsu. “Text-Free Prosody-Aware Generative Spoken Language Modeling.” Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 8666–8681, 2022.

@inproceedings{kharitonov-etal-2022-text,
  title     = {Text-Free Prosody-Aware Generative Spoken Language Modeling},
  author    = {Kharitonov, Eugene and Lee, Ann and Polyak, Adam and Adi, Yossi
               and Copet, Jade and Lakhotia, Kushal and Nguyen, Tu Anh
               and Riviere, Morgane and Mohamed, Abdelrahman
               and Dupoux, Emmanuel and Hsu, Wei-Ning},
  booktitle = {Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  pages     = {8666--8681},
  year      = {2022}
}