TR2025-038

ESPnet-SpeechLM: An Open Speech Language Model Toolkit

    •  Tian, J., Shi, J., Chen, W., Arora, S., Masuyama, Y., Takashi, M., Wu, Y., Peng, J., Bharadwaj, S., Zhao, Y., Cornell, S., Peng, Y., Yue, X., Yang, C.-H.H., Neubig, G., Watanabe, S., "ESPnet-SpeechLM: An Open Speech Language Model Toolkit", NAACL-HLT (the system demonstration track), March 2025.
      BibTeX TR2025-038 PDF
      • @inproceedings{Tian2025mar,
      • author = {Tian, Jinchuan and Shi, Jiatong and Chen, William and Arora, Siddhant and Masuyama, Yoshiki and Takashi, Maekaku and Wu, Yihan and Peng, Junyi and Bharadwaj, Shikhar and Zhao, Yiwen and Cornell, Samuele and Peng, Yifan and Yue, Xiang and Yang, Chao-Han H. and Neubig, Graham and Watanabe, Shinji},
      • title = {{ESPnet-SpeechLM: An Open Speech Language Model Toolkit}},
      • booktitle = {NAACL-HLT (the system demonstration track)},
      • year = 2025,
      • month = mar,
      • url = {https://www.merl.com/publications/TR2025-038}
      • }
  • MERL Contact:
  • Research Areas:

    Artificial Intelligence, Machine Learning, Speech & Audio

Abstract:

We present ESPnet-SpeechLM, an open toolkit designed to democratize the development of speech language models (SpeechLMs) and voice-driven agentic applications. The toolkit standardizes speech processing tasks by framing them as universal sequential modeling prob- lems, encompassing a cohesive workflow of data preprocessing, pre-training, inference, and task evaluation. With ESPnet-SpeechLM, users can easily define task templates and configure key settings, enabling seamless and stream- lined SpeechLM development. The toolkit ensures flexibility, efficiency, and scalability by offering highly configurable modules for every stage of the workflow. To illustrate its capabilities, we provide multiple use cases demonstrating how competitive SpeechLMs can be constructed with ESPnet-SpeechLM, including a 1.7B-parameter model pre-trained on both text and speech tasks, across diverse benchmarks.

 

  • Related Publication

  •  Tian, J., Shi, J., Chen, W., Arora, S., Masuyama, Y., Takashi, M., Wu, Y., Peng, J., Bharadwaj, S., Zhao, Y., Cornell, S., Peng, Y., Yue, X., Yang, C.-H.H., Neubig, G., Watanabe, S., "ESPnet-SpeechLM: An Open Speech Language Model Toolkit", arXiv, February 2025.
    BibTeX arXiv
    • @article{Tian2025feb,
    • author = {Tian, Jinchuan and Shi, Jiatong and Chen, William and Arora, Siddhant and Masuyama, Yoshiki and Takashi, Maekaku and Wu, Yihan and Peng, Junyi and Bharadwaj, Shikhar and Zhao, Yiwen and Cornell, Samuele and Peng, Yifan and Yue, Xiang and Yang, Chao-Han H. and Neubig, Graham and Watanabe, Shinji},
    • title = {{ESPnet-SpeechLM: An Open Speech Language Model Toolkit}},
    • journal = {arXiv},
    • year = 2025,
    • month = feb,
    • url = {https://arxiv.org/abs/2502.15218}
    • }