Detail publikačního výsledku

Speech and Language Recognition with Low-rank Adaptation of Pretrained Models

PRASAD, A.; MADIKERI, S.; KHALIL, D.; MOTLÍČEK, P.; SCHUEPBACH, C.

Originální název

Speech and Language Recognition with Low-rank Adaptation of Pretrained Models

Anglický název

Speech and Language Recognition with Low-rank Adaptation of Pretrained Models

Druh

Stať ve sborníku v databázi WoS či Scopus

Originální abstrakt

Finetuning large pretrained models demands considerable computational resources, posing practical constraints. Major- ity of the total number of parameters in these models are used by fully connected layers. In this work, we consider applying a semi-orthogonal constraint, followed by full finetuning to the fully connected layers reduces model parameters significantly without sacrificing efficacy in downstream tasks. Specifically, we consider wav2vec2.0 XLS-R and Whisper models for Auto- matic Speech Recognition and Language Recognition. Our re- sults show that we can reduce the model size by approximately 24% during both training and inference time with 0.7% absolute drop in performance for XLS-R and no drop in performance for Whisper for ASR. In combination with performance-efficient training with low-rank adapters, the resource requirements for training can be further reduced by up to 90%.

Anglický abstrakt

Finetuning large pretrained models demands considerable computational resources, posing practical constraints. Major- ity of the total number of parameters in these models are used by fully connected layers. In this work, we consider applying a semi-orthogonal constraint, followed by full finetuning to the fully connected layers reduces model parameters significantly without sacrificing efficacy in downstream tasks. Specifically, we consider wav2vec2.0 XLS-R and Whisper models for Auto- matic Speech Recognition and Language Recognition. Our re- sults show that we can reduce the model size by approximately 24% during both training and inference time with 0.7% absolute drop in performance for XLS-R and no drop in performance for Whisper for ASR. In combination with performance-efficient training with low-rank adapters, the resource requirements for training can be further reduced by up to 90%.

Klíčová slova

parameter reduction, language identification, speech recognition, wav2vec2.0

Klíčová slova v angličtině

parameter reduction, language identification, speech recognition, wav2vec2.0

Autoři

PRASAD, A.; MADIKERI, S.; KHALIL, D.; MOTLÍČEK, P.; SCHUEPBACH, C.

Rok RIV

2025

Vydáno

01.09.2024

Nakladatel

International Speech Communication Association

Místo

Kos Island

Kniha

Proceedings of Interspeech

ISSN

1990-9772

Periodikum

Proceedings of Interspeech

Svazek

2024

Číslo

9

Stát

Francouzská republika

Strany od

2825

Strany do

2829

Strany počet

5

URL

https://www.isca-archive.org/interspeech_2024/prasad24_interspeech.html

BibTex

@inproceedings{BUT193370,
  author="PRASAD, A. and MADIKERI, S. and KHALIL, D. and MOTLÍČEK, P. and SCHUEPBACH, C.",
  title="Speech and Language Recognition with Low-rank Adaptation of Pretrained Models",
  booktitle="Proceedings of Interspeech",
  year="2024",
  journal="Proceedings of Interspeech",
  volume="2024",
  number="9",
  pages="2825--2829",
  publisher="International Speech Communication Association",
  address="Kos Island",
  doi="10.21437/Interspeech.2024-2187",
  issn="1990-9772",
  url="https://www.isca-archive.org/interspeech_2024/prasad24_interspeech.html"
}

Dokumenty

prasad_2024_interspeech