Fine-Tuning Self-Supervised Models for Language Identification Using Orthonormal Constraint

Fine-Tuning Self-Supervised Models for Language Identification Using Orthonormal Constraint

Paper in proceedings (conference paper)

Self-supervised models trained with high linguistic diversity, such as the XLS-R model, can be effectively fine-tuned for the language recognition task. Typically, a back-end classifier followed by statistics pooling layer are added during train- ing. Commonly used back-end classifiers require a large num- ber of parameters to be trained, which is not ideal in limited data conditions. In this work, we explore smaller parame- ter back-ends using factorized Time Delay Neural Network (TDNN-F). The TDNN-F architecture is also integrated into Emphasized Channel Attention, Propagation and Aggregation- TDNN (ECAPA-TDNN) models, termed ECAPA-TDNN-F, reducing the number of parameters by 30 to 50% absolute, with competitive accuracies and no change in minimum cost. The results show that the ECAPA-TDNN-F can be extended to tasks where ECAPA-TDNN is suitable. We also test the effectiveness of a linear classifier and a variant, the Orthonor- mal linear classifier, previously used in x-vector type systems. The models are trained with NIST LRE17 data and evalu- ated on NIST LRE17, LRE22 and the ATCO2 LID datasets. Both linear classifiers outperform conventional back-ends with improvements in accuracy between 0.9% and 9.1%

Self-supervised models trained with high linguistic diversity, such as the XLS-R model, can be effectively fine-tuned for the language recognition task. Typically, a back-end classifier followed by statistics pooling layer are added during train- ing. Commonly used back-end classifiers require a large num- ber of parameters to be trained, which is not ideal in limited data conditions. In this work, we explore smaller parame- ter back-ends using factorized Time Delay Neural Network (TDNN-F). The TDNN-F architecture is also integrated into Emphasized Channel Attention, Propagation and Aggregation- TDNN (ECAPA-TDNN) models, termed ECAPA-TDNN-F, reducing the number of parameters by 30 to 50% absolute, with competitive accuracies and no change in minimum cost. The results show that the ECAPA-TDNN-F can be extended to tasks where ECAPA-TDNN is suitable. We also test the effectiveness of a linear classifier and a variant, the Orthonor- mal linear classifier, previously used in x-vector type systems. The models are trained with NIST LRE17 data and evalu- ated on NIST LRE17, LRE22 and the ATCO2 LID datasets. Both linear classifiers outperform conventional back-ends with improvements in accuracy between 0.9% and 9.1%

Language Identification, Transformers, Wav2Vec2, fine-tuning, low-resource, out-of-domain,

Language Identification, Transformers, Wav2Vec2, fine-tuning, low-resource, out-of-domain,

PRASAD, A.; CAROFILIS, A.; VANDERREYDT, G.; KHALIL, D.; MADIKERI, S.; MOTLÍČEK, P.; SCHUEPBACH, C.

2025

14.04.2024

IEEE Signal Processing Society

Seoul

979-8-3503-4485-1

ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings

11921

11925

5

https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10446751

prasad_icassp2024_fine-tuning