Multilingual Sequence-to-Sequence Speech Recognition: Architecture, Transfer Learning, and Language Modeling

Multilingual Sequence-to-Sequence Speech Recognition: Architecture, Transfer Learning, and Language Modeling

Paper in proceedings (conference paper)

Sequence-to-sequence (seq2seq) approach for low-resourceASR is a relatively new direction in speech research. The approachbenefits by performing model training without using lexicon andalignments. However, this poses a new problem of requiring moredata compared to conventional DNN-HMM systems. In this work,we attempt to use data from 10 BABEL languages to build a multilingualseq2seq model as a prior model, and then port them towards4 other BABEL languages using transfer learning approach. We alsoexplore different architectures for improving the prior multilingualseq2seq model. The paper also discusses the effect of integrating arecurrent neural network language model (RNNLM) with a seq2seqmodel during decoding. Experimental results show that the transferlearning approach from the multilingual model shows substantialgains over monolingual models across all 4 BABEL languages.Incorporating an RNNLM also brings significant improvements interms of %WER, and achieves recognition performance comparableto the models trained with twice more training data.

Sequence-to-sequence (seq2seq) approach for low-resourceASR is a relatively new direction in speech research. The approachbenefits by performing model training without using lexicon andalignments. However, this poses a new problem of requiring moredata compared to conventional DNN-HMM systems. In this work,we attempt to use data from 10 BABEL languages to build a multilingualseq2seq model as a prior model, and then port them towards4 other BABEL languages using transfer learning approach. We alsoexplore different architectures for improving the prior multilingualseq2seq model. The paper also discusses the effect of integrating arecurrent neural network language model (RNNLM) with a seq2seqmodel during decoding. Experimental results show that the transferlearning approach from the multilingual model shows substantialgains over monolingual models across all 4 BABEL languages.Incorporating an RNNLM also brings significant improvements interms of %WER, and achieves recognition performance comparableto the models trained with twice more training data.

Automatic speech recognition (ASR), sequence tosequence, multilingual setup, transfer learning, language modeling

Automatic speech recognition (ASR), sequence tosequence, multilingual setup, transfer learning, language modeling

CHO, J.; BASKAR, M.; LI, R.; WIESNER, M.; MALLIDI, S.; YALTA, N.; KARAFIÁT, M.; WATANABE, S.; HORI, T.

2020

18.12.2018

IEEE Signal Processing Society

Athens

978-1-5386-4334-1

Proceedings of 2018 IEEE WORKSHOP ON SPOKEN LANGUAGE TECHNOLOGY (SLT 2018)

521

527

7

https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=8639655

cho_slt2018_08639655