Analysis of Multilingual Sequence-to-Sequence Speech Recognition Systems

Analysis of Multilingual Sequence-to-Sequence Speech Recognition Systems

Stať ve sborníku v databázi WoS či Scopus

This paper investigates the applications of various multilingualapproaches developed in conventional deep neural network -hidden Markov model (DNN-HMM) systems to sequence-tosequence(seq2seq) automatic speech recognition (ASR). Weemploy a joint connectionist temporal classification-attentionnetwork as our base model. Our main contribution is separatedinto two parts. First, we investigate the effectiveness ofthe seq2seq model with stacked multilingual bottle-neck featuresobtained from a conventional DNN-HMM system on theBabel multilingual speech corpus. Second, we investigate theeffectiveness of transfer learning from a pre-trained multilingualseq2seq model with and without the target language includedin the original multilingual training data. In this experiment,we also explore various architectures and training strategiesof the multilingual seq2seq model by making use of knowledgeobtained in the DNN-HMM based transfer-learning. Althoughboth approaches significantly improved the performancefrom a monolingual seq2seq baseline, interestingly, we foundthe multilingual bottle-neck features to be superior to multilingualmodels with transfer learning. This finding suggests thatwe can efficiently combine the benefits of the DNN-HMM systemwith the seq2seq system through multilingual bottle-neckfeature techniques.

This paper investigates the applications of various multilingualapproaches developed in conventional deep neural network -hidden Markov model (DNN-HMM) systems to sequence-tosequence(seq2seq) automatic speech recognition (ASR). Weemploy a joint connectionist temporal classification-attentionnetwork as our base model. Our main contribution is separatedinto two parts. First, we investigate the effectiveness ofthe seq2seq model with stacked multilingual bottle-neck featuresobtained from a conventional DNN-HMM system on theBabel multilingual speech corpus. Second, we investigate theeffectiveness of transfer learning from a pre-trained multilingualseq2seq model with and without the target language includedin the original multilingual training data. In this experiment,we also explore various architectures and training strategiesof the multilingual seq2seq model by making use of knowledgeobtained in the DNN-HMM based transfer-learning. Althoughboth approaches significantly improved the performancefrom a monolingual seq2seq baseline, interestingly, we foundthe multilingual bottle-neck features to be superior to multilingualmodels with transfer learning. This finding suggests thatwe can efficiently combine the benefits of the DNN-HMM systemwith the seq2seq system through multilingual bottle-neckfeature techniques.

multilingual ASR, sequence-to-sequence,language-transfer, multilingual bottle-neck feature

multilingual ASR, sequence-to-sequence,language-transfer, multilingual bottle-neck feature

KARAFIÁT, M.; BASKAR, M.; WATANABE, S.; HORI, T.; WIESNER, M.; ČERNOCKÝ, J.

2020

15.09.2019

International Speech Communication Association

Graz

Proceedings of Interspeech

1990-9772

Proceedings of Interspeech

2019

9

Francouzská republika

2220

2224

5

https://www.isca-speech.org/archive/Interspeech_2019/pdfs/2355.pdf

karafiat_is2019_192355