Jointly Trained Transformers Models for Spoken Language Translation

Jointly Trained Transformers Models for Spoken Language Translation

Stať ve sborníku v databázi WoS či Scopus

End-to-End and cascade (ASR-MT) spoken language translation(SLT) systems are reaching comparable performances, however,a large degradation is observed when translating the ASR hypothesisin comparison to using oracle input text. In this work, degradationin performance is reduced by creating an End-to-End differentiablepipeline between the ASR and MT systems. In this work, we trainSLT systems with ASR objective as an auxiliary loss and both thenetworks are connected through the neural hidden representations.This training has an End-to-End differentiable path with respectto the final objective function and utilizes the ASR objective forbetter optimization. This architecture has improved the BLEU scorefrom 41.21 to 44.69. Ensembling the proposed architecture withindependently trained ASR and MT systems further improved theBLEU score from 44.69 to 46.9. All the experiments are reported onEnglish-Portuguese speech translation task using the How2 corpus.The final BLEU score is on-par with the best speech translationsystem on How2 dataset without using any additional training dataand language model and using fewer parameters.

End-to-End and cascade (ASR-MT) spoken language translation(SLT) systems are reaching comparable performances, however,a large degradation is observed when translating the ASR hypothesisin comparison to using oracle input text. In this work, degradationin performance is reduced by creating an End-to-End differentiablepipeline between the ASR and MT systems. In this work, we trainSLT systems with ASR objective as an auxiliary loss and both thenetworks are connected through the neural hidden representations.This training has an End-to-End differentiable path with respectto the final objective function and utilizes the ASR objective forbetter optimization. This architecture has improved the BLEU scorefrom 41.21 to 44.69. Ensembling the proposed architecture withindependently trained ASR and MT systems further improved theBLEU score from 44.69 to 46.9. All the experiments are reported onEnglish-Portuguese speech translation task using the How2 corpus.The final BLEU score is on-par with the best speech translationsystem on How2 dataset without using any additional training dataand language model and using fewer parameters.

Spoken Language Translation, Transformers, Jointtraining, How2 dataset, Auxiliary loss, ASR objective, Coupled decoding, End-to-End differentiable pipeline.

Spoken Language Translation, Transformers, Jointtraining, How2 dataset, Auxiliary loss, ASR objective, Coupled decoding, End-to-End differentiable pipeline.

VYDANA, H.; KARAFIÁT, M.; ŽMOLÍKOVÁ, K.; BURGET, L.; ČERNOCKÝ, J.

2022

06.06.2021

IEEE Signal Processing Society

Toronto, Ontario

978-1-7281-7605-5

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

7513

7517

5

https://www.fit.vut.cz/research/publication/12522/

vydana_icassp2021_09414159