Advanced automation support developed in Wave 1 of SESAR IR includes using of automatic speech recognition (ASR) to reduce the amount of manual data inputs by air-traffic controllers. Evaluation of controllers feedback has been subdued due to the limited recognition performance of the commercial of the shell ASR engines that were used, even in laboratory conditions. The reasons for the unsatisfactory conclusions include e.g. inability to distinguish controllers accents, deviations from standard phraseology and limited real-time recognition performance. Past exploratory research funded project MALORCA, however, has shown (on restricted use-cases) that satisfactory performance can be reached with novel datadriven machine learning approaches. Based on the results of MALORCA HAAWAII project aims to research and develop a reliable, error resilient and adaptable solution to automatically transcribe voice commands issued by both air-traffic controllers and pilots. The project will build on very large collection of data, organized with a minimum expert effort to develop a new set of models for complex environments of Icelandic en-route and London TMA. HAAWAII aims to perform proof-of-concept trials in challenging environments, i.e. to be directly connected with real-life data from ops room. As pilot read-back error detection is the main application, HAAWAII aims to significantly enhance the validity of the speech recognition models. The proposed work goes far beyond the work planned for the Wave 2 IR programme and will improve both safety and reduce controllers workload. The digitization of controller and pilot voice utterances can be used for a wide variety of safety and performance related benefits including, but not limiting to pre-fill entries into electronic flight strips and CPDLC messages. Another application demonstrated during proof-of-concept will be to objectively estimate controllers workload utilising digitized voice recordings of the complex London TMA.

Description in Czech
Cílem projektu je výzkum a vývoj spolehlivého, proti chybám odolného a přizpůsobitelného řešení pro automatický přepis hlasových příkazů vyřčených jak pracovníky řízení letového provozu, tak piloty.

Keywords
Artificial Intelligence , Machine Learning, Air-Traffic Control, Natural Language Processing, Automatic Speech Recognition, 

Key words in Czech
umělá inteligence, strojové učení, řízení letového provozu, zpracování přirozeného jazyka, automatické rozpoznávání řeči

H2020-SESAR-2019-2

English

Smrž Pavel, doc. RNDr., Ph.D. - principal person responsible
Doležal Jan, Ing. - fellow researcher
Dytrych Jaroslav, Ing., Ph.D. - fellow researcher
Hradiš Michal, Ing., Ph.D. - fellow researcher
Jírovec Martin, Ing. - fellow researcher
Musil Martin, Ing., Ph.D. - fellow researcher
Otrusina Lubomír, Ing. - fellow researcher
Veselý Karel, Ing., Ph.D. - fellow researcher

Department of Computer Graphics and Multimedia
- responsible department (28.1.2020 - not assigned)
Knowledge Technology Research Group
- internal (28.1.2020 - 30.11.2022)
Department of Computer Graphics and Multimedia
- beneficiary (28.1.2020 - 30.11.2022)

PRASAD, A.; ZULUAGA-GOMEZ, J.; MOTLÍČEK, P.; SARFJOO, S.; NIGMATULINA, I.; OHNEISER, O.; HELMKE, H. Grammar Based Speaker Role Identification for Air Traffic Control Speech Recognition. Proceedings of the 12th SESAR Innovation Days. Budapest: 2022. p. 1-9.
Detail

KLEINERT, M.; HELMKE, H.; SHETTY, S.; OHNEISER, O.; EHR, H.; PRASAD, A.; MOTLÍČEK, P.; HARFMANN, J. Automated Interpretation of Air Traffic Control Communication: The Journey from Spoken Words to a Deeper Understanding of the Meaning. In Proceedings of DASC 2021. San Antonio, Texas: Institute of Electrical and Electronics Engineers, 2021. p. 1-9. ISBN: 978-1-6654-3420-1.
Detail

MOTLÍČEK, P.; PRASAD, A.; NIGMATULINA, I.; HELMKE, H.; OHNEISER, O.; KLEINERT, M. Automatic Speech Analysis Framework for ATC Communication in HAAWAII. In SESAR Innovation Days. SESAR Innovation Days. Seville: SESAR Joint Undertaking, 2023. no. 11, p. 1-9. ISSN: 0770-1268.
Detail

PRASAD, A.; ZULUAGA-GOMEZ, J.; MOTLÍČEK, P.; SARFJOO, S.; NIGMATULINA, I.; VESELÝ, K. Speech and Natural Language Processing Technologies for Pseudo-Pilot Simulator. Proceedings of the 12th SESAR Innovation Days. Budapest: 2022. p. 1-9.
Detail

ZULUAGA-GOMEZ, J.; SARFJOO, S.; PRASAD, A.; NIGMATULINA, I.; MOTLÍČEK, P.; ONDŘEJ, K.; OHNEISER, O.; HELMKE, H. BERTraffic: BERT-based Joint Speaker Role and Speaker Change Detection for Air Traffic Control Communications. In IEEE Spoken Language Technology Workshop, SLT 2022 - Proceedings. Doha: IEEE Signal Processing Society, 2023. p. 633-640. ISBN: 978-1-6654-7189-3.
Detail

HELMKE, H.; ONDŘEJ, K.; SHETTY, S.; KLEINERT, M.; OHNEISER, O.; EHR, H.; ZULUAGA-GOMEZ, J.; SMRŽ, P. Readback Error Detection by Automatic Speech Recognition and Understanding - Results of HAAWAII project for Isavia's Enroute Airspace. SESAR Innovation Days 2022. Budapest: 2022. p. 1-9.
Detail

NIGMATULINA, I.; ZULUAGA-GOMEZ, J.; PRASAD, A.; SARFJOO, S.; MOTLÍČEK, P. A Two-Step Approach to Leverage Contextual Data: Speech Recognition in Air-Traffic Communications. In ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings. Singapore: IEEE Signal Processing Society, 2022. p. 6282-6286. ISBN: 978-1-6654-0540-9.
Detail

HELMKE, H.; SHETTY, S.; KLEINERT, M.; OHNEISER, O.; EHR, H.; MOTLÍČEK, P.; PRASAD, A.; WINDISCH, C. Measuring Speech Recognition And Understanding Performance in Air Traffic Control Domain Beyond Word Error Rates. In Proceedings of 11th SESAR Innovation Days 2021. Belgie: 2021. p. 1-8.
Detail

HELMKE, H.; KLEINERT, M.; SHETTY, S.; OHNEISER, O.; EHR, H.; PRASAD, A.; MOTLÍČEK, P.; VESELÝ, K.; ONDŘEJ, K.; SMRŽ, P.; HARFMANN, J.; WINDISCH, C. Readback Error Detection by Automatic Speech Recognition to Increase ATM Safety. In Proceedings of ATM Seminar. on-line: EUROPEAN ORGANISATION FOR THE SAFETY OF AIR NAVIGATION, 2021. p. 1-10.
Detail

KOCOUR, M.; ŽMOLÍKOVÁ, K.; ONDEL YANG, L.; ŠVEC, J.; DELCROIX, M.; OCHIAI, T.; BURGET, L.; ČERNOCKÝ, J. Revisiting joint decoding based multi-talker speech recognition with DNN acoustic model. In Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH. Proceedings of Interspeech. Incheon: International Speech Communication Association, 2022. no. 9, p. 4955-4959. ISSN: 1990-9772.
Detail

ZULUAGA-GOMEZ, J.; NIGMATULINA, I.; PRASAD, A.; MOTLÍČEK, P.; VESELÝ, K.; KOCOUR, M.; SZŐKE, I. Contextual Semi-Supervised Learning: An Approach to Leverage Air-Surveillance and Untranscribed ATC Data in ASR Systems. In Proceedings Interspeech 2021. Proceedings of Interspeech. Brno: International Speech Communication Association, 2021. no. 8, p. 3296-3300. ISSN: 1990-9772.
Detail

KOCOUR, M.; VESELÝ, K.; BLATT, A.; ZULUAGA-GOMEZ, J.; SZŐKE, I.; ČERNOCKÝ, J.; KLAKOW, D.; MOTLÍČEK, P. Boosting of Contextual Information in ASR for Air-Traffic Call-Sign Recognition. In Proceedings Interspeech 2021. Proceedings of Interspeech. Brno: International Speech Communication Association, 2021. no. 8, p. 3301-3305. ISSN: 1990-9772.
Detail

SZŐKE, I.; KESIRAJU, S.; NOVOTNÝ, O.; KOCOUR, M.; VESELÝ, K.; ČERNOCKÝ, J. Detecting English Speech in the Air Traffic Control Voice Communication. In Proceedings Interspeech 2021. Proceedings of Interspeech. Brno: International Speech Communication Association, 2021. no. 8, p. 3286-3290. ISSN: 1990-9772.
Detail

Responsibility: Smrž Pavel, doc. RNDr., Ph.D.