Multi-Channel Speech Separation with Cross-Attention and Beamforming

Multi-Channel Speech Separation with Cross-Attention and Beamforming

Paper in proceedings (conference paper)

Originally, single-channel source separation gained more research interest. It resulted in immense progress. Multichannel (MC) separation comes with new challenges posed by adverse indoor conditions making it an important field of study. We seek to combine promising ideas from the two worlds. First, we build MC models by extending current single-channel time-domain separators relying on their strength. Our approach allows reusing pre-trained models by inserting designed lightweight reference channel attention (RCA) combiner, the only trained module. It comprises two blocks: the former allows attending to different parts of other channels w.r.t. the reference one, and the latter provides an attention-based combination of channels. Second, like many successful MC models, our system incorporates beamforming and allows for the fusion of the network and beamformer outputs. We compare our approach with the SOTA models on the SMS-WSJ dataset and show better or similar performance.

Originally, single-channel source separation gained more research interest. It resulted in immense progress. Multichannel (MC) separation comes with new challenges posed by adverse indoor conditions making it an important field of study. We seek to combine promising ideas from the two worlds. First, we build MC models by extending current single-channel time-domain separators relying on their strength. Our approach allows reusing pre-trained models by inserting designed lightweight reference channel attention (RCA) combiner, the only trained module. It comprises two blocks: the former allows attending to different parts of other channels w.r.t. the reference one, and the latter provides an attention-based combination of channels. Second, like many successful MC models, our system incorporates beamforming and allows for the fusion of the network and beamformer outputs. We compare our approach with the SOTA models on the SMS-WSJ dataset and show better or similar performance.

multi-channel source separation, cross-channel attention, beamforming

multi-channel source separation, cross-channel attention, beamforming

MOŠNER, L.; PLCHOT, O.; PENG, J.; BURGET, L.; ČERNOCKÝ, J.

2024

20.08.2023

International Speech Communication Association

Dublin

Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH

1990-9772

Proceedings of Interspeech

2023

08

French Republic

1693

1697

5

https://www.isca-speech.org/archive/interspeech_2023/mosner23_interspeech.html

mosner23_interspeech2023_multi-channel