Investigation of Speaker Representation for Target-Speaker Speech Processing

Investigation of Speaker Representation for Target-Speaker Speech Processing

Paper in proceedings outside WoS and Scopus

Target-speaker speech processing (TS) tasks, such as target-speaker automatic speech recognition (TS-ASR), target speech extraction (TSE), and personal voice activity detection (p-VAD), are important for extracting information about a desired speaker's speech even when it is corrupted by interfering speakers. While most studies have focused on training schemes or system architectures for each specific task, the auxiliary network for embedding target-speaker cues has not been investigated comprehensively in a unified cross- task evaluation. Therefore, this paper aims to address a fundamental question: what is the preferred speaker embedding for TS tasks? To this end, for the TS-ASR, TSE, and p-VAD tasks, we compare pre-trained speaker encoders (i.e., self-supervised or speaker recog- nition models) that compute speaker embeddings from pre-recorded enrollment speech of the target speaker with ideal speaker embed- dings derived directly from the target speaker's identity in the form of a one-hot vector. To further understand the properties of ideal speaker embedding, we optimize it using a gradient-based approach to improve performance on the TS task. Our analysis reveals that speaker verification performance is somewhat unrelated to TS task performances, the one-hot vector outperforms enrollment-based ones, and the optimal embedding depends on the input mixture.

Target-speaker speech processing (TS) tasks, such as target-speaker automatic speech recognition (TS-ASR), target speech extraction (TSE), and personal voice activity detection (p-VAD), are important for extracting information about a desired speaker's speech even when it is corrupted by interfering speakers. While most studies have focused on training schemes or system architectures for each specific task, the auxiliary network for embedding target-speaker cues has not been investigated comprehensively in a unified cross- task evaluation. Therefore, this paper aims to address a fundamental question: what is the preferred speaker embedding for TS tasks? To this end, for the TS-ASR, TSE, and p-VAD tasks, we compare pre-trained speaker encoders (i.e., self-supervised or speaker recog- nition models) that compute speaker embeddings from pre-recorded enrollment speech of the target speaker with ideal speaker embed- dings derived directly from the target speaker's identity in the form of a one-hot vector. To further understand the properties of ideal speaker embedding, we optimize it using a gradient-based approach to improve performance on the TS task. Our analysis reveals that speaker verification performance is somewhat unrelated to TS task performances, the one-hot vector outperforms enrollment-based ones, and the optimal embedding depends on the input mixture.

peaker representation, target-speaker automatic speech recognition, target speech extraction, personal voice activity detection, self-supervised learning

peaker representation, target-speaker automatic speech recognition, target speech extraction, personal voice activity detection, self-supervised learning

ASHIHARA, T.; MORIYA, T.; HORIGUCHI, S.; PENG, J.; OCHIAI, T.; DELCROIX, M.; MATSUURA, K.; SATO, H.

IEEE Signal Processing Society

Macao

979-8-3503-9225-8

Proc. 2024 IEEE Spoken Language Technology Workshop (SLT)

423

430

8

https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10832160

ashihara_slt2024_Investigation_of_Speaker_Representation_for_Target-Speaker_Speech_Processing