Detail publikačního výsledku

Target Speaker ASR with Whisper

POLOK, A.; KLEMENT, D.; WIESNER, M.; KHUDANPUR, S.; ČERNOCKÝ, J.; BURGET, L.

Originální název

Target Speaker ASR with Whisper

Anglický název

Target Speaker ASR with Whisper

Druh

Stať ve sborníku v databázi WoS či Scopus

Originální abstrakt

We propose a novel approach to enable the use of large, single-speaker ASR models, such as Whisper, for target speaker ASR. The key claim of this method is that it is much easier to model relative differences among speakers by learning to condition on frame-level diarization outputs than to learn the space of all speaker embeddings. We find that adding even a single bias term per diarization output type before the first transformer block can transform single-speaker ASR models into target-speaker ASR models. Our approach also supports speaker-attributed ASR by sequentially generating transcripts for each speaker in a diarization output. This simplified method outperforms baseline speech separation and diarization cascade by 12.9% absolute ORC-WER on the NOTSOFAR-1 dataset.

Anglický abstrakt

We propose a novel approach to enable the use of large, single-speaker ASR models, such as Whisper, for target speaker ASR. The key claim of this method is that it is much easier to model relative differences among speakers by learning to condition on frame-level diarization outputs than to learn the space of all speaker embeddings. We find that adding even a single bias term per diarization output type before the first transformer block can transform single-speaker ASR models into target-speaker ASR models. Our approach also supports speaker-attributed ASR by sequentially generating transcripts for each speaker in a diarization output. This simplified method outperforms baseline speech separation and diarization cascade by 12.9% absolute ORC-WER on the NOTSOFAR-1 dataset.

Klíčová slova

target-speaker ASR, diarization conditioning, multi-speaker ASR, Whisper

Klíčová slova v angličtině

target-speaker ASR, diarization conditioning, multi-speaker ASR, Whisper

Autoři

POLOK, A.; KLEMENT, D.; WIESNER, M.; KHUDANPUR, S.; ČERNOCKÝ, J.; BURGET, L.

Rok RIV

2026

Vydáno

06.05.2025

Nakladatel

IEEE Signal Processing Society

Místo

Hyderabad

ISBN

979-8-3503-6874-1

Kniha

ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings

Strany od

1

Strany do

5

Strany počet

5

URL

https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10887683

BibTex

@inproceedings{BUT198049,
  author="Alexander {Polok} and Dominik {Klement} and Matthew {Wiesner} and Sanjeev {Khudanpur} and Jan {Černocký} and Lukáš {Burget}",
  title="Target Speaker ASR with Whisper",
  booktitle="ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings",
  year="2025",
  pages="1--5",
  publisher="IEEE Signal Processing Society",
  address="Hyderabad",
  doi="10.1109/ICASSP49660.2025.10887683",
  isbn="979-8-3503-6874-1",
  url="https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10887683"
}

Dokumenty

Target_Speaker_ASR_with_Whisper