Approaching Dialogue State Tracking via Aligning Speech Encoders and LLMs

Approaching Dialogue State Tracking via Aligning Speech Encoders and LLMs

Stať ve sborníku v databázi WoS či Scopus

In this work, we approach spoken Dialogue State Tracking (DST) by bridging the representation spaces of speech encoders and LLMs via a small connector module, with a focus on fully open-sourced and open-data components (WavLM-large, OLMo). We focus on ablating different aspects of such systems including full/LoRA adapter fine-tuning, the effect of agent turns in the dialogue history, as well as fuzzy matching-based output post-processing, which greatly improves performance of our systems on named entities in the dialogue slot values. We conduct our experiments on the SpokenWOZ dataset, and additionally utilize the Speech-Aware MultiWOZ dataset to augment our training data. Ultimately, our best-performing WavLM + connector + OLMo-1B aligned models achieve state of the art on the SpokenWOZ test set (34.66% JGA), and our system with Gemma-2-9B-instruct further surpasses this result, reaching 42.17% JGA on SpokenWOZ test.

In this work, we approach spoken Dialogue State Tracking (DST) by bridging the representation spaces of speech encoders and LLMs via a small connector module, with a focus on fully open-sourced and open-data components (WavLM-large, OLMo). We focus on ablating different aspects of such systems including full/LoRA adapter fine-tuning, the effect of agent turns in the dialogue history, as well as fuzzy matching-based output post-processing, which greatly improves performance of our systems on named entities in the dialogue slot values. We conduct our experiments on the SpokenWOZ dataset, and additionally utilize the Speech-Aware MultiWOZ dataset to augment our training data. Ultimately, our best-performing WavLM + connector + OLMo-1B aligned models achieve state of the art on the SpokenWOZ test set (34.66% JGA), and our system with Gemma-2-9B-instruct further surpasses this result, reaching 42.17% JGA on SpokenWOZ test.

dialogue state tracking | speech LLMs | task-oriented dialogue

dialogue state tracking | speech LLMs | task-oriented dialogue

SEDLÁČEK, Š.; YUSUF, B.; ŠVEC, J.; HEGDE, P.; KESIRAJU, S.; PLCHOT, O.; ČERNOCKÝ, J.

2026

17.08.2025

International Speech Communication Association

Rotterdam, The Netherlands

Proceedings of the Annual Conference of the International Speech Communication Association Interspeech

Interspeech

Nizozemsko

1748

1752

5

https://www.isca-archive.org/interspeech_2025/sedlacek25_interspeech.pdf

sedlacek_interspeech_2025