Approaching Dialogue State Tracking via Aligning Speech Encoders and LLMs

Approaching Dialogue State Tracking via Aligning Speech Encoders and LLMs

Paper in proceedings (conference paper)

In this work, we approach spoken Dialogue State Tracking (DST) by bridging the representation spaces of speech encoders and LLMs via a small connector module, with a focus on fully open-sourced and open-data components (WavLM-large, OLMo). We focus on ablating different aspects of such systems including full/LoRA adapter fine-tuning, the effect of agent turns in the dialogue history, as well as fuzzy matching-based output post-processing, which greatly improves performance of our systems on named entities in the dialogue slot values. We conduct our experiments on the SpokenWOZ dataset, and additionally utilize the Speech-Aware MultiWOZ dataset to augment our training data. Ultimately, our best-performing WavLM + connector + OLMo-1B aligned models achieve state of the art on the SpokenWOZ test set (34.66% JGA), and our system with Gemma-2-9B-instruct further surpasses this result, reaching 42.17% JGA on SpokenWOZ test.

In this work, we approach spoken Dialogue State Tracking (DST) by bridging the representation spaces of speech encoders and LLMs via a small connector module, with a focus on fully open-sourced and open-data components (WavLM-large, OLMo). We focus on ablating different aspects of such systems including full/LoRA adapter fine-tuning, the effect of agent turns in the dialogue history, as well as fuzzy matching-based output post-processing, which greatly improves performance of our systems on named entities in the dialogue slot values. We conduct our experiments on the SpokenWOZ dataset, and additionally utilize the Speech-Aware MultiWOZ dataset to augment our training data. Ultimately, our best-performing WavLM + connector + OLMo-1B aligned models achieve state of the art on the SpokenWOZ test set (34.66% JGA), and our system with Gemma-2-9B-instruct further surpasses this result, reaching 42.17% JGA on SpokenWOZ test.

dialogue state tracking | speech LLMs | task-oriented dialogue

dialogue state tracking | speech LLMs | task-oriented dialogue

SEDLÁČEK, Š.; YUSUF, B.; ŠVEC, J.; HEGDE, P.; KESIRAJU, S.; PLCHOT, O.; ČERNOCKÝ, J.

01.01.2025

International Speech Communication Association

Rotterdam, The Netherlands

Proceedings of the Annual Conference of the International Speech Communication Association Interspeech

Interspeech

Kingdom of the Netherlands

1748

1752

5

https://www.isca-archive.org/interspeech_2025/sedlacek25_interspeech.pdf

sedlacek_interspeech_2025