MultiFlipFormer: A Multimodal Transformer for Emotion Flip Reasoning and Instigator Detection in Therapeutic Conversations

MultiFlipFormer: A Multimodal Transformer for Emotion Flip Reasoning and Instigator Detection in Therapeutic Conversations

Stať ve sborníku v databázi WoS či Scopus

Understanding the dynamics of emotional change within therapeutic conversations is a key challenge in computational mental health. We present MultiFlipFormer, a multimodal transformer-based architecture designed to model emotion flip reasoning and instigator detection. Unlike prior models that focus on static emotional state classification, MultiFlipFormer captures temporal emotional transitions across dialogue turns and identifies multimodal cues responsible for these shifts. The architecture comprises four core components: (1) emotion transition analysis to model evolving emotional trajectories (e.g., sadness → anger → neutral), (2) multimodal instigator detection that integrates visual cues, textual content, and conversational strategies to uncover emotion flip triggers, (3) cross-modal attention networks to fuse text and visual information, and (4) trajectory prediction to anticipate future emotional states. Evaluated on a large-scale MESC dataset of 23,126 therapeutic samples spanning 7 emotions, 41 flip types, 15 scenarios, and 10 intervention strategies, MultiFlipFormer achieves a final weighted F1-score of 0.828 across tasks. This framework provides a clear and reliable approach that can genuinely support therapy and help in planning better interventions.

Understanding the dynamics of emotional change within therapeutic conversations is a key challenge in computational mental health. We present MultiFlipFormer, a multimodal transformer-based architecture designed to model emotion flip reasoning and instigator detection. Unlike prior models that focus on static emotional state classification, MultiFlipFormer captures temporal emotional transitions across dialogue turns and identifies multimodal cues responsible for these shifts. The architecture comprises four core components: (1) emotion transition analysis to model evolving emotional trajectories (e.g., sadness → anger → neutral), (2) multimodal instigator detection that integrates visual cues, textual content, and conversational strategies to uncover emotion flip triggers, (3) cross-modal attention networks to fuse text and visual information, and (4) trajectory prediction to anticipate future emotional states. Evaluated on a large-scale MESC dataset of 23,126 therapeutic samples spanning 7 emotions, 41 flip types, 15 scenarios, and 10 intervention strategies, MultiFlipFormer achieves a final weighted F1-score of 0.828 across tasks. This framework provides a clear and reliable approach that can genuinely support therapy and help in planning better interventions.

Emotion Flip Reasoning, Therapeutic Dialogue Modelling, Emotional Trajectory Prediction, Transformer

Emotion Flip Reasoning, Therapeutic Dialogue Modelling, Emotional Trajectory Prediction, Transformer

SHARMA, A.; FATIMA, N.; SIKORA, P.; MYŠKA, V.; FROLKA, J.; DUTTA, M.

2026

05.11.2025

IEEE

Florence, Italy

979-8-3315-7675-2

2025 17th International Congress on Ultra Modern Telecommunications and Control Systems and Workshops (ICUMT)

International Congress on Ultra Modern Telecommunications and Workshops

Spojené státy americké

187

192

6

https://ieeexplore.ieee.org/document/11268690/keywords#keywords