Finite-State Controllers for (Hidden-Model) POMDPs using Deep Reinforcement Learning

Finite-State Controllers for (Hidden-Model) POMDPs using Deep Reinforcement Learning

Stať ve sborníku v databázi WoS či Scopus

Solving partially observable Markov decision processes (POMDPs) requires computing policies under imperfect state information. Despite recent advances, the scalability of existing POMDP solvers remains limited. Moreover, many settings require a policy that is robust across multiple POMDPs, further aggravating the scalability issue. We propose the Lexpop framework for POMDP solving. Lexpop (1) employs deep reinforcement learning to train a neural policy, represented by a recurrent neural network, and (2) constructs a finite-state controller mimicking the neural policy through efficient extraction methods. Crucially, unlike neural policies, such controllers can be formally evaluated, providing performance guarantees. We extend Lexpop to compute robust policies for hidden-model POMDPs (HM-POMDPs), which describe finite sets of POMDPs. We associate every extracted controller with its worst-case POMDP. Using a set of such POMDPs, we iteratively train a robust neural policy and consequently extract a robust controller. Our experiments show that on problems with large state spaces, Lexpop outperforms state-of-the-art solvers for POMDPs as well as HM-POMDPs.

Solving partially observable Markov decision processes (POMDPs) requires computing policies under imperfect state information. Despite recent advances, the scalability of existing POMDP solvers remains limited. Moreover, many settings require a policy that is robust across multiple POMDPs, further aggravating the scalability issue. We propose the Lexpop framework for POMDP solving. Lexpop (1) employs deep reinforcement learning to train a neural policy, represented by a recurrent neural network, and (2) constructs a finite-state controller mimicking the neural policy through efficient extraction methods. Crucially, unlike neural policies, such controllers can be formally evaluated, providing performance guarantees. We extend Lexpop to compute robust policies for hidden-model POMDPs (HM-POMDPs), which describe finite sets of POMDPs. We associate every extracted controller with its worst-case POMDP. Using a set of such POMDPs, we iteratively train a robust neural policy and consequently extract a robust controller. Our experiments show that on problems with large state spaces, Lexpop outperforms state-of-the-art solvers for POMDPs as well as HM-POMDPs.

POMDPs, Deep Reinforcement Learning, Planning, Finite-State Controllers, Robustness, Model Uncertainty, Synthesis, Verification

POMDPs, Deep Reinforcement Learning, Planning, Finite-State Controllers, Robustness, Model Uncertainty, Synthesis, Verification

HUDÁK, D.; GALESLOOT, M.; TAPPLER, M.; KURECKA, M.; JANSEN, N.; ČEŠKA, M.

01.01.2026

International Foundation for Autonomous Agents and Multiagent Systems

Richland

979-8-4007-2317-9

AAMAS '26: Proceedings of the 25th International Conference on Autonomous Agents and Multiagent Systems

1201

1210

10