Reinforcement Learning for Mathematical Reasoning in Small-Scale Language Models with Structured Policy Optimization

Reinforcement Learning for Mathematical Reasoning in Small-Scale Language Models with Structured Policy Optimization

Stať ve sborníku v databázi WoS či Scopus

Advancing mathematical reasoning in small-scale language models remains a challenge due to their limited capacity and the high computational demands of standard reinforcement learning methods like Proximal Policy Optimization (PPO). To address this, a resource-efficient training pipeline based on Group Relative Policy Optimization (GRPO) is proposed, enabling fine-tuning of compact models under strict memory constraints. The proposed method introduces structured prompting to explicitly separate reasoning steps from final answers and applies a dual reward system to jointly optimize for format adherence and mathematical correctness. The training incorporates low-overhead techniques such as 8-bit optimization, mixed-precision training, gradient checkpointing, and accelerated decoding for efficient rollout and policy updates. Experimental results show a 50.95% accuracy on a benchmark reasoning dataset—GSM8K, outperforming several larger models—while training entirely on a single GPU. These findings demonstrate that small-scale models, when trained with structured reinforcement learning, can achieve competitive performance in mathematical reasoning tasks. The approach offers a practical pathway for deploying interpretable, reasoning-capable models in low-resource environments.

Advancing mathematical reasoning in small-scale language models remains a challenge due to their limited capacity and the high computational demands of standard reinforcement learning methods like Proximal Policy Optimization (PPO). To address this, a resource-efficient training pipeline based on Group Relative Policy Optimization (GRPO) is proposed, enabling fine-tuning of compact models under strict memory constraints. The proposed method introduces structured prompting to explicitly separate reasoning steps from final answers and applies a dual reward system to jointly optimize for format adherence and mathematical correctness. The training incorporates low-overhead techniques such as 8-bit optimization, mixed-precision training, gradient checkpointing, and accelerated decoding for efficient rollout and policy updates. Experimental results show a 50.95% accuracy on a benchmark reasoning dataset—GSM8K, outperforming several larger models—while training entirely on a single GPU. These findings demonstrate that small-scale models, when trained with structured reinforcement learning, can achieve competitive performance in mathematical reasoning tasks. The approach offers a practical pathway for deploying interpretable, reasoning-capable models in low-resource environments.

Emotion Flip Reasoning, Therapeutic Dialogue Modelling, Emotional Trajectory Prediction, Transformer

Emotion Flip Reasoning, Therapeutic Dialogue Modelling, Emotional Trajectory Prediction, Transformer

TYAGI, N.; JOSHI, R.; DAS, S.; SIKORA, P.; MYŠKA, V.; DUTTA, M.

2026

05.11.2025

IEEE

Florence, Italy

979-8-3315-7675-2

2025 17th International Congress on Ultra Modern Telecommunications and Control Systems and Workshops (ICUMT)

International Congress on Ultra Modern Telecommunications and Workshops

Spojené státy americké

272

277

6

https://ieeexplore.ieee.org/document/11268643