Pre-trained large language models require post-training to become useful problem solvers, and DeepSeek’s Group Relative Policy Optimization (GRPO) has emerged as the dominant reinforcement learning algorithm for this purpose. However, current GRPO research is primarily focused on larger models, while many practical applications may be better served by smaller specialized models that can run locally on consumer-grade hardware. This thesis investigates how a GRPO-based training pipeline can be modified to work effectively at the small-model scale, using multi-digit multiplication as a controlled environment with cheap, verifiable, binary rewards.
Starting from Qwen2.5-1.5B-Instruct, we train and compare five LoRA-adapted models on a single NVIDIA A100 GPU through Google Colab. Our baseline GRPO run improves substantially over the base model, but suffers from mid-training instability and stagnant performance on harder problems. We then propose and evaluate three modifications: zeroing the KL coefficient on uninformative prompts whose rollout group produces no reward variation, adopting the DAPO modifications proposed by ByteDance, and adding a multi-armed bandit-inspired data sampler that shifts training budget toward problem types at the frontier of the current model’s ability. The first and third modifications produce substantial gains. Our best model reaches 100%, 94%, 92%, 69%, and 80% accuracy on 2×2, 3×3, 4×4, 5×5, and out-of-distribution 6×3 multiplication problems respectively, up from 89%, 30%, 0%, 0%, and 1% for the base model. We then investigate each model’s responses to better understand the strategies they have learned and confirm that the final models are robust to variation in prompt format and therefore are not overfitting.