Designing and Building Custom Reinforcement Learning Environments for Fine-tuning LLMs - Niels Bantilan, Union.ai Techniques like GRPO, combined with reinforcement learning with verifiable rewards (RLVR), have shown success in fine-tuning LLMs for reasoning tasks with a strong inherent reward, such as math and coding. However, can these techniques be more generally applied to other real-world tasks without a clear reward signal? This session will dive into the design considerations and practical challenges associated with building RL environments for fine-tuning reasoning models for such cases. Using a “Wikipedia Maze” environment as a case study, I’ll demonstrate how to cast reasoning tasks into multi-turn episodic RL environments that have deterministic terminal conditions and clear rewards. I’ll generalize this particular case study into a graph traversal problem that can apply to many tasks, including multi-step agentic tasks. Finally, I’ll present some optimizations that can reduce bottlenecks during the training process, such as using inference frameworks like vLLM to generate trajectories efficiently and exploring off-policy reinforcement learning techniques to generate trajectories from a higher capacity model in order to update the weights of a lower capacity model.