Soft Value Iteration for Bellman Equations via Maximum Entropy Reinforcement Learning

This work evaluates the effectiveness of entropy-regularized Reinforcement Learning (RL) by contrasting Soft Value Iteration with conventional Bellman-based approaches. Based on the Maximum Entropy principle, our new Soft RL method showcases a clear performance gain in state value estimations. Experimentation within the FrozenLake-v1 environment identifies that Soft RL consistently returns higher and more consistent state values between around 2.5 and 3.5, as opposed to the 0.0 and 1.0 range of Traditional RL. Visualizations of state evaluation using heatmap representations further demonstrate the difference, with Soft RL displaying uniformly higher state assessments, reflecting robustness as well as stability. Difference analysis exemplifies consistent gaps in values among states with peaks of up to 2.9 in key regions, reflecting the higher expressiveness of the entropy-regularized approach. The findings confirm that Soft Value Iteration, based on the maximization of entropy, provides an improved, more robust framework for solving the Bellman equations for reinforcement learning problems.