Tree Search Distillation utilizes Proximal Policy Optimization (PPO) to enhance language models by integrating a test-time search mechanism similar to that used in game-playing neural networks like AlphaZero. The method aims to distill a stronger, augmented policy back into the language model, addressing the limitations observed in previous attempts with Monte Carlo Tree Search (MCTS).

Chess engines like AlphaZero and lc0 use reinforcement learning by having the engine play itself multiple times to train the model on game outcomes. A combination of a weaker model and strong search capabilities can outperform a stronger model alone, as the search can significantly enhance performance.

Reinforcement learning from human feedback (RLHF) is a key technique for deploying advanced machine learning systems. A new book provides an introduction to the core methods of RLHF for readers with a quantitative background.