Probing models like Claude and GPT with curated requests reveals insights into their training processes and knowledge cutoffs. Techniques such as "Incompressible Knowledge Probes" help estimate model parameters and analyze dataset mixtures by measuring token breakdowns.

MegaTrain is a memory-centric system that enables the full precision training of large language models with over 100 billion parameters on a single GPU. It utilizes host memory to store parameters and optimizer states, treating GPUs as transient computation units.

Reinforcement learning from human feedback (RLHF) is a key technique for deploying advanced machine learning systems. A new book provides an introduction to the core methods of RLHF for readers with a quantitative background.