Latest stories

Google Research researchers Ali Behrouz and Vahab Mirrokni introduced a continual-learning “sleep framework” for large language models that alternates active interaction with offline consolidation. During the offline phase, the system unlocks pre-allocated Mixture-of-Experts memory blocks and updates numerical weights using self-generated practice examples, while retaining the model’s fixed architecture and human-defined training rules. The framework is designed to move short-term contextual information into long-term parameters, addressing fixed knowledge cutoffs, limited context retention, and catastrophic forgetting from sequential parameter updates. The framework has two stages: memory consolidation and dreaming. Consolidation uses knowledge seeding, on-policy distillation, semantic rewards, and parameter expansion to preserve selected information without overwriting prior knowledge. Dreaming uses random expert activation and reward-based selection of synthetic samples to diversify and prioritize learning signals. Google researchers reported near-zero performance loss for continual translation in previously unseen languages including Manchu and Kalamang, where ICL, Cartridges, and supervised fine-tuning showed severe forgetting. They also reported near-perfect scaling to 10 million tokens on BABILong, ahead of GPT-4 and Titans; 48.9% single-task and 46.2% continued-pretraining results for factual incorporation on SQuAD; and reasoning scores of 79.2%, 69.0%, and 46.1% on AIME 24, AIME 25, and HMMT 25, respectively.