Continual learning
Latest stories covering Continual learning, with summaries and links to related coverage.
1 stories · 1 reports · page 1 of 1
1 story · 1 report in the last 7 days
Latest report · Counts as of (UTC)
Models in this coverage
No tracked models are linked to this topic’s reports in the last 30 days.
No established topics share multiple stories with this topic yet.
Latest stories
Sleep framework: A new way for LLMs to retain knowledge in continual learning
research
5 min read
Google Research researchers Ali Behrouz and Vahab Mirrokni introduced a continual-learning “sleep framework” for large language models that alternates active interaction with offline consolidation. During the offline phase, the system unlocks pre-allocated Mixture-of-Experts memory blocks and updates numerical weights using self-generated practice examples, while retaining the model’s fixed architecture and human-defined training rules. The framework is designed to move short-term contextual information into long-term parameters, addressing fixed knowledge cutoffs, limited context retention, and catastrophic forgetting from sequential parameter updates. The framework has two stages: memory consolidation and dreaming. Consolidation uses knowledge seeding, on-policy distillation, semantic rewards, and parameter expansion to preserve selected information without overwriting prior knowledge. Dreaming uses random expert activation and reward-based selection of synthetic samples to diversify and prioritize learning signals. Google researchers reported near-zero performance loss for continual translation in previously unseen languages including Manchu and Kalamang, where ICL, Cartridges, and supervised fine-tuning showed severe forgetting. They also reported near-perfect scaling to 10 million tokens on BABILong, ahead of GPT-4 and Titans; 48.9% single-task and 46.2% continued-pretraining results for factual incorporation on SQuAD; and reasoning scores of 79.2%, 69.0%, and 46.1% on AIME 24, AIME 25, and HMMT 25, respectively.