Google's SLEEP: moving new information from context into model parameters

On October 27, 2026, Google Research presented SLEEP, a research approach to continual learning in language models. The system periodically stops accepting new data and trains on information it has already received. The authors propose using this process to consolidate new knowledge while preserving more of the model's existing abilities.

The October announcement has a longer history. The paper Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories appeared on arXiv on June 2, 2026; an updated version followed on July 10. In a note, the authors state that a version has been available on OpenReview since September 2025. The researchers are Ali Behrouz, Farnoosh Hashemi, Adel Javanmard and Vahab Mirrokni. Google's post introduces a wider audience to existing work; it does not announce a user service with defined access terms.

A language model can use a document supplied with a question. While that text remains in context—the input sequence on which the model bases its answer—its contents can influence the result. Producing that answer usually leaves the model's numerical parameters unchanged. A subsequent, independent conversation therefore does not automatically inherit everything stated in the previous one.

Sequential training on new data can worsen performance on earlier tasks. Researchers call this catastrophic forgetting: measured quality falls sharply after parameter updates. SLEEP addresses two requirements together—acquiring new abilities and preserving those already demonstrated.

SLEEP builds on the earlier approach Nested Learning and the Hope architecture. This architecture describes model memory through components with different update frequencies. Some change often and adapt quickly to incoming data; others receive updates less frequently. Frequency here concerns parameter changes during training. It does not refer to text-generation speed, processor frequency or how long an individual file is stored.

SLEEP adds periodic knowledge transfer between these components. During the active phase, the system receives inputs and performs tasks. During a separate consolidation phase, it uses the state already accumulated to train more stable components. Understanding the cycle requires distinguishing the production of an answer from computations that change the model itself.

To consolidate information, the authors use a mixture of experts, or MoE: a neural-network block containing several computational modules, with a routing mechanism selecting which ones to use. During knowledge transfer, the system activates an additional low-rank expert—a compact set of trainable matrices. Only the new parameters are trained; previously consolidated parameters remain unchanged during that transfer. The sequence is described in detail in Section 3 of the study.

“Expansion” needs clarification. In the mathematical description, the number of available parameters grows. An implementation can allocate them in advance, excluding them from computation and updates until needed. The authors consider this approach to avoid continually changing array sizes in the software. Either way, retaining new information consumes additional parameter capacity. The design does not provide unlimited memory with fixed resources.

Transfer takes place through distillation: one version of the model provides a training signal to another. In SLEEP, the initial version holds temporarily acquired information, while the expanded version gains additional parameters in which to consolidate it. The researchers call this procedure Knowledge Seeding. It involves two computational states of a model; a person need not participate in every transfer.

Training combines two signals. The first compares the versions' probabilities for subsequent tokens—pieces of text—on sequences produced by the version being trained. The second asks it to continue a beginning supplied by the original model and assesses the continuation's similarity to that model's answer. Both meaning and literal sequence similarity are considered. After consolidation, the faster-changing component's low-rank parameters are reset, freeing it for another cycle. Information is retained in the modified numerical parameters, also called weights; the procedure does not create a separate verbatim copy of the original context.

During the Dreaming stage, the model generates training examples, selects useful ones and learns from them. The algorithm description sets out the testable operations behind that idea, including further parameter updates and measurement of the outcome.

When generating synthetic examples, the MoE router additionally selects a random expert. The system then selects some examples using a gradient-related signal, which indicates how an example would change the trainable parameters. Some random examples are retained for diversity. For a selected example, a separate model state is created and fine-tuned, after which performance on the target task is evaluated. Improvement becomes a reward signal for the generation procedure.

“Learning by itself” therefore has a precise, bounded meaning here. Researchers define the architecture, selection procedure, update rule and success criterion in advance. Within those conditions, the system automates the creation and testing of training data. The experiment did not test an ability to independently establish the truth of any new claim. A synthetic example remains a model-generated result, and its usefulness is assessed through the chosen task.

One SLEEP experiment uses BABILong. The test's authors embed information needed to solve tasks within large amounts of unrelated natural text. The evaluation checks whether a model can connect scattered facts, draw an inference, count items or perform another specified operation. Increasing the surrounding text can extend the input sequence.

The SLEEP researchers report nearly perfect Hope performance at lengths of up to 10 million tokens. Their long-context and sequential-learning experiments tested consolidation without the Dreaming stage. However, Appendix B.2 specifies a decisive condition: the small models in this comparison were fine-tuned under BABILong's official protocol. The authors also note a substantial drop in performance for all small models without fine-tuning. Comparing them with large models tested without such training brings together different preparation regimes.

The figure of 10 million describes the length of an input sequence in a particular test. It does not measure how many arbitrary documents a model could reproduce without error after prolonged use. Answering a planted question correctly also differs from checking that every sentence has been retained. The length is measured in tokens; their number depends on the language and how text is segmented, so there is no universal conversion into words.

In the language experiment, the system encounters material in Manchu and Kalamang sequentially, then translates sentences into English. The authors compare learning each language separately with learning both in sequence. Additional consolidation stages help Hope preserve more of its initial performance; the three-stage variant approaches the separately trained result. The experiment tested two languages in sequence; retaining performance over a longer series of updates remains an open question. The conditions appear in Section 4.1.

The same paper reports Qwen3-8B's mathematical results: after SLEEP, it scores 79.2 on AIME-24, 69.0 on AIME-25 and 46.1 on HMMT-25. The original instruction-tuned version scores 73.8, 68.1 and 42.4 respectively. The table is labelled average@16: results are averaged over 16 generations. These figures cannot be directly converted into the probability of a correct answer in an ordinary chat with different questions and settings. They are the authors' measurements on selected tasks and settings; this article has not independently reproduced them.

Separate training has a cost. In the efficiency appendix, the researchers write that conventional supervised fine-tuning on prepared examples with correct answers, or SFT, is four times as efficient as their method at the same number of steps. To reach the same quality on AIME-24, AIME-25 and HMMT-25, however, conventional SFT takes 4.3, 3.6 and 4.8 times as long respectively in their measurements. The comparisons answer different questions: cost at a fixed step count, and time to reach a given result. The paper makes no single promise of acceleration across all tasks.

These materials do not describe a user-facing SLEEP feature that would switch on continual learning for a chat. They present a research cycle with specified rules, additional parameters and separate computation. Its strongest substantive result is retaining task-relevant information after transferring it into weights. The reported experiments leave open how long the cycle can withstand accumulating new, contradictory and erroneous information.