A 752 KB model that remembers, and still fails its gate: three versions of a fast-weight memory
17 November 2026
A decision model in this project reads one frame of a game, answers, and forgets the frame. I wanted to know whether a memory could change that under a megabyte, and whether letting the model "sleep" on what it has seen improves it. The third of three versions remembers, measurably, and still, rightly, fails the test I wrote for it.
The verdict: failed
s6-sleep3-nano failed its pre-registered gate. Its memory is used: on its probe's test episodes it answers 0.4892 of the questions right, against 0.2440 for the same training without a memory. But sleep adds nothing measurable (+0.0030), and the model plays two of its four games worse with it. The file is published as a result, not as a part to use.
The context: skycmd
skycmd gathers very small "decision models" with ternary weights (each is −1, 0 or +1), written in Rust, run natively or in the browser through WebAssembly, and trained on one MacBook (Apple M5) in a braked mode, so budgets count working minutes.
Pre-registration and the gate
Before an experiment runs, I commit the test, the bar to clear and what a failure would mean. The bar is the "gate": conditions that must all hold. The test is read once, a failure is published like a success, and the rule cannot be adjusted afterwards.
Two more terms. A control arm is the same training without the thing tested, here the memory, so any difference belongs to the memory. Episodes come from numbered seeds: dev seeds are read to take decisions, test seeds once, at the end.
The idea: a memory, and time to sleep on it
s6-nano, the base model, answers each frame alone. The paper Do Language Models Need Sleep? (arXiv 2605.26099) gives a model a fixed-size memory of fast weights: a small matrix the model writes into as it reads and reads back later, unlike its ordinary weights, which change only in training. It also lets the model rerun its layers over what it just read while no answer is asked; there, accuracy rises with the number of passes.
Does that hold for a ternary model of 1.6 million parameters reading game frames? A sleep pass, here: every 8 frames, the model replays them N − 1 more times, writing to its memory, with no decision asked. N = 1 means no sleep.
The probe: a question with a known answer
The model's four games might never need more than one frame, and then a memory would show nothing. So the work began with a probe: a game built to measure one thing, with a known right answer at every frame.
It is called recall. An episode has 64 frames and three doors. On a show frame a door gets one of four colours, visible on that frame only. Later an ask frame names a door; the right answer is its last colour, and nothing on that frame says it. Other frames are answered wait. The delay between show and question falls in four buckets, from 1–4 frames to 17–32. Without memory, a model is right one time in four.
v1: a memory that was never used
The first pre-registration began with a pilot at the small pico size: three memory runs, trained three ways, and a control, 3,684 steps each. The memory had to beat the control by at least 0.10 on dev episodes. They scored 0.2460, 0.2565 and 0.2430, the control 0.2650: chance, and unchanged at one, two or four passes. No larger run took place.
A diagnosis read the code and the saved runs. The memory was wired correctly, and every memory weight had moved. But the memory runs' training loss matched the control's within ±0.004, where solving recall was worth about 0.069, and each model answered nearly the same colour every time: two memory runs were both right on only 15 of 4,000 questions, where independent guessers would share about 250. The memory was trained, and ignored.
v2: stopped by its own checks, before any GPU time
The second version changed how the memory starts and is first trained, and added checks to pass on the CPU, on training episodes only, before any GPU minute.
C1 took three amendments in one day, its decoder having proved biased. C2, the colour held at the start, read 0.390 against a bar of 0.40 I had set by judgement; I went on, as a stated deviation, and let the next check decide.
C4 decided: 300 CPU training steps with the memory, and 300 with its read forced to zero. The loss on question frames had to reach 1.30 or less, and 0.05 below the zeroed run. Three allowed attempts: 1.3867 against 1.3873, 1.3876 against 1.3879, 1.3871 against 1.3873, all at chance (ln 4, about 1.386). The memory's gradient fell from 2.8e-2 to about 1e-11 by step 52.
v2 ended there, with no GPU run. That is a result too: training did not learn to use the memory, and no GPU time went into those odds.
v3: hold the gates, and give the memory its own reason
A second diagnosis found why. The write gate, a learned number setting how strongly a frame is written, read an unnormalised state and was trained ten times faster than the base: its mean fell from 0.96 at step 7 to 3e-7 at step 35. A closed gate writes nothing and gets no gradient to reopen. Held open, the gradient lived, yet the loss stayed at chance. The rest of the model was not the obstacle: with the colour written into the question frame itself, the base learned to answer in about thirty steps.
A literature review of 43 cited sources agreed. Published fast-weight memories do not start with a learned write gate that can close. And a network that can answer without its memory tends to ignore it; one counter-measure is an auxiliary loss asking the memory to give back what it stored.
The third version is v2 with two changes:
- The gates are held, as constants: write strength 0.5, decay 0.9975 per frame.
- An auxiliary recall loss, in training only. At each question, a linear head reads what the memory retrieves and must name the colour. The head is not exported: at inference, no help.
A last CPU check, C5, passed on two seeds: 1.0000 and 0.9997 of the questions right, 0.2386 and 0.2628 with the memory off; its bars were set after the diagnosis, so it only showed the script worked. Then the pico pilot, on fresh dev episodes: 1.0000 against 0.2542 for the control, and 0.2567 with the memory zeroed.
A run stopped by the memory guard
The first nano run stopped at step 203: the machine's memory guard, which halts a training above 13 GB, found it at 16 GB on a 16 GB laptop. The fix cut each batch of 16 sequences into parts of 2 and summed their gradients before the single update: the same computation, up to the order of floating-point sums. The run restarted from scratch with the same seed, and all six runs, memory and control at seeds 0, 1 and 2, went through in the night of 16 to 17 November.
The test, read once
The pre-registered rule chose N = 1 on dev episodes (0.4823, against 0.4911 at N = 2: within 0.01 of the best). The test: 1,000 recall episodes, the four games over 200 episodes each, and typed_decisions_bench, the language test, averaged over three seeds. Brackets are 95% bootstrap intervals: where the difference most likely lies, estimated by resampling the episodes 10,000 times.
| Hypothesis | Measured | Bar | Verdict |
|---|---|---|---|
H0, the memory is used: recall at N = 1 minus the control | +0.2452 [+0.2370, +0.2532] (0.4892 against 0.2440) | ≥ +0.10 | holds |
H1, sleep helps: recall at N = 4 minus N = 1 | +0.0030 [−0.0010, +0.0068] | lower bound above 0 | not shown |
| H2, the games do not lose against the control | Flappy 15.87 against 71.05; Dodge 5.89 against 7.94; Rescue and the Doom-like within 0.1% | no game more than 2% under | not shown |
| H3, the language tasks do not lose | 0.4897 against 0.4808 | at most 0.01 under | holds |
recall, by delay since the show | 1–4 frames | 5–8 | 9–16 | 17–32 |
|---|---|---|---|---|
| with memory, N = 1 | 0.6147 | 0.5590 | 0.4530 | 0.3302 |
| with memory, N = 4 (sleep) | 0.6290 | 0.5910 | 0.4620 | 0.2867 |
| control, no memory | 0.2420 | 0.2398 | 0.2518 | 0.2423 |
recall, three seeds pooled, by frames since the door was shown (axis from zero; dashed line: chance). The memory, in colour, is well above the control at short delays and near chance at 17 to 32 frames; four sleep passes gain a little up to 16 frames and lose beyond.The exported file also had to pass s6-nano's own gate. Its language part holds (0.499 on typed_decisions_bench, bar 0.398; calibration error 0.0578, bar 0.08). Its arena, against the earlier s3-pico, does not: Flappy 2.10 against 200.00, Dodge 6.90 against 9.47. The shipped engine matches the training code, natively and in WebAssembly.
What the failure says
- The memory is real, and it fades: 0.6147 a few frames after the show, 0.3302 after 17 to 32.
- Sleep did nothing measurable: +0.0030, an interval that contains zero; at the longest delays four passes score under one.
- The memory cost two games. The control itself plays Flappy at 71.05, where its base reached 200 in another reader; this training may lower Flappy on its own, but that is not measured.
- The pilot did not carry over: 1.0000 at the pico size, 0.4892 at nano. On record, the nano runs' first phase had at most 233
recallsteps, the pilot's 526; whether that is the cause is not measured. Note added on 19 November 2026: a fourth version, s6-sleep v4, tried to answer that on training sequences and is closed. Its first check, K3 (the base model frozen), failed on both seeds; its second, K1 (everything trained, morerecallsteps), met its bar on one seed of two (0.8442 and 1.0000 against 0.95 on both, read on training batches), so v4 stopped there with no model and no test read (docs/s6-sleep4-prereg.md§16). That narrows the question without settling it. - One probe, one size, three seeds. A synthetic probe says nothing about real tasks, nor about the paper, whose memory spans tokens, not frames.
What I take from it
A probe with a known answer and a control arm tell you whether a mechanism is used at all. Without recall, v1's unused memory would have vanished in the game scores; without the control and the zeroed reading, the pilot's 1.0000 could have been a better guess.
Spend CPU minutes before GPU hours. v2's checks stopped a pilot before it started; v3's diagnosis found the closing gate on training episodes, without reading one dev or test seed.
Write down what a failure means before the run. v1's pre-registration says it in one sentence: "a model that remembers but does not gain from sleeping is not s6-sleep-nano." Committed in advance, it stops "the memory works" from being reported as "the model works". The gate asked for a memory, a gain from sleep and intact games; it got the first.
Publish the failure in a form people can run, so anyone can watch the memory work and fade, as the secrets verifier was published with its failed gate.
Try it
The Memory page plays recall with the 752 KB file and its 715 KB control, memory on, zeroed, or the control, at one, two or four sleep passes, next to the test reading.
Check it
Everything is public: the pre-registrations of v1, v2 and v3, each with its amendments and result; the diagnoses after v1 and v2, v2's design study and the literature review; the test report; the model page; the paper, arXiv 2605.26099; all the results, the models and the code.