Preprint · 2026

History Replays Itself: Unleashing Long-Context Capability in Linear Attention

Li Nanbo^†, Firas Laakom^†, Yilan Zhang, Piotr Piękos, Yimeng Chen, Liangyu Wang, Yifan Xu, Mateusz Ostaszewski, Dandan Guo, Wenyi Wang, Yanning Dai, Yuhui Wang, Jürgen Schmidhuber

† Equal contribution

Project

NoteBank separates recurrent computation from long-term storage. It saves selected original text before future questions are known, retrieves relevant notes using the backbone’s native features, and replays the evidence through the same frozen model. No external embedding model, separately trained retriever, or separate neural reader is required.

Illustrative demo

This fictional, scripted example follows three suspect interviews and scene records through six stages: interviews, scene, question, retrieval, replay, and answer. Use Pause, Restart, the stage buttons, or the timeline to explore the animation. On small screens, swipe horizontally within the diagram.

Results at a glance

At one million tokens, NoteBank raises Gated DeltaNet’s mean accuracy across six RULER retrieval tasks from 1.7% to 70.3%. Its added analytical FLOPs are ≤0.025%, while persistent context storage is approximately 2,600× smaller than a full multi-head-attention KV cache at matched width, depth, and precision. These are analytical compute and storage comparisons, not latency measurements; storage excludes model weights and workspace.

Experiments also extend to Mamba-2-8B and post-trained Qwen3.5-9B, with gains on LongBench v1/v2 and MemoryAgentBench.

See the NoteBank project page for the method, detailed results, and citation.