Recurrent Looped Transformer

Yifan Zhang; Jichen Feng; Shihan Qin

Abstract

Recurrent Looped Transformer (RLT-1) feeds the previous token's final decoder state into the current token's decoder input through a gated merge. A causal encoder supplies token representations and global key–value memory, while the decoder maintains local sliding-window attention (SWA) caches. This recurrence extends the computation path with sequence length at a fixed number of block evaluations per token, but requires sequential decoder updates during both training and prompt prefill. We evaluate five eight-layer encoder–decoder splits against a decoder-only Transformer on six algorithmic tasks, using three initialization seeds and shared training and test data. Several RLT-1 splits learn parity earlier, and the 5+3 and 7+1 splits retain 100% accuracy at 256 bits in all three seeds. On swaps-based permutation tracking at length 512, 4+4 reaches 55.70±25.78% final-state accuracy, compared with 0.85±0.30% for the Transformer (mean ± sample standard deviation). The gains depend on the task and depth split: teacher-forced addition accuracy falls beyond trained operand widths, flat modular arithmetic is sensitive to initialization, and standard permutation tracking remains difficult.

Architecture and execution

01 / REASONING

Depth that grows with the sequence.

Each token extends the recurrent path through the full decoder. After \(t\) tokens, that path traverses \(tL_D\) decoder blocks while the per-token block count stays fixed.

02 / HARDWARE

Parallel work around a recurrent core.

Batch known-token encoder work and independent decoder updates. Reuse weights and memory, and checkpoint activations while preserving the reference computation.

03 / RL

One transition from sampling to replay.

Rebuild the full history under current parameters, including prompt states and decoder SWA KV. Keep recorded behavior probabilities tied to the actual sampler.

The complete state matters.

RLT state updates across prompt and response. — The merge reads the previous final output; each SWA layer reads its own cache. Global memory is restricted to the current prefix.

H_t=(s_t,C_t^D),\qquad H_0=(s_\star,\varnothing).

(s_t,C_t^D)=D_\phi\!\left(\operatorname{Merge}(e_t,s_{t-1});M_{\le t},C_{t-1}^D,t\right).

p_\Theta(x_{t+1}\mid x_{1:t})=\operatorname{softmax}\!\left(W_o\operatorname{RMSNorm}_o(s_t)\right)_{x_{t+1}}.

Here \(M_{\le t}\) is global encoder memory, \(s_t\) is the recurrent output, and \(C_t^D\) contains layerwise decoder KV. A SWA window of \(W\) includes the current token and retains at most \(W-1\) historical entries for the next update.

Compatible encoder and decoder attention and FFN weights can be shared. A tied 48+48 layout illustrates this option in the report; the experiments below use untied eight-layer layouts.

Detailed RLT architecture

RLT-0: no hidden-state feedback

RLT-0 removes the previous-output feedback path, learned initial state, state normalization, gated merge and feedback projection. It feeds \(z_t^0=e_t\) directly into the decoder while retaining global cross-attention and per-layer SWA.

Known tokens run in parallel within each decoder layer during training and prefill; generation proceeds one token at a time. The 4+4 control is implemented for matched ablations; no RLT-0 results are included below.

Architecture of RLT-0

RLT without hidden-state feedback: direct encoder input, shared global KV and layerwise SWA. — Each layer processes known positions together under causal masks. Layers follow depth order. Vector PDF ↗

RLT-2: chunk-level feedback

RLT-2 is a proposed extension that holds the feedback state fixed within a chunk and updates it from the last decoder output at a complete chunk boundary. Known positions can run together within each decoder layer, using causal SWA and prefix-restricted encoder memory. Chunk boundaries are anchored at BOS and continue across prompt, response and message boundaries; a partial chunk preserves the previous boundary state. With chunk size B = 1, the equations recover RLT-1 at the same weights. The paper specifies the algorithms and execution costs; the reported experiments contain no RLT-2 training or throughput results.

RLT-0, RLT-1 and RLT-2 decoder schedules for known tokens.

Chunk architecture · Schedule PDF

Depth-eight experiments

The September 17, 2026 snapshot compares RLT-1 4+4, 5+3, 6+2, 7+1, 8+0 and Transformer 8 on six algorithmic tasks. All 108 runs completed 2,000 optimizer steps, using initialization seeds 42, 43 and 44 for every task. Training examples and held-out sets are fixed across initializations. All aggregates report mean ± sample standard deviation (n = 3, ddof = 1).

Models use width 512, FFN width 1,365, four attention heads, global batch 512, microbatch 32 and the same AdamW schedule. RLT-1 uses an SWA window of eight, one shared encoder-memory group, feedback scale 0.1 and TBPTT 128, which covers every training sequence and cuts no gradients here. RLT-1 has 26.10–28.73M parameters; Transformer 8 has 25.31M. The 8+0 variant has no decoder blocks but still applies the gated recurrent merge.

Validation accuracy after 2,000 steps

Values are percentages, reported as mean ± sample SD across three initializations. Each run has consumed 1,024,000 training examples. Addition measures teacher-forced answer-token accuracy, including answer formatting and EOS, excluding prompt and padding positions. Parity and mod-5 score the final label; S5 scores the final state.

Task	RLT-1 4+4	RLT-1 5+3	RLT-1 6+2	RLT-1 7+1	RLT-1 8+0	Transformer 8
Addition	100.00±0.00	100.00±0.00	100.00±0.00	100.00±0.00	100.00±0.00	100.00±0.00
Parity	100.00±0.00	100.00±0.00	100.00±0.00	100.00±0.00	98.83±1.92	94.84±3.43
Mod 5, no brackets	45.36±46.43	94.18±7.29	69.62±33.01	70.01±43.15	60.33±35.70	64.02±37.64
Mod 5, brackets	70.53±10.75	74.35±6.71	74.87±3.72	74.78±3.01	75.17±6.78	73.87±9.07
S5, swaps	100.00±0.00	100.00±0.00	100.00±0.00	99.35±0.81	99.61±0.68	99.09±0.23
S5, standard	0.78±0.68	2.47±1.26	2.21±1.48	2.08±0.98	1.82±1.19	0.52±0.23

Validation accuracy on six tasks, mean and sample SD across three initialization seeds.

All curves extend through step 2,000. Bands show sample SD, clipped to the accuracy range. Standard S5 uses a narrower vertical scale. PDF

Learning speed and initialization

At step 500, RLT-1 6+2 reaches 99.44 ± 0.98% parity accuracy, compared with 48.48 ± 0.53% for Transformer 8. All three 6+2 seeds reach 100% by step 600. At step 2,000, splits 4+4 through 7+1 reach 100% in all three seeds, while Transformer 8 reaches 94.84 ± 3.43%.

Parity accuracy at steps 500 and 2,000, with individual seeds, means and sample SD.

Both panels use the same 768 validation examples. They correspond to 256,000 and 1,024,000 training examples per seed. SD whiskers can extend beyond 100%. PDF

Flat mod-5 varies strongly with initialization: RLT-1 5+3 reaches 94.18 ± 7.29%, compared with 64.02 ± 37.64% for Transformer 8. The three 4+4 seeds score 17.58%, 19.53% and 98.96%. Bracketed mod-5 model means are closer, spanning 70.53–75.17% for RLT-1 versus 73.87 ± 9.07% for the Transformer. The generators differ in operator structure and label distribution; parentheses also occupy token positions.

Length generalization

Each run selects its lowest in-distribution validation-loss checkpoint over the full training history, taking the earliest step on ties. Test results do not enter selection. At every task and length, all models and seeds receive the same 256 addition pairs or 1,024 formal-task sequences. The evaluation covers 846 model–seed–length combinations, summarized as 282 three-seed means and sample SDs.

Parity: 5+3 and 7+1 retain 100% accuracy at 256 bits in every seed; Transformer 8 reaches 50.07 ± 1.63%.
Swaps-S5: at 512 operations, 4+4 reaches 55.70 ± 25.78% final-state accuracy and 91.16 ± 6.09% prefix-token accuracy, versus 0.85 ± 0.30% and 9.33 ± 0.11% for Transformer 8.
Addition: 7+1 reaches 68.05 ± 3.64% teacher-forced token accuracy at nine digits per operand. At 32 digits, all model means fall to 14.89–16.84%.
Modular arithmetic: flat mod-5 approaches its 20% uniform reference at length 255; bracketed mod-5 also loses accuracy at longer lengths.

Length generalization on all six tasks with three-seed error bars.

Gray regions mark training lengths. Addition uses teacher-forced answer tokens; other tasks use final labels or states. Error bars show sample SD across initialization seeds. PDF

Accuracy at the longest tested length

All entries are percentages, mean ± sample SD across three initializations. Addition lengths count digits per operand; formal-task lengths count input symbols or operations, excluding boundary markers.

Task (test length)	RLT-1 4+4	RLT-1 5+3	RLT-1 6+2	RLT-1 7+1	RLT-1 8+0	Transformer 8
Addition (32)	15.30±0.44	14.89±2.27	15.74±1.58	15.59±1.57	15.31±0.93	16.84±1.45
Parity (256)	66.76±28.78	100.00±0.00	84.05±27.63	100.00±0.00	68.91±27.39	50.07±1.63
Mod 5, no brackets (255)	18.00±0.39	20.57±0.62	21.42±0.49	19.34±0.54	20.44±0.91	20.35±2.01
Mod 5, brackets (256)	25.20±1.71	25.81±4.34	21.58±1.86	22.04±1.72	22.30±1.13	25.07±2.05
S5, standard (512)	1.24±0.30	1.43±0.60	1.30±0.31	0.88±0.20	0.72±0.31	0.81±0.06
S5, swaps (512)	55.70±25.78	34.86±6.10	22.14±20.95	0.85±0.06	0.85±0.20	0.85±0.30

S5 length generalization under prefix-token, final-state and whole-sequence scoring.

Whole-sequence accuracy requires every prefix prediction to be correct. Every model uses the same 1,024 sequences at each length. Each panel labels its vertical scale. PDF

The preferred encoder–decoder split depends on the task. These comparisons change feedback, attention structure, parameter count and compute together; matched RLT-0 runs are needed to isolate hidden-state feedback. Hardware-throughput and RL performance have not been measured in this study.

Supplementary results

Training loss, individual parity seeds, fixed-length token accuracy, and token-level length generalization provide additional diagnostics. Training loss is unsmoothed and measured before the optimizer update; all six tasks use three seeds.

The paper also reports a separate seed-42 sixteen-layer addition study from September 16, 2026, with four unfinished runs. It evaluates greedy exact-answer accuracy and teacher-forced token accuracy on shared operand pairs. All nine RLT-1 layouts produce zero exact answers at every tested width from nine to 32 digits; Transformer 16 produces 3/256 at nine digits and zero at longer widths. See the sixteen-layer figure and 4+4 versus 8+8 comparison for checkpoint steps and unequal training budgets.

Experiment figures and protocol

Independent community experiments

Earlier state-tracking results contributed by @AradhyeAgarwal use a separate implementation with about 79K parameters, three seeds and training length 32. Evaluation uses 2,048 test programs per task and length, extending to 128 operations.

Community length-extrapolation results

Independent community comparison on parity and five-state transitions across sequence lengths. — Points show seed means and whiskers show seed minima and maxima. RLT reaches 60.8% parity accuracy and 20.7% five-state accuracy at 128 operations; chance levels are 50% and 20%. Parameter and data budgets were matched; FLOPs were not.

Per-length tables and measurement notes

One execution across training and inference

Known tokens can be encoded in a causal batch. Decoder updates still proceed in token order, constructing both recurrent outputs and decoder SWA caches.

Reference execution schedules
Mode	Encoder	Decoder
Prompt prefill	Causal batch	Update complete state through every prompt token.
Generation	Incremental	Sample from the preceding state, then consume each token exactly once.
Pretraining	Causal batch	Full BPTT over all valid next-token targets.
SFT	Causal batch	Assistant-target loss; all context tokens update differentiable state.
RL replay	Rebuild with current weights	Replay the complete history and SWA caches; score each action before consuming it.

Forward consistency and complete gradients are separate requirements. Full BPTT includes paths through recurrent outputs, decoder KV, and encoder memory. Detaching any of these changes the gradient. Parameter updates invalidate old caches for exact current-policy replay.

Behavior log-probabilities must describe the actual sampling distribution. Exact importance sampling additionally requires support coverage. Shared transitions remove structural prompt-boundary mismatch; numerical kernel parity and off-policy estimation remain separate concerns.

For the execution-level distinction, see the prefill–decode kernel mismatch note.

Read the full report ↗

Citation

If you find this work useful, please cite:

@techreport{zhang2026recurrentlooped,
  title  = {Recurrent Looped Transformer},
  author = {Zhang, Yifan and Feng, Jichen and Qin, Shihan},
  year   = {2026},
  month  = sep,
  url    = {https://github.com/yifanzhang-pro/recurrent-looped-tranformer}
}