Mohamed El Khoudimi · Sequence modeling research

Hybrid sequence models

How Much Attention Do You Actually Need?

I am measuring when a hybrid model genuinely needs softmax attention, and when the recurrent update matters more.

Modern sequence models increasingly mix a small number of attention layers into larger stacks of recurrent or state-space layers. That saves compute, but it leaves a basic question unanswered: which capabilities actually need those expensive attention layers? I built a controlled experiment to measure the response curve rather than assume one ratio works for everything.

Variable
attention mixers / all sequence mixers
Ratios
0, 1/8, 1/4, 1/2, 1
Families
Mamba-2 · Gated DeltaNet · negative-eigenvalue control
Probes
retrieval · copy · needle · state tracking · induction

Research status: the full capability sweep has not run yet. The infrastructure is complete and tested; CPU pilots are measured and reported below as pilots, not final results.

What the pilots already changed Source and results Experiment design
01 The question

Attention is powerful. It is also the part that gets more expensive as context grows.

A recurrent or state-space layer compresses the past into a running state. An attention layer can look back at individual tokens directly. Hybrid models combine the two: most layers move information recurrently, while occasional attention layers restore direct content lookup.

The useful quantity is not “does attention help?” It is the shape of capability versus attention ratio: does performance improve gradually, jump after one attention layer, or remain flat because the recurrent update already solves the task?

Interactive architecture sketch · not a result

Sixteen sequence mixers, with attention inserted at a chosen ratio

Change the ratio and placement. The expensive layers are the dark ones; the research asks what capability is lost when I remove them.

Attention ratio
Placement

4 attention layers + 12 recurrent layers · r = 1/4

softmax attention recurrent / state-space mixer
02 The design

I separate five capabilities instead of collapsing everything into one language-model loss.

The main sweep compares matched hybrid stacks across two recurrent families and five attention ratios. Parameters are matched within 3% across families and ratios. Learning rate is selected on a validation split; every reported capability number comes from a separate test split.

RetrievalMulti-query associative recall

Can the model retrieve the value associated with a key as the number of bindings grows?

CopyExact copying

Can information survive over a gap without drift or interference?

Long rangeNeedle in a haystack

Can a sparse target be recovered among distractors as context grows?

StateFinite-group tracking

Can the update rule maintain an exact evolving state beyond the training length?

InductionIn-context bigram copying

When does a repeated pattern become usable, and does one attention layer suffice?

What would count as a threshold

I only report a critical ratio if a logistic transition beats linear and constant alternatives strongly enough: the logistic must beat linear by ΔAIC > 4 in at least 80% of bootstrap resamples. With only five coarse ratios, any transition is then refined with 16-layer points rather than treated as exact.

I also test the simpler competing explanation directly: maybe attention layer count, not ratio, is what matters. One attention layer in an 8-layer model and one in a 16-layer model separate those hypotheses.

1Match models

same parameter budget, different sequence mixer

2Tune fairly

validation-only learning-rate selection per cell

3Replicate

three training seeds are the unit of replication

4Test separately

disjoint examples and deterministic generators

5Refine only where needed

add ratios inside a detected transition

03 Pilot findings

The pilots have not answered the main question. They have already killed two tempting stories.

These runs use one seed and small CPU-trainable models. I use them to debug the experiment, not to claim the final attention ratio. Two lessons are strong enough to change the design because they expose confounds that would otherwise produce believable wrong conclusions.

Pilot 01 · MQAR · one seed

An apparent Mamba-2 failure disappeared when I widened the learning-rate grid.

My first pass made pure Mamba-2 look dramatically worse than attention and Gated DeltaNet. That result was not architectural. The grid stopped too low.

At P = 8, pure Mamba-2 moved from 0.264 token accuracy in the first table to 0.984 after selecting LR 3e-2. Pure attention preferred 3e-3 and collapsed at 1e-2. The models wanted learning rates roughly an order of magnitude apart.

Final pilot reading: at L = 32 and P ≤ 8, every architecture reaches at least 0.973 after tuning. This regime is too easy to measure attention need.

Measured MQAR pilot token accuracy across architectures and attention ratios.
Measured pilot figure from the repository. One seed; test split. The experiment log preserves the original misleading reading and both learning-rate corrections.

Read the full MQAR experiment log →

Pilot 02 · parity · one seed

On parity, adding attention did not fix length generalization. Changing the recurrent update did.

Models trained at length 64 were evaluated at 64, 128 and 256. Vanilla Mamba-2, vanilla Gated DeltaNet, their hybrids, and pure attention all failed to extrapolate once the sequence extended past the training regime.

A Gated DeltaNet variant allowed negative eigenvalues through β ∈ (0, 2). With zero attention, it reached exact-sequence accuracy 1.00 / 1.00 / 0.996 at lengths 64 / 128 / 256. Its r = 1/2 hybrid reached 1.00 / 1.00 / 1.00.

In this pilot, attention was neither necessary nor sufficient for length-general parity. The recurrent update rule determined the qualitative behavior.

Measured parity pilot showing the first error position across model families and test lengths.
Measured pilot figure. One seed, four layers, d = 64. The project does not treat this as a universal claim about transformers or large language models.

Read the full parity experiment log →

04 Engineering

The experiment is built to make architecture mistakes difficult to hide.

I implemented the sequence mixers in pure PyTorch so the full stack can run on CPU and remain inspectable. The point is not kernel speed; it is having a reference implementation whose behavior I can compare against equations and upstream references before spending GPU compute.

Mamba-2

Chunked implementation checked against token-by-token recurrence and the official minimal SSD reference.

Gated DeltaNet

Chunked form checked against its recurrence, including the negative-eigenvalue state-tracking control.

Matched hybrids

8- and 16-layer schedules with explicit attention placement and parameter matching within 3%.

Deterministic probes

Generators test label correctness, answer leakage, and train/validation/test disjointness.

Experiment registry

Run configs, metrics, compute accounting, selected learning rates, and analysis artifacts are persisted.

Statistical pipeline

Hierarchical bootstrap, threshold model comparison, and guardrails around edge-of-grid hyperparameters.

53registered CPU runs
448.5Mtraining tokens recorded
7.5 hregistered CPU wall time
$0billed compute so far

Totals from PROJECT_STATUS.md and the compute summary.

05 Where it stands

The main answer is still pending, and the site says exactly what remains.

The project is blocked on GPU access for the approved calibration stage. The current hard maximum is $25. No paid GPU compute has been run.

CompleteLiterature verification

Production ratios and nearby prior work checked; novelty claim narrowed where prior sweeps already exist.

CompleteExperiment design

Hypotheses, controls, stopping rules, validity risks, and statistical decision rules are written down.

CompleteReference implementations

Mixers, hybrid stack, probes, training loop, registry, analysis and CI are built.

MeasuredCPU pilots

MQAR and parity pilots exposed optimization and state-transition confounds.

NextGPU calibration

Benchmark throughput, verify trainability, then decide GO / NO-GO for the main ratio sweep.

PendingMain capability curves

No critical attention ratio is claimed until multi-seed GPU results exist.

What would falsify the thesis

If every probe reaches ceiling with the same small number of attention layers, if apparent differences disappear under fair learning-rate tuning or state matching, or if the critical ratio does not move with context or memory load, that is the result.

06 My role

I am treating the repository as a research record, not a benchmark screenshot.

I framed the capability-level question, narrowed the literature claim, implemented and checked the reference models, built the deterministic probes and evaluation pipeline, recorded failed pilot interpretations, widened the hyperparameter search when it was unfair, and changed the experimental design when the pilots exposed confounds.

The repo keeps the original pilot conclusions next to their corrections instead of rewriting history after the fact.

07 Reproduce

Everything on this page points back to the research record.

Testsmake test
CPU smoke sweepmake smoke
CPU pilotmake train-small
Model matchingmake match-report
Compute accountingmake compute-summary

This page deliberately separates measured pilot findings from hypotheses and planned GPU work. No production-model capability claim is inferred from the CPU pilots.