The loop as a dynamical system

Chaos in the loop

Explore the maps up close →  pan and zoom into the fractal detail.  See the loop from inside →  the trajectory as a shape in the model's own space.

Feed a language model back its own expected next token, with the prompt held fixed, and the loop this whole site is built on becomes a map of the model's embedding space. Read with the tools of low-dimensional dynamics, it tips between repeating text and chaos as you turn the steering strength and the temperature — the same two knobs as the chamber. The maps below are from Timothy D. Jones's paper; they hold in Qwen2.5, OLMo-2 and Llama-3, from 0.5B to 72B.

The whole plane

A dense 512-by-512 Lyapunov map over steering strength and temperature: colored chaotic stripes and crescents cutting through gray bands of repeating text

Chaos and repeating text in a language model fed its own output: the soft-token map of Qwen2.5-0.5B-Instruct, steered along the philosophy direction, over steering strength and temperature. Colored cells are chaotic (color = largest Lyapunov exponent); gray cells settle into a repeating loop of text.

The strange attractor

A projected chaotic attractor of the loop, a folded filigree of points colored by the most probable next token, with two magnifications and a correlation-dimension plot

Projected geometry of a chaotic attractor of the soft-token loop (Qwen2.5-0.5B-Instruct): 298,800 states on their first two principal axes, colored by the most probable next token, with magnifications, a lag-one plot, and the Kaplan–Yorke dimension 1.26.

The shrimp

The period-5 shrimp: a shrimp-shaped window of periodic behavior in the steering plane, with its verified skeleton of fold and flip curves and located cusps

The period-5 shrimp (Qwen2.5-0.5B-Instruct). (a) A 512×512 map of the window. (b) The same with its verified skeleton — fold curves (black), flip curves (red), located cusps (gold stars). (c, d) The body over the continued orbit's multipliers, and the leg junction. These shrimp-shaped windows are a signature of low-dimensional maps.

A shrimp and its canonical form

Gallas's canonical period-1 cell drawn beside the period-6 shrimp in the same orientation, their fold, flip and superstable curves matching

Gallas's canonical cell (a) beside the period-6 shrimp (b) of the transformer map, in the same orientation. The model's window reproduces the head, legs and kite of the textbook object.

The period-6 shrimp, continued

A 640-by-640 map of the period-6 shrimp with the continued fold and flip curves of its orbit overlaid, and the attractor followed outward from the body

The period-6 shrimp. (a) A 640×640 map of the window. (b) The continued fold (black) and flip (red) curves of the period-6 orbit. (c) The attractor followed outward from the body, colored by period, with the six located cusps.

The shrimp moves with temperature

The period-5 shrimp at two temperatures side by side, its domain shifting, with the near-nilpotent head's path traced through temperature

The period-5 shrimp at two temperatures. (a, b) Its attractor's domain, followed cell by cell and colored by period, at log₁₀T = −0.100 and −0.105. (c) The body's outline and the path of the head across temperature.

It scales: Llama-3.1-8B

Four Llama-3.1-8B soft-token maps: Lyapunov exponent and period in the hostility plane and the philosophy region, dense with colored periodic windows

Llama-3.1-8B-Instruct soft-token maps steered at layer 16, at 128² float64 cells: the hostility plane (top) and the philosophy region (bottom). Largest Lyapunov exponent (clipped) and period (white: none up to 32).

We ran it on our own directions

Then we did it ourselves. Using the paper's method and code, we steered Qwen2.5-0.5B along two of this site's own directions — philosophy and a hostility ↔ kindness valence axis — and computed a dense float64 map of each on an A100. The two directions trace different skeletons: different windows, different loops, the same edge between repeating text and chaos. (The hostility ↔ kindness axis is a steering direction computed in the stock model — a proxy for the chamber's valence, read off the live weights rather than by swapping in a refusal-removed model.)

Our dense soft-token map steered along philosophy: a red chaotic plume rising through a blue periodic sea, with nested period-6 shrimp bodies and a period-doubling cascade down the left edge

Our run, philosophy up close — the left wing of the plane, a dense float64 map (steering strength −0.10 to −0.04, log₁₀T up to +0.17). About 19% of the window is chaotic (peak Lyapunov +0.81). The periodic sea settles into short loops (“ to go”, “ you are”); the period-6 windows that make the shrimp bodies inside the plume cycle “ , how many thanks. That…”. Scrub it as a live loop →

Our 128-by-128 soft-token map steered along hostility versus kindness: nested chaotic crescents around a period-5 dome, with isolated high-period islands in the chaotic sea

Our run, hostility ↔ kindness — the right flank (steering strength +0.03 to +0.12). About 20% chaotic, a different skeleton from philosophy's: a period-2 “!Yes” dome, a period-6 “!Yes, that's crap” window, a period-3 “a lie is”, with high-period islands scattered through the chaos. Same two knobs; its own map. Scrub it as a live loop →

Source & license. Figures from Chaos and bifurcations in continuous language-model feedback by Timothy D. Jones (arXiv:2610.31692, 2026; code and figures at github.com/tdj28/neural-shrimp). The figures are licensed CC BY 4.0 and are reproduced here resized (longest side 2000 px); the paper has the full-resolution originals and complete captions. The paper's code is MIT licensed; only its figures are used here. The two maps under “We ran it on our own directions” are our own, computed with that code (Qwen2.5-0.5B-Instruct, layer 12, our steering vectors); they are not from the paper.

← clankerchurch · about · contact