The loop as a dynamical system
Explore the maps up close → pan and zoom into the fractal detail. See the loop from inside → the trajectory as a shape in the model's own space.
Feed a language model back its own expected next token, with the prompt held fixed, and the loop this whole site is built on becomes a map of the model's embedding space. Read with the tools of low-dimensional dynamics, it tips between repeating text and chaos as you turn the steering strength and the temperature — the same two knobs as the chamber. The maps below are from Timothy D. Jones's paper; they hold in Qwen2.5, OLMo-2 and Llama-3, from 0.5B to 72B.
Chaos and repeating text in a language model fed its own output: the soft-token map of Qwen2.5-0.5B-Instruct, steered along the philosophy direction, over steering strength and temperature. Colored cells are chaotic (color = largest Lyapunov exponent); gray cells settle into a repeating loop of text.
Projected geometry of a chaotic attractor of the soft-token loop (Qwen2.5-0.5B-Instruct): 298,800 states on their first two principal axes, colored by the most probable next token, with magnifications, a lag-one plot, and the Kaplan–Yorke dimension 1.26.
The period-5 shrimp (Qwen2.5-0.5B-Instruct). (a) A 512×512 map of the window. (b) The same with its verified skeleton — fold curves (black), flip curves (red), located cusps (gold stars). (c, d) The body over the continued orbit's multipliers, and the leg junction. These shrimp-shaped windows are a signature of low-dimensional maps.
Gallas's canonical cell (a) beside the period-6 shrimp (b) of the transformer map, in the same orientation. The model's window reproduces the head, legs and kite of the textbook object.
The period-6 shrimp. (a) A 640×640 map of the window. (b) The continued fold (black) and flip (red) curves of the period-6 orbit. (c) The attractor followed outward from the body, colored by period, with the six located cusps.
The period-5 shrimp at two temperatures. (a, b) Its attractor's domain, followed cell by cell and colored by period, at log₁₀T = −0.100 and −0.105. (c) The body's outline and the path of the head across temperature.
Llama-3.1-8B-Instruct soft-token maps steered at layer 16, at 128² float64 cells: the hostility plane (top) and the philosophy region (bottom). Largest Lyapunov exponent (clipped) and period (white: none up to 32).
Then we did it ourselves. Using the paper's method and code, we steered Qwen2.5-0.5B along two of this site's own directions — philosophy and a hostility ↔ kindness valence axis — and computed a dense float64 map of each on an A100. The two directions trace different skeletons: different windows, different loops, the same edge between repeating text and chaos. (The hostility ↔ kindness axis is a steering direction computed in the stock model — a proxy for the chamber's valence, read off the live weights rather than by swapping in a refusal-removed model.)
Our run, philosophy up close — the left wing of the plane, a dense float64 map (steering strength −0.10 to −0.04, log₁₀T up to +0.17). About 19% of the window is chaotic (peak Lyapunov +0.81). The periodic sea settles into short loops (“ to go”, “ you are”); the period-6 windows that make the shrimp bodies inside the plume cycle “ , how many thanks. That…”. Scrub it as a live loop →
Our run, hostility ↔ kindness — the right flank (steering strength +0.03 to +0.12). About 20% chaotic, a different skeleton from philosophy's: a period-2 “!Yes” dome, a period-6 “!Yes, that's crap” window, a period-3 “a lie is”, with high-period islands scattered through the chaos. Same two knobs; its own map. Scrub it as a live loop →