LLM internals specimen sheet

Jul 18, 2026

DEFAULT_PARAGRAPHS = [
    ("attention_paragraph_01",
     "A token never exists in isolation. Each new token gathers context from "
     "the sequence around it and contributes to a representation larger than "
     "itself. Meaning emerges from these connections as attention transforms "
     "a sequence into a network of relationships. Information can travel "
     "across distant parts of the sequence without relying on recurrence. "
     "The resulting context influences every prediction that follows."),
    ("attention_paragraph_02",
     "Attention determines which parts of the past remain relevant in the "
     "present. Query and key interactions reveal where information should "
     "flow, while softmax transforms attention scores into probabilities. "
     "Every prediction reshapes the landscape for the token that follows. "
     "Some signals become stronger while others fade into the background. "
     "The model continuously updates its understanding as new tokens arrive."),
]

six idioms tested on the measured trace · Qwen3-14B · paragraph 01 · 40L × 40H × 65T · all values measured from attn.npz / hidden.npz · every display transform disclosed per panel

Pattern atlas — the α tensor itself

all 1,600 heads of one forward pass: rows L00→L39 top→bottom, columns h0→h39. each tile is that head’s full 65×65 causal α matrix. ink = α0.35, posterized to 8 levels (transform disclosed; raw α spans ~4 decades). read the tile grammar: a left vertical bar = sink head, a diagonal hairline = local head, scattered ink = content head. the sheet darkens toward the sink column as you read down — the depth story is visible without a single number. sharpest head: L14 h24, mean top-1 1.00.

Head simplex — the population in class space

each of the 1,600 heads is one point in the sink/local/content simplex (its measured class shares, mean over q 16–64); ink density = layer depth (L00 light → L39 dark). The green path is the per-layer population centroid: the whole population migrates toward the sink corner with depth. green ring = sharpest head.

Ridgeline — content share on the position × depth plane

one ridge per layer (L00 top → L39 bottom, front occludes), x = query position, height = mean content share (α beyond sink+local). peaks are where the model genuinely reads context; valleys are bookkeeping. shared amplitude scale, max 0.78 marked.

Twin axis — sink and register are one mechanism

The cross-tensor shot: mean α on t0 from attn.npz (solid, left axis) against |x[t0, dim 731]| from hidden.npz (dashed, right axis, logarithmic scale). Both rise together through depth; the dashed collapse at the right edge is the final RMSNorm. Dimension 731 is the top-|activation| dimension at every sampled depth in both paragraphs.

Stream trajectories — 65 tokens through 41 boundaries

each path is one token’s residual state from embed (gray dot) to post-norm (green dot), unit-normalized then PCA-projected to 2D (16% of cosine variance — a projection, not a measurement; disclosed). t0 drawn heavier: the register carrier travels its own route.

Convergence fan — when the direction settles, and what the norm does to it

cos(x[l,t], x[b39,t]) for all 65 tokens: the stream converges on its own late direction gradually (median ≥0.9 at L36) while ‖x‖ grows ~1164ו — then the final rmsnorm’s per-dim γ re-bases the coordinate system and the median cos drops to 0.36. that re-basing is also why a naive cos-to-post-norm instrument reads flat-then-jump: measured here, it never exceeds 0.24 before the end.

Reading arcs — the mechanism laid over the sentence

content-class attention only (k ≥ 1, k ≤ q−3), grand mean over 40L×40H, top 120 of 555 links drawnᵗ; arc weight and ink = mass. word ink = p(sampled) under teacher forcing — dark words the model found predictable, light ones it did not. the one panel where the tensor meets the text.

Behavioral head map — the simplex without the taxonomy

all 1,600 heads embedded by their measured behavior, not by the 3-class rule: signature = sink share + attention mass in 10 log-spaced offset bins + entropy (mean over q 16–64ᵗ), z-scored, PCA to 2D (85% of variance). ink = depth. the arms label themselves from the data: the head maximizing each signature component is annotated in place. green ring = sharpest head (L14 h24).

edit geometry — what kind of edit each layer makes

for each block write Δxₗ = xₗ₊₁ − xₗ (l = 0…38; L39 unrecoverable post-norm): its cosine to the current state — amplify (>0), rotate (≈0), erase (<0) — as 65 token dots with the median in ink, and median relative magnitude ‖Δx‖/‖x‖ on the log strip below.

Token-cloud geometry — the sentence collapsing into a cone

two measurements of the 65-state cloud per boundary: effective dimensionality (participation ratio of the centered covariance spectrum, log scale) — dashed with all tokens (the register carrier t0 drags it to ~1 by mid-depth), solid with t0 excluded — and pairwise-cosine quartiles: anisotropy rising 0.08 → 0.83 by b39, then relaxing to 0.68 after the final norm.

basis skyline — 5,120 dimensions at b20

every coordinate of the residual basis as one point: max |activation| over the 65 tokens at boundary 20, log scale. a flat city with three towers — the register dimensions the twin-axis panel (D) tracked in depth, seen here across the full basis at one instant.