Energy Function · Pattern Completion · Capacity Limit
🧠 Middle school → college · Interactive simulation
§ 01
Interactive Simulation
Network state — 100 neurons (■ = +1 firing, □ = −1 silent). Click or drag to paint.
Energy E during recall (it can only go down)
Stored memories — bar = overlap m with the current state
Energy E
—
−½ Σ wijsisj
Best overlap m
—
—
Stored P
0
patterns
Load P/N
0.00
limit ≈ 0.138
Flips
0
sweep 0
Status
—
Try it: store, smudge, recall
Three letters are already stored as memories. Press Add noise to smudge the letter on the grid, then Run recall. Each neuron looks at the weighted votes of the other 99 and flips to agree with the majority — watch the energy fall until the grid snaps back to a clean letter.
Recall
Noise to add25%
Speed4 neurons/frame
Memories
Capacity experiment: how many memories fit in 100 neurons?
For each number of stored random patterns P, the test smudges a stored pattern by 10%, lets the network recall it, and records the final overlap (1 = perfect). The dashed line marks the theoretical limit P = 0.138 N ≈ 14.
§ 02
The Idea, Step by Step
▸ From a smudged photo to a stored memory
You catch a glimpse of a friend across a foggy street and instantly know who it is, even though you only saw half a face. Your memory took a partial, noisy input and "snapped" it to the closest complete thing it already knew. A Hopfield network does exactly this trick with numbers: feed it a corrupted pattern and it settles down into the nearest stored memory. That single behaviour — pattern completion — is the whole point.
START · middle school
Picture a grid of tiny lights, each either ON or OFF — like black-and-white pixels making a face. The network "remembers" a picture by gently wiring the lights together: lights that were ON together get a friendly link, lights that disagreed get a wary one. Show it a messy version with a few wrong pixels and each light politely flips to match what most of its friends are doing. After a few rounds the mess cleans itself up into the saved picture.
BUILD · high school
Give each neuron a value $s_i = +1$ (on) or $-1$ (off). A memory is a pattern $\xi^\mu$. We set the connection strengths by Hebb's rule — "neurons that fire together, wire together": $w_{ij}=\frac{1}{N}\sum_{\mu}\xi_i^\mu\xi_j^\mu$. To recall, each neuron takes a weighted vote of all the others and copies the majority: $s_i \leftarrow \text{sign}\!\left(\sum_j w_{ij}s_j\right)$. Store one 100-pixel pattern, flip 10 pixels, and the votes drag those 10 right back — the memory is restored.
DEEPEN · AP / intro-college
Every configuration has an energy $E=-\tfrac{1}{2}\sum_{i\neq j}w_{ij}s_is_j$. Each allowed update can only lower $E$, like a ball rolling downhill, and with asynchronous updates $E$ is a Lyapunov function — the network is guaranteed to roll to the bottom of a valley and stop. Each stored memory carves out its own valley (an attractor), and the set of starting points that drain into it is its basin. The catch is capacity: you can carve at most $P_{max}\approx 0.138\,N$ stable valleys before crosstalk noise (standard deviation $\sqrt{P/N}$) merges them into useless spurious states.
CLOSE · picture it as the sim runs
Hold these three experiments in mind while watching the simulation above: (1) start from a pattern with a few "pixels" wrong and watch the energy fall step by step until it lands in a valley — that descent is the recall. (2) Keep adding stored patterns past $0.138N$ and the clean valleys collapse into spurious mixtures, so the network "remembers" something that was never stored. (3) Compare a lightly-corrupted start (lands in the right memory) with a heavily-corrupted one (rolls into the wrong basin) — that boundary is exactly how far a memory's basin reaches.
At each update step: ΔE = -(Δs_i)(h_i) ≤ 0 since s_i → sign(h_i). E bounded below → converges to a fixed point (stored pattern or spurious state). Proof: energy is a Lyapunov function for asynchronous (one-neuron-at-a-time) updates; fully synchronous updates can instead settle into a period-2 cycle.
STEP 2 — Capacity
Amit, Gutfreund & Sompolinsky (1987): replica method gives P_max = 0.138N. For P > P_c: spurious mixture states dominate, recall fails. Signal-to-noise ratio at pattern μ: SNR ≈ 1/√(P/N) — the crosstalk noise has standard deviation √(P/N), so recall stays reliable while the load P/N is below α_c ≈ 0.138 and degrades beyond it.
STEP 3 — Modern Hopfield Networks
Ramsauer et al. (2020): modern Hopfield with energy E = -logΣ_μ exp(ξ_μ·s) has exponential capacity P ~ exp(αN). The softmax update rule IS the Transformer attention mechanism. Hopfield & Hinton — Nobel Prize Physics 2024.
▸ Primary References
Hopfield (1982) PNAS 79:2554; Ramsauer et al. (2020) arXiv:2008.02217
🧠 ConceptualWhat is a Hopfield network, in simple terms?▼
A Hopfield network is a group of N model neurons, each either firing (+1) or silent (−1), where every neuron is connected to every other one with a symmetric weight. Memories are not kept in any single neuron: they are written into the whole pattern of weights by Hebb's rule, wij = (1/N) Σμ ξiμξjμ. To recall, you start the network from a partial or noisy cue and let each neuron repeatedly copy the sign of its weighted input. The state slides downhill in energy until it stops in the nearest stored pattern. Because you retrieve a memory from part of its content rather than from an address, it is called content-addressable or associative memory.
Key: Store = Hebb's rule on the weights. Recall = let the state roll downhill in energy to the nearest memory.
🧠 ConceptualWhy does the energy always go down during recall?▼
Define E = −½ Σi≠j wijsisj. When one neuron i updates, it takes the sign of its input field hi = Σj wijsj, and the energy change is ΔE = −Δsi·hi. If the neuron flips, Δsi has the same sign as hi, so ΔE < 0; if it does not flip, ΔE = 0. This needs symmetric weights (wij = wji), no self-connections and one-neuron-at-a-time (asynchronous) updates. Since there are only 2N states, the energy cannot fall forever, so the network must stop in a local minimum. Tick “Synchronous updates” in the simulation to see what happens without the one-at-a-time rule: the network can flip-flop between two states forever.
Key: Asynchronous updates + symmetric weights ⇒ E never increases ⇒ the network always settles.
📐 ComputationalHow many memories can a Hopfield network store?▼
For random patterns and a large network, about 0.138 N patterns (Amit, Gutfreund & Sompolinsky, 1985). Below that load the stored patterns are stable with only about 1–2% of bits wrong; above it, recall collapses for all memories at once. If you demand that every memory be recalled with no wrong bits at all, the limit is much smaller, roughly N/(4 ln N) (McEliece et al., 1987) — about 5 patterns for 100 neurons. The capacity test in the simulation shows both effects: near-perfect recall for a handful of patterns, a gradual decline around 0.138 N ≈ 14, and poor recall beyond. Correlated patterns such as letters of the alphabet interfere much sooner than random ones.
Key: Random patterns: ≈ 0.138 N. Perfect recall of every bit: ≈ N / (4 ln N). Similar patterns: fewer.
💡 Non-ObviousWhat are spurious states and inverse memories?▼
Every stored pattern ξ comes with its mirror image −ξ, which has exactly the same energy because E depends only on products sisj. Start closer to the inverse (more than half the pixels wrong) and the network falls into it. The network also creates false memories it was never taught. The best known are mixture states such as sign(ξ1 + ξ2 + ξ3), a blend of three memories that sits in its own shallow valley. At high load a huge number of glassy spurious minima appear. Try storing the letters and then pressing “Random start” a few times: you will often land in a blend rather than a clean letter.
Key: Energy valleys exist for each memory, its inverse, and for blends of memories the network never saw.
🌍 AppliedHow do Hopfield networks relate to the brain and to modern AI?▼
In the brain, attractor networks are a leading model for memory circuits with strong recurrent connections, especially area CA3 of the hippocampus, where a partial cue is thought to be completed into a full memory. In AI, “modern” Hopfield networks (dense associative memories; Krotov & Hopfield 2016, Ramsauer et al. 2020) use a sharper energy function and can store exponentially many patterns. Their update rule is mathematically the same as the attention step in Transformer models. John Hopfield shared the 2024 Nobel Prize in Physics with Geoffrey Hinton for foundational work on neural networks, including this model.
Key: Attractor memory is a model for hippocampal recall, and modern Hopfield updates equal Transformer attention.
§ 04 Resources
Gerstner et al. — Neuronal Dynamics. Free: neuronaldynamics.epfl.ch
Dayan & Abbott — Theoretical Neuroscience, MIT Press, 2001
Izhikevich — Dynamical Systems in Neuroscience. Free: dynamicalsystems.org
Neuromatch Academy — neuromatch.io (free comp neuro tutorials)
§ 05
Misconceptions & Common Errors
Sub-block A — Conceptual
❌"Each memory is stored in its own neuron or its own connection."
✅Every memory is spread over all N×N weights, and every weight takes part in every memory. That is why the network can recover a memory from any sufficiently large piece of it — and why adding too many memories damages all of them at once.
✅Recall returns the bottom of whichever valley the starting state is in. With a heavy smudge that can be the inverted pattern, and with similar or too many memories it can be a spurious blend. Nothing in the dynamics “knows” which memory you wanted.
❌"More training makes a Hopfield network remember better, like a deep network."
✅There is no training loop. The weights are written in one shot by Hebb's rule (an outer product per pattern). Storing a pattern twice only deepens its valley. The fixed ≈ 0.138 N limit comes from crosstalk between patterns, not from too little training.
❌Updating all neurons at the same time is just a faster way to run recall.
✅The energy guarantee holds for asynchronous (one-at-a-time) updates. With synchronous updates the network can enter a 2-cycle, flipping between two states forever — tick “Synchronous updates” in the simulation to see it.
❌Keeping the self-connections wii or using asymmetric weights does no harm.
✅Both break the proof that energy decreases. Asymmetric weights can produce endless cycles or chaos, and self-connections create extra stable states that are not memories. The standard model sets wii = 0 and wij = wji.