This is Chapter 2 of the BDH Explainer. The previous chapter set out six design requirements, and argued that short- and long-term memory should be structurally aligned and parts of the same fabric, rather than separating a growing KV cache and fixed parameters. To show how BDH works, this chapter derives the architecture directly from the attention equation.

Overview: Deriving BDH from the attention equation

Standard Transformer attention treats memory as a sequence of key-value pairs. BDH reorganizes those pairs into a fixed-size, high-dimensional matrix, gives both axes of that matrix a neuron interpretation, and reads the result as an evolving graph of synaptic connections.

The derivation comes in two parts. Part 2.1 builds BDH as an idealized graph, the version with a clean neuron and synapse reading. Part 2.2 turns it into BDH-GPU, the practical form we train.

Symbols and definitions used in this chapter

The table below collects the symbols used throughout this chapter.

2.1 BDH: attention as an evolving graph of synapses

Step 1. Attention as a memory mechanism

In our quest for the graph hidden in the LLM, we start with the attention mechanism. Attention mixes current token information with past context, stored as keys $k_{\tau,l}$ and values $v_{\tau,l}$, which are queried using a query $q_{t,l}$. Softmax attention matches the current query to past keys with $\tau \leq t$, normalizes the matching scores, and uses them to compute a weighted average of past values:

Typically $\operatorname{sim}(q,k)=\exp(qk^{\top})$ (omitting the scale factor), and queries and keys have a fairly small dimensionality, often a few hundred. That is a compressed space rather than a space of neurons.

Step 2. From a nonlinear similarity to a high dimensional space

It is interesting to consider a separable similarity function. For readers with a machine learning background, this is the inverse of the kernel trick in SVMs. If the similarity admits a feature-map representation, we can replace it with a dot product in a highly dimensional space:

We are looking for a graph over neurons, so a highly dimensional activation space is welcome, because it means that our system has many neurons and can exhibit desirable network properties. Assuming keys and queries already live in $\mathbb{R}^N$, we do not need to raise their dimension further with $\phi$:

where $Z$ is a normalizing constant. Since the system applies another normalization such as layernorm, the normalization by $Z$ is spurious and we omit it from further derivations, remembering to add layernorms.

One caution matters here. This is not the same as swapping softmax attention for linear attention in a small space and changing nothing else, otherwise we would lose capacity. The construction becomes meaningful only because the space is large, sparse, non-negative, and interpretable as neuron activity.

Step 3. Identifying the neurons

Let us simplify further by assuming the system has no separate queries and keys, and instead has $N$ units serving both roles. Substituting $k_{t,l}=q_{t,l}=x_{t,l}$:

The coordinates of $x_{t,l}$ are now our neurons. Read concretely, keys and queries are the current and past activity patterns of the network, while the values store what happened next, so each memory entry is a before and after snapshot of the system.

Step 4. Identifying the synapses

To have a graph we also need a transition matrix. Observe that $\sum_{\tau\leq t}x_{\tau,l}^{\top}v_{\tau,l}$ is a matrix, since every term $x_{\tau,l}^{\top}v_{\tau,l}$ is an outer product forming an $N\times D_v$ matrix, with $D_v$ the dimensionality of the values. Setting $D_v=N$ and defining the synaptic state gives attention written as a graph:

The matrix $\sigma\in\mathbb{R}^{N\times N}$ is large, and a practical system will need a smart way to compress it, which is the subject of 2.2. Conceptually, both axes now enumerate neurons, so $\sigma$ is a connectivity matrix, and every token adds a small sparse outer product update to it. Because a connection is strengthened when the before and after activations co-occur, this update has exactly the form of a Hebbian-like outer-product write.

Step 5. What the equations do, seen as a graph

Assume for a moment that all $x$ and $v$ vectors are non-negative. Then $\sigma$ also has only non-negative entries. We can read $x_{t,l}$ as an unnormalized probability distribution over $N$ neurons, so each neuron carries some probability mass, and the multiplication $x_{t,l}\sigma_{t,l}$ redistributes the mass of each neuron over its neighbors in proportions set by $\sigma$.

This is a local operation, since every neuron sends its mass to its neighbors in the graph given by $\sigma$, and a new activation vector is formed by neurons integrating all incoming masses and optionally thresholding them. The behavior is very similar to simple models of real neurons, where an activated neuron sends electrical charge to the neurons connected to it via synapses, and those neurons integrate the charge they receive and may in turn become active.

Step 6. From attention to a graph-powered LLM

So far we have defined how attention works. We now add the elements that turn it into an LLM, namely two transition matrices $G_x\in\mathbb{R}^{N\times N}$ and $G_y\in\mathbb{R}^{N\times N}$, used to transform activations in a preceding layer into keys $x$ and values $v$ in the next layer:

An interpretation of what is happening: $y_{t,l-1}$ is a distribution over neurons, and since neurons are sparsely activated they are specialized, in the sense of monosemanticity. We propagate this through $G_x$ to gather everything that resonates with these concepts from what the model knows. We then use this for a lookup into the context, propagate it again through $G_y$, and finally constrain it to $x$. The process works like a fuzzy beam search: start with a beam, expand it through the graph, then contract it.

Filling in the blanks with ReLUs and positional information:

Here, $U\in\mathbb{R}^{N\times N}$ is a diagonal or block-diagonal operator that advances the synaptic state by one time step. A diagonal $U$ can damp older information, as in an ALiBi-like decay, while $2\times2$ rotation blocks can encode relative position in a RoPE-like way.

The same treatment applies to the MLP, which can also be read as a graph traversal. Both attention and the MLP therefore become transitions from the space of all neurons into the space of all neurons, so the model has two graphs rather than one, which is still much closer to a network view than a Transformer where the network is not visible at all.

2.2 BDH-GPU: making the graph practical

We have talked previously about how the brain has about $10^{11}$ neurons and $10^{15}$ synapses that connect them, so like any other efficient scale-free network it is a rather sparsely connected graph. There are two ways to achieve the same ratio between the number of neurons, meaning the activation vector dimensionality, and the number of connections, meaning the parameters if we treat $G_x$ and $G_y$ as trainable weights:

1. Directly assume sparse connectivity.
2. Approximate sparse connections with low-rank decompositions and thresholding.

The first strategy leads to a model optimal for brain-like hardware, which thrives on sparse connections. Current deep-learning hardware prefers operating on dense arrays, which favors the second approach. In section 5 of the accompanying paper we formally analyze graphs whose connection matrices are ReLU-thresholded low-rank factorizations.

Intuitively, low-rank matrices are dense, since the outer product of two vectors of all ones yields a matrix filled with ones, and sparse matrices can be full rank, since the identity is very sparse and full rank at the same time. The product of two random vectors, however, gives a matrix with random entries, and thresholding those entries at some level produces matrices with the desired sparsity.

Let us replace transition matrices $G_x$ and $G_y$ by their rank-$D$ factorization:

Using this factorization we never need to materialize the large matrices, because we simply do two chained multiplications:

Notice how $y(E_xD_x)$ requires materializing an $N\times N$ matrix in memory, while $(yE_x)D_x$ first down-projects the $y$ vector into $D$ dimensions and then up-projects back into $N$, saving on memory and compute. Moreover, we have experimentally validated that we can simplify even further, since we only need one $E$ matrix:

This allows us to write a GPU-friendly model:

Defining $\rho_{t,l}=\sigma_{t,l}E$ and simplifying, we finally obtain:

Four equations, three parameter matrices shared across layers, and a rectangular state. The attention state has the shape of a parameter matrix, so it keeps what we could call the platonic idea of being a graph while only the factorized form is ever stored. Part of that structure is model parameters and part of it is short-term storage.

Where this leaves us

Two views of the same computation are now in hand. One is a GPU-friendly tensor program that can be trained at scale. The other is a conceptual graph machine whose nodes and edges can be interpreted as neurons and synapses, with local propagation and synaptic state updates. Low-rank factorization with thresholding is the bridge between them, which means the graph is never materialized in memory yet remains recoverable for analysis.

We summarize what has changed between BDH and the Transformer in the table below:

The next chapter puts this construction to the test: what actually emerges inside BDH models once they are trained.