This is Chapter 3 of the BDH Explainer. The previous chapter derived BDH equations from attention and set out how it differs from a Transformer. This chapter is empirical: what actually appears inside these models once they are trained. This chapter puts that theoretical picture to the test, showing that the intended structures do indeed emerge in trained models.
3.1 A clear, recoverable graph
BDH behaves like a population of connected, cooperating neurons. The modular structure of the neuron network is not “engineered” into blocks serving different functions but emerges spontaneously during training as shown in the picture below. The network interpretation of BDH, and the dynamics of its neurons during inference, can be analyzed both formally and empirically. We have exhibited a number of properties of how neurons interact with each other (they notably rely on a form of voting mechanism given by the replicator equation, see Definition 1 and the discussion that follows in the BDH paper). The system exhibits a kind of local “winner-take-all” dynamics as strong connections usually get stronger and weak connections get weaker.
While the precise structure of the network depends on the training methods, the neurons appear to form connections resembling those in real-world scale-free networks, with a form of core-periphery structure and fat-tailed degree distributions. Here is a sample image of what neuron connections in a trained model look like. This structure emerges on its own during training, without enforced regularization.
Although BDH-GPU, the tensor formulation of BDH, is trained in a factorized form, its underlying connectivity can be reconstructed and studied. The recovered graph shows the properties the model was designed to develop:
- Power-law degree distribution: The graph is sparse and scale-free, with a small number of highly connected neurons and many neurons with relatively few connections. This structure is associated with efficient signal propagation in networks such as the brain and the World Wide Web.
- Modularity: The network organizes itself into clusters of neurons that are more strongly connected to one another than to the rest of the graph. These modules are learned rather than explicitly designed, allowing specialized components to emerge spontaneously.
- Locality of information processing: Neurons process information received from nearby connected neurons, while synapses update according to the activity at their endpoints. As a result, the information required for computation is stored close to where it is being processed.
3.2 Monosemanticity: the currency synapse
Individual synapses in BDH-GPU encode semantic concepts, not tokens, making interpretability inherent. This monosemanticity (each synapse consistently representing a single, coherent concept rather than an opaque mixture) is statistically verified (), giving proof of interpretability even at small scales.
With the synaptic network encoding what the model remembers and reasons about, we can try to peek at individual synapses to see if they encode meaningful and easily accessible patterns w.r.t. current inputs. As shown below, a trained BDH model has semantically meaningful synapses that give us hints at how the model is reasoning about a given input.
We choose for visualization one specific synapse, which we call the “currency synapse”. Whenever the model sees a word linked to a currency, like dollar, the same synapse lights up (“potentiates”) and we can easily observe it. More intense color means more activity in a given neuron.
The specific synapse pictured reacts to all mentions of currencies, illustrating its monosemanticity. It is active only when the two neurons co-fire, which is Hebbian learning in practice, and it then remembers for a long time that the context is the US dollar as a currency rather than the US as a country.
This makes BDH both performant and interpretable: reasoning in BDH arises from explicit local graph dynamics with well-defined limit behavior, enabling mechanistic interpretability of attention (based on monosemanticity) and reasoning processes rather than post-hoc explanations. By monitoring synapse values in real time, users may be able to identify the concepts and associations most active during a decision, diagnose unexpected behavior, and intervene more precisely when a model begins to follow an unsafe or unhelpful path.
3.3 Neurons get bored
Similarly to the brain, BDH is “lazy” in the good sense. When the incoming data is repetitive or low-signal, the model keeps its internal state largely stable rather than “churning” unnecessarily. This means a more efficient use of context: BDH has no fixed context-window boundary and can process an unbounded sequence, efficiently compressing all that was seen. We can measure its size in information content and not in (perhaps repetitive) token count. As an analogy, a Transformer-based model is a security camera that holds the past 30 minutes, regardless of what happened, while BDH holds the last 300 relevant events. This behavior is well-suited to deal with very long contexts by ignoring noisy, low-value data.
As demonstrated on the image below, BDH-GPU’s neuron activity is sparse and depends on predictability: fewer neurons are active when the input becomes predictable. Sparsity is not just a compression trick but a behavior of the inference dynamics (neurons “quiet down” after warm-up / once the repeating pattern is learned). It also translates to less compute and energy use, analogous to how biological systems such as the brain conserve energy by avoiding unnecessary activity.
3.4 BDH works and scales: Transformer-like performance and scaling
BDH can be trained with an autoregressive objective like any other large language model, with natural support for long context.
We also show below an architecture comparison using a sample problem which originally motivated the introduction of the Transformer: natural language translation. We provide the appropriate scaling curves for the BDH architecture presented in the paper (BDH-GPU), an extended variant developed and used internally by Pathway (BDH-GPU'), and a TransformerXL baseline in a GPT2-architecture (GPTXL). All the models were trained with backpropagation through time on sequences 2048 tokens long, and carry their state for at least 4096 tokens. We observe that BDH-GPU' matches the GPT Transformer.
Transformer scaling typically involves jointly increasing both width (embedding/hidden dimension) and depth (number of layers), and this multi-axis scaling often requires re-tuning optimization and regularization settings at each model size, leading to complex hyperparameter search, and weeks of tuning per model size.
By contrast, BDH models can be scaled largely by changing a single primary parameter - the number of neurons - while holding the rest of the hyperparameters fixed, which makes scaling behavior more controlled and operationally straightforward. This leads to predictable, reproducible scaling, with only days of tuning per model size.
This makes BDH-GPU particularly attractive when optimizing for makespan (end-to-end training time) and when wanting predictable scaling behavior with fewer knobs to tune.
3.5 Sharding, scaling, and gluing models
BDH is easy to shard, to scale, and to combine, and this follows directly from its shape. All the large tensors have shape where is the low-rank technical dimension and stays constant. The model is grown by expanding it along , the neuron dimension. To shard it, we cut along that same dimension, place one shard per GPU, and let the shards exchange only the low-dimensional representation, which keeps most communication local.
BDH’s modularity is very handy and unlocks new possibilities: surgically removing or adding shards, merging models to obtain additional capabilities (e.g. mastery of different languages, see Figure 15 in the BDH paper), flexible distributions of model parameters and states on GPU clusters.
This uniformity is also what makes efficient inference straightforward to write. Because we know which parts of the model talk to each other, there are not several sharding paradigms to reconcile. Parameters and states follow the same layout, so the same kernel handles everything, as illustrated in the figure below.
Moreover, in BDH, important data is located adjacent to the sites where it is being processed, which minimizes communication and alleviates the most painful bottleneck for reasoning models during inference, memory-to-core bandwidth. BDH thus opens the door to high inference speeds on new generations of AI processor hardware.
Once trained, BDH reveals a sparse, modular, scale-free graph whose structure can be recovered and inspected. Individual synapses encode coherent concepts, while neurons quiet down for predictable inputs, making computation both sparse and interpretable. At the system level, Transformer-like scaling and uniform sharding along the neuron dimension make BDH practical to train, distribute, and extend.
Together, these results connect the series’ argument: the design requirements established in Chapter 1 lead to the architecture derived in Chapter 2, and the intended graph, memory, and interpretability properties remain visible after training. The conclusion considers what these results could mean for persistent memory, latent reasoning, continual learning, and the future of Post-Transformer AI.
