This is Chapter 1 of the BDH Explainer, Pathway’s guide to the Dragon Hatchling architecture. BDH is a Post-Transformer language model that turns attention into synaptic memory. This chapter explains what motivated its design: where the Transformer architecture falls short, what natural networks do differently, and the design requirements BDH had to satisfy.

1.1 What Transformers can do, and where they fall short

Transformers are the backbone of most AI systems today. They are very good at replicating a wide range of human-generated artifacts, including text, code, and structured analysis. Part of their success was due to:

1. Efficient leverage of current hardware (GPU). In other words, they won the hardware lottery, as it happened that we could run attention and MultiLayer Perceptron (MLP)-based computations very efficiently on GPU.
2. Scaling laws. Pour more data into more parameters and it keeps improving. More precisely, greater model capacity enables better memorization of pretraining data, and their reasoning and instruction-following abilities also seem to improve [Kimi K3].
3. Strong expressiveness. Theoretically, we can show that Transformers become Turing-complete (they’re able to execute any computation) when equipped with chain-of-thought reasoning, which provides a form of short-term memory that they can modify iteratively. Theoretical models can also characterize task-length generalization, as studied through RASP, a formal language for expressing Transformer-compatible algorithms and bounding the attention heads and layers required to implement them [Weiss et al., 2021].

However, while the Transformer has many strengths, failure modes such as hallucinations, brittle reasoning, and limited interpretability still hinder the adoption of the Transformer in safety-critical and regulated industries. We identify three bottlenecks that are strongly tied to the Transformer architecture and justify actively looking for improvements in this direction:

1. The token bottleneck. Chain-of-thought helps by allowing the model to externalize intermediate information in tokens, but it also means that reasoning is routed through language. A Transformer can spend billions of operations per step, but is ultimately forced to compress the result into the next token. In other words, the model’s internal computation is rich, but what it can carry forward from one step to the next through discrete tokens is extremely bandwidth-limited. This is one reason “thinking longer” in token space becomes expensive: the model repeatedly re-derives intermediate structures that it cannot persist internally at high bandwidth.
2. The memory bottleneck. The Transformer has two structurally different memory systems: long-term memory stored in fixed weights, and short-term memory stored as a token-indexed KV cache (the list of past key/value entries). The KV cache can grow with sequence length, which makes long-context inference expensive and pushes systems toward heuristic context-engineering techniques, like summarization, compression, or retrieval. Because these representations differ in kind, there’s no evident pathway for context to consolidate into parameters. Furthermore, the cache’s append-only growth conflicts with what effective memory needs: to weaken or delete low-value information (decay), compress repeated experiences into durable summaries (consolidation), and retain only the most useful items (sparse forgetting).
3. The scientific unknown. The Transformer is naturally implemented as a tensor program rather than a distributed system of local interactions on a sparse graph, so no formal rule connects its microscopic operations to the macroscopic reasoning behaviors we observe. Without that connection, we cannot predict how a model’s stability or failure modes will change when we alter hyperparameters or scale, and each new regime must be tested empirically rather than derived in advance. Moreover, although the Transformer can appear to reason, it does so through mechanisms that differ fundamentally from those of the human brain, relying on dense connectivity, backpropagation, and large-scale statistical learning. As a result, it offers limited insight into the underlying mechanisms of human reasoning.

The Transformer has fueled innovation in AI over the past decade, enabling the development of systems with a strong command of language as well as other modalities like images or audio. Moving forward requires new properties: models that continuously learn, remember, or reason in latent space. With BDH, we innovate at the architecture level by drawing inspiration from the most powerful observable system that encompasses these properties: the brain.

1.2 How artificial neural networks drifted away from the brain

With BDH, we want to reconnect microscopic dynamics, like those of the brain, with macroscopic reasoning. Modern neural networks began by modeling biological neurons but gradually shifted toward dense tensor computations.

One of AI’s earliest debates concerned where intelligence resides: in explicit symbols and rules, or in the learned interactions among many simple units. Symbolists focused on representing and accumulating knowledge; connectionists instead sought systems that could learn, taking their central inspiration from the brain’s networks of neurons.

The canonical abstraction dates to the work of McCulloch and Pitts [McCulloch & Pitts, 1943], which modeled a neuron as a threshold unit that fires when the weighted sum of its inputs exceeds a threshold

Fifteen years later, Rosenblatt introduced the well-known perceptron [Rosenblatt, 1958], adding a learning rule as a crucial component—an idea later extended through backpropagation [Rumelhart, 1986]. At the time, the update rule was framed more directly as a correction based on an error signal:

This update is also closely related to another attempt to model learning from the same period: Donald Hebb’s 1949 book The Organization of Behavior, which gave rise to the principle later known as the Hebbian rule. This rule is summarized as “neurons that fire together, wire together,” meaning that the connection between two neurons strengthens in proportion to their correlated activity.

Over time, the field’s dominant training paradigm shifted toward backpropagation, in which a model calculates an overall error and uses it to update weights throughout the network. Modern deep learning came to be understood as large mathematical operations on dense arrays of numbers rather than as an evolving, sparse communication network. Modern AI accelerators, including GPUs and TPUs, reinforced this shift by favoring structured operations on large rectangular blocks of data.

The Transformer represents the culmination of the shift toward dense, globally coordinated computation. It is highly effective for sequence modeling but lacks the sparse communication, local adaptation, and connection-based memory of biological networks. Recovering these principles could enable models that update continuously, preserve information within their evolving structure, scale through local computation, and make their behavior easier to trace from individual interactions.

Quoting the BDH paper:

BDH’s motivation is to reopen that connection.

1.3 The power of scale-free networks

Defining a brain-inspired model like BDH requires rethinking both the “neural” part of “neural networks” and the “network” part. The brain exhibits interesting graph properties that we also observe in many kinds of networks called “scale-free networks”. Informally, scale-free networks are defined as networks in which most nodes have few connections, while a small number of highly connected hubs have many. In particular, the scale-free property makes transportation and utility systems (airline routes, road networks, communication networks, water and waste systems) very efficient and scalable.

Network science describes a broad family of such systems as scale-free or power-law-like networks. The point is not that every system follows a perfect power law under strict statistical tests, but that many exhibit four important structural properties.

1. Sparse connectivity. The number of edges grows much more slowly than the number of possible pairwise connections. A large network can therefore operate without paying the cost of linking every node to every other node.
2. Short navigable paths. Despite sparsity, any two nodes reach each other in a few hops. Airline networks are the intuitive example: there is no direct flight between most cities, yet a trip usually takes one or two layovers rather than ten.
3. Highly unequal node degrees. There is no single “typical” node. A few hubs concentrate connectivity while many peripheral nodes remain lightly connected. This heterogeneity is what supports short paths in sparse networks. The term scale-free comes from exactly this, since the typical degree of a node is not well defined.
4. Local growth. These networks grow and optimize through local decisions rather than global redesign. Locality enables decentralization: parts of the network can improve without forcing full synchronization.

These properties are crucial for scaling architectures because a system whose computation and optimization stay local can grow without a central bottleneck. Locality also makes systems easier to analyze, optimize, and shard, since the “units” do not require global coordination at every step.

1.4 The brain as nature’s scale-free network

The brain is an instance of a large, sparse information transport network. With roughly $10^{11}$ neurons and $10^{15}$ synapses, it contains far fewer connections than a densely connected system of comparable size. A stack of a hundred dense layers at the same neuron count would require on the order of $10^{18}$ connections. Instead, a typical neuron connects to about 10,000 others, not to every neuron in the brain.

Two principles from this picture guide BDH directly:

1. Connections encode function and memory. Neurons perform transient computation, but continuity comes from the network: which neurons connect, and how synapses change. Synaptic plasticity provides a mechanism for memory at multiple timescales; activations are transient, but connections evolve. The fruit fly connectome illustrates how much of the brain’s function comes from wiring alone. Using a simple neuron model, built largely from measured connectivity and neurotransmitter identity, researchers can simulate a fly that reproduces recognizable behaviors such as walking, grooming, and feeding responses. This indicates that computation is encoded in the topology of the network and not only the individual units.
2. Computation is local. A neuron integrates incoming signals and determines whether or not to fire, based on a threshold. A synapse changes based on activity at its endpoints, following the Hebbian rule that neurons that fire together wire together. The update depends primarily on what happens at that edge, not on a globally propagated error signal.

1.5 The design requirements for a brain-inspired model

Choosing the brain as the source of inspiration for BDH is a methodological choice. We do not claim that AI must necessarily reproduce the brain. Rather, we believe that the brain remains the strongest evidence that efficient, general intelligence can emerge by establishing proper particle dynamics under severe constraints on energy, locality, and communication.

What we take from it are the two principles above: that connections carry function and memory, and that computation stays local. The difficulty is preserving them on hardware built for dense tensor operations. BDH’s internal organization admits a neuron-synapse interpretation and local dynamics, as well as a tensor-program interpretation to run efficiently on GPU.

That produces six design requirements:

1. Neurons as identifiable computational units. The model should have units that behave like neurons in a concrete sense, meaning identifiable coordinates in the computation.
2. Synapses as identifiable connections. There should be explicit or recoverable edges whose strengths carry meaning and whose updates have a plausible local interpretation.
3. Network-centric memory. Memory should live in the network, not as a separate, ever-growing list of token-indexed traces.
4. State tracking on synapses. The time-varying state should be carried by synapses (or synapse-like objects), aligning short-term adaptation with the same structures that carry long-term learned parameters.
5. Local computation. The model’s core operations should be interpretable as local message passing with global behavior emerging from local interactions.
6. Practical implementation on modern hardware. The conceptual model favors sparse graphs while GPUs favor dense tensor operations. In our case, the architecture must preserve graph interpretability without explicitly materializing a gigantic sparse adjacency matrix.

1.6 Why synaptic memory is the right memory architecture

For an AI system, memory is a trade-off. It needs enough short-term memory to follow the current conversation or task, but it also needs long-term knowledge that remains useful without storing every past detail forever.

Today’s leading language models handle short-term memory by growing a KV-cache. This can give them strong recall but inference becomes slower, more costly, and harder to manage as the conversation gets longer. Older recurrent models had the opposite problem: they kept a compact internal summary, which was efficient but had very limited capacity.

Denoting $N$ the dimensionality of the activations, we observe two very different regimes in RNN and softmax-attention Transformers, also justifying the success of the latter:

The $O(N^2)$ parameter count comes from dense matrix computations (e.g., MLP) whereas RNNs only maintain a single $N$-dimensional vector as latent states. Although the Transformer also relies on dense matrix computations (hence the $O(N^2)$), short-term memory is stored in a KV cache that grows as $O(LC)$, $C$ being the latent dimension of the model and $L$ the sequence length. This gives the model very high recall accuracy but it becomes costly and inefficient as input length grows.

With BDH, we want to strike a balance between long-context performance and token generation complexity. We aim at aligning the structure of the parameters and that of the states, obtaining an optimal ratio of parameter and state dimensionality:

As we will see later, $D$ is the neuronal dimension, across which we compute inter-neuronal connectivity, or how neurons communicate with each other, with $N \gg D > \log(N)$ (typically $D=256$).

Because parameters and state now share the same structure, namely synaptic connections, moving information from short-term reasoning and memory dynamics (“fast weights”) toward long-term structure (“slow weights”) is easier to conceive. N.B.: This is outside the scope of both the 2025 BDH paper and this blog.

BDH processes information through two types of memory working in parallel:

• Dynamic memory updates with every interaction – every time two connected neurons activate, that connection strengthens.
• Fixed long-term memory provides the foundational rules established during training.

In the next chapter, we derive from the attention equations a system that integrates both forms of memory while adhering to our design principles.