Moore's Law distracted us from the real bottleneck: moving data.

For decades, the semiconductor industry told itself a comforting story: make transistors smaller, place more of them on a chip, and computation will become faster, cheaper, and more abundant.

For a long time, that story was mostly true.

Moore’s Law gave engineers more transistors approximately every process generation. Dennard scaling allowed those transistors to switch faster without proportionally increasing power density. Higher clock frequencies improved serial performance, while larger transistor budgets funded deeper pipelines, wider execution engines, larger caches, more sophisticated branch predictors, and eventually more cores.

The visible result was an apparently simple relationship:

$$\text{More transistors} \Rightarrow \text{More performance}$$

But this relationship concealed the physical problem that would eventually dominate computer architecture.

A processor does not merely calculate. It must continuously move operands toward execution units and move results somewhere they can be used again.

Instructions move through pipelines. Data moves between registers, caches, memory controllers, DRAM devices, storage systems, accelerators, and network interfaces. Addresses move through translation structures. Cache lines move through coherence networks. Intermediate values move across buses, crossbars, meshes, and physical wires.

The arithmetic is often the easy part.

The expensive part is getting the operands into the right place at the right time.

Moore’s Law increased the number of places where computation could happen. It did not eliminate the cost of moving information between them.

That is the real bottleneck.

---

Transistors Became Cheap. Distance Did Not.

A transistor is a local device. It performs useful work on signals physically close to it.

A processor, however, is not a collection of isolated transistors. It is a communication system made from transistors connected by wires.

As fabrication technology improved, transistor dimensions shrank dramatically. Local switching became faster and denser. But the physical dimensions of complete chips did not shrink at the same rate. A signal travelling from one side of a processor to the other still had to cross a meaningful physical distance.

For an unbuffered wire, propagation delay is approximately related to its resistance-capacitance product:

$$t_{\text{wire}} \propto RC$$

Since both resistance and capacitance increase with wire length, the delay of a long wire can approximately scale as:

$$t_{\text{wire}} \propto L^2$$

Repeaters can reduce the delay growth, but they consume transistors, routing area, and energy. The wire no longer behaves like a free connection simply because more transistors are available.

This created an architectural inversion.

Originally, logic was expensive and communication was relatively simple. Modern processors increasingly live in a world where logic is abundant but communication is expensive.

An integer addition may require only a small local circuit. Delivering its operands may involve register renaming, dependency tracking, scheduler lookup, register-file access, bypass selection, wire traversal, and clock distribution.

The arithmetic operation is only one event inside a much larger transportation system.

---

The Memory Wall Was a Movement Problem

Processor performance improved much faster than main-memory latency.

This did not mean DRAM stopped improving. Memory capacity increased enormously, and bandwidth improved through wider interfaces, higher transfer rates, multiple channels, prefetching, burst transfers, and parallel banks.

But latency did not fall at the same rate as processor cycle time.

The number of processor cycles lost to one memory access is:

$$N_{\text{cycles}} = t_{\text{memory}} f_{\text{clock}}$$

Consider a hypothetical processor running at $4\,\text{GHz}$ and a memory access taking $100\,\text{ns}$:

$$N_{\text{cycles}} = 100 \times 10^{-9} \times 4 \times 10^9 = 400$$

The exact numbers vary across systems, but the architectural consequence remains the same: a single missing operand can leave an execution pipeline waiting for hundreds of cycles.

Adding another arithmetic unit does not solve this problem.

Doubling the number of multipliers does not help when the multipliers have no operands. Increasing instruction width does not help when independent instructions cannot be found. Adding cores does not help when all of them compete for the same memory channels.

The memory wall is therefore not merely a problem of slow storage. It is a problem of physical separation.

The data exists.

The processor simply cannot bring it close enough, quickly enough, and cheaply enough.

---

Bandwidth and Latency Are Different Constraints

Memory systems are often described using bandwidth alone, but bandwidth and latency limit different behaviours.

A simplified transfer time is:

$$T = T_{\text{latency}} + \frac{D}{B}$$

where:

• $T_{\text{latency}}$ is the startup delay,
• $D$ is the amount of data transferred,
• $B$ is the available bandwidth.

For a large sequential transfer, the bandwidth term dominates. For a small dependent access, latency dominates.

This distinction explains why a system can advertise enormous memory bandwidth while still performing badly on pointer-heavy workloads.

A streaming kernel can issue many independent requests and keep the memory interface busy. A linked-list traversal cannot request the next node until the current node reveals its address.

The first workload exposes transfer capacity.

The second exposes movement dependency.

Bandwidth can be increased through parallelism. Latency is much harder to hide when the address of the next operation depends on data that has not yet arrived.

This is why data structures matter physically.

Two algorithms with similar operation counts can perform very differently because one traverses contiguous arrays while the other chases unpredictable pointers across memory.

From an abstract algorithmic perspective, both may perform $O(n)$ work.

From the machine’s perspective, one is a predictable stream of cache lines. The other is a sequence of compulsory journeys through the memory hierarchy.

---

The Cache Hierarchy Is a Data-Movement Machine

Caches are commonly introduced as small, fast memories placed between the processor and DRAM.

That description is correct but incomplete.

A cache hierarchy is fundamentally a mechanism for avoiding movement.

Its purpose is not merely to store data. Its purpose is to keep useful data physically close to where it will be consumed.

The average memory access time is commonly represented as:

$$T_{\text{avg}} = T_{\text{hit}} + r_{\text{miss}}T_{\text{penalty}}$$

For multiple cache levels, the structure expands:

$$T_{\text{avg}} = T_{L1} + r_{L1} \left( T_{L2} + r_{L2} \left( T_{L3} + r_{L3}T_{\text{memory}} \right) \right)$$

The important term is not only the latency of each level. It is the miss rate multiplying the cost of travelling farther away.

A value found in an L1 cache has barely moved. A value fetched from DRAM may travel through a memory bank, an on-package or board-level interface, a memory controller, an interconnect, a last-level cache, and finally into the requesting core.

The hierarchy works because programs tend to reuse nearby information.

Temporal locality means recently used data is likely to be used again.

Spatial locality means data near a recently used address is likely to be used soon.

Caches convert these statistical properties into reduced movement.

A cache hit is not merely a faster lookup.

It is a long journey that never had to occur.

---

Performance Is Limited by Arithmetic Intensity

The clearest mathematical description of the movement bottleneck comes from the roofline model.

Let arithmetic intensity be:

$$I = \frac{\text{operations}}{\text{bytes transferred}}$$

Then attainable performance is bounded by:

$$P \leq \min(P_{\text{peak}}, B_{\text{memory}}I)$$

where:

• $P_{\text{peak}}$ is the machine’s maximum computational throughput,
• $B_{\text{memory}}$ is available memory bandwidth,
• $I$ is arithmetic intensity.

This equation separates workloads into two regions.

When arithmetic intensity is high, performance may be limited by execution units. The processor has enough work to perform on every byte it fetches.

When arithmetic intensity is low, performance is limited by data movement. The execution hardware consumes operands faster than memory can supply them.

Consider vector addition:

$$C_i = A_i + B_i$$

Each element requires roughly one arithmetic operation but requires reading two values and writing one value. Even with simplified accounting, the ratio of computation to traffic is low.

Adding more floating-point units will eventually stop helping. The kernel becomes constrained by memory bandwidth.

Matrix multiplication behaves differently:

$$C_{ij} = \sum_k A_{ik}B_{kj}$$

With effective blocking, values loaded from memory can be reused many times. The arithmetic intensity increases with block size. The processor performs more operations for every byte brought into local storage.

This is why matrix multiplication can approach peak floating-point throughput while a simpler vector operation cannot.

The more mathematically complicated operation may run closer to the machine’s advertised peak because it reuses data more effectively.

The relevant metric is not merely operations per second.

It is useful operations per byte moved.

---

Locality Is a Form of Performance

Traditional complexity analysis focuses on the number of operations.

For example:

$$T(n) = O(n)$$

or:

$$T(n) = O(n\log n)$$

This abstraction is essential, but it does not describe where the operands are located.

On modern hardware, a more realistic cost model must include communication:

$$T_{\text{total}} = T_{\text{compute}} + T_{\text{data movement}} + T_{\text{synchronisation}}$$

Two implementations with the same asymptotic operation count can have radically different physical costs.

A cache-aware algorithm may outperform a theoretically elegant alternative because it accesses memory in predictable blocks.

A structure-of-arrays layout may outperform an array-of-structures layout because each cache line contains a higher fraction of immediately useful fields.

Loop interchange can improve performance without changing the mathematical algorithm.

Loop fusion can eliminate intermediate arrays.

Tiling can turn distant reuse into local reuse.

Compression can reduce movement even when decompression requires additional arithmetic.

Recomputation can be cheaper than retrieving a previously computed value from a distant memory level.

That last point is particularly revealing.

For years, programmers were trained to avoid repeating calculations. On modern systems, repeating a cheap calculation may cost less than loading its result.

The optimisation target has changed.

Compute is often cheap enough to waste.

Movement is not.

---

The Energy Cost Is Even More Severe

Performance is only one consequence of data movement. Energy is another.

The total energy required by a computation can be expressed as:

$$E_{\text{total}} = E_{\text{operations}} + E_{\text{memory}} + E_{\text{interconnect}} + E_{\text{control}}$$

In many workloads:

$$E_{\text{move}} \gg E_{\text{compute}}$$

Moving a value across a large chip structure can cost far more energy than performing a simple arithmetic operation on it.

The reason is physical.

A moving signal must charge and discharge capacitance along wires, buses, multiplexers, clocked storage elements, and interfaces. Longer distances and larger structures generally involve more capacitance.

Dynamic energy is approximately:

$$E_{\text{dynamic}} = \alpha CV^2$$

where:

• (\alpha) is the switching activity,
• $C$ is capacitance,
• $V$ is voltage.

Data movement increases switching across distributed structures. Reducing voltage helps, but communication remains expensive because the physical capacitance of wires and large storage arrays does not disappear.

This is why enormous monolithic register files are difficult to scale. A wider processor requires more read and write ports. More ports increase area, wiring complexity, access time, and energy.

It is also why large shared caches are not equivalent to small private caches. Capacity may improve hit rate, but larger arrays are slower and more expensive to access.

Every level of the hierarchy is a compromise between capacity, latency, bandwidth, and energy.

The cheapest value is usually the one already present in a nearby register.

The most expensive value is the one that must be recovered from far away.

---

More Cores Made Communication More Visible

When frequency scaling slowed, the industry shifted toward multicore processors.

More transistors could still produce more throughput by replicating execution engines.

But additional cores also increased the number of agents requesting data.

If $N$ cores each require memory bandwidth $b$, the ideal demand is:

$$B_{\text{required}} = Nb$$

If the available bandwidth is $B_{\text{max}}$, scaling begins to collapse when:

$$Nb > B_{\text{max}}$$

At that point, the cores do not operate independently. They queue behind a shared transportation system.

Coherence makes the problem more complex.

When multiple cores cache the same memory locations, the machine must track ownership, propagate invalidations, transfer modified lines, order conflicting accesses, and preserve the shared-memory abstraction.

A cache line written by one core may need to migrate to another core before the second core can modify it.

This creates a phenomenon often called cache-line bouncing.

The cores may perform almost no useful arithmetic. Most of the activity consists of repeatedly moving ownership of the same block through the coherence network.

False sharing makes this worse. Two threads may modify logically independent variables that happen to occupy the same cache line. The hardware moves the entire line because coherence operates at cache-line granularity, not variable granularity.

The software sees independent data.

The hardware sees one contested physical block.

Parallelism therefore does not remove movement.

It creates more endpoints between which movement must be coordinated.

---

Accelerators Win by Controlling Where Data Lives

GPUs, tensor processors, systolic arrays, and other accelerators are often described as successful because they contain many arithmetic units.

That is only part of the explanation.

Their real advantage is that they organise computation around explicit data reuse.

A GPU uses massive threading to tolerate latency and keep many memory operations in flight. It also provides local storage, shared memory, caches, and registers so that fetched data can participate in many operations before being discarded.

A systolic array moves values through a regular grid of processing elements. Data is reused as it travels rather than repeatedly returning to a central register file or memory.

Matrix engines perform large numbers of multiply-accumulate operations on tiles stored close to the arithmetic units.

The machine is not merely calculating faster.

It is reducing the average distance each operand travels per operation.

For matrix multiplication, the communication lower bound for a fast memory of size $M$ is commonly expressed as:

$$Q = \Omega\left(\frac{n^3}{\sqrt{M}}\right)$$

This result matters because it demonstrates that data movement is not always an implementation accident. Some amount of communication is mathematically unavoidable.

Hardware and software cannot eliminate all movement.

They can only approach the lower bound by maximising reuse inside local storage.

This is why accelerator programming revolves around tiling, blocking, fusion, batching, and memory layout.

The central question is rarely just:

How many operations can the unit execute?

The more important question is:

How many times can each operand be reused before it must move again?

---

Storage Capacity Does Not Equal Accessibility

Modern systems can store enormous amounts of data, but storage capacity says little about how quickly that data can influence computation.

A value may exist in:

• a register,
• a private cache,
• a shared cache,
• local DRAM,
• remote memory,
• persistent storage,
• another machine.

All of these locations preserve information. They do not provide equivalent access.

A rough model is:

$$\text{Cost} \propto \text{distance} \times \text{traffic} \times \text{contention}$$

The logical address space hides these distinctions.

A program may load two addresses using identical instructions even though one address is already in the L1 cache and the other triggers a page walk, a cache miss, and a remote memory transaction.

The instruction set presents a uniform abstraction.

The physical machine experiences a radically non-uniform geography.

Virtual memory deepens the abstraction. A pointer appears to identify a location directly, but the processor may need to translate it through multiple page-table levels before accessing the data.

Translation lookaside buffers exist to cache these translations because translating addresses is itself another data-movement problem.

The machine must move information describing where other information is located.

---

Prefetching Is Prediction About Movement

When data cannot be delivered instantly, one strategy is to begin moving it before it is requested.

That is the purpose of prefetching.

A prefetcher observes memory access patterns and predicts future addresses. Correct predictions convert demand latency into background traffic. Incorrect predictions waste bandwidth, cache capacity, and energy.

This produces a familiar architectural trade-off:

$$\text{Benefit} = \frac{\text{latency hidden}}{\text{resources wasted}}$$

Sequential accesses are relatively easy to predict. Regular strides are also manageable. Irregular graphs, pointer chains, hash tables, and data-dependent traversals are much harder.

The prefetcher is not predicting computation.

It is predicting where the computation will need data next.

This is another sign that movement has become the limiting concern. The machine spends hardware resources speculating about future transportation requirements.

Branch prediction guesses which instructions will be needed.

Prefetching guesses which data will be needed.

Both exist because waiting for certainty would leave expensive execution hardware idle.

---

Software Optimisation Is Increasingly About Traffic

Many low-level optimisations can be interpreted as attempts to reduce, regularise, or hide data movement.

Tiling

Tiling divides a large problem into blocks that fit inside a nearby memory level.

The goal is to increase reuse before eviction.

Loop fusion

Loop fusion combines passes over data so that intermediate values remain in registers or caches instead of being written to and reloaded from memory.

Data-layout transformation

Changing an array of structures into a structure of arrays can increase useful bytes per cache line and improve vectorisation.

Batching

Batching groups work so that setup costs and data transfers are amortised across more operations.

Compression

Compression decreases the number of bytes transferred. It may improve performance even when it adds compute overhead.

A simplified condition is:

$$T_{\text{compress}} + T_{\text{transfer compressed}} + T_{\text{decompress}} < T_{\text{transfer original}}$$

Operator fusion

Machine-learning compilers fuse adjacent operators to prevent intermediate tensors from being written to external memory.

Recomputation

A value may be recalculated rather than stored and fetched later when:

$$E_{\text{recompute}} < E_{\text{reload}}$$

or:

$$T_{\text{recompute}} < T_{\text{reload}}$$

These optimisations appear different at the source-code level.

Physically, they pursue the same objective:

Keep useful information close to the hardware that consumes it.

---

The Real Resource Is Not FLOPS

Computer systems are frequently marketed using peak arithmetic throughput.

But peak throughput describes what the machine can do only when every execution unit receives operands continuously.

A processor capable of $P_{\text{peak}}$ operations per second may sustain only a fraction of that performance when memory cannot supply enough bytes.

The utilisation is approximately:

$$U = \frac{P_{\text{sustained}}}{P_{\text{peak}}}$$

For a bandwidth-bound workload:

$$P_{\text{sustained}} \approx B_{\text{memory}}I$$

Therefore:

$$U \approx \frac{B_{\text{memory}}I}{P_{\text{peak}}}$$

As computational throughput grows faster than memory bandwidth, utilisation falls unless arithmetic intensity also increases.

This explains the strange reality of modern computing: a machine can contain an extraordinary amount of arithmetic capacity while spending much of its time waiting, scheduling, buffering, predicting, and transporting.

The execution units are not necessarily the scarce resource.

The scarce resource is the ability to feed them.

---

Moore’s Law Did Not Fail. Our Interpretation Did.

Moore’s Law remains one of the most important technological trends in computing history. More transistors enabled larger caches, multicore processors, vector units, integrated memory controllers, sophisticated predictors, specialised accelerators, and advanced interconnects.

The mistake was not believing that transistor density mattered.

The mistake was treating transistor density as synonymous with useful computation.

A transistor contributes to performance only when the surrounding system can supply it with information and carry away its results.

Beyond a certain point, adding execution units without improving locality, bandwidth, or communication merely creates more hardware waiting for data.

The modern processor is therefore not best understood as an arithmetic engine.

It is a constrained information-transport machine.

Its caches are movement filters.

Its prefetchers are movement predictors.

Its coherence system is a movement protocol.

Its memory controller is a movement scheduler.

Its interconnect is a movement network.

Its register-renaming logic moves names away from false dependencies.

Its out-of-order engine rearranges time so useful work can continue while other data is moving.

Even speculative execution is partly an attempt to avoid waiting for information to arrive.

The entire machine is organised around the fact that data is rarely where it needs to be.

---

The Next Era Will Be Defined by Proximity

Future performance improvements will increasingly come from reducing the distance between storage and computation.

This includes larger on-chip memories, stacked memory, chiplets, wider package-level interfaces, near-memory computing, processing-in-memory, domain-specific accelerators, and software designed around explicit locality.

But none of these approaches makes movement free.

Larger local memories consume area and energy.

Three-dimensional integration creates thermal challenges.

Chiplets require high-bandwidth die-to-die links.

Processing near memory complicates programmability and consistency.

Specialised accelerators sacrifice generality.

Every solution changes the geography of computation, but the underlying constraint remains physical.

Information has a location.

Using it somewhere else requires time and energy.

---

Conclusion

Moore’s Law taught us to think of computation as the scarce resource.

It is not.

Modern chips can perform enormous numbers of additions, multiplications, comparisons, and logical operations. What they struggle with is delivering enough operands to those circuits without exceeding latency, bandwidth, power, and thermal limits.

The real performance hierarchy is not defined only by operation count.

It is defined by distance.

A value in a register is cheap.

A value in a nearby cache is manageable.

A value in main memory is expensive.

A value on another device or machine is an architectural event.

The most important optimisation is often not reducing the number of instructions. It is reducing the number of times information must move, reducing how far it travels, and increasing how much useful work is performed before it moves again.

Moore’s Law gave us more transistors.

The harder problem was always deciding where to place the data.