Compute-in-Memory
Compute-in-memory (CIM), also called processing-in-memory (PIM), is a processor architecture that performs computation directly inside or right next to the memory that holds the data, instead of shuttling operands back and forth to a separate arithmetic unit. By doing the math where the weights already sit, it attacks the dominant cost of modern AI hardware — moving bytes — rather than the arithmetic itself, and is especially suited to the dense multiply-accumulate (MAC) operations at the heart of neural networks.
It breaks the von Neumann separation of memory and compute. A conventional machine keeps memory and the ALU apart and streams data across a bus between them; for memory-bound workloads that data movement dominates energy and latency, and the compute unit spends much of its time waiting. Compute-in-memory collapses that split: the storage array itself becomes the compute engine, so a weight never has to travel to a distant multiplier. The bottleneck the design targets is the bus, not the FLOPs.
Analog crossbars compute a dot-product with physics. In the most striking form, each memory cell stores a weight as its conductance in a grid (a crossbar of SRAM, ReRAM, PCM, or flash). Drive a row with an input voltage and Ohm's law makes each cell pass a current equal to voltage times conductance; Kirchhoff's law then sums all currents on a shared column. One column therefore reads out an entire vector dot-product — a full MAC over many weights — in a single analog step, with no weights fetched and no per-element multiply. A whole matrix-vector product happens in the array at once.
| Von Neumann | Compute-in-memory | |
|---|---|---|
| Where math happens | separate ALU | inside the memory array |
| Data movement | weights stream over bus | weights stay put |
| MAC cost | fetch + multiply + writeback | one analog column read |
| Dominant limit | memory bandwidth / bus energy | ADC/DAC, analog noise |
| Best fit | general-purpose, exact | dense MAC, tolerant precision |
The costs are precision, conversion, and generality. Analog computation is noisy: device variation, drift, and limited cell states cap effective precision, and every array needs digital-to-analog drivers on the inputs and analog-to-digital converters on the outputs, whose energy and area can eat much of the savings. Write endurance and non-linearity limit which memory technologies work, and the paradigm fits dense, low-precision MAC-heavy layers far better than exact or control-heavy code. So CIM is deployed as a specialized accelerator for neural inference, not a general CPU replacement.
Read compute-in-memory through a quant lens rather than a 'faster memory' lens: it moves the operating point on the roofline by driving the bytes-moved term toward zero — the MAC is paid for in physics inside the array instead of in fetched operands — so the figure of merit becomes MACs per joule and per mm² including the ADC/DAC overhead, at the effective bit-precision the analog array can hold. The design question is how much of that data-movement energy you can eliminate before conversion and noise give it back, which is exactly why CIM wins on dense low-precision inference and loses on exact general compute.