Microsoft Maia 300: Why moving data is becoming a major challenge for AI chips

Image credit: Microsoft (https://blogs.microsoft.com/blog/2026/10/12/microsoft-to-expand-azure-maia-system-to-multi-gigawatt-deployments)

There is a problem with building increasingly powerful AI processors. Adding more computing capability doesn't necessarily make them faster. The processor still needs a constant supply of data, and moving that data around a chip consumes time, energy and memory bandwidth. If the computing hardware can process information faster than memory can supply it, much of that additional processing power goes unused.

It's a particular concern for AI inference, where large models repeatedly access billions of parameters while generating responses. Longer conversations and more complicated reasoning tasks increase the amount of information that needs to be handled, especially when thousands of requests are being processed simultaneously.

Microsoft has designed its new Maia 300 accelerator with these challenges in mind. Announced on October 12, the third-generation processor is intended to become a central part of the company's AI infrastructure from 2027, with plans to scale the Maia family to multi-gigawatt deployments. Rather than concentrating solely on raw computing performance, Microsoft has given considerable attention to how information moves between computing resources, memory and other accelerators.

Why Maia 300 needs 18 TB/s of memory bandwidth

The Maia 300 carries eight HBM4 memory stacks, providing 288 GB of capacity and a peak bandwidth of 18 TB/s. That's 2.5 times the memory bandwidth of the previous Maia 200, despite the new accelerator operating within a 950 W thermal design power.

The bandwidth figure is particularly interesting. During inference, a processor continually retrieves model weights from memory to perform calculations. If the computing engines can process data faster than memory can supply it, some of that processing capability sits idle. Simply adding more arithmetic units does little to address this bottleneck, which is why memory performance has become such an important part of AI processor design.

Microsoft's approach involves more than fitting faster memory. Maia 300 uses what the company calls a Software Defined Locally Accessed Dataflow Architecture, or SDLA, which gives software explicit control over where data is placed and how it travels between different parts of the processor.

The latest design also moves away from the uniform memory arrangement used in Maia 200. Instead, it adopts a non-uniform architecture across its chiplets, placing memory resources closer to the computing engines that need them. Transferring data across a chip requires energy, and repeatedly accessing external memory can become expensive when the same information is needed thousands of times.

On-die SRAM is used as a software-managed scratchpad, while lower-precision formats such as MXFP4 reduce the amount of information that needs to be stored and transferred. Microsoft also supports sub-3-bit storage formats, although the usefulness of reduced precision depends on the model and the calculations involved. A smaller numerical representation saves memory, but it isn't automatically suitable for every operation.

Why Microsoft has moved to chiplets

Maia 300 also changes how the processor itself is manufactured. Rather than placing every function on one large piece of silicon, Microsoft has separated compute, memory-controller and I/O functions into different chiplets. The compute dies use TSMC's N2P process, while the remaining functions can be implemented using technologies better suited to their requirements.

There is a practical reason for doing this. Advanced manufacturing processes are expensive, and not every part of a processor benefits equally from the smallest available transistors. Memory interfaces and I/O circuits have different design constraints from the compute engines responsible for performing AI calculations. Splitting those functions allows Microsoft to concentrate its most advanced manufacturing technology where it makes the greatest difference.

Of course, chiplets introduce another problem. Data must now move between separate pieces of silicon, making the connections between them an important part of the design. Microsoft's emphasis on controlling data movement is particularly relevant here, since the company has developed the chiplet arrangement and software architecture together rather than treating communication between the dies as a separate concern.

Fitting 96 accelerators into a 167 kW rack

The challenge becomes considerably larger when Maia 300 moves beyond an individual processor. Microsoft plans to install 96 accelerators in each rack, arranged across 24 trays containing four accelerators apiece. A complete rack has a 167 kW power envelope, with 27.6 TB of HBM capacity and an aggregate memory bandwidth of approximately 1.7 PB/s.

That's an enormous amount of memory bandwidth concentrated in one system, but the accelerators still need to communicate with each other. Large AI models can be distributed across multiple processors, with different devices responsible for different parts of a calculation. Intermediate results then have to pass between them, and even exceptionally fast individual chips can be held back by communication delays.

Microsoft is addressing this using high-bandwidth interconnects and a custom transport built around Ethernet networking. Its architecture extends the same software-controlled approach to data movement beyond the chip and across the rack, with the company describing configurations extending to 12,288 accelerators within a single scale-up domain.

Power and cooling become equally important at this point. A 167 kW rack represents a substantial thermal load, and increasing processor density only works if the surrounding infrastructure can remove that heat reliably. Microsoft has integrated host computing, memory, storage and cooling within each four-accelerator tray, rather than allocating separate trays to host processors. The arrangement doubles accelerator density compared with Maia 200.

Can Maia 300 outperform Nvidia GPUs?

Microsoft expects Maia 300 to deliver 40% more tokens per megawatt than leading GPU-based systems. It also claims a complete rack will achieve up to seven times the token throughput of a Maia 200 rack. Those are substantial improvements, although the comparison needs some care.

Microsoft hasn't established that Maia 300 will outperform every competing GPU across every AI workload. The efficiency figures come from the company's own projections, and results will depend on which models, precision formats and operating conditions are used. There is also an important difference between building a general-purpose AI accelerator and designing hardware around the workloads of one enormous cloud operator.

Nvidia's GPUs support a broad range of applications, from model training to inference and scientific computing. Microsoft has the advantage of knowing which models its own infrastructure runs, how they behave and where inefficiencies appear. Its Maia 200 is already serving production workloads, including Microsoft Copilot, and the experience gained from operating that hardware has fed into Maia 300's design.

That doesn't mean Microsoft can dispense with Nvidia. Custom silicon requires major investment, and specialised accelerators aren't necessarily the best choice for every workload. But when a company operates AI services at Microsoft's scale, improving the amount of useful computation delivered by each megawatt can be worth pursuing.

Maia 300 is scheduled to enter production deployment in 2027, so the claimed efficiency advantages still need to be demonstrated at scale. What makes the design interesting is where Microsoft has concentrated its engineering effort. Faster compute remains part of the equation, but the real focus is on keeping that compute supplied with data, whether it comes from nearby SRAM, HBM4 or another accelerator elsewhere in the system. With 96 processors sharing a rack and potentially thousands communicating across a data center, moving information efficiently may prove just as important as processing it.