Modular and Qualcomm: Same code, new silicon

Today during the ModCon keynote, we shared a major milestone in Modular's multi-silicon work: Qualcomm Technologies' data center AI accelerators are now integrated into the Modular Platform. Qualcomm Technologies has spent decades building high-performance, power-efficient compute, and has directed that expertise at purpose-built inference silicon designed for the best performance per dollar per watt. Developers can now reach this hardware through the same software stack they already run on NVIDIA and AMD GPUs. This milestone represents an early step towards a common software foundation spanning an even broader range of industry accelerators, from edge to cloud.

Every time a new AI accelerator enters the market, developers face the same question: how much of our stack do we have to rewrite? Modular’s unified compute platform is making that answer clear: none.

Following Qualcomm's acquisition of Modular Inc, we're excited to share that Qualcomm Technologies' data center AI accelerator roadmap, beginning with the Qualcomm® Cloud AI 100 and extending to Qualcomm Dragonfly™ AI 200 and future generations, is being integrated into Modular's vertically integrated AI software platform with Mojo and MAX.

This means the same code that runs on NVIDIA and AMD GPUs can now target Qualcomm Technologies' silicon. Same abstractions. Same developer experience. Best in-class performance. Built for Agents. This marks the first ASIC (NPU) to be integrated into the Modular stack and an additional proof point of the portability of MAX and Mojo.

Two companies, one thesis

Qualcomm Technologies has spent decades building high-performance, power-efficient compute at massive scale across mobile, PCs, automotive, IoT, and networking. Now the company brings that expertise to purpose-built AI accelerators for data centers: high-throughput, low-power acceleration designed to scale across modern hyperscale workloads.

Modular has been building the software layer for that same future. MAX provides a complete high-performance AI framework, from compiler to runtime to model serving, where a single model implementation can target multiple hardware backends at native performance. Mojo gives developers Python-level usability with systems-level control, making it possible to write highly optimized kernels that expose the capabilities of the underlying hardware without fragmenting the programming model. Modular Cloud unlocks true hardware portability giving developers and enterprises the ability to run high-performance endpoints for all the most important AI models.

Both companies share a conviction: developers and enterprises should be able to choose the best hardware for their workload without rewriting their software stack.

"Qualcomm Dragonfly AI Inference solutions leverage decades of compute expertise and bring innovative solutions to solve some of the hardest problems in data center AI. The 'portability-by-design' of Modular's software stack enables rapid integration with Qualcomm Technologies' NPU-based AI accelerator and can free developers from rewriting models, enabling best-in-class AI inference solutions and lower total cost of ownership. Paired with the Qualcomm Dragonfly portfolio, the Modular AI software stack gives customers a silicon-agnostic path to efficient, day-zero performance, real choice in how they deploy AI."

"We built Modular's software stack to be portable by design, so that when new hardware shows up, developers don't have to start over. Qualcomm Technologies' AI accelerators are exactly the kind of purpose-built inference silicon we designed MAX and Mojo for, with a roadmap that gives our users real choice in the data center that’s powered by Modular Cloud. We couldn't be more excited to continue building together."

Native bring-up in weeks

Over the past several months, engineers from Modular and Qualcomm Technologies have been working side by side to integrate the Qualcomm Cloud AI 100 Ultra into MAX and Mojo.

The integration moved quickly. Within weeks of getting matrix multiplication kernels running on the Qualcomm Cloud AI 100, the team had GPT-2 executing end to end through MAX, using the same high-level model definition and host code paths used for NVIDIA and AMD GPUs. From there we scaled to a modern model: measured from the first Gemma 4 kernel work to a live serving endpoint, that bring-up took under six months.

This is different from a typical hardware bring-up: the integration is native, not bolted on. Mojo compiles to the Qualcomm® Hexagon™ LLVM NPU backend. Hardware capabilities are mapped directly into the Mojo standard library through the same abstractions used for GPU programming. Through MAX's device driver connector, the Qualcomm Cloud AI 100 plugs into the graph compiler, serving framework, and runtime the same way NVIDIA, AMD, and Apple Metal GPUs do.

The result is portable AI infrastructure in practice: new hardware joins the platform, while the developer experience remains consistent. The integration of the Qualcomm AI 100 is the first proof point of a broader collaboration across Qualcomm Technologies' AI accelerator roadmap, including the Qualcomm Dragonfly AI200, AI250, and AI300.

The rest of this post goes deeper into how we got there. The bring-up, and the sections that follow, are structured in stages: a foundation phase to enable the Mojo compiler and graph compiler and write the first kernels for the NPU; a phase to build a GPT-2 "Hello World" pipeline with reproducible numbers; a phase that scaled all the way to a Gemma 4 31B model running across four chips, served live via max serve; and a final phase where new models and features went from weeks to days. We went from Mojo works to Gemma 4 serving in under a quarter, and we are confident the next models we enable can be done even faster. We are not copying CUDA; we are building a custom path into a portable stack.

The Qualcomm Cloud AI 100 hardware

The Qualcomm Cloud AI 100 Ultra is a performance- and cost-optimized AI inference solution, purpose-designed for generative AI and large language models (LLMs).

The Qualcomm Cloud AI 100 Ultra has 4 devices with 32 GB of RAM, each one of those made up of 16 NSPs. The NSPs are multi-threaded, which we leverage to orchestrate the different vector (HVX), DMA and matrix (HMX) units building software pipelines. Two units matter most for model work: the HVX vector engine, which handles wide SIMD vector and elementwise work and maps to the upstream LLVM Hexagon vector intrinsics, and the HMX matrix engine, a dedicated matrix-multiply unit that does the heavy lifting for GEMM and attention.

The memory model is also different. There is no unified memory as on a modern GPU; the device works with separate on-chip VTCM and off-chip DDR, and data is staged between them explicitly with DMA. Getting good performance means tiling work into VTCM and moving data with DMA deliberately, rather than relying on a cache hierarchy to hide it.

The HMX unit requires a specific layout for the data it is going to operate on (called crouton layout), and the conversion has a cost, so it is important from the developer perspective to make sure conversion costs are minimized. Sometimes it is better to stay on the HVX vector engine than pay the conversion cost.

Stage 0: Laying out the groundwork

Building the foundations is the key element that underpins all the later model bring-up work, so our team spent time setting up the Mojo platform on the Qualcomm Cloud AI 100 series. This initial validation and enablement work focused on getting the compiler backend, the various passes, and the APIs for the device-specific features working on the target.

The Mojo compiler leverages natively the upstream LLVM, thus, for the Qualcomm Cloud AI 100 platform we leverage the existing LLVM Hexagon DSP backend, so that our SIMD types and operations lower to the right Hexagon vector intrinsics (HVX). HMX and DMA are exposed through bindings to the Mojo programming model, so expert kernel engineers can extract maximum performance from the device.

We implemented our DeviceContext API (the interface that currently targets NVIDIA and AMD GPUs) to target the Qualcomm Cloud AI 100 JIT API (the JIT interface used to dispatch kernels to devices) when running on a Qualcomm Cloud AI 100 Ultra card. The QAIC JIT API offers the ability to create command buffers on each NSP and to enqueue operations that copy data in and out of the host, which we can map to equivalent DeviceContext functionality. There were no fundamental changes to the DeviceContext, except for adapting to a single-stream dispatch from the multi-dispatch nature of GPU APIs, and adapting the block/grids from GPUs to NPUs.

One of the main challenges, however, was the difference in expected kernel signatures. Kernels in Mojo typically receive their arguments as typed arguments, whereas the QAIC JIT API expects kernel arguments as a contiguous pointer array in memory. To convert Mojo kernels into QAIC-supported entry points, a typical AI software stack would need dedicated compiler passes to massage the function arguments from one format to another. Instead, by leveraging the reflection and compile-time capabilities of the Mojo language, we implemented the conversion of a Mojo kernel signature into the QAIC JIT signature purely in Mojo. This significantly simplified the implementation and improved compilation times.

Once basic kernel dispatch was working, it was straightforward to get a graph-compiler proof of concept where an elementwise op maps to a single elementwise kernel generated by the graph compiler. That first op validates that the rest of the stack will be functional on the device, so we could shift focus to optimization and performance.

The NPU is fundamentally different from a GPU in many respects. One key difference is kernel dispatch: GPUs dispatch in terms of grids and blocks, whereas the NPU works in terms of number of cores and number of threads. And, as noted above, the Qualcomm Cloud AI 100 is a SIMD device rather than SIMT, so the kernel programming model is fundamentally different.

# Create the NPU device context.
ctx = DeviceContext(device_id=0, api="qaic")

# Allocate device buffers.
var lhs_device = ctx.enqueue_create_buffer[float_dtype](VECTOR_SIZE)
var rhs_device = ctx.enqueue_create_buffer[float_dtype](VECTOR_SIZE)
var result_device = ctx.enqueue_create_buffer[float_dtype](VECTOR_SIZE)

# Copy inputs host -> device.
ctx.enqueue_copy(lhs_device, lhs_host)
ctx.enqueue_copy(rhs_device, rhs_host)
ctx.synchronize()

# Compile: compile_function auto-generates the QAIC entry-point wrapper
# from the kernel's typed signature.
var kernel_func = ctx.compile_function[vector_add_kernel]()

# Launch across the NPU cores/threads.
ctx.enqueue_function(
    kernel_func,
    lhs_device,
    rhs_device,
    result_device,
    Dim(NUM_CORES),
    Dim(THREAD_MASK),
)
ctx.synchronize()

# Copy result device -> host.
ctx.enqueue_copy(result_host, result_device)
ctx.synchronize()

The snippet below shows a basic vector addition Mojo kernel for the Qualcomm Cloud AI 100 NPU. The npu_work_range function is a convenience helper that splits the total size of the vector equally across all NSPs, provided as a mojo function for kernels Native thread and core ID functions are available as well.

def vector_add_kernel(
    lhs: UnsafePointer[Float32, MutAnyOrigin],
    rhs: UnsafePointer[Float32, MutAnyOrigin],
    result: UnsafePointer[Float32, MutAnyOrigin],
):
    """Vector addition kernel."""
    # Split the total work across all NSP threads; each gets a sub-range.
    var (start, end) = npu_work_range(VECTOR_SIZE)

    for i in range(start, end):
        result[i] = lhs[i] + rhs[i]

It would have been possible to implement a SIMT (GPU-like) programming model for the NPU but it is not required. The Modular stack is able to leverage all the hardware features available without imposing a specific programming model. For hardware and algorithms that require SIMT we provide abstractions that make programming at that level easier. In general, we classify the different operations in terms of patterns (map, reduction, etc.) and provide abstractions that split the compute (function to be mapped onto) from the schedule (the loop iteration schedule). As a result we get both abstractions while not getting sub-par performance.

For heavy-duty kernels like matmul or attention, kernel engineers write optimized kernels for the target using hardware-specific APIs and concepts, reusing parts of the existing infrastructure where possible via compile-time conditions.

Because this is the first time we brought up a non-GPU target, and given the QAIC JIT API is relatively new, we spent significant time with Qualcomm Technologies engineers co-designing features, working through performance bottlenecks, and making our interfaces more stable and production-ready. Onboarding the next NPU will be significantly faster as a result, and we are already working on a next iteration of our DeviceContext API that is more generic and easier for Hardware Companies to leverage

Takeaway: The modularity of our software stack enabled us to focus on what was different for the NPU, so we can re-use the rest. Building on standard technologies, like LLVM and MLIR, means we work from solid technical foundations.

Stage 1: GPT-2 end to end, and the first real numbers

With Mojo and the graph path online, GPT-2 124M became the first complete model supported on the Qualcomm Cloud AI 100. GPT-2 is a "Hello World" for LLMs, small and easy to test, but enough to exercise the entire model pipeline, the graph compiler and a variety of operators.

Our kernel engineers wrote an optimized GEMM kernel using the HMX unit, supporting arbitrary M/K/N values with VTCM tiling and DMA support. They also implemented all the basic building blocks for GPT-2, such as LayerNorm, softmax, attention, and the elementwise patterns mentioned earlier. These Mojo building blocks are re-usable across multiple models and validated independently.

We then plugged those kernels through the graph compiler, enabling them one after another until the graph compiler built the whole model. Our first run was roughly 1.6× slower than the PyTorch implementation of Qualcomm Cloud AI 100 we were using as a baseline.

Through roughly three weeks of optimization work we reached 6.7× the baseline, comfortably faster than that baseline. Among other things, we:

• implemented custom operations to fuse each layer, reducing per-op kernel-launch overhead and increasing the number of fused operations beyond elementwise;
• wrote a dedicated HVX GEMV kernel for the M=1 decode shape, faster than the HMX matmul here because it avoids the crouton layout conversion;
• added device-info caching and asynchronous function dispatch to hide driver overhead;
• and applied various runtime caching and dispatch optimizations to further reduce driver cost.

Enabling the model only took a few days. We took the model defined by our LLM handbook and then created a variant with fused operators for the Qualcomm Cloud AI 100 platform. We spent a few weeks then working on optimizing the software stack, the kernels and the overall testing infrastructure.

import max.experimental.functional as F
from max.driver import CPU, Accelerator, Device
# ... Model definition ...

# ... Model runner ...
dev = Accelerator(0)
with F.lazy(), default_device(dev), default_dtype(DType.float16):
     model = GPT2LMHeadModel(config)

The snippet above shows how the GPT-2 model is instantiated from MAX. The code is the same for all accelerators, and the underlying library routes the implementation to the active target.

Because we had the Mojo compiler working on the device, while part of the team was working on GPT-2, some engineers were already working on individual kernels for the next stage, Gemma 4. Using a test harness, the engineers can write and optimize the kernels in Mojo without having to wait for the model to be up and running. This is a great advantage of the Modular software stack.

Takeaway: Using GPT-2 as an initial hardware bring-up tool enabled us to quickly optimize the system side of the implementation, enable the graph compiler and design fusion strategies, while the kernel engineers continued working on plain Mojo kernels. And all the work is re-usable across multiple models.

Stage 2: Getting Gemma 4 dense to max serve

Once we were confident GPT-2 was stable and well optimized, the rest of the team moved to work on Gemma 4, and the kernels were plugged into the model definition, enabling the graph compiler to call them as part of the model pipeline.

From the perspective of the Modular stack, the main difference between GPT-2 and Gemma 4 is the complexity of the kernels and the communications (Gemma 4 with KV Cache in fp16 does not fit on a single SoC). The fundamentals of the system validated with GPT-2 remain the same, and we could move to focus on the differences for Gemma 4.

More complex kernels

Gemma 4 needs a richer set of operations than GPT-2, most notably a real attention path. We built a paged flash-attention kernel (the first on the Qualcomm Cloud AI 100) inspired by the Flash Attention 4 design using thread-specialization and software pipelining. We also implemented all the other kernels a modern decoder requires (rotary embeddings, RMSNorm, and so on), with the graph compiler fusing adjacent operations automatically where it can.

The more interesting decisions were hardware-shaped. The HMX matrix unit expects weights in a packed "crouton" layout, and paying that conversion cost only makes sense when the matrix multiply is large enough to amortize it. For the memory-bound decode step at batch size 1 we therefore route through a GEMV kernel on the vector unit and skip the conversion entirely, switching to the HMX path only at larger batch sizes where it pays off. These types of choices are driven by kernel profiling and parametrization, which enables quickly validating a series of mojo kernels for different inputs.

Scaling beyond one device

A Qualcomm Cloud AI 100 Ultra card contains 4 SoCs (each SoC is referred to as a device), and each SoC has 32 GB of memory. Gemma 4 31B in fp16 needs roughly 60 GB for weights alone, nearly double a single SoC's 32 GB. The model must be sharded across the four devices with tensor parallelism (TP=4).  We leverage the  QCCL library which exposes all-reduce, broadcast and all-gather collectives via a host interface

Concretely, tensor parallelism shards the model across the four chips: the QKV and MLP-up projections are column-parallel, and the attention-output and MLP-down projections are row-parallel. Each layer therefore needs a collective step (all-reduce) to put together the partial results back together before proceeding. The integration work started early, but it wasn't fully validated until we had the distributed matmul running across all four SoCs with weights and activations sharded across chips.

The Qualcomm Cloud AI 100 communication library (QCCL) is relatively recent, and relies on host-driven communication (as opposed to device-initiated), so it naturally has additional overheads compared to other solutions. Reducing that overhead is an ongoing area of work, but the integration of the library into our stack was straightforward.

It runs, ergo, it serves

We validated the model with an engine benchmark, confirming the Gemma 4 31B output matched the reference fp16 model within tolerance.

We then switched to max serve, which loads the same model architecture inside our inference-server environment. We didn't need to change anything about the model or the serving environment; we simply pointed max serve at the model architecture file. That lets us reuse all the serving tooling, in particular max benchmark, which we use to validate different workloads and extract performance results.

Our Gemma4 model performance is on par with the existing software solutions for Qualcomm Cloud AI 100

Once we had the model serving and were able to run our different benchmarks (like arxiv or sharegpt) we were able to identify further optimizations. For example, we implemented MXFP6 weights (converting the weights before loading the model) and an int8-based KV cache, roughly halving the KV-cache usage without significant accuracy loss.

Takeaway: going from one device to four required only enabling the communication layer, no further changes to the code. And, once the model is enabled, running it through max serve is trivial.

Stage 3: From weeks to days, new models and features

Once we got the dense model serving and implemented general optimizations, it became increasingly simple to support new models or optimizations. In particular, over the last 3 weeks of the bring-up we enabled speculative decoding and the Gemma 4 MoE variant, both re-using the existing infrastructure and added with very little customization for the Qualcomm Cloud AI 100.

Speculative decoding

Gemma 4, like any other LLM, generates one token at a time, and each step is memory-bound. We leverage speculative decoding to produce faster responses. Speculative decoding uses a small fast draft model that proposes several tokens ahead (currently configured as 3), and then the large Gemma 4 model verifies them in a single forward pass, accepting the ones that it agrees with. When the draft model is correct, several tokens are produced for the cost of one large-model step.

Our implementation uses the smaller Gemma 4 "assistant" weights as the draft model, and verifies against the 31B dense model.

One of the reasons speculative decoding gives great performance improvements on the device with Qualcomm Cloud AI 100 is that we are able to increase our M size for the dense model (from 1 to 4), so the decode path and the flash-attention kernel can now work using the HMX unit by grouping the GEMV onto a single GEMM, whereas the smaller draft model can still use the fast GEMV implementation for M=1 on HVX.

Adding speculative decoding didn't require any changes to the serving architecture. We use the same max serve path, the same tensor-parallel sharding and the same kernels. We simply schedule smaller weights as part of the model pipeline.

Mixture of Experts variant

Google also provides a Mixture of Experts (MoE) variant of Gemma 4, a 26B model where only 3.8B parameters are active for any given token, routed through a set of MLPs.

Adding the MoE variant was a straightforward task with the Modular stack. Most of the architecture is shared with the dense model (attention, RMSNorm, RoPE, paged KV cache, etc.). We even re-used the speculative decode implementation for the MoE model.

We implemented the router kernel (the top-k step) that decides which experts each token goes to, and grouped matmul so the selected experts can be evaluated efficiently. This is all an incremental change from the existing implementation, so we only needed to enable the model for the target, and we re-used all the existing infrastructure (e.g. graph compiler optimizations).

The current implementation of QCCL is host-driven (i.e. the communication primitives are synchronized via the host). The in-kernel variant of the communication primitives is not available yet, so we opted to serve MoE using tensor parallelism instead of expert parallelism. This way we didn't have to rely on a potentially slow all-to-all for the experts. Once QCCL supports in-kernel communications we can easily implement an EP variant, and we can explore different configurations to tune for performance under different workloads.

Takeaway: Enabling advanced features or model variants becomes an incremental task where most of the effort is reduced. It only took a couple of days to enable and tune each of them.

Where we are and where we go next

The headline number (Gemma 4 in less than a quarter) is only the teaser; what's important is how we managed it. Three elements are worth highlighting.

First, the stack is portable by construction. We didn't fork the model, the graph compiler, or the serving layer to reach a new class of hardware. All the hardware targets live in the same repository and share the vast majority of the code. The same DeviceContext abstraction that targets NVIDIA and AMD GPUs now targets an NPU with a completely different execution model, and the same model definition that runs on one device runs across four and is then available in max serve.

Second, the hard problems were de-risked early. Starting from foundations (Mojo support on the target, GPT-2 as a "Hello World") meant we could de-risk the project early and build strong foundations for Gemma 4, and it will be even easier to bring other models onto the chip from now on.

Third, co-design with Qualcomm Technologies engineers accelerated the work. Our implementation regularly stressed the JIT API, and working through those functionality and performance issues together shortened the turnaround on each one. Qualcomm Technologies shared the target-hardware detail our kernel engineers needed through regular syncs and technical sessions, and in return they saw how the Modular stack exercises their chip. Documentation and a direct line to the vendor team turned a functional backend into a performant implementation.

The result is a software stack serving tokens in Modular cloud today, with room to take on new hardware and new models next. We are heavily leveraging agentic tools and skills to speed up our workflow, using agents to optimize kernels, to port models to MAX or to run benchmarks and analyze results automatically. AI Models can get context really quickly from the codebase and the documentation and agentic knowledge base. We have now the first knowledge base for the Qualcomm Dragonfly series, making our iteration speed faster now that our tools have the right context.

The next step is already ongoing: Qualcomm Dragonfly AI 200 is the evolution of the Qualcomm Cloud AI 100 for the LLM era. We already have Mojo compiler support working on the device, and we can run GPT-2 on it. We are working on validating and optimizing the rest of the stack for larger memory and much higher matrix throughput. Qualcomm Dragonfly AI 200 brings native fp8 and bf16 support, so models trained on those formats can run efficiently on it, and our stack already supports both types natively.   There is still work ahead, driving down the all-reduce cost in the decode step, pushing context length further, and bringing up new models beyond Gemma 4. But the core result stands: a modern model, on a new class of hardware, brought to a live serving endpoint in under six months, on a stack that was built to be portable. At Modular, we continue iterating and making our stack more portable and efficient: the next version of DeviceContext puts backend bring-up in the hands of hardware companies.

Snapdragon Qualcomm branded products are products of Qualcomm Technologies, Inc. and/or its subsidiaries. Snapdragon, Qualcomm, Qualcomm Dragonfly, and Hexagon are trademarks or registered trademarks of Qualcomm Incorporated.