Engineering Articles (X)

Measuring state of the art GPU performance compared to vLLM on Modular's MAX 24.6

We are pleased to announce the release of a new tool aimed at simplifying the management of stacked pull requests (PRs) on GitHub - stack-pr. This tool is still in its early development days, but we are excited to share it with the community and welcome your contributions.

Developer tooling is a big priority for Mojo and MAX, we want to vastly improve the debugging experience compared to the traditional Python, C++, and CUDA stack. Machine learning often requires inspecting the state of a program after a long running process, requiring more control than what "print debugging" gives you. Over time this tooling will extend to GPUs, allowing you to step through CPU code into GPU calls with the same developer experience.

Last week, Chris Lattner sat down for an interview on the Developer Voices podcast with Kris Jenkins. It was a wide-ranging episode that explored a variety of topics, including the motivations behind creating Mojo, what it offers to both Python and non-Python programmers alike, how it is built for performance, and which performance features actually matter. This post recaps a number of highlights from the podcast, edited for clarity and brevity. You can find the full 90 minute interview on YouTube.

Mojo as a language offers three main benefits, namely the 3 P’s: Performance, Programmability and Portability. It enables users to write fast code, do so easier than many alternative languages, and allows code to be run across different CPU platforms, with GPU support on the roadmap.

Open any introductory programming book and you’ll find several pages dedicated to structured programming concepts, i.e. making use of loops, conditions and functions extensively for better clarity and maintainability for your code. It helps you express your ideas and solutions neatly and elegantly in code. However, these benefits come at a cost: performance overhead. 

Did you catch Fabian's presentation at SINTEF? If you missed out, he'll be doing it again in Oslo, Norway on the 15th of February!By popular request, we're publishing the slides here, and will soon have video to go with it. Make sure to subscribe to our YouTube channel to get the alert!

Jeff, Abdul, and Chris presented a talk about Mojo and how it relates to MLIR and LLVM, at the LLVM Developer Conference 2023.

Today’s AI infrastructure is difficult to evaluate - so many converge on simple and quantifiable metrics like QPS, Latency and Throughput. This is one reason why today’s AI industry is rife with bespoke tools that provide high performance on benchmarks but have significant usability challenges in real-world AI deployment scenarios.

In this post, we describe Modular’s approach to solving this problem and its game-changing benefits, including a new standard in state-of-the-art (SOTA) performance on CPU as compared to existing solutions.