Problem

Hippocratic AI builds safety-focused AI health agents that converse with patients, helping to close the global shortfall of 15 million healthcare workers. Their Polaris system orchestrates dozens of specialized models in parallel to ensure every interaction is clinically safe, with error rates lower than human clinicians. Hippocratic AI’s systems scale to contacting tens of thousands of patients daily and build trust that AI products can be used in highly regulated industries.

Every millisecond matters in real-time voice, and at Hippocratic AI's scale latency gains compound directly into better patient experience and per-node efficiency. Production deployments run across multiple frameworks, including SGLang and vLLM, with ongoing evaluation of emerging frameworks for additional latency headroom, alongside a hardware roadmap spanning NVIDIA, AMD, and future-generation accelerators.

Solution

Our partnership with Hippocratic AI is a joint effort where both teams worked together to integrate Modular's MAX framework into Hippocratic AI's inference pipelines with NVIDIA B300 GPUs. The evaluation benchmarked MAX against an existing SGLang deployment on 400B+ parameter models, with particular focus on tail latency and on the future portability of the underlying architecture to the heterogeneous hardware.

Modular has rebuilt the AI infrastructure stack from the ground up. From highly optimized, portable kernels written in Mojo, to model serving infrastructure with MAX, to cloud orchestration that can be deployed in Modular's cloud or yours. This vertically integrated approach, built over years of deep infrastructure investment, gives Modular an edge to extract performance against existing frameworks.

MAX delivered across every dimension that matters:

• Keep every conversation instant. MAX delivers sub-500ms mean time to first token (TTFT) and holds total generation time tight even at high concurrency, supporting responsive, natural interactions.
• Eliminate latency spikes that break trust. In healthcare, the worst-case interaction matters as much as the average one. MAX achieved approximately 30% faster P99 end-to-end latency in the evaluation for a critical dense production model, addressing the tail-latency spikes that would cause noticeable pauses mid-conversation.
• Scale to more patients per node. MAX delivered approximately 22% faster mean end-to-end latency at scale for a specific workload, contributing to the per-node efficiency gains of Hippocratic AI targets across its production stack.

Results

By adding MAX to its inference stack, Hippocratic AI opens up a heterogeneous deployment strategy across vendor hardware. The collaboration between Hippocratic AI and Modular is ongoing. Because MAX's portability comes from its optimized kernel library and scheduling architecture rather than vendor-specific glue, the same benefits extend to the large reasoning models becoming central to production AI deployments: supporting flexible, hardware-agnostic deployment for the frontier LLMs used in production.

About Hippocratic AI

Hippocratic AI has developed the safest generative AI Agents for healthcare. The company believes that generative AI has the ability to bring healthcare abundance to every person in the world. The company focuses on building non-diagnostic patient-facing clinical AI agents and does not allow its agents to be used to prescribe or diagnose. Hippocratic AI has received a total of $404 million in funding and is backed by leading investors, including Andreessen Horowitz, General Catalyst, Kleiner Perkins, Avenir, NVIDIA’s NVentures, Premji Invest, SV Angel, Google’s CapitalG, and numerous health systems. Learn more at https://hippocraticai.com/.

Request a demo of this use case

If you're deploying large language models for inference, request a demo today. Excited to chat!