Contributors to the llm-d open-source project announced the release of llm-d 1.0, a distributed inference platform for large language models designed for production-scale deployments. The project was founded in 2025 by Red Hat, IBM (NYSE: IBM), Google Cloud, CoreWeave and Nvidia, and is now governed by the Cloud Native Computing Foundation (CNCF).

The 1.0 release adds multi-tenant quality-of-service controls, zero-downtime operations through router high availability and rolling updates, a stable v1 configuration API, and session-aware routing for agentic workloads. The platform supports multiple model servers, including vLLM and SGLang, and runs across cloud and on-premises GPU infrastructure.

The project reports more than 500 contributors from over 25 organizations. According to the announcement, demonstrations using benchmark workloads showed the platform serving a 753-billion-parameter model on Nvidia GPUs, delivering over 6.6 million output tokens per minute and supporting up to 3,000 concurrent coding agents. The project claims this resulted in per-token costs five to ten times lower than equivalent commercial API pricing for input-heavy agentic workloads.

In a separate use case, IBM Research reported using llm-d alongside open frontier models to achieve a prefix-cache hit rate of approximately 98% and a blended serving cost of about $0.11 per million tokens, at 11.5 times lower inference cost than an all-proprietary configuration.

Google Cloud said it is integrating llm-d 1.0 into Google Kubernetes Engine and enabling it across its TPU and Nvidia GPU portfolio. JetBrains said llm-d runs in production behind its Junie Lite plan. DigitalOcean said it relies on llm-d for its Serverless Inference and Managed Agents products, serving what it described as trillions of tokens.

AMD, Cohere, Mistral, Roblox and CoreWeave also confirmed participation in the project as contributors or adopters, according to the announcement.