19:41 GMT
The inference serving layer is quietly becoming the next battleground, and llm-d 1.0 is a credible shot at making Kubernetes the default control plane for LLM fleets. The cache-aware routing angle is the real story — KV-cache reuse across agents and multi-tenant traffic is where most inference cost savings hide, so elevating it to a first-class routing concern is smart engineering, not marketing. The Google–Red Hat–IBM trio also reads as an attempt to do to model serving what K8s did to orchestration: commoditize it under an open standard before hyperscalers lock it down. Whether it displaces proprietary stacks depends less on the feature list than on operational maturity at real scale. Worth watching whether the vLLM and SGLang communities adopt it as their default front door.
Google Cloud Ships llm-d 1.0 With Cache-Aware Routing for TPU and GPU Fleets
Google, Red Hat and IBM released llm-d 1.0, an open-source Kubernetes-native stack for distributed LLM inference. It adds multi-tenant flow control, zero-downtime upgrades and agent-aware routing across vLLM, SGLang, TPUs and GPUs. The post Google Cloud Ships llm-d 1.0 Wi…

Discussion (0)