Plain-AyI
← All projects
Running AI at scale

llm-d

Helps organisations serve large language models quickly and reliably to many users on Kubernetes server clusters.

What it does

llm-d is a set of tools for running large language models (the kind of AI behind chatbots) for lots of users at once. It runs on Kubernetes, a system for managing many servers together. It works on top of model servers like vLLM and SGLang, which do the actual work of running the models.

It adds smart traffic routing, so each request goes to the server best placed to answer it. It also manages the model's short-term memory (called the KV cache) by moving it to regular memory or disk. Other features split very large models across many graphics chips, scale servers up and down with demand, and handle big batches of offline jobs.

The project offers tested guides and setup recipes for different hardware and cloud providers. Its makers share benchmark results from partners showing faster responses and more work done per chip compared with simpler setups.

Who it's for

Engineering teams at companies or cloud providers who run AI models in production on their own server clusters.

Can I use it without coding?

Developers only. It needs a Kubernetes cluster, server hardware with AI accelerators and infrastructure know-how to deploy. New to this? Start here.

Can I run it?

Cost
FreeNothing to pay or sign up for.
Runs on
Server, Cloud
Computer
Made for servers or data centres
Privacy
Runs on your own device: your data stays with you

Try it

Official download page

Before you download

Plain-AyI is an independent guide: we link to each project's official pages, but we don't host, review or endorse the software, and we can't accept responsibility for how it works on your device. Please download and install at your own discretion.

Using Claude Code, Cursor or another AI coding assistant? Paste the prompt and it will walk you through the setup.New to this?

Before you install

  • Only download from the official GitHub page or the official download page above. Copies with similar names exist for many popular projects.
  • Plain-AyI is an independent guide and doesn't host or review the software it lists. Please check each project's official page before installing.
  • License (Apache-2.0): Permissive: free to use, change and share, including for business.
  • More on staying safe

Where people are talking about it