What it does
When an AI language model reads a prompt, it builds up a kind of short-term memory called a KV cache. Normally this is thrown away or kept only in the graphics card's limited memory. LMCache stores it and lets the server reuse it for later requests that share the same text.
It can move this cached data to regular memory, local disks or remote storage, and share it between several AI servers. Its makers say this cuts repeated work and shortens the wait for the first word of an answer.
It plugs into popular open-source model-serving software and includes monitoring tools for teams running it in production.
Who it's for
Engineering teams that host large language models for many users.
Can I use it without coding?
Developers only. It's server software that needs technical setup alongside an AI serving engine. New to this? Start here.
Can I run it?
- Cost
- FreeNothing to pay or sign up for.
- Runs on
- Server
- Computer
- Made for servers or data centres
- Privacy
- Runs on your own device: your data stays with you
- API key
- Not needed
Try it
Using Claude Code, Cursor or another AI coding assistant? Paste the prompt and it will walk you through the setup.New to this?
Before you install
- Only download from the official GitHub page. Copies with similar names exist for many popular projects.
- License (Apache-2.0): Permissive: free to use, change and share, including for business.
- More on staying safe
Where people are talking about it
- LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM InferenceResearch paper · Oct 2025
- Lossless LLM 3x Throughput Increase by LMCacheHacker News · Jun 2025