ovg-project/kvcached
Virtualized Elastic KV Cache for Dynamic GPU Sharing and Beyond
- Stars
- 1.9k
- Forks
- 227
- Open issues
- 53
- Last push
- today
About kvcached
kvcached is a free, open-source AI inference tool by ovg-project, written mainly in Python. Virtualized Elastic KV Cache for Dynamic GPU Sharing and Beyond. It has 1.9k GitHub stars and 227 forks, and was last updated today. It's released under the Apache-2.0 license, a standard open-source license.
- License
- Apache-2.0
- Language
- Python
- Stars
- 1,890
- Last update
- today
- Runs on Mac
- Not stated
- GPU
- Not stated
- Paid API key
- Not required
- Self-hostable
- Not stated
Similar free tools
vllm95k stars · A high-throughput and memory-efficient inference and serving engine for LLMs
TensorRT-LLM15k stars · TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT
BentoML8.9k stars · The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!
sglang38k stars · SGLang is a high-performance serving framework for large language models and multimodal models.
Mooncake6.9k stars · Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.
kvcached FAQ
Is kvcached free?
Yes. kvcached is open source and free to use. It's released under the Apache-2.0 license, a standard open-source license.
Does kvcached run on a Mac?
The project doesn't say. Check its installation guide before you start.
Does kvcached need a GPU?
The project doesn't state hardware needs. AI models generally run much faster with a GPU.
Is kvcached still maintained?
Yes. It was last updated today and has 53 open issues on GitHub.
What are alternatives to kvcached?
Similar free, open-source tools include vllm, TensorRT-LLM, BentoML, OpenLLM.