Inference & serving

vllm-project/vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

Really freeApache-2.0Runs on MacPython

Open on GitHub Website

Stars: 95k
Forks: 24k
Open issues: 2.7k
Last push: today

Related topics

gpt llm pytorch model-serving transformer llm-serving inference llama