Best free open-source LLM inference and serving tools
The 60 most-starred inference & serving repos on GitHub. Stars and activity are verified from the GitHub API and updated every night.
Updated · data refreshed nightly from GitHub
LLM inference in C/C++
Tensors and Dynamic neural networks in Python with strong GPU acceleration
A high-throughput and memory-efficient inference and serving engine for LLMs
Code at the speed of thought – Zed is a high-performance, multiplayer code editor from the creators of Atom and Tree-sitter.
The Cloud Native Application Proxy
Deep Learning for humans
Learn how to develop, deploy and iterate on production-grade ML applications.
🦍 The API and AI Gateway
Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.
Making large AI models cheaper, faster and more accessible
SGLang is a high-performance serving framework for large language models and multimodal models.
Visualizer for neural network, deep learning and machine learning models
Pretrain, finetune ANY AI model of ANY size on 1 or 10,000+ GPUs with zero code changes.
See, hear, and interact on your local devices.
The easiest, and most secure way to access and protect all of your infrastructure.
DeepInfer: High-performance, large-scale inference framework for DeepSeek models
Fast and accurate AI powered file content types detection
Welcome to the Llama Cookbook! This is your go to guide for Building with Llama: Getting started with Inference, Fine-Tuning, RAG. We also show you how to solve end to end problems using Llama model f
The container platform tailored for Kubernetes multi-cloud, datacenter, and edge management ⎈ 🖥 ☁️
Nano vLLM
TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT
Unified framework for building enterprise RAG pipelines with small, specialized models
Scalene: a high-performance, high-precision CPU, GPU, and memory profiler for Python with AI-powered optimization proposals
✍🏻 Source Code Deep Dives, System Design & Engineering Blogs | Halfrost-Field 冰霜之地：源码解析、系统设计与工程实践笔记
Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
Open Source API and AI Gateway supporting REST, GraphQL, TCP, gRPC and MCP (Model Context Protocol)
The AI Compute Platform for frontier teams. SkyPilot turns fragmented AI compute into one AI supercomputer, so frontier AI teams build custom intelligence faster.
Python bindings for llama.cpp
Build, Manage and Deploy AI/ML Systems
High-speed Large Language Model Serving for Local Deployment
A fast, scalable, high performance Gradient Boosting on Decision Trees library, used for ranking, classification, regression and other machine learning tasks for Python, R, Java, C++. Supports computa
A 2.78-trillion-parameter Kimi K3 running inference on a single CPU in 8.24 GB of RAM. Portable C99: no BLAS, no framework, no GPU.
The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!
Giving Kubernetes Superpowers to everyone
LMDeploy is a toolkit for compressing, deploying, and serving LLMs.
Open-source observability for your GenAI or LLM application, based on OpenTelemetry
An AI-powered next-generation open source real-time observability system.
A framework for efficient model inference with omni-modality models
Plano is an AI-native proxy server and data plane for agentic apps. Smart LLM routing, observability, agent orchestration, and guardrails so you stay focused on your agents core logic.
Gemma 4 26B-A4B inference in ~2 GB of RAM on any M-series MacBook
ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution
Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.
《深入理解 AI Infra：量化分析与系统设计》（李博杰 著）开源书稿：从硬件约束和模型架构出发，量化推导 LLM 推理与训练系统设计。含全书正文、PDF、配套计算工具与实验
Standardized Distributed Generative and Predictive AI Inference Platform for Scalable, Multi-Framework Deployment on Kubernetes
Reliable model swapping for any local OpenAI/Anthropic compatible server - llama.cpp, vllm, etc
Observability and enforcement for AI agent harnesses. Capture every run and runtime reliability with policy enforcement.
⚡ Pure-Rust WebGPU inference engine — OpenAI-API compatible, GGUF native, runs on any GPU. No Python. No llama.cpp. Single binary.
WebGPU components for React, Vue, Svelte, Solid, JS & Framer
cube studio开源云原生一站式机器学习/深度学习/大模型AI平台，mlops算法链路全流程，算力租赁平台，notebook在线开发，拖拉拽任务流pipeline编排，多机多卡分布式训练，超参搜索，推理服务VGPU虚拟化，边缘计算，标注平台自动化标注，deepseek等大模型sft微调/奖励模型/强化学习训练，vllm/ollama/mindie大模型多机推理，私有知识库，AI模型市场，支持
A blazing fast inference solution for text embeddings models
AITemplate is a Python framework which renders neural network into high performance CUDA/HIP C++ code. Specialized for FP16 TensorCore (NVIDIA GPU) and MatrixCore (AMD GPU) inference.
An OBS plugin for removing background in portrait images (video), making it easy to replace the background when recording or streaming.
A repository for storing models that have been inter-converted between various frameworks. Supported frameworks are TensorFlow, PyTorch, ONNX, OpenVINO, TFJS, TFTRT, TensorFlowLite (Float32/16/INT8),
LightLLM is a Python-based LLM (Large Language Model) inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance.
eBPF Observability - Distributed Tracing and Profiling
Generative AI reference workflows optimized for accelerated infrastructure and microservice architecture.
Bulk port forwarding Kubernetes services for local development.
State of the Art Natural Language Processing