thushan/olla
High-performance lightweight proxy and load balancer for LLM infrastructure. Intelligent routing, automatic failover and unified model discovery across local and remote inference backends.
- Stars
- 320
- Forks
- 58
- Open issues
- 23
- Last push
- 6d ago
About olla
olla is a free, open-source AI inference tool by thushan, written mainly in Go. High-performance lightweight proxy and load balancer for LLM infrastructure. Intelligent routing, automatic failover and unified model discovery across local and remote inference backends. It has 320 GitHub stars and 58 forks, and was last updated 6d ago. It's released under the Apache-2.0 license, a standard open-source license.
- License
- Apache-2.0
- Language
- Go
- Stars
- 320
- Last update
- 6d ago
- Runs on Mac
- Not stated
- GPU
- Not stated
- Paid API key
- Not required
- Self-hostable
- Yes
Similar free tools
vllm95k stars · A high-throughput and memory-efficient inference and serving engine for LLMs
llama.cpp132k stars · LLM inference in C/C++
llama-cookbook19k stars · Welcome to the Llama Cookbook! This is your go to guide for Building with Llama: Getting started with Inference, Fine-Tuning, RAG. We also show you how to solve end to end problems using Llama model f
plano7.1k stars · Plano is an AI-native proxy server and data plane for agentic apps. Smart LLM routing, observability, agent orchestration, and guardrails so you stay focused on your agents core logic.
BentoML8.9k stars · The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!
TensorRT-LLM15k stars · TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT
olla FAQ
Is olla free?
Yes. olla is open source and free to use. It's released under the Apache-2.0 license, a standard open-source license.
Does olla run on a Mac?
The project doesn't say. Check its installation guide before you start.
Does olla need a GPU?
The project doesn't state hardware needs. AI models generally run much faster with a GPU.
Is olla still maintained?
Yes. It was last updated 6d ago and has 23 open issues on GitHub.
What are alternatives to olla?
Similar free, open-source tools include vllm, llama.cpp, llama-cookbook, plano.