bentoml/OpenLLM
Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.
- Stars
- 13k
- Forks
- 851
- Open issues
- 6
- Last push
- today
About OpenLLM
OpenLLM is a free, open-source AI inference tool by bentoml, written mainly in Python. Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud. It has 13k GitHub stars and 851 forks, and was last updated today. It's released under the Apache-2.0 license, a standard open-source license.
- License
- Apache-2.0
- Language
- Python
- Stars
- 12,560
- Last update
- today
- Runs on Mac
- Not stated
- GPU
- Not stated
- Paid API key
- Not required
- Self-hostable
- Yes
Similar free tools
vllm95k stars · A high-throughput and memory-efficient inference and serving engine for LLMs
llama-cookbook19k stars · Welcome to the Llama Cookbook! This is your go to guide for Building with Llama: Getting started with Inference, Fine-Tuning, RAG. We also show you how to solve end to end problems using Llama model f
llm-action25k stars · 本项目旨在分享大模型相关技术原理以及实战经验（大模型工程化、大模型应用落地）
BentoML8.9k stars · The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!
TensorRT-LLM15k stars · TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT
llama.cpp132k stars · LLM inference in C/C++
OpenLLM FAQ
Is OpenLLM free?
Yes. OpenLLM is open source and free to use. It's released under the Apache-2.0 license, a standard open-source license.
Does OpenLLM run on a Mac?
The project doesn't say. Check its installation guide before you start.
Does OpenLLM need a GPU?
The project doesn't state hardware needs. AI models generally run much faster with a GPU.
Is OpenLLM still maintained?
Yes. It was last updated today and has 6 open issues on GitHub.
What are alternatives to OpenLLM?
Similar free, open-source tools include vllm, llama-cookbook, llm-action, BentoML.