InftyAI/llmaz
☸️ Easy, advanced inference platform for large language models on Kubernetes. 🌟 Star to support our work!
- Stars
- 315
- Forks
- 48
- Open issues
- 39
- Last push
- 1mo ago
About llmaz
llmaz is a free, open-source AI inference tool by InftyAI, written mainly in Go. ☸️ Easy, advanced inference platform for large language models on Kubernetes. 🌟 Star to support our work!. It has 315 GitHub stars and 48 forks, and was last updated 1mo ago. It's released under the Apache-2.0 license, a standard open-source license.
- License
- Apache-2.0
- Language
- Go
- Stars
- 315
- Last update
- 1mo ago
- Runs on Mac
- Not stated
- GPU
- Not stated
- Paid API key
- Not required
- Self-hostable
- Not stated
Similar free tools
vllm95k stars · A high-throughput and memory-efficient inference and serving engine for LLMs
BentoML8.9k stars · The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!
TensorRT-LLM15k stars · TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT
text-embeddings-inference5.1k stars · A blazing fast inference solution for text embeddings models
shimmy6.0k stars · ⚡ Pure-Rust WebGPU inference engine — OpenAI-API compatible, GGUF native, runs on any GPU. No Python. No llama.cpp. Single binary.
llmaz FAQ
Is llmaz free?
Yes. llmaz is open source and free to use. It's released under the Apache-2.0 license, a standard open-source license.
Does llmaz run on a Mac?
The project doesn't say. Check its installation guide before you start.
Does llmaz need a GPU?
The project doesn't state hardware needs. AI models generally run much faster with a GPU.
Is llmaz still maintained?
It was last updated 1mo ago. Check recent issues on GitHub to judge how active it is.
What are alternatives to llmaz?
Similar free, open-source tools include vllm, BentoML, TensorRT-LLM, text-embeddings-inference.