b4rtaz/distributed-llama
Distributed LLM inference. Connect home devices into a powerful cluster to accelerate LLM inference. More devices means faster inference.
- Stars
- 3.1k
- Forks
- 258
- Open issues
- 46
- Last push
- 9d ago
About distributed-llama
distributed-llama is a free, open-source AI inference tool by b4rtaz, written mainly in C++. Distributed LLM inference. Connect home devices into a powerful cluster to accelerate LLM inference. More devices means faster inference. It has 3.1k GitHub stars and 258 forks, and was last updated 9d ago. It's released under the MIT license, a standard open-source license. According to the project, it works on macOS.
- License
- MIT
- Language
- C++
- Stars
- 3,111
- Last update
- 9d ago
- Runs on Mac
- Yes (as stated by the project)
- GPU
- Not stated
- Paid API key
- Not stated
- Self-hostable
- Not stated
Similar free tools
llm-action25k stars · 本项目旨在分享大模型相关技术原理以及实战经验（大模型工程化、大模型应用落地）
OpenLLM13k stars · Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.
TensorRT-LLM15k stars · TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT
vllm95k stars · A high-throughput and memory-efficient inference and serving engine for LLMs
ai-infra-book6.9k stars · 《深入理解 AI Infra：量化分析与系统设计》（李博杰 著）开源书稿：从硬件约束和模型架构出发，量化推导 LLM 推理与训练系统设计。含全书正文、PDF、配套计算工具与实验
distributed-llama FAQ
Is distributed-llama free?
Yes. distributed-llama is open source and free to use. It's released under the MIT license, a standard open-source license.
Does distributed-llama run on a Mac?
Yes. According to the project, distributed-llama works on macOS.
Does distributed-llama need a GPU?
The project doesn't state hardware needs. AI models generally run much faster with a GPU.
Is distributed-llama still maintained?
Yes. It was last updated 9d ago and has 46 open issues on GitHub.
What are alternatives to distributed-llama?
Similar free, open-source tools include llm-action, OpenLLM, TensorRT-LLM, vllm.