Helldez/BigMoeOnEdge
Run MoE models bigger than your RAM. Frontier-size MoE on a phone, laptop, edge. Lossless, on stock llama.cpp. An adaptive streaming optimizer. Any hardware, any OS, any model, any quantization, behin
- Stars
- 649
- Forks
- 71
- Open issues
- 19
- Last push
- today
About BigMoeOnEdge
BigMoeOnEdge is a free, open-source AI inference tool by Helldez, written mainly in C++. Run MoE models bigger than your RAM. Frontier-size MoE on a phone, laptop, edge. Lossless, on stock llama.cpp. An adaptive streaming optimizer. Any hardware, any OS, any model, any quantization, behin. It has 649 GitHub stars and 71 forks, and was last updated today. It's released under the Apache-2.0 license, a standard open-source license. According to the project, it works on macOS.
- License
- Apache-2.0
- Language
- C++
- Stars
- 649
- Last update
- today
- Runs on Mac
- Yes (as stated by the project)
- GPU
- Not stated
- Paid API key
- Not required
- Self-hostable
- Not stated
Similar free tools
llama.cpp132k stars · LLM inference in C/C++
vllm95k stars · A high-throughput and memory-efficient inference and serving engine for LLMs
llama-cookbook19k stars · Welcome to the Llama Cookbook! This is your go to guide for Building with Llama: Getting started with Inference, Fine-Tuning, RAG. We also show you how to solve end to end problems using Llama model f
llm-action25k stars · 本项目旨在分享大模型相关技术原理以及实战经验（大模型工程化、大模型应用落地）
BentoML8.9k stars · The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!
ai-infra-book6.9k stars · 《深入理解 AI Infra：量化分析与系统设计》（李博杰 著）开源书稿：从硬件约束和模型架构出发，量化推导 LLM 推理与训练系统设计。含全书正文、PDF、配套计算工具与实验
BigMoeOnEdge FAQ
Is BigMoeOnEdge free?
Yes. BigMoeOnEdge is open source and free to use. It's released under the Apache-2.0 license, a standard open-source license.
Does BigMoeOnEdge run on a Mac?
Yes. According to the project, BigMoeOnEdge works on macOS.
Does BigMoeOnEdge need a GPU?
The project doesn't state hardware needs. AI models generally run much faster with a GPU.
Is BigMoeOnEdge still maintained?
Yes. It was last updated today and has 19 open issues on GitHub.
What are alternatives to BigMoeOnEdge?
Similar free, open-source tools include llama.cpp, vllm, llama-cookbook, llm-action.