Michael-A-Kuykendall/shimmy
⚡ Pure-Rust WebGPU inference engine — OpenAI-API compatible, GGUF native, runs on any GPU. No Python. No llama.cpp. Single binary.
- Stars
- 6.0k
- Forks
- 587
- Open issues
- 9
- Last push
- 3d ago
About shimmy
shimmy is a free, open-source AI inference tool by Michael-A-Kuykendall, written mainly in Rust. ⚡ Pure-Rust WebGPU inference engine — OpenAI-API compatible, GGUF native, runs on any GPU. No Python. No llama.cpp. Single binary. It has 6.0k GitHub stars and 587 forks, and was last updated 3d ago. It's released under the Apache-2.0 license, a standard open-source license. According to the project, it works on macOS.
- License
- Apache-2.0
- Language
- Rust
- Stars
- 5,988
- Last update
- 3d ago
- Runs on Mac
- Yes (as stated by the project)
- GPU
- Not stated
- Paid API key
- Not required
- Self-hostable
- Not stated
Similar free tools
vllm95k stars · A high-throughput and memory-efficient inference and serving engine for LLMs
llama-cookbook19k stars · Welcome to the Llama Cookbook! This is your go to guide for Building with Llama: Getting started with Inference, Fine-Tuning, RAG. We also show you how to solve end to end problems using Llama model f
BentoML8.9k stars · The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!
ColossalAI41k stars · Making large AI models cheaper, faster and more accessible
TensorRT-LLM15k stars · TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT
Discussions
shimmy FAQ
Is shimmy free?
Yes. shimmy is open source and free to use. It's released under the Apache-2.0 license, a standard open-source license.
Does shimmy run on a Mac?
Yes. According to the project, shimmy works on macOS.
Does shimmy need a GPU?
The project doesn't state hardware needs. AI models generally run much faster with a GPU.
Is shimmy still maintained?
Yes. It was last updated 3d ago and has 9 open issues on GitHub.
What are alternatives to shimmy?
Similar free, open-source tools include vllm, llama-cookbook, BentoML, ColossalAI.