guoqingbao/xinfer
Blazing-fast LLM inference in pure Rust. No PyTorch and Python runtime.
- Stars
- 339
- Forks
- 48
- Open issues
- 47
- Last push
- 2d ago
About xinfer
xinfer is a free, open-source AI inference tool by guoqingbao, written mainly in Rust. Blazing-fast LLM inference in pure Rust. No PyTorch and Python runtime. It has 339 GitHub stars and 48 forks, and was last updated 2d ago. It's released under the MIT license, a standard open-source license.
- License
- MIT
- Language
- Rust
- Stars
- 339
- Last update
- 2d ago
- Runs on Mac
- Not stated
- GPU
- Not stated
- Paid API key
- Not stated
- Self-hostable
- Not stated
Similar free tools
vllm95k stars · A high-throughput and memory-efficient inference and serving engine for LLMs
TensorRT-LLM15k stars · TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT
BentoML8.9k stars · The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!
failproofai6.0k stars · Observability and enforcement for AI agent harnesses. Capture every run and runtime reliability with policy enforcement.
nano-vllm16k stars · Nano vLLM
xinfer FAQ
Is xinfer free?
Yes. xinfer is open source and free to use. It's released under the MIT license, a standard open-source license.
Does xinfer run on a Mac?
The project doesn't say. Check its installation guide before you start.
Does xinfer need a GPU?
The project doesn't state hardware needs. AI models generally run much faster with a GPU.
Is xinfer still maintained?
Yes. It was last updated 2d ago and has 47 open issues on GitHub.
What are alternatives to xinfer?
Similar free, open-source tools include vllm, TensorRT-LLM, BentoML, failproofai.