patchy631/time-to-first-token
A 10-week, 30-minutes-a-day roadmap for LLM inference serving and optimization. vLLM, SGLang, quantization, speculative decoding, benchmarking.
- Stars
- 1.0k
- Forks
- 111
- Open issues
- 0
- Last push
- 2mo ago
About time-to-first-token
time-to-first-token is a free, open-source AI inference tool by patchy631, written mainly in HTML. A 10-week, 30-minutes-a-day roadmap for LLM inference serving and optimization. vLLM, SGLang, quantization, speculative decoding, benchmarking. It has 1.0k GitHub stars and 111 forks, and was last updated 2mo ago. It's released under the Apache-2.0 license, a standard open-source license.
- License
- Apache-2.0
- Language
- HTML
- Stars
- 1,001
- Last update
- 2mo ago
- Runs on Mac
- Not stated
- GPU
- Not stated
- Paid API key
- Not required
- Self-hostable
- Not stated
Similar free tools
vllm95k stars · A high-throughput and memory-efficient inference and serving engine for LLMs
llm-action25k stars · 本项目旨在分享大模型相关技术原理以及实战经验（大模型工程化、大模型应用落地）
BentoML8.9k stars · The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!
nano-vllm16k stars · Nano vLLM
TensorRT-LLM15k stars · TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT
time-to-first-token FAQ
Is time-to-first-token free?
Yes. time-to-first-token is open source and free to use. It's released under the Apache-2.0 license, a standard open-source license.
Does time-to-first-token run on a Mac?
The project doesn't say. Check its installation guide before you start.
Does time-to-first-token need a GPU?
The project doesn't state hardware needs. AI models generally run much faster with a GPU.
Is time-to-first-token still maintained?
It was last updated 2mo ago. Check recent issues on GitHub to judge how active it is.
What are alternatives to time-to-first-token?
Similar free, open-source tools include vllm, llm-action, BentoML, OpenLLM.