huggingface/tokenizers
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
- Stars
- 11k
- Forks
- 1.2k
- Open issues
- 115
- Last push
- today
About tokenizers
tokenizers is a free, open-source LLM chat app by huggingface, written mainly in Rust. 💥 Fast State-of-the-Art Tokenizers optimized for Research and Production. It has 11k GitHub stars and 1.2k forks, and was last updated today. It's released under the Apache-2.0 license, a standard open-source license.
- License
- Apache-2.0
- Language
- Rust
- Stars
- 11,190
- Last update
- today
- Runs on Mac
- Not stated
- GPU
- Not stated
- Paid API key
- Not stated
- Self-hostable
- Yes
Similar free tools
Hands-On-Large-Language-Models30k stars · Official code repo for the O'Reilly Book - "Hands-On Large Language Models"
gpt_academic72k stars · 为GPT/GLM等LLM大语言模型提供实用化交互接口，特别优化论文阅读/润色/写作体验，模块化设计，支持自定义快捷按钮&函数插件，支持Python和C++等项目剖析&自译解功能，PDF/LaTex论文翻译&总结功能，支持并行问询多种LLM模型，支持chatglm3等本地模型。接入通义千问, deepseekcoder, 讯飞星火, 文心一言, llama2, rwkv, claude2, moss
Qwen22k stars · The official repo of Qwen (通义千问) chat & pretrained large language model proposed by Alibaba Cloud.
gpt4free67k stars · The official gpt4free repository | various collection of powerful language models | opus 4.6 gpt 5.3 kimi 2.5 deepseek v3.2 gemini 3
RWKV-LM15k stars · RWKV (pronounced RwaKuv) is an RNN with great LLM performance, which can also be directly trained like a GPT transformer (parallelizable). We are at RWKV-7 "Goose". So it's combining the best of RNN a
MOSS12k stars · An open-source, tool-augmented conversational language model from Fudan University
tokenizers FAQ
Is tokenizers free?
Yes. tokenizers is open source and free to use. It's released under the Apache-2.0 license, a standard open-source license.
Does tokenizers run on a Mac?
The project doesn't say. Check its installation guide before you start.
Does tokenizers need a GPU?
The project doesn't state hardware needs. AI models generally run much faster with a GPU.
Is tokenizers still maintained?
Yes. It was last updated today and has 115 open issues on GitHub.
What are alternatives to tokenizers?
Similar free, open-source tools include Hands-On-Large-Language-Models, gpt_academic, Qwen, gpt4free.