zurawiki/tiktoken-rs
Ready-made tokenizer library for working with GPT and tiktoken
- Stars
- 408
- Forks
- 76
- Open issues
- 2
- Last push
- 28d ago
About tiktoken-rs
tiktoken-rs is a free, open-source AI developer tool by zurawiki, written mainly in Rust. Ready-made tokenizer library for working with GPT and tiktoken. It has 408 GitHub stars and 76 forks, and was last updated 28d ago. It's released under the MIT license, a standard open-source license.
- License
- MIT
- Language
- Rust
- Stars
- 408
- Last update
- 28d ago
- Runs on Mac
- Not stated
- GPU
- Not stated
- Paid API key
- Not stated
- Self-hostable
- Not stated
Similar free tools
litellm62k stars · The fastest, litest AI Gateway. Rust core with Python SDK. Call 100+ LLM APIs in OpenAI (or native) format with cost tracking, guardrails, load balancing, and logging [Bedrock, Azure, OpenAI, Anthropi
openai-cookbook77k stars · Examples and guides for using the OpenAI API
polars40k stars · Extremely fast Query Engine for DataFrames, written in Rust
tiktoken-rs FAQ
Is tiktoken-rs free?
Yes. tiktoken-rs is open source and free to use. It's released under the MIT license, a standard open-source license.
Does tiktoken-rs run on a Mac?
The project doesn't say. Check its installation guide before you start.
Does tiktoken-rs need a GPU?
The project doesn't state hardware needs. AI models generally run much faster with a GPU.
Is tiktoken-rs still maintained?
Yes. It was last updated 28d ago and has 2 open issues on GitHub.
What are alternatives to tiktoken-rs?
Similar free, open-source tools include litellm, openai-cookbook, openai-python, openai-node.