openai/tiktoken
tiktoken is a fast BPE tokeniser for use with OpenAI's models.
- Stars
- 20k
- Forks
- 1.7k
- Open issues
- 59
- Last push
- 2mo ago
About tiktoken
tiktoken is a free, open-source AI developer tool by openai, written mainly in Python. tiktoken is a fast BPE tokeniser for use with OpenAI's models. It has 20k GitHub stars and 1.7k forks, and was last updated 2mo ago. It's released under the MIT license, a standard open-source license.
- License
- MIT
- Language
- Python
- Stars
- 19,504
- Last update
- 2mo ago
- Runs on Mac
- Not stated
- GPU
- Not stated
- Paid API key
- Not stated
- Self-hostable
- Yes
Similar free tools
All Dev tools & SDKs tools →
tiktoken FAQ
Is tiktoken free?
Yes. tiktoken is open source and free to use. It's released under the MIT license, a standard open-source license.
Does tiktoken run on a Mac?
Its README doesn't say. Check the installation guide on GitHub before you start.
Does tiktoken need a GPU?
Its README doesn't state hardware needs. AI models generally run much faster with a GPU.
Is tiktoken still maintained?
It was last updated 2mo ago. Check recent issues on GitHub to judge how active it is.
What are alternatives to tiktoken?
Similar free, open-source tools include tiktoken-rs, SharpToken.