huggingface/lighteval
Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends
- Stars
- 2.6k
- Forks
- 588
- Open issues
- 244
- Last push
- 16d ago
About lighteval
lighteval is a free, open-source AI developer tool by huggingface, written mainly in Python. Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends. It has 2.6k GitHub stars and 588 forks, and was last updated 16d ago. It's released under the MIT license, a standard open-source license.
- License
- MIT
- Language
- Python
- Stars
- 2,566
- Last update
- 16d ago
- Runs on Mac
- Not stated
- GPU
- Not stated
- Paid API key
- Not stated
- Self-hostable
- Not stated
Similar free tools
deepeval19k stars · The LLM Evaluation Framework
lm-evaluation-harness14k stars · A framework for few-shot evaluation of language models.
RagaAI-Catalyst16k stars · Python SDK for Agent AI Observability, Monitoring and Evaluation Framework. Includes features like agent, llm and tools tracing, debugging multi-agentic system, self-hosted dashboard and advanced anal
helicone6.2k stars · 🧊 Open source LLM observability platform. One line of code to monitor, evaluate, and experiment. YC W23 🍓
giskard-oss5.9k stars · 🐢 Open-Source Evaluation & Testing library for LLM Agents
lmms-eval4.5k stars · One-for-All Multimodal Evaluation Toolkit Across Text, Image, Video, and Audio Tasks
lighteval FAQ
Is lighteval free?
Yes. lighteval is open source and free to use. It's released under the MIT license, a standard open-source license.
Does lighteval run on a Mac?
The project doesn't say. Check its installation guide before you start.
Does lighteval need a GPU?
The project doesn't state hardware needs. AI models generally run much faster with a GPU.
Is lighteval still maintained?
Yes. It was last updated 16d ago and has 244 open issues on GitHub.
What are alternatives to lighteval?
Similar free, open-source tools include deepeval, lm-evaluation-harness, RagaAI-Catalyst, helicone.