openai/evals
Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.
- Stars
- 20k
- Forks
- 3.1k
- Open issues
- 180
- Last push
- 6mo ago
About evals
evals is a free, open-source AI developer tool by openai, written mainly in Python. Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks. It has 20k GitHub stars and 3.1k forks, and was last updated 6mo ago. GitHub doesn't detect a standard license, so check the repository before using it commercially.
- License
- Not detected
- Commercial use
- No license
- Language
- Python
- Stars
- 19,653
- Last update
- 6mo ago
- Runs on Mac
- Not stated
- GPU
- Not stated
- Paid API key
- Required
- Self-hostable
- Yes
Similar free tools
promptfoo26k stars · Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line
opik23k stars · Monitor and evaluate LLM applications and AI agents with comprehensive tracing and dashboards.
giskard-oss5.9k stars · 🐢 Open-Source Evaluation & Testing library for LLM Agents
solon2.8k stars · 🔥 Java enterprise application development framework for full scenario: Restrained, Efficient, Open, Ecologicalll!!! 700% higher concurrency 50% memory savings Startup is 10 times faster. Packing 90%
llm-gateway7.6k stars · Open Source LLM Gateway 🦚
litellm62k stars · Unified interface for calling multiple LLM providers with a single API.
evals FAQ
Can I use evals commercially?
Unclear. No license was detected, which legally means all rights are reserved. Ask the author before using it commercially. This is a plain-English summary, not legal advice.
Is evals free?
evals is open source, but it needs a paid third-party API key to work. GitHub doesn't detect a standard license, so check the repository before using it commercially.
Does evals run on a Mac?
The project doesn't say. Check its installation guide before you start.
Does evals need a GPU?
The project doesn't state hardware needs. AI models generally run much faster with a GPU.
Is evals still maintained?
It was last updated 6mo ago. Check recent issues on GitHub to judge how active it is.
What are alternatives to evals?
Similar free, open-source tools include promptfoo, opik, giskard-oss, solon.