modelscope/evalscope
A streamlined and customizable framework for efficient large model (LLM, VLM, AIGC) evaluation and performance benchmarking.
- Stars
- 3.6k
- Forks
- 532
- Open issues
- 11
- Last push
- today
About evalscope
evalscope is a free, open-source AI developer tool by modelscope, written mainly in Python. A streamlined and customizable framework for efficient large model (LLM, VLM, AIGC) evaluation and performance benchmarking. It has 3.6k GitHub stars and 532 forks, and was last updated today. It's released under the Apache-2.0 license, a standard open-source license.
- License
- Apache-2.0
- Language
- Python
- Stars
- 3,576
- Last update
- today
- Runs on Mac
- Not stated
- GPU
- Not stated
- Paid API key
- Required
- Self-hostable
- Yes
Similar free tools
deepeval19k stars · The LLM Evaluation Framework
lm-evaluation-harness14k stars · A framework for few-shot evaluation of language models.
RagaAI-Catalyst16k stars · Python SDK for Agent AI Observability, Monitoring and Evaluation Framework. Includes features like agent, llm and tools tracing, debugging multi-agentic system, self-hosted dashboard and advanced anal
giskard-oss5.9k stars · 🐢 Open-Source Evaluation & Testing library for LLM Agents
helicone6.2k stars · 🧊 Open source LLM observability platform. One line of code to monitor, evaluate, and experiment. YC W23 🍓
lmms-eval4.5k stars · One-for-All Multimodal Evaluation Toolkit Across Text, Image, Video, and Audio Tasks
evalscope FAQ
Is evalscope free?
evalscope is open source, but it needs a paid third-party API key to work. It's released under the Apache-2.0 license, a standard open-source license.
Does evalscope run on a Mac?
The project doesn't say. Check its installation guide before you start.
Does evalscope need a GPU?
The project doesn't state hardware needs. AI models generally run much faster with a GPU.
Is evalscope still maintained?
Yes. It was last updated today and has 11 open issues on GitHub.
What are alternatives to evalscope?
Similar free, open-source tools include deepeval, lm-evaluation-harness, RagaAI-Catalyst, giskard-oss.