EvolvingLMMs-Lab/lmms-eval
One-for-All Multimodal Evaluation Toolkit Across Text, Image, Video, and Audio Tasks
- Stars
- 4.5k
- Forks
- 676
- Open issues
- 27
- Last push
- today
About lmms-eval
lmms-eval is a free, open-source AI developer tool by EvolvingLMMs-Lab, written mainly in Python. One-for-All Multimodal Evaluation Toolkit Across Text, Image, Video, and Audio Tasks. It has 4.5k GitHub stars and 676 forks, and was last updated today. GitHub doesn't detect a standard license, so check the repository before using it commercially.
- License
- Not detected
- Language
- Python
- Stars
- 4,462
- Last update
- today
- Runs on Mac
- Not stated
- GPU
- Not stated
- Paid API key
- Not stated
- Self-hostable
- Not stated
Similar free tools
deepeval19k stars · The LLM Evaluation Framework
lm-evaluation-harness14k stars · A framework for few-shot evaluation of language models.
helicone6.2k stars · 🧊 Open source LLM observability platform. One line of code to monitor, evaluate, and experiment. YC W23 🍓
dsh-our-free-model35k stars · 在 dsh 里装上这个插件即可，无需登录、注册或填 API Key，就能使用包括 DeepSeek V4.1 Flash、Kimi K3 在内的前沿模型——完全免费，不限量。 All you do is install this plugin in dsh: no login, no sign-up, no API key — the frontier models are just there,
VLMEvalKit4.5k stars · Open-source evaluation toolkit of large multi-modality models (LMMs), support 220+ LMMs, 80+ benchmarks
RagaAI-Catalyst16k stars · Python SDK for Agent AI Observability, Monitoring and Evaluation Framework. Includes features like agent, llm and tools tracing, debugging multi-agentic system, self-hosted dashboard and advanced anal
lmms-eval FAQ
Is lmms-eval free?
Yes. lmms-eval is open source and free to use. GitHub doesn't detect a standard license, so check the repository before using it commercially.
Does lmms-eval run on a Mac?
The project doesn't say. Check its installation guide before you start.
Does lmms-eval need a GPU?
The project doesn't state hardware needs. AI models generally run much faster with a GPU.
Is lmms-eval still maintained?
Yes. It was last updated today and has 27 open issues on GitHub.
What are alternatives to lmms-eval?
Similar free, open-source tools include deepeval, lm-evaluation-harness, helicone, dsh-our-free-model.