EleutherAI/lm-evaluation-harness
A framework for few-shot evaluation of language models.
- Stars
- 14k
- Forks
- 3.7k
- Open issues
- 657
- Last push
- 1mo ago
About lm-evaluation-harness
lm-evaluation-harness is a free, open-source AI developer tool by EleutherAI, written mainly in Python. A framework for few-shot evaluation of language models. It has 14k GitHub stars and 3.7k forks, and was last updated 1mo ago. It's released under the MIT license, a standard open-source license. According to its README, it works on macOS.
- License
- MIT
- Language
- Python
- Stars
- 14,297
- Last update
- 1mo ago
- Runs on Mac
- Yes (per README)
- GPU
- Not stated
- Paid API key
- Not required
- Self-hostable
- Yes
Similar free tools
deepeval19k stars · The LLM Evaluation Framework
dsh-our-free-model35k stars · 在 dsh 里装上这个插件即可，无需登录、注册或填 API Key，就能使用包括 DeepSeek V4.1 Flash、Kimi K3 在内的前沿模型——完全免费，不限量。 All you do is install this plugin in dsh: no login, no sign-up, no API key — the frontier models are just there,
lmms-eval4.5k stars · One-for-All Multimodal Evaluation Toolkit Across Text, Image, Video, and Audio Tasks
helm3.0k stars · Holistic Evaluation of Language Models (HELM) is an open source Python framework created by the Center for Research on Foundation Models (CRFM) at Stanford for holistic, reproducible and transparent e
Foundation-Models-Framework-Lab1.2k stars · A practical lab for building, testing, and evaluating apps with Apple's Foundation Models framework.
helicone6.2k stars · 🧊 Open source LLM observability platform. One line of code to monitor, evaluate, and experiment. YC W23 🍓
lm-evaluation-harness FAQ
Is lm-evaluation-harness free?
Yes. lm-evaluation-harness is open source and free to use. It's released under the MIT license, a standard open-source license.
Does lm-evaluation-harness run on a Mac?
Yes. According to its README, lm-evaluation-harness works on macOS.
Does lm-evaluation-harness need a GPU?
Its README doesn't state hardware needs. AI models generally run much faster with a GPU.
Is lm-evaluation-harness still maintained?
It was last updated 1mo ago. Check recent issues on GitHub to judge how active it is.
What are alternatives to lm-evaluation-harness?
Similar free, open-source tools include deepeval, dsh-our-free-model, lmms-eval, helm.