open-compass/VLMEvalKit
Open-source evaluation toolkit of large multi-modality models (LMMs), support 220+ LMMs, 80+ benchmarks
- Stars
- 4.5k
- Forks
- 778
- Open issues
- 226
- Last push
- today
About VLMEvalKit
VLMEvalKit is a free, open-source AI developer tool by open-compass, written mainly in Python. Open-source evaluation toolkit of large multi-modality models (LMMs), support 220+ LMMs, 80+ benchmarks. It has 4.5k GitHub stars and 778 forks, and was last updated today. It's released under the Apache-2.0 license, a standard open-source license.
- License
- Apache-2.0
- Language
- Python
- Stars
- 4,450
- Last update
- today
- Runs on Mac
- Not stated
- GPU
- Not stated
- Paid API key
- Not stated
- Self-hostable
- Not stated
Similar free tools
openai-cookbook77k stars · Examples and guides for using the OpenAI API
new-api50k stars · A unified AI model hub for aggregation & distribution. It supports cross-converting various LLMs into OpenAI-compatible, Claude-compatible, or Gemini-compatible formats. A centralized gateway for pers
litellm62k stars · The fastest, litest AI Gateway. Rust core with Python SDK. Call 100+ LLM APIs in OpenAI (or native) format with cost tracking, guardrails, load balancing, and logging [Bedrock, Azure, OpenAI, Anthropi
deepeval19k stars · The LLM Evaluation Framework
dsh-our-free-model35k stars · 在 dsh 里装上这个插件即可，无需登录、注册或填 API Key，就能使用包括 DeepSeek V4.1 Flash、Kimi K3 在内的前沿模型——完全免费，不限量。 All you do is install this plugin in dsh: no login, no sign-up, no API key — the frontier models are just there,
chatgpt-java3.6k stars · ChatGPT Java SDK。支持 GPT-4o、 GPT-5 API。开箱即用。An unofficial Java SDK for seamless integration with ChatGPT's GPT-5 and GPT-4 APIs. Ready-to-use, simple setup, and efficient for building AI-powered applic
VLMEvalKit FAQ
Is VLMEvalKit free?
Yes. VLMEvalKit is open source and free to use. It's released under the Apache-2.0 license, a standard open-source license.
Does VLMEvalKit run on a Mac?
The project doesn't say. Check its installation guide before you start.
Does VLMEvalKit need a GPU?
The project doesn't state hardware needs. AI models generally run much faster with a GPU.
Is VLMEvalKit still maintained?
Yes. It was last updated today and has 226 open issues on GitHub.
What are alternatives to VLMEvalKit?
Similar free, open-source tools include openai-cookbook, new-api, litellm, deepeval.