paulpierre/markdown-crawler
A multithreaded 🕸️ web crawler that recursively crawls a website and creates a 🔽 markdown file for each page, designed for LLM RAG
- Stars
- 477
- Forks
- 53
- Open issues
- 5
- Last push
- 3mo ago
About markdown-crawler
markdown-crawler is a free, open-source LLM chat app by paulpierre, written mainly in Python. A multithreaded 🕸️ web crawler that recursively crawls a website and creates a 🔽 markdown file for each page, designed for LLM RAG. It has 477 GitHub stars and 53 forks, and was last updated 3mo ago. It's released under the MIT license, a standard open-source license.
- License
- MIT
- Language
- Python
- Stars
- 477
- Last update
- 3mo ago
- Runs on Mac
- Not stated
- GPU
- Not stated
- Paid API key
- Not stated
- Self-hostable
- Yes
Similar free tools
open-webui156k stars · User-friendly AI Interface (Supports Ollama, OpenAI API, ...)
md13k stars · ✍ WeChat Markdown Editor | 一款高度简洁的微信 Markdown 编辑器：支持 Markdown 语法、自定义主题样式、内容管理、多图床、AI 助手等特性
khoj38k stars · Your AI second brain. Self-hostable. Get answers from the web or your docs. Build custom agents, schedule automations, do deep research. Turn any online or local LLM into your personal, autonomous AI
Open-LLM-VTuber14k stars · Talk to any LLM with hands-free voice interaction, voice interruption, and Live2D avatar running locally across platforms
llm-scraper6.9k stars · Turn any webpage into structured data using LLMs
happy-llm35k stars · 📚 从零开始构建大模型
markdown-crawler FAQ
Is markdown-crawler free?
Yes. markdown-crawler is open source and free to use. It's released under the MIT license, a standard open-source license.
Does markdown-crawler run on a Mac?
The project doesn't say. Check its installation guide before you start.
Does markdown-crawler need a GPU?
The project doesn't state hardware needs. AI models generally run much faster with a GPU.
Is markdown-crawler still maintained?
It was last updated 3mo ago. Check recent issues on GitHub to judge how active it is.
What are alternatives to markdown-crawler?
Similar free, open-source tools include open-webui, md, khoj, Open-LLM-VTuber.