crawler-user-agents
✅ Recently maintainedSyntactic patterns of HTTP user-agents used by bots / robots / crawlers / scrapers / spiders. pull-request welcome :star:
Go · MIT · Last push 2026-10-31
Official repository & installation ↗GitValue · Value evidence
Unscored · Evidence coverage 0%
Provisional rubric; documentation and runtime testing are separate. Missing evidence does not imply low value.
Star history
View historical data
| Date | Stars |
|---|---|
| 2026-10-05 | 1,407 |
| 2026-10-06 | 1,407 |
| 2026-10-07 | 1,407 |
| 2026-10-08 | 1,408 |
| 2026-10-09 | 1,408 |
| 2026-10-10 | 1,408 |
| 2026-10-11 | 1,408 |
| 2026-10-12 | 1,408 |
| 2026-10-13 | 1,409 |
| 2026-10-14 | 1,409 |
| 2026-10-15 | 1,411 |
| 2026-10-16 | 1,411 |
| 2026-10-17 | 1,411 |
| 2026-10-18 | 1,411 |
| 2026-10-19 | 1,411 |
| 2026-10-20 | 1,411 |
| 2026-10-21 | 1,411 |
| 2026-10-22 | 1,411 |
| 2026-10-23 | 1,411 |
| 2026-10-24 | 1,411 |
| 2026-10-25 | 1,411 |
| 2026-10-26 | 1,411 |
| 2026-10-27 | 1,411 |
| 2026-10-28 | 1,412 |
| 2026-10-29 | 1,412 |
| 2026-10-30 | 1,412 |
| 2026-10-31 | 1,412 |
| 2026-11-01 | 1,412 |
| 2026-11-02 | 1,413 |
| 2026-11-03 | 1,413 |
Get the project
git clone https://github.com/monperrus/crawler-user-agents.gitCloning downloads the source. For dependencies and installation, follow the project README.
Related projects & alternative candidates
Related domains do not imply feature parity. Review official documentation and licenses.
Crawls and scrapes web pages into clean Markdown for AI agents and LLM pipelines.
★ 198,099crawl4aiOpen-source LLM-friendly web crawler and data extractor for AI pipelines.
★ 86,808obscuraRust-based antidetect headless browser compatible with Playwright and Puppeteer for stealthy AI agent web browsing and data scraping.
★ 30,231