Crawling Framework

3 projects

Showing 3 of 3 projects

An open-source Java web crawler that provides a simple interface for multi-threaded web crawling.

An open-source, extensible, web-scale, archival-quality web crawler from the Internet Archive.

An async Python web scraping micro-framework built on asyncio and aiohttp for fast, extensible crawling.

Community-curated · Updated weekly · 100% open source

Found a gem we're missing?

Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.