Spider
Showing 7 of 7 projects
A fast and elegant scraping and crawling framework for Go, designed for extracting structured data from websites.
An incredibly fast web crawler designed for OSINT (Open Source Intelligence) data extraction.
A Node.js web crawler with server-side jQuery, rate limiting, and proxy support for efficient scraping.
An open-source intelligence (OSINT) tool for crawling and analyzing websites on the dark web and beyond.
A PHP class for detecting bots, crawlers, and spiders via user agent and HTTP headers.
An async Python web scraping micro-framework built on asyncio and aiohttp for fast, extensible crawling.
A preconfigured web crawler for backing up websites, producing WARC files with a live dashboard and dynamic ignore patterns.
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.