Spider
Showing 13 of 13 projects
A fast and elegant scraping and crawling framework for Go, designed for extracting structured data from websites.
An incredibly fast web crawler designed for OSINT (Open Source Intelligence) data extraction.
A Node.js web crawler with server-side jQuery, rate limiting, and proxy support for efficient scraping.
An open-source intelligence (OSINT) tool for crawling and analyzing websites on the dark web and beyond.
A PHP class for detecting bots, crawlers, and spiders via user agent and HTTP headers.
An async Python web scraping micro-framework built on asyncio and aiohttp for fast, extensible crawling.
A preconfigured web crawler for backing up websites, producing WARC files with a live dashboard and dynamic ignore patterns.
An advanced Cross-Site Request Forgery (CSRF) audit and exploitation toolkit for security testing.
A high-level web crawling and scraping framework for Elixir, designed for data extraction and processing.
A high-performance web crawler and scraper built in Elixir with worker pooling and rate limiting.
A versatile Ruby web spidering library for crawling sites, domains, or specific links with extensive filtering and callback support.
A tool that extracts all GraphQL endpoints from a given domain using subdomain enumeration, script analysis, and brute force.
A reliable, flexible, and fast Rust framework for web crawling and request-response services.
Related Tags
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.