Web Crawling
Showing 22 of 22 projects
A Python library and CLI tool for web crawling, scraping, and extracting main text, metadata, and comments from web pages.
A lightweight, efficient, and fast high-level web crawling and scraping framework for .NET.
A framework-agnostic Ruby gem for generating XML sitemaps with Rails integration and support for multiple sitemap extensions.
A high-level Ruby API for controlling Chrome/Chromium browsers directly via the Chrome DevTools Protocol.
A Python security analysis tool that automatically discovers and reports comprehensive information about a given domain.
A high-level web crawling and scraping framework for Elixir, designed for data extraction and processing.
A versatile Ruby web spidering library for crawling sites, domains, or specific links with extensive filtering and callback support.
A fast, powerful, and extensible web crawling and scraping framework for Go, inspired by Scrapy.
A self-hosted URL to PNG generator with parallel rendering via Playwright and configurable storage caching.
A Go library for generating various types of XML sitemaps with support for search engine pinging and cloud storage.
Python command-line tools and libraries for handling, validating, and converting WARC and ARC web archive files.
Serverless data pipeline for crawling PDFs from the web and extracting structured data using AWS Textract.
A reliable, flexible, and fast Rust framework for web crawling and request-response services.
An Elixir library for generating sitemap.xml files with support for news, image, video, and mobile sitemaps.
A native Rust port of Google's robots.txt parser and matcher library, preserving all original behavior.
Converts HTTrack website crawls into standardized WARC files for web archiving and preservation.
A Ruby gem for customizable text tokenization, useful for web crawling and natural language processing.
A CLI tool that verifies the current status of URIs in text files like markdown documentation.
A fast high-level web crawling and scraping framework for Elixir, built on Broadway.
A Go library for parsing XML sitemaps, robots.txt, and gzipped sitemaps with configurable rules and concurrent fetching.
A Go library for generating XML sitemaps with a simple, fluent API.
A .NET library for generating and managing XML sitemaps and sitemap indexes for websites.
Related Tags
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.