Crawler
Showing 15 of 51 projects
A Go tool and library for downloading URLs and files from Common Crawl and Wayback Machine web archives.
A high-fidelity, user-scriptable archival web crawler using Chrome/Chromium to preserve JavaScript-rendered content.
An open-source price tracker that automatically monitors product prices from e-commerce sites and sends alerts when prices drop.
A reliable, flexible, and fast Rust framework for web crawling and request-response services.
A Go library that parses and deserializes HTML pages into structs using goquery and struct tags for web crawlers.
A command-line tool for archiving web pages into WARC files and replaying them locally.
A Node.js middleware that pre-renders JavaScript web apps for search engines and social media bots, improving SEO and link previews.
A GitHub pinned gist that displays your Medium follower count and latest articles with claps.
A fast high-level web crawling and scraping framework for Elixir, built on Broadway.
A Python DNS crawler that finds identical domain names across different TLDs by querying authoritative SOA records.
A .NET library for programmatically logging into Backpack.tf, a trading site for Team Fortress 2, CS:GO, and Dota 2.
A Python CLI tool for converting web pages to clean, readable Markdown format, optimized for feeding documentation into LLMs.
A Node.js package to search and fetch Super Mario Maker course data from the official Nintendo bookmark site.
A .NET Core API wrapper for TF2 Outpost, enabling automated trading for Team Fortress 2, CS:GO, and Dota 2 items.
A Node.js package to fetch player profile data from the Super Mario Maker bookmark site using Nintendo IDs.
Related Tags
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.