storm-crawler
A scalable, mature, and versatile web crawler built on Apache Storm for building low-latency, distributed crawling systems.
What is storm-crawler?
Apache StormCrawler is an open-source framework for building scalable, low-latency web crawlers on Apache Storm. It provides a collection of resources and tools to create distributed crawling systems that can efficiently process large volumes of web data in real-time. The project solves the problem of building custom, high-performance web crawlers without reinventing core infrastructure.
Developers and engineers building large-scale web crawling systems, data extraction pipelines, or search engine components that require distributed, real-time processing capabilities.
Developers choose StormCrawler for its mature, production-ready architecture built on Apache Storm, offering proven scalability and flexibility for custom crawling needs. Its modular design and comprehensive documentation make it a versatile alternative to building crawlers from scratch.
Overview
A scalable, mature and versatile web crawler based on Apache Storm
Use Cases
Best For
- Building distributed web crawlers for large-scale data collection
Related Projects
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
