Heritrix
An open-source, extensible, web-scale, archival-quality web crawler from the Internet Archive.
What is Heritrix?
Heritrix is an open-source, extensible web crawler developed by the Internet Archive for large-scale, archival-quality web harvesting. It is designed to capture and preserve digital artifacts with high fidelity, respecting website politeness policies like `robots.txt` and META nofollow tags. The project addresses the need for responsible, scalable tools to archive web content for future research and cultural preservation.
Archivists, researchers, and institutions focused on digital preservation, web archiving, and large-scale data collection who require a robust, configurable crawling solution.
Developers choose Heritrix for its proven scalability in web-scale archiving, its commitment to respectful crawling practices, and its extensible architecture that allows deep customization. As a project from the Internet Archive, it benefits from real-world use in preserving cultural heritage.
Overview
Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.
Use Cases
Best For
- Large-scale web archiving projects by libraries and museums
Related Projects
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
