webarchive-indexing
MITPython
MapReduce tools for bulk indexing of web archive WARC/ARC files into ZipNum sharded CDX clusters on Hadoop, EMR, or local systems.
46 stars12 forks0 contributors
Overview
Tools for bulk indexing of WARC/ARC files on Hadoop, EMR or local file system.
Related Projects
wikiteamTools for downloading and preserving wikis. We archive wikis, from Wikipedia to tiniest wikis. As of 2026, WikiTeam has preserved more than 600,000 wikis.
Stars848
Forks175
Last commit5 months ago
warcdbWarcDB: Web crawl data as SQLite databases.
Stars405
Forks10
Last commit1 year ago
Go Get CrawlExtract web archive data using Wayback Machine and Common Crawl
Stars180
Forks17
Last commit1 year ago
MemGatorA Memento Aggregator CLI and Server in Go
Stars80
Forks13
Last commit2 months ago
Community-curated · Updated weekly · 100% open source
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.