Categories Alternatives Stacks Self-Hosted Explore

© 2026 Open-Awesome. Curated for the developer elite.

Terms Privacy About GitHub RSS

webarchive-indexing — WARC File Indexing Tools | Open Awesome

Home
Web Archiving
webarchive-indexing

webarchive-indexing

MITPython

MapReduce tools for bulk indexing of web archive WARC/ARC files into ZipNum sharded CDX clusters on Hadoop, EMR, or local systems.

46 stars13 forks0 contributors

Overview

Tools for bulk indexing of WARC/ARC files on Hadoop, EMR or local file system.

Quick Stats

Stars46

Forks13

Contributors0

Open Issues4

Last commit

Related Projects

Community-curated · Updated weekly · 100% open source

Found a gem we're missing?

Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.

Submit a project Star on GitHub

8 years ago

CreatedSince 2015

Tags

#mapreduce #warc-indexing #python #web-archiving #amazon-emr #hadoop #bulk-processing

Built With

Included in

Web Archiving2.5k

Auto-fetched 13 hours ago

Internet Archive Library

A Python and Command-Line Interface to Archive.org

Last commit12 days ago

Tools for downloading and preserving wikis. We archive wikis, from Wikipedia to tiniest wikis. As of 2026, WikiTeam has preserved more than 600,000 wikis.

Last commit7 months ago

WarcDB: Web crawl data as SQLite databases.

Last commit2 years ago

Extract web archive data using Wayback Machine and Common Crawl

Last commit1 year ago