Web Archiving
Showing 8 of 80 projects
A splitable Hadoop InputFormat for processing concatenated GZIP files and web archive (*.warc.gz) data efficiently in distributed systems.
A Java command-line application for downloading web archive (WARC) files from the WASAPI (Web Archiving Systems API).
A command-line tool for performing various gzip, ARC, WARC, and XML tasks on web archive files.
A command-line tool and Go package for archiving webpages to IPFS with Wayback Machine integration.
A Node.js library for parsing CDXJ files produced by web archiving tools like Pywb.
Intelligent web page comparison tool with Wayback Machine artifact cleaning, visual regression testing, and significance scoring.
Converts bag-nabit datasets from ZIP archives into full-content WARC files for web archiving.
A Hadoop/MapReduce tool that splits and partitions web archive records in (W)ARC files by MIME type and year.
Related Tags
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.