Training/Documentation
14 projects
A comprehensive open educational resource for teaching web archiving concepts, practices, and tools.
A Python tutorial demonstrating how to access and process Common Crawl's web archive datasets (WARC, WET, WAT) using tools like warcio, cdx_toolkit, and DuckDB.
Resources for Web Publishers
2 projectsTools & Software
2 projects
A compilation of research materials on data resilience, interactivity, and related topics for the Data Together community.
A comprehensive curated list of open-source and hosted tools for monitoring and detecting changes on websites.
Acquisition
41 projects
Open-source self-hosted web archiving tool that saves websites in multiple durable formats like HTML, PDF, and WARC.
A Python tool to automatically archive web content (videos, images, social media) from Google Sheets and other sources in a secure, verifiable way.
A standalone Docker container for high-fidelity, browser-based web archiving crawls using Puppeteer and Brave.
NPM package and CLI tool for saving web pages as single HTML files, implemented in TypeScript.
An offline-first web browser that archives, searches, and crawls websites for personal use.
A command-line tool for archiving web pages into WARC files and replaying them locally.
Offline full-text search and archiving tool for Chromium-based browsers that saves and indexes every page you visit.
A command-line tool and Python library for archiving Facebook data via the Graph API, supporting recursive retrieval of nodes and edges.
A preconfigured web crawler for backing up websites, producing WARC files with a live dashboard and dynamic ignore patterns.
An open-source, extensible, web-scale, archival-quality web crawler from the Internet Archive.
An open-source, extensible, web-scale, archival-quality web crawler from the Internet Archive.
A virtual machine and walkthrough for setting up and using the Heritrix web crawler for web archiving.
A Python script that converts offline web resources into a single WARC file for archiving.
CLI tool and library for saving complete web pages as a single, self-contained HTML file.
A Go package and CLI tool that saves web pages as single HTML files with all assets embedded.
A high-fidelity, browser-based web archiving library and CLI for capturing single web pages with provenance.
A high-fidelity, user-scriptable archival web crawler using Chrome/Chromium to preserve JavaScript-rendered content.
A command line tool and Python library for collecting and archiving Twitter JSON data via the Twitter API.
A graphical desktop application that simplifies web archiving by providing a one-click interface to preserve and replay web pages using Heritrix and OpenWayback.
A dockerized, queued web archiver using Chrome headless to create high-fidelity WARC files from URLs.
A privacy-focused web archiving tool with an IM-style interface that captures pages to multiple archival services.
A Python package and CLI tool for interacting with the Wayback Machine's Save, CDX, and Availability APIs.
A customizable Scala crawler for creating personal web archives in WARC/CDX format.
Replay
6 projectsA distributed and persistent web archive replay system that uses IPFS to store and serve WARC files.
Legacy web archive replay engine for accessing historical web content from WARC files.
Converts WARC web archive files to static HTML with relative links for offline browsing or rehosting.
Search & Discovery
13 projects
A research-driven web crawler for building and analyzing curated web corpora as networks of web entities.
A Chrome extension that integrates live web browsing with archived copies using the Memento protocol.
A desktop application for retrieving, normalizing, and exploring document collections from scientific, institutional, and web sources without coding.
A command-line tool to playback archived webpages from the Wayback Machine using GitHub as a source.
A toolkit for indexing and exploring web archive content from ARC and WARC files using OpenSearch/Elasticsearch.
A prototype web archive exploration UI powered by Solr for searching and browsing archived web content.
A web application for searching, browsing, and analyzing archived web content (ARC/WARC files) with a Solr backend.
A Rails engine for discovering web archives in WARC and ARC formats with faceted search and advanced discovery options.
A personal web archive and search system that runs in a Docker container, allowing you to browse and search archived web pages.
A toolkit for indexing and exploring web archive content from ARC and WARC files using OpenSearch/Elasticsearch.
Utilities
28 projects
A Python toolkit for extracting, filtering, and analyzing data from web archives, JSON files, and imageboards.
Converts bag-nabit datasets from ZIP archives into full-content WARC files for web archiving.
A DuckDB extension to query web archive CDX APIs (Wayback Machine & Common Crawl) directly from SQL with smart query pushdown.
A Go tool and library for downloading URLs and files from Common Crawl and Wayback Machine web archives.
A BadgerDB-based server for indexing and serving WARC file contents with CDX capture index support.
A CLI tool to test URL availability and retrieve Internet Archive snapshots, with output in JSON, CSV, or BoltDB.
Converts HTTrack website crawls into standardized WARC files for web archiving and preservation.
A framework for profiling web archives to summarize their holdings using compact SURT-based maps.
A portable concurrent Memento aggregator CLI and server for retrieving archived web pages from multiple sources.
A Node.js library for parsing CDXJ files produced by web archiving tools like Pywb.
A high-performance, RocksDB-based capture index server for web archives, supporting OpenWayback and PyWb protocols.
A Python command-line client for downloading web archive (WARC) files from Archive-It and Webrecorder WASAPI Data Transfer APIs.
Extracts hyperlinks from files using Apache Tika for batch processing and web archiving workflows.
A Java command-line application for downloading web archive (WARC) files from the WASAPI (Web Archiving Systems API).
WarcDB is an SQLite-based file format that makes web crawl data easier to share and query.
A resilient and configurable Python tool for exploring, analyzing, transforming, and extracting data from WARC (Web ARChive) files.
A Python tool that scans WARC web archive files for viruses and NSFW content using AI and antivirus detection.
A Hadoop/MapReduce tool that splits and partitions web archive records in (W)ARC files by MIME type and year.
Java-based web archive deduplication tool that identifies duplicates and converts them to reference records in WARC files.
MapReduce tools for bulk indexing of web archive WARC/ARC files into ZipNum sharded CDX clusters on Hadoop, EMR, or local systems.
A set of Python tools for downloading and preserving wikis, including MediaWiki wikis and Wikimedia projects.
WARC I/O Libraries
13 projects
A collection of robust and fast Python tools for parsing, extracting, and analyzing web archive data, including a high-performance WARC parser.
A splitable Hadoop InputFormat for processing concatenated GZIP files and web archive (*.warc.gz) data efficiently in distributed systems.
A Java library for reading and writing WARC files with a typed, extensible API and high-performance NIO-based parsing.
A command-line tool for performing various gzip, ARC, WARC, and XML tasks on web archive files.
A Node.js library for parsing and creating Web ARChive (WARC) files with support for Chrome, Puppeteer, and Electron.
A Scala/Spark library for efficient processing, extraction, and derivation of web archive data (CDX/WARC).
A Python CLI tool for extracting and validating WARC and WACZ web archive files.
A Rust library for reading and writing WARC (Web ARChive) files.
A command-line tool and Rust library for handling Web ARChive (WARC) files.
Python command-line tools and libraries for handling, validating, and converting WARC and ARC web archive files.
A Go library for reading and parsing WARC and ARC web archive formats with specialized utilities for web archiving workflows.
Analysis
9 projectsA job server for distributed compute analysis of web archive (WARC) collections.
An Apache Spark framework for efficient data processing, extraction, and derivation from web archives and archival collections.
Example notebooks for analyzing web archives using the Archives Unleashed Toolkit.
An open-source toolkit for analyzing web archives at scale using Apache Spark.
A collection of Jupyter notebooks for analyzing Common Crawl web archive data using columnar indexes and webgraph datasets.
An open-source toolkit for analyzing line-oriented JSON Twitter archives using Apache Spark.
Quality Assurance
12 projects
A powerful, open-source screenshot tool with built-in annotation and editing capabilities for Linux, Windows, and macOS.
A command-line tool for simulating keyboard/mouse input and automating window management on X11 systems.