Web Archiving

ArchiveBox

Open-source self-hosted web archiving tool that saves websites in multiple durable formats like HTML, PDF, and WARC.

Auto Archiver

A Python tool to automatically archive web content (videos, images, social media) from Google Sheets and other sources in a secure, verifiable way.

Browsertrix Crawler

A standalone Docker container for high-fidelity, browser-based web archiving crawls using Puppeteer and Brave.

Chronicler

An offline-first web browser that archives, searches, and crawls websites for personal use.

DiskerNet

Offline full-text search and archiving tool for Chromium-based browsers that saves and indexes every page you visit.

F(b)arc

A command-line tool and Python library for archiving Facebook data via the Graph API, supporting recursive retrieval of nodes and edges.

grab-site

A preconfigured web crawler for backing up websites, producing WARC files with a live dashboard and dynamic ignore patterns.

Heritrix

An open-source, extensible, web-scale, archival-quality web crawler from the Internet Archive.

Heritrix Q&A

An open-source, extensible, web-scale, archival-quality web crawler from the Internet Archive.

monolith

CLI tool and library for saving complete web pages as a single, self-contained HTML file.

Obelisk

A Go package and CLI tool that saves web pages as single HTML files with all assets embedded.

Scoop

A high-fidelity, browser-based web archiving library and CLI for capturing single web pages with provenance.

Squidwarc

A high-fidelity, user-scriptable archival web crawler using Chrome/Chromium to preserve JavaScript-rendered content.

twarc

A command line tool and Python library for collecting and archiving Twitter JSON data via the Twitter API.

WAIL

A graphical desktop application that simplifies web archiving by providing a one-click interface to preserve and replay web pages using Heritrix and OpenWayback.

Wayback

A privacy-focused web archiving tool with an IM-style interface that captures pages to multiple archival services.

Waybackpy

A Python package and CLI tool for interacting with the Wayback Machine's Save, CDX, and Availability APIs.

InterPlanetary Wayback (ipwb)

A distributed and persistent web archive replay system that uses IPFS to store and serve WARC files.

OpenWayback

Legacy web archive replay engine for accessing historical web content from WARC files.

hyphe

A research-driven web crawler for building and analyzing curated web corpora as networks of web entities.

webarchive-discovery

A toolkit for indexing and exploring web archive content from ARC and WARC files using OpenSearch/Elasticsearch.

SolrWayback

A web application for searching, browsing, and analyzing archived web content (ARC/WARC files) with a Solr backend.

here

A toolkit for indexing and exploring web archive content from ARC and WARC files using OpenSearch/Elasticsearch.

ArchiveTools

A Python toolkit for extracting, filtering, and analyzing data from web archives, JSON files, and imageboards.

Go Get Crawl

A Go tool and library for downloading URLs and files from Common Crawl and Wayback Machine web archives.

MemGator

A portable concurrent Memento aggregator CLI and server for retrieving archived web pages from multiple sources.

warcdb

WarcDB is an SQLite-based file format that makes web crawl data easier to share and query.

wikiteam

A set of Python tools for downloading and preserving wikis, including MediaWiki wikis and Wikimedia projects.

FastWARC

A collection of robust and fast Python tools for parsing, extracting, and analyzing web archive data, including a high-performance WARC parser.

node-warc

A Node.js library for parsing and creating Web ARChive (WARC) files with support for Chrome, Puppeteer, and Electron.

warctools

Python command-line tools and libraries for handling, validating, and converting WARC and ARC web archive files.

ArchiveSpark

An Apache Spark framework for efficient data processing, extraction, and derivation from web archives and archival collections.

Archives Unleashed Toolkit

An open-source toolkit for analyzing web archives at scale using Apache Spark.

Common Crawl Jupyter notebooks

A collection of Jupyter notebooks for analyzing Common Crawl web archive data using columnar indexes and webgraph datasets.

FlameShot

A powerful, open-source screenshot tool with built-in annotation and editing capabilities for Linux, Windows, and macOS.

xDoTool

A command-line tool for simulating keyboard/mouse input and automating window management on X11 systems.