cdx-toolkit

0 stars0 forks0 contributors

Overview

Library and CLI to consult cdx indexes and create WARC extractions of subsets. Abstracts away Common Crawl's unusual crawl structure. (Stable)

Related Projects

Community-curated · Updated weekly · 100% open source

Found a gem we're missing?

Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.

Submit a project Star on GitHub

Last commit

Created

Links & Resources

Website

Included in

Web Archiving2.5k

wikiteam

Tools for downloading and preserving wikis. We archive wikis, from Wikipedia to tiniest wikis. As of 2026, WikiTeam has preserved more than 600,000 wikis.

Stars846

Forks175

Last commit4 months ago

warcdb

WarcDB: Web crawl data as SQLite databases.

Stars405

Forks10

Last commit1 year ago

Go Get Crawl

Extract web archive data using Wayback Machine and Common Crawl

Stars179

Forks17

Last commit1 year ago

httpreserve.info

Service to return the status of a web page or save it to the Internet Archive. HTTPreserve includes disambiguation of well-known short link services. It returns JSON via the browser or command line via CURL using GET. Describes web sites using earliest and latest dates in the Internet Archive and demonstrates the construction of Robust Links in its output using that range. (Golang). (Stable)

Stars0

Forks0

Last commit