warcdedupe

0 stars0 forks0 contributors

Overview

WARC deduplication tool (and WARC library) written in Rust. (In Development)

Related Projects

Community-curated · Updated weekly · 100% open source

Found a gem we're missing?

Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.

Submit a project Star on GitHub

Created

Links & Resources

Website

Included in

Web Archiving2.5k

wikiteam

Tools for downloading and preserving wikis. We archive wikis, from Wikipedia to tiniest wikis. As of 2026, WikiTeam has preserved more than 600,000 wikis.

Stars844

Forks175

Last commit4 months ago

warcdb

WarcDB: Web crawl data as SQLite databases.

Stars405

Forks10

Last commit1 year ago

Go Get Crawl

Extract web archive data using Wayback Machine and Common Crawl

Stars179

Forks17

Last commit1 year ago

Warchaeology

Warchaeology is a collection of tools for inspecting, manipulating, deduplicating and validating WARC-files. (Stable)

Stars0

Forks0

Last commit