An Apache Spark framework for easy data processing, extraction as well as derivation for web archives and archival collections, developed at Internet Archive.

Stars161

Forks19

Last commit8 months ago

Archives Unleashed Toolkit

The Archives Unleashed Toolkit is an open-source toolkit for analyzing web archives.

Stars158

Forks34

Last commit6 months ago

Common Crawl Columnar Index

SQL-queryable index, with CDX info plus language classification. (Stable)

Structured data extracted from Common Crawl. (Stable)

Stars0

Forks0

Last commit

Common Crawl Jupyter notebooks

Overview

Related Projects

Found a gem we're missing?

Tags

Built With

Included in