Common Crawl Columnar Index

0 stars0 forks0 contributors

Overview

SQL-queryable index, with CDX info plus language classification. (Stable)

Related Projects

Community-curated · Updated weekly · 100% open source

Found a gem we're missing?

Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.

Submit a project Star on GitHub

Created

Links & Resources

Website

Included in

Web Archiving2.5k

ArchiveSpark

An Apache Spark framework for easy data processing, extraction as well as derivation for web archives and archival collections, developed at Internet Archive.

Stars160

Forks19

Last commit7 months ago

Archives Unleashed Toolkit

The Archives Unleashed Toolkit is an open-source toolkit for analyzing web archives.

Stars157

Forks34

Last commit5 months ago

Web Data Commons

Structured data extracted from Common Crawl. (Stable)

Stars0

Forks0

Last commit

Common Crawl Web Graph

A host or domain-level graph of the web, with ranking information. (Stable)

Stars0

Forks0

Last commit