A Whirlwind Tour of Common Crawl's Datasets using Python
Apache-2.0Python
A Python tutorial demonstrating how to access and process Common Crawl's web archive datasets (WARC, WET, WAT) using tools like warcio, cdx_toolkit, and DuckDB.
45 stars9 forks0 contributors
Overview
A whirlwind tour of Common Crawl's data using Python
Related Projects
GLAM Workbench: Web ArchivesSee also this related blog post on 'Asking questions with web archives'
Stars0
Forks0
Last commit
IIPC and DPC Training materials: module for beginners (8 sessions)Stars0
Forks0
Last commit
Tutorial for Humanities researchers about how to explore Arquivo.ptStars0
Forks0
Last commit
warc-specificationscommunity HTML version of the official specification and hub for new proposals
Stars0
Forks0
Last commit
Community-curated · Updated weekly · 100% open source
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.