A Whirlwind Tour of Common Crawl's Datasets using Python
Apache-2.0Python
A Python tutorial demonstrating how to access and process Common Crawl's web archive datasets (WARC, WET, WAT) using tools like warcio, cdx_toolkit, and DuckDB.
45 stars9 forks0 contributors
Overview
A whirlwind tour of Common Crawl's data using Python
Related Projects
Community-curated · Updated weekly · 100% open source
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
