Common Crawl
Showing 5 of 5 projects
A Go tool and library for downloading URLs and files from Common Crawl and Wayback Machine web archives.
A collection of Jupyter notebooks for analyzing Common Crawl web archive data using columnar indexes and webgraph datasets.
A Python tutorial demonstrating how to access and process Common Crawl's web archive datasets (WARC, WET, WAT) using tools like warcio, cdx_toolkit, and DuckDB.
A DuckDB extension to query web archive CDX APIs (Wayback Machine & Common Crawl) directly from SQL with smart query pushdown.
German language versions of GPT-2, trained on the CC-100 corpus and initialized from English GPT-2 weights.
Related Tags
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.