Commoncrawl
3 projects
Showing 3 of 3 projects
ELI5Python
Scripts and tools to recreate the ELI5 dataset for long-form question answering research.
#dataset-creation#transformer-models#research-tools
Stars324
Forks42
Last commit4 years ago
Go Get CrawlGo
A Go tool and library for downloading URLs and files from Common Crawl and Wayback Machine web archives.
#crawler#go-library#wayback-machine
Stars179
Forks17
Last commit1 year ago
Common Crawl Jupyter notebooksJupyter Notebook
A collection of Jupyter notebooks for analyzing Common Crawl web archive data using columnar indexes and webgraph datasets.
#warc-files#data-science#web-archive-analysis
Stars66
Forks11
Last commit
Community-curated · Updated weekly · 100% open source
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.