Warc Files
4 projects
Showing 4 of 4 projects
node-warcJavaScript
A Node.js library for parsing and creating Web ARChive (WARC) files with support for Chrome, Puppeteer, and Electron.
#stream-processing#web-archives#warc-files
Stars105
Forks22
Last commit1 year ago
Common Crawl Jupyter notebooksJupyter Notebook
A collection of Jupyter notebooks for analyzing Common Crawl web archive data using columnar indexes and webgraph datasets.
#warc-files#data-science#web-archive-analysis
Stars66
Forks11
Last commit
WarcworkerPython
A dockerized, queued web archiver using Chrome headless to create high-fidelity WARC files from URLs.
#squidwarc#high-fidelity-preservation#warc-files
Stars62
Forks9
Last commit2 years ago
warc-safePython
A Python tool that scans WARC web archive files for viruses and NSFW content using AI and antivirus detection.
#warc-safe#ai-classification#warc-files
Stars18
Forks1
Last commit9 days ago
Related Tags
Community-curated · Updated weekly · 100% open source
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.