FastWARC
A collection of robust and fast Python tools for parsing, extracting, and analyzing web archive data, including a high-performance WARC parser.
What is FastWARC?
ChatNoir Resiliparse is a Python toolkit for robust and fast processing of web archive data. It provides utilities for parsing, extracting, and cleaning web content, along with a high-performance WARC parser called FastWARC, designed to handle large-scale analytics efficiently.
Researchers, data scientists, and developers working with web archives, Common Crawl data, or large-scale web analytics who need reliable and performant tools for parsing and extracting information from noisy web data.
Developers choose Resiliparse for its optimized performance, resilience against malformed data, and comprehensive toolset specifically tailored for web archive processing, including the fast FastWARC parser which outperforms alternatives in speed.
Overview
A robust web archive analytics toolkit
Use Cases
Best For
- Parsing and analyzing WARC files from web archives like Common Crawl
- Extracting main content and removing boilerplate from web pages at scale
Related Projects
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
