ArchiveSpark
An Apache Spark framework for efficient data processing, extraction, and derivation from web archives and archival collections.
What is ArchiveSpark?
ArchiveSpark is an Apache Spark framework specifically designed for processing, extracting, and deriving data from web archives and other archival collections. It solves the problem of efficiently accessing and transforming raw archival data into more usable formats while maintaining data lineage. The framework enables researchers and developers to create derived datasets through filtering and extraction operations.
Data scientists, researchers, and developers working with web archives or archival collections who need to process, analyze, and extract value from large-scale historical data. This includes digital humanities researchers, web archivists, and data engineers working with temporal web data.
Developers choose ArchiveSpark because it provides a specialized, efficient framework for archival data processing built on Apache Spark's distributed computing capabilities. Its unique modular architecture with customizable data specifications allows it to work with diverse archival collections while maintaining data lineage—a critical feature for reproducible research.
Overview
An Apache Spark framework for easy data processing, extraction as well as derivation for web archives and archival collections, developed at Internet Archive.
Use Cases
Best For
Related Projects
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
