Data Collection
Showing 36 of 37 projects
Open-source data integration platform for building ELT pipelines from APIs, databases, and files to data warehouses, lakes, and lakehouses.
An open-source log collector that unifies logging infrastructure by collecting events from various sources and routing them to multiple destinations.
Open-source customer data infrastructure that collects, validates, and enriches behavioral event data for AI and analytics.
Open-source data pipelines for cloud asset inventory, CSPM, FinOps, and vulnerability management across AWS, Azure, GCP, and 70+ sources.
Open-source data pipelines to sync cloud infrastructure metadata from AWS, Azure, GCP, and 70+ sources into your data warehouse.
An open source iOS framework for creating apps for medical and other research studies.
An open-source Java web crawler that provides a simple interface for multi-threaded web crawling.
A curated collection of publicly accessible JSON datasets across diverse topics like government, finance, climate, and entertainment.
A deprecated tool for collecting, processing, and delivering data from multiple sources with Go and Lua plugin support.
A distributed service for efficiently collecting, aggregating, and moving large amounts of log-like data.
An open-source Python library for low-code data preparation, offering fast EDA, data cleaning, and collection from APIs and databases.
A modular PowerShell framework for enterprise incident response and breach hunting using remote data collection.
A preconfigured web crawler for backing up websites, producing WARC files with a live dashboard and dynamic ignore patterns.
A command line tool and Python library for collecting and archiving Twitter JSON data via the Twitter API.
A suite of tools for collecting, processing, and analyzing NetFlow, IPFIX, and sFlow data from network devices.
A high-performance, multithreaded command-line tool for downloading images from webpages.
An open-source distributed IoT platform based on Zabbix for collecting, analyzing, and storing data from millions of devices.
A writable Node.js stream that collects all data chunks and concatenates them into a single buffer or array.
A modular autonomous driving platform for developing and testing AV components on CARLA simulator and real-world vehicles.
An iOS client library for Segment that integrates analytics into any iOS application with minimal hassle.
A research-driven web crawler for building and analyzing curated web corpora as networks of web entities.
A collection of prepackaged InfluxDB configurations for quickly collecting and analyzing time series data from various sources.
A passive BLE scanner with a graphical UI that collects and persistently stores device data in an SQLite database on an SD card for ESP32-based hardware.
An R package for accessing Facebook's Graph API to retrieve and analyze social media data programmatically.
A framework for orchestrating forensic collection, processing, and data export through modular recipes.
A Rails engine gem for creating and managing dynamic surveys with multiple question types and result aggregation.
A command-line tool to fetch and gather data from software repositories and development platforms using modular backends.
A full-featured generic SNMP data collector with a web administration interface for InfluxDB.
An Android mobile app for aid workers to collect and share child information to speed up family tracing and reunification in emergencies.
A deprecated threat intelligence platform for collecting, processing, and sharing security indicators.
A Ruby SDK for integrating applications with Apache PredictionIO's Event Server and Engine APIs.
A curated guide to R packages for web scraping, APIs, web services, and web technologies.
A platform-independent lightweight Python library for designing and conducting timing-critical behavioral and neuroimaging experiments.
A crowdsourced database of interesting aircraft (governments, military, historic, distinctive) formatted as CSV for use with plane tracking software.
Collect, validate, and send ROS 2 data to build APIs and dashboards with reliable data pipelines.
An open-source Android app for easy, efficient, and collaborative FIRST robotics competition scouting.
Related Tags
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.