ThreatIngestor
An extendable Python tool to extract and aggregate Indicators of Compromise (IOCs) from various threat intelligence feeds.
What is ThreatIngestor?
ThreatIngestor is an open-source Python framework that automates the extraction and aggregation of Indicators of Compromise (IOCs) from various online threat feeds. It solves the problem of manually monitoring disparate intelligence sources by programmatically collecting data from Twitter, RSS feeds, websites, and more, then forwarding the extracted IOCs to security analysis platforms. This enables security teams to centralize threat intelligence and improve their detection and response capabilities.
Security analysts, threat intelligence teams, and SOC (Security Operations Center) personnel who need to automate the collection and processing of IOCs from public and private feeds. It's also suitable for developers building custom threat intelligence pipelines.
Developers choose ThreatIngestor for its extensible plugin architecture, which allows seamless integration with existing security tools like MISP and ThreatKB, and its ability to fit into diverse workflows without requiring major changes. Its out-of-the-box support for multiple sources and outputs reduces development time for custom threat intelligence aggregation.
Overview
Extract and aggregate threat intelligence.
Use Cases
Best For
- Automating IOC collection from Twitter security lists and RSS feeds
- Integrating threat intelligence into MISP or ThreatKB instances
- Building custom security monitoring pipelines with SQS or databases
- Extracting IOCs from images using OCR technology
- Aggregating threat data from GitHub repositories and gists
- Creating a centralized IOC database for security analysis
Not Ideal For
- Real-time threat intelligence with instant alerts (polls every 15 minutes by default)
- Projects requiring a GUI for configuration and management (YAML/CLI-only)
- Teams lacking Python development resources for custom plugins or troubleshooting
- Large-scale, high-volume IOC processing without significant customization
Pros & Cons
Pros
Uses modular source and operator plugins, allowing seamless integration with custom workflows and tools like SQS or databases, as highlighted in the documentation.
Supports diverse inputs including Twitter, RSS, web pages, and images via OCR, centralizing threat data from multiple feeds without manual effort.
Includes native operators for MISP, ThreatKB, and SQL databases, reducing development time for common security platforms.
Leverages OpenCV and Tesseract to extract IOCs from images, a unique feature that addresses embedded threat data in visual content.
Cons
Requires YAML configuration and managing optional Python dependencies (e.g., for image extraction), which can be daunting for new users without a streamlined onboarding process.
Focuses on aggregation and forwarding; lacks native alerting mechanisms for immediate notifications on new IOCs, relying on external systems for response.
Some sources like Twitter need API keys and developer accounts, adding administrative overhead, and the Docker setup has known issues requiring manual fixes per the README.
Frequently Asked Questions
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.