How do I add a new data source to Machinae?

As per the README, adding new sites is documented as 'COMING SOON,' so there's no official guide. You'll likely need to reverse-engineer existing YAML configurations and Python code, which can be complex for non-developers.

Machinae vs Automater: which is better for OSINT?

Machinae improves on Automater with Python 3 support, YAML configs, and JSON output, making it more modern and maintainable. However, Automater might be preferred if you need a tool with a longer history or specific features not yet in Machinae, like more output options.

How to configure API keys for VirusTotal in Machinae?

Create a YAML auth file with your VirusTotal API key under keys like 'vt_ip' and 'vt_domain,' then reference it using the -a flag. You also need to enable these sites in the main configuration file, as they might be disabled by default.

Can Machinae output results to CSV or Excel?

No, Machinae does not support CSV or Excel output. Current formats are limited to normal text, dot-escaped, JSON, and short yes/no/error outputs, which may require additional scripting for spreadsheet integration.

Does Machinae support real-time monitoring or alerts?

Machinae is designed for batch queries from the command line and lacks built-in real-time alerting or monitoring features. It's best for manual or scripted intelligence gathering, not continuous threat detection.

How to use Machinae with a proxy server?

Use the -H command-line option to specify an HTTP proxy, or set the HTTP_PROXY/HTTPS_PROXY environment variables. Machinae will automatically pick up proxy settings from these variables for all requests.

Is Machinae good for large-scale batch processing of observables?

While it supports batch files via the -i option, it may not handle high-volume queries efficiently due to rate limits from data sources and lack of advanced features like parallel processing or error handling for failed requests.

Open-Awesome

Machinae

MITPython1.4.8

A Python tool for collecting security intelligence from public feeds about IPs, domains, URLs, emails, hashes, and SSL fingerprints.

GitHub

539 stars101 forks0 contributors

What is Machinae?

Machinae is a Python-based command-line tool designed to collect security intelligence from public websites and feeds. It automates the process of looking up information about security observables such as IP addresses, domain names, URLs, email addresses, file hashes, and SSL certificates across multiple sources. It solves the problem of manual, repetitive queries by providing a unified, configurable interface for threat intelligence gathering.

Target Audience

Security analysts, threat intelligence researchers, and incident responders who need to quickly gather data from public OSINT sources during investigations.

Value Proposition

Developers choose Machinae for its clean Python 3 code, easy-to-read YAML configuration, out-of-the-box support for many data sources, and flexible output formats like JSON. It improves upon older tools with better maintainability and extensibility.

Overview

Machinae Security Intelligence Collector

Use Cases

Best For

Investigating suspicious IP addresses across multiple reputation feeds
Checking domain names against malware and phishing blocklists
Looking up file hashes on VirusTotal and other malware databases
Analyzing SSL certificate fingerprints for malicious associations
Gathering OSINT data during incident response or threat hunting
Automating repetitive security intelligence queries from the command line

Not Ideal For

Real-time threat intelligence operations requiring low-latency data streams and live alerts
Teams needing integrated dashboards or GUI interfaces for data visualization and collaboration
Organizations with strict compliance requirements where detailed audit logs and data provenance are mandatory
Users who want a fully configured, plug-and-play tool without manually enabling disabled data sources

Pros & Cons

Pros

Modern Python 3 Codebase

Built from scratch for Python 3 with a more pythonic design, making it easier to maintain and extend compared to legacy tools like Automater.

Flexible YAML Configuration

Uses human-readable YAML with config merging, allowing overrides in local files without modifying the base configuration, which simplifies updates and customization.

Wide Data Source Coverage

Pre-configured with over 25 public security feeds, including VirusTotal, Shodan, and IPVoid, covering IPs, domains, hashes, and more for comprehensive OSINT gathering.

Automatic Observable Detection

Automatically identifies input types like IP addresses, domains, and file hashes, reducing manual effort and speeding up investigations.

Cons

Many Sources Disabled by Default

Several data sources, such as Fortinet Category and TotalHash, are disabled in the default configuration, requiring manual YAML edits to enable them, adding setup overhead.

API Key Management Overhead

Premium sources require API keys configured in separate YAML auth files, which can be cumbersome to manage and secure, especially for teams with multiple credentials.

Limited Output Formats

Only supports a few output types like JSON and normal text, with no CSV or structured formats, limiting integration with other tools despite plans for future additions.

Known Bugs and Incomplete Features

The README admits issues like double-encoded HTML entities on IPvoid and lacks documentation for adding new sites, which is marked as 'COMING SOON,' hindering extensibility.

Frequently Asked Questions

Related Projects

dnstwist

Domain name permutation engine for detecting homograph phishing attacks, typo squatting, and brand impersonation

Stars5,683

Forks845

Last commit1 year ago

mailchecker

:mailbox: Cross-language temporary (disposable/throwaway) email detection library. Covers 55 734+ fake email providers.

Stars1,883

Forks303

Last commit24 days ago

URLhaus

A project from abuse.ch with the goal of sharing malicious URLs that are being used for malware distribution

Threat intelligence tracker, with IP/domain/hash search