How does Substation compare to Logstash for log processing?

Substation is a modern alternative with native AWS serverless deployment and over 100 transformation functions, using Jsonnet for expressive configurations. It aims for higher performance and lower costs than Logstash, but Logstash has a larger community and more plugins for diverse integrations.

How to deploy Substation on AWS using Terraform?

Substation includes Terraform modules for deploying resources like Lambda functions and S3 buckets securely. The README provides example configurations that define KMS encryption, AppConfig for management, and least-privilege IAM roles, making it easy to set up serverless pipelines.

Can Substation process logs in real-time?

Yes, it supports real-time processing through AWS services like Kinesis and SQS, handling over 100,000 events per second. However, latency depends on pipeline complexity and AWS service configurations, so it may not suit ultra-low-latency use cases.

What are the costs of running Substation compared to Cribl?

Substation is open-source with costs based on AWS usage, typically a few cents per GB, while Cribl is commercial and can cost up to 10x more for similar throughput, as noted in the README. This makes Substation a budget-friendly option for scale.

How to write custom transformations in Go for Substation?

You can extend functionality by writing Go code that implements Substation's transformation interfaces. The project includes a development guide and examples, but this requires Go expertise and familiarity with the codebase, adding complexity.

Does Substation support multi-cloud deployments?

While it runs on any Go-supported platform, its built-in routing and deployment tools are optimized for AWS. For other clouds, you may need to rely on HTTP endpoints or custom integrations, as native connectors are limited.

Open-Awesome

Substation

MITGov2.8.0

A serverless toolkit for routing, normalizing, and enriching security event and audit logs in AWS.

Visit Website GitHub

399 stars31 forks0 contributors

What is Substation?

Substation is an open-source toolkit for building data pipelines that route, normalize, and enrich security event and audit logs. It solves the problem of processing heterogeneous log data at scale by transforming it into standardized schemas, enriching it with external intelligence, and routing it to various destinations, all within a serverless AWS environment.

Target Audience

Security engineers, DevOps teams, and platform engineers who need to process and normalize security logs from multiple sources, especially those operating in AWS environments and seeking to reduce costs compared to commercial log management solutions.

Value Proposition

Developers choose Substation for its high performance, low cost, and extensibility. It offers a serverless, maintainable alternative to tools like Logstash and Fluentd, with native AWS integration, over 100 transformation functions, and the ability to run anywhere Go is supported.

Overview

Substation is a toolkit for routing, normalizing, and enriching security event and audit logs.

Use Cases

Best For

Transforming Zeek or Suricata logs into Elastic Common Schema (ECS) format
Building a cost-effective log enrichment pipeline using threat intelligence APIs
Creating a serverless log router between AWS Kinesis, S3, and Lambda
Reducing spend on commercial observability pipelines like Cribl or Datadog
Developing custom data transformation microservices in Go
Deploying log normalization pipelines using Infrastructure as Code (Terraform)

Not Ideal For

Teams deploying log pipelines on non-AWS cloud platforms like Google Cloud or Azure
Simple use cases requiring only log forwarding without transformation or enrichment
Organizations lacking Go programming expertise for custom pipeline extensions
Projects demanding sub-millisecond latency for real-time event processing

Pros & Cons

Pros

Extensive Transformation Library

Offers over 100 built-in functions for data manipulation, enabling complex log processing without custom code, as highlighted in the README's feature list.

Cost-Effective Serverless Deployment

Deploys as a serverless application in AWS using Terraform, reducing operational costs compared to commercial solutions like Cribl or Datadog, which can cost up to 10x more.

Schema Compliance Made Easy

Supports normalization to standards like ECS and OCSF, with examples showing transformation of Zeek logs to ECS format, simplifying security log management.

High Performance at Scale

Capable of processing over 100,000 events per second with low cloud costs, making it suitable for high-volume security event pipelines.

Cons

AWS-Centric Architecture

Heavily integrated with AWS services; while it can run elsewhere, optimal deployment and routing features are tailored for AWS, limiting flexibility for multi-cloud setups.

Complex Configuration with Jsonnet

Uses Jsonnet for configuration, which is powerful but has a steeper learning curve compared to simpler DSLs or YAML, as admitted in the README's comparison tables.

Limited Third-Party Integrations

As a newer project, it lacks the extensive plugin ecosystem of established tools like Logstash, potentially requiring custom development for non-standard integrations.

Open Source Alternative To

Substation is an open-source alternative to the following products:

Cribl

Cribl is a data engine for IT and security teams that provides observability pipeline solutions to collect, reduce, enrich, and route log and metric data.

Datadog Observability Pipelines

Datadog Observability Pipelines is a feature that collects, processes, and routes observability data across various sources and destinations within the Datadog platform.

Frequently Asked Questions

Related Projects

MediaPipe

Cross-platform, customizable ML solutions for live and streaming media.

Apache Kafka - A distributed event streaming platform

Stars32,615

Forks15,196