How do I set up CloudWatch Logs subscription consumer for AWS Lambda logs?

Choose 'AWS Lambda' in the CloudFormation template's LogFormat parameter, which automatically applies the filter pattern [timestamp=*Z, request_id="*-*", event]. JSON logs from Lambda functions are indexed directly in Elasticsearch, and sample dashboards are provided.

CloudWatch Logs subscription consumer vs. Amazon Kinesis Data Firehose: which should I use?

This tool offers more customization with pre-built Elasticsearch integrations and dashboards, but requires managing EC2 instances. Firehose is fully managed and serverless but less flexible for custom destinations. Choose based on control vs. convenience.

Can I extend this to send logs to a custom destination like a database?

Yes, it's built on the Amazon Kinesis Connector Library, so you can implement custom connectors. However, this requires Java development and modifying the codebase, as only Elasticsearch and S3 connectors are included out-of-the-box.

How secure is the Elasticsearch cluster from the CloudFormation template?

It only provides basic IP-based security groups and HTTP Basic Auth via nginx. The README warns this is insufficient for production data, so you must manually add HTTPS, authentication, or other measures to meet security standards.

What does it cost to run this stack on AWS?

The CloudFormation template creates EC2 instances, a Kinesis stream, and an Elastic Load Balancer, so costs vary with usage. You'll incur ongoing charges for these resources, unlike serverless alternatives that scale with log volume.

Is Kibana 4 in this project compatible with newer Elasticsearch versions?

No, Kibana 4.1 is outdated and may not work with Elasticsearch 7.x or above. You'd need to upgrade Kibana manually, which isn't covered in the setup, potentially breaking pre-configured dashboards.

Open-Awesome

cloudwatch-logs-subscription-consumer

NOASSERTIONJava

A specialized Amazon Kinesis stream reader that delivers CloudWatch Logs data to other systems like Elasticsearch and S3 in near real-time.

GitHub

397 stars143 forks0 contributors

What is cloudwatch-logs-subscription-consumer?

CloudWatch Logs Subscription Consumer is a specialized Amazon Kinesis stream reader that delivers data from Amazon CloudWatch Logs to external systems like Elasticsearch and S3 in near real-time. It solves the problem of centralized log analysis by enabling seamless streaming of AWS service logs (e.g., VPC Flow Logs, Lambda, CloudTrail) to customizable dashboards and storage solutions.

Target Audience

AWS users, DevOps engineers, and SREs who need to analyze and monitor logs from AWS services in real-time, particularly those using Elasticsearch for log aggregation.

Value Proposition

Developers choose this tool because it provides a ready-to-use, extensible pipeline for CloudWatch Logs with pre-built Kibana dashboards, reducing the complexity of setting up log analysis infrastructure from scratch.

Overview

A specialized Amazon Kinesis stream reader (based on the Amazon Kinesis Connector Library) that can help you deliver data from Amazon CloudWatch Logs to any other system in near real-time using a CloudWatch Logs Subscription Filter.

Use Cases

Best For

Streaming AWS VPC Flow Logs to Elasticsearch for network traffic analysis
Real-time monitoring of AWS Lambda function logs with custom dashboards
Centralizing AWS CloudTrail events for security and compliance auditing
Building custom log analysis pipelines for CloudWatch Logs data
Deploying a quick Elasticsearch and Kibana stack for log visualization
Extending log delivery to custom destinations using the Kinesis Connector framework

Not Ideal For

Organizations with strict security compliance needing advanced authentication (e.g., SAML, OAuth) for log data
Projects not using Amazon CloudWatch Logs as their primary log source or in non-AWS environments
Teams already invested in modern log management platforms like Elastic Stack 7+ or commercial SIEMs
Use cases requiring fully serverless, managed log pipelines without EC2 maintenance

Pros & Cons

Pros

Real-Time Log Streaming

Uses Amazon Kinesis for near real-time delivery of CloudWatch Logs data, enabling immediate analysis and monitoring as highlighted in the README's emphasis on streaming.

Pre-Built AWS Integrations

Includes built-in connectors for Elasticsearch and S3, plus sample Kibana dashboards for VPC Flow Logs, Lambda, and CloudTrail, reducing setup effort.

One-Click Deployment

Offers a CloudFormation template that deploys a full Elasticsearch cluster with Kibana in about 10 minutes, simplifying initial setup.

Flexible Data Parsing

Supports JSON, fixed-column, and unstructured log data with customizable filter patterns, allowing for easy indexing in Elasticsearch without extra configuration for JSON logs.

Cons

Inadequate Security Controls

The README admits that basic IP restrictions and HTTP Basic Auth are insufficient for confidential data, requiring manual enhancements like HTTPS or client authentication for production.

Outdated Kibana Versions

Bundles Kibana 3 and 4.1, which are legacy and lack features, updates, and security patches of modern versions, potentially limiting functionality.

AWS-Centric Lock-In

Heavily dependent on AWS services (CloudWatch, Kinesis, EC2), making it less portable for hybrid or multi-cloud log management scenarios.

Frequently Asked Questions

Related Projects

awslogs

AWS CloudWatch logs for Humans™

Stars4,988

Forks342

Last commit2 years ago

newrelic_aws_cloudwatch_plugin

New Relic AWS Cloudwatch Plugin

Stars152

Forks83

Last commit7 years ago

ecs-cloudwatch-logs

This repository provides the assets referred to in the blog post on using Amazon ECS and Amazon CloudWatch logs.

Stars70

Forks67

Last commit11 years ago

logstash-output-cloudwatchlogs

A logstash plugin that allows to send logs to AWS CloudWatch Logs service.

Stars35

Forks31

Last commit8 years ago

Community-curated · Updated weekly · 100% open source

Found a gem we're missing?

Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.

Submit a project Star on GitHub