How do I install fluent-plugin-kinesis on td-agent?

Use 'td-agent-gem install fluent-plugin-kinesis' for td-agent v4 or lower, or 'fluent-gem install' for v5+, as specified in the Installation section. This ensures compatibility with your Fluentd package.

What's the difference between kinesis_streams and kinesis_firehose plugins?

kinesis_streams sends data to Kinesis Data Streams for real-time processing, while kinesis_firehose sends to Kinesis Data Firehose for loading into data stores like S3 or Redshift. Firehose handles delivery and transformation, whereas Streams require more management.

How to configure IAM roles for EKS with fluent-plugin-kinesis?

Use the section in your Fluentd config, providing role_arn, role_session_name, and web_identity_token_file, as detailed in the credentials documentation. This leverages AWS IAM roles for service accounts in Kubernetes.

Does fluent-plugin-kinesis support data compression?

Yes, it supports zlib and gzip compression via the compression parameter in the format section, which can reduce data size and transfer costs. This is optional and configurable based on your needs.

How to handle retries for failed batch requests in fluent-plugin-kinesis?

Configure retries_on_batch_request and related settings like reset_backoff_if_success. By default, it retries 8 times with exponential backoff, and can drop or retry failed records based on drop_failed_records_after_batch_request_retries.

fluent-plugin-kinesis vs fluent-plugin-s3 for AWS logging?

fluent-plugin-kinesis is for streaming logs to Kinesis for real-time analytics, while fluent-plugin-s3 writes directly to S3 for batch storage. Choose based on whether you need immediate processing with Kinesis or archival with S3.

What buffer settings should I use for high throughput with fluent-plugin-kinesis?

Adjust parameters like flush_interval, chunk_limit_size, and flush_thread_count in the buffer section. The README suggests starting with defaults and tuning based on system performance, as detailed in Configuration: Performance.

Open-Awesome

aws-fluent-plugin-kinesis

Apache-2.0Rubyv0.4.0

A Fluentd output plugin for sending log events to Amazon Kinesis Data Streams and Amazon Data Firehose.

Visit Website GitHub

289 stars96 forks0 contributors

What is aws-fluent-plugin-kinesis?

aws-fluent-plugin-kinesis is an open-source Fluentd output plugin developed by AWS Labs that sends log events from Fluentd to Amazon Kinesis Data Streams and Amazon Data Firehose. It solves the problem of efficiently streaming log data from Fluentd into AWS Kinesis for real-time analytics, processing, or data lake ingestion. The plugin handles authentication, batching, retries, and format transformations to ensure reliable data transfer.

Target Audience

DevOps engineers, SREs, and developers who use Fluentd for log aggregation and need to forward logs to AWS Kinesis for streaming analytics, monitoring, or data pipeline integration.

Value Proposition

Developers choose this plugin because it's an official AWS-supported solution that provides robust integration with Kinesis services, extensive configuration options for production use, and compatibility with various AWS authentication methods. It simplifies the setup of log streaming pipelines from Fluentd to AWS without custom code.

Overview

Amazon Kinesis output plugin for Fluentd

Use Cases

Best For

Forwarding application logs from Fluentd to Kinesis Data Streams for real-time processing
Sending log data to Kinesis Data Firehose for loading into S3, Redshift, or Elasticsearch
Building log analytics pipelines on AWS with Fluentd as the collector
Handling high-volume log streams with configurable buffering and retry logic
Using IAM roles or web identity tokens for authentication in EKS environments
Implementing dynamic log routing to different Kinesis streams based on tags or metadata

Not Ideal For

Projects using log aggregation tools other than Fluentd, such as Logstash or Vector.
Environments that require sending logs to non-AWS streaming services like Google Cloud Pub/Sub or Azure Event Hubs.
Simple logging setups where direct AWS SDK integration or file outputs suffice without Fluentd middleware.
Teams needing serverless log ingestion without managing Fluentd instances, preferring AWS native services like Kinesis Data Firehose direct sources.

Pros & Cons

Pros

Multiple Kinesis Outputs

Includes three distinct plugins for Kinesis Data Streams, Firehose, and aggregated streams, allowing flexible data routing based on use cases, as highlighted in the Key Features and README.

Comprehensive AWS Authentication

Supports various credential methods including instance profiles, IAM roles, and web identity tokens for EKS, ensuring secure integration in diverse AWS environments, detailed in the Credentials configuration section.

Advanced Buffering & Retry

Configurable buffer sections and batch request retries with exponential backoff optimize throughput and handle failures reliably, evidenced in the Performance configuration with parameters like retries_on_batch_request.

Dynamic Stream Routing

Allows dynamic stream names using Fluentd placeholders based on tags, time, or custom fields, facilitating log segmentation, as described in the stream_name parameter with examples.

Cons

Configuration Complexity

The plethora of options for buffering, batching, and credentials requires significant expertise to tune properly, which can be overwhelming for quick deployments, as seen in the extensive Configuration sections.

AWS Vendor Lock-in

Exclusively designed for AWS Kinesis, making it unsuitable for multi-cloud strategies or environments without AWS integration, limiting flexibility compared to more generic log forwarders.

Version Compatibility Hassles

The README notes that v3 is 'almost compatible' with v2, indicating potential breaking changes and migration hurdles that need careful management, especially with Fluentd version requirements.

Frequently Asked Questions

Related Projects

amazon-kinesis-client

Client library for Amazon Kinesis

Stars664

Forks486

Last commit2 days ago

amazon-kinesis-producer

Amazon Kinesis Producer Library

Stars416

Forks342

Last commit1 day ago

amazon-kinesis-client-python

Amazon Kinesis Client Library for Python

Stars376

Forks228

Last commit1 month ago

amazon-kinesis-scaling-utils

The Kinesis Scaling Utility is designed to give you the ability to scale Amazon Kinesis Streams in the same way that you scale EC2 Auto Scaling groups – up or down by a count or as a percentage of the total fleet. You can also simply scale to an exact number of Shards. There is no requirement for you to manage the allocation of the keyspace to Shards when using this API, as it is done automatically.

Stars335

Forks84

Last commit2 years ago

Community-curated · Updated weekly · 100% open source

Found a gem we're missing?

Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.

Submit a project Star on GitHub