A Fluentd output plugin for sending log events to Amazon Kinesis Data Streams and Amazon Data Firehose.
aws-fluent-plugin-kinesis is an open-source Fluentd output plugin developed by AWS Labs that sends log events from Fluentd to Amazon Kinesis Data Streams and Amazon Data Firehose. It solves the problem of efficiently streaming log data from Fluentd into AWS Kinesis for real-time analytics, processing, or data lake ingestion. The plugin handles authentication, batching, retries, and format transformations to ensure reliable data transfer.
DevOps engineers, SREs, and developers who use Fluentd for log aggregation and need to forward logs to AWS Kinesis for streaming analytics, monitoring, or data pipeline integration.
Developers choose this plugin because it's an official AWS-supported solution that provides robust integration with Kinesis services, extensive configuration options for production use, and compatibility with various AWS authentication methods. It simplifies the setup of log streaming pipelines from Fluentd to AWS without custom code.
Amazon Kinesis output plugin for Fluentd
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
Includes three distinct plugins for Kinesis Data Streams, Firehose, and aggregated streams, allowing flexible data routing based on use cases, as highlighted in the Key Features and README.
Supports various credential methods including instance profiles, IAM roles, and web identity tokens for EKS, ensuring secure integration in diverse AWS environments, detailed in the Credentials configuration section.
Configurable buffer sections and batch request retries with exponential backoff optimize throughput and handle failures reliably, evidenced in the Performance configuration with parameters like retries_on_batch_request.
Allows dynamic stream names using Fluentd placeholders based on tags, time, or custom fields, facilitating log segmentation, as described in the stream_name parameter with examples.
The plethora of options for buffering, batching, and credentials requires significant expertise to tune properly, which can be overwhelming for quick deployments, as seen in the extensive Configuration sections.
Exclusively designed for AWS Kinesis, making it unsuitable for multi-cloud strategies or environments without AWS integration, limiting flexibility compared to more generic log forwarders.
The README notes that v3 is 'almost compatible' with v2, indicating potential breaking changes and migration hurdles that need careful management, especially with Fluentd version requirements.