How does Log-Pilot compare to running Fluentd or Filebeat directly?

Log-Pilot simplifies deployment by managing a single instance per host with automatic container discovery via Docker labels, whereas direct Fluentd/Filebeat often require manual configuration per container. However, for advanced custom plugins or fine-grained control, direct usage might be better.

Can I use Log-Pilot with Kubernetes?

Log-Pilot is designed for Docker and may not work out-of-the-box with Kubernetes pods unless Docker is the container runtime. For Kubernetes-native logging, tools like Fluent Bit or DaemonSets are more suitable, as Log-Pilot relies on Docker-specific features.

How to configure Log-Pilot to send logs to AWS CloudWatch?

Use the Filebeat plugin with Log-Pilot and set up AWS credentials in the configuration, similar to standard Filebeat setups. Specify output in Log-Pilot's settings, and use Docker labels on containers to define log sources for CloudWatch integration.

Does Log-Pilot handle log rotation?

Log-Pilot can collect logs from rotated files if specified with wildcards in Docker labels, such as 'localhost_access_log.*.txt', but it doesn't manage rotation itself. Log rotation must be handled by the application or Docker log drivers.

How to add custom tags to logs in Log-Pilot?

Add tags via Docker labels like 'aliyun.logs.tags' on containers, which attaches metadata to logs for filtering in systems like Elasticsearch. This is documented in the feature section for easier organization and querying.

Is Log-Pilot good for high-volume production environments?

While efficient, Log-Pilot's performance depends on the underlying plugin (Fluentd or Filebeat) and host resources. For very high volumes, tuning and monitoring are necessary, and it may add overhead due to extra container and host mounts.

Open-Awesome

log-pilot

Apache-2.0Gov0.9.7

A Docker log collection tool that gathers container stdout and internal log files, sending them to centralized logging systems.

Visit Website GitHub

1.4k stars397 forks0 contributors

What is log-pilot?

Log-Pilot is a Docker log collection tool that automatically gathers logs from Docker containers, including both stdout and internal log files. It solves the problem of fragmented log management in containerized environments by providing a unified way to ship logs to centralized systems like Elasticsearch, Graylog, or AWS CloudWatch Logs.

Target Audience

DevOps engineers and platform teams managing Docker-based infrastructure who need reliable, automated log aggregation without per-container manual configuration.

Value Proposition

Developers choose Log-Pilot for its declarative configuration via Docker labels, support for both stdout and file logs, and flexibility with multiple logging backends—all without requiring separate log collector processes per container.

Overview

Collect logs for docker containers

Use Cases

Best For

Collecting logs from Docker containers in development or production environments
Shipping container logs to Elasticsearch for analysis in Kibana
Aggregating Docker stdout and application log files simultaneously
Setting up log collection for Dockerized microservices with minimal configuration
Integrating Docker logs with Graylog or AWS CloudWatch Logs
Implementing a centralized logging strategy for containerized applications

Not Ideal For

Environments using non-Docker container runtimes like Kubernetes with containerd or CRI-O
Teams requiring extensive in-flight log parsing, enrichment, or complex transformation
Projects where minimizing container count and host-level permissions is critical for security or simplicity

Pros & Cons

Pros

Multi-Plugin Flexibility

Supports both Fluentd and Filebeat plugins without separate processes per container, allowing seamless integration with various logging backends like Elasticsearch and AWS CloudWatch.

Comprehensive Log Capture

Collects both Docker container stdout and log files from inside containers, addressing limitations of Docker's native logging that only handles stdout.

Declarative Label Configuration

Uses Docker labels to specify logs to collect, minimizing manual setup and enabling easy scaling in containerized environments as per the quickstart example.

Broad Backend Compatibility

Outputs to multiple centralized systems including Elasticsearch, Graylog2, and AWS CloudWatch Logs, providing flexibility in log management tooling.

Cons

Docker-Only Limitation

Exclusively designed for Docker environments, making it incompatible with other container runtimes or orchestrators without Docker, which restricts use in heterogeneous setups.

Complex and Insecure Setup

Requires mounting the host filesystem (/) and granting SYS_ADMIN capabilities, as shown in the run command, which can introduce security risks and deployment complexity.

Limited Log Processing

Focuses primarily on collection and shipping with basic tagging, lacking built-in features for advanced parsing, filtering, or transformation compared to dedicated log processors.

Frequently Asked Questions

Related Projects

ansible

Ansible is a radically simple IT automation platform that makes your applications and systems easier to deploy and maintain. Automate everything from code deployment to network configuration to cloud management, in a language that approaches plain English, using SSH, with no agents to install on remote systems. https://docs.ansible.com.

Stars69,669

Forks24,186

Last commit23 hours ago

localstack

💻 A fully functional local AWS cloud stack. Develop and test your cloud & Serverless apps offline

Stars65,152

Forks4,771

Last commit4 months ago

traefik

The Cloud Native Application Proxy

Making Docker and Kubernetes management easy.

Stars38,033

Forks2,870