How to deploy White Elephant in production?

Build a WAR file using 'ant war' with a config path, then deploy it to Tomcat. You must specify Hadoop JARs, configuration directories, and security settings like principal and keytab in config.yml, as detailed in the server deployment section.

White Elephant vs Apache Ambari for Hadoop monitoring?

White Elephant focuses on log aggregation and utilization visualization with detailed charts, while Ambari offers broader cluster management and monitoring. White Elephant provides deeper analytics but is limited to Hadoop 1.0.x, whereas Ambari supports newer versions and real-time features.

Does White Elephant process logs in real-time?

No, it processes logs in batches with hourly aggregation via Hadoop jobs. The incremental job handles new data efficiently, but it's not designed for real-time streaming; it's better for periodic analysis of usage patterns.

What Hadoop versions are compatible with White Elephant?

It is compiled and tested against Hadoop 1.0.3 and should work with any 1.0.x version, as stated in the README. Hadoop 2.0 is not yet supported, so check your cluster version before use.

How to configure White Elephant for multiple clusters?

Set up different log paths in the 'logs.root' parameter in base.properties for each cluster, and adjust the server configuration to load data from these paths. The multi-cluster support is designed through configurable patterns in the Hadoop jobs.

Can White Elephant handle large-scale log data efficiently?

Yes, through incremental processing that only deals with new logs, but the in-memory HyperSQL database might limit scalability for very large datasets. Consider the batch nature and potential database constraints for high-volume environments.

Open-Awesome

White Elephant

NOASSERTIONJava

A Hadoop log aggregator and dashboard for visualizing cluster utilization across users.

GitHub

190 stars60 forks0 contributors

What is White Elephant?

White Elephant is a Hadoop log aggregator and dashboard that processes Hadoop job logs to visualize cluster utilization across users. It transforms raw log data into structured Avro format and provides interactive charts showing resource consumption patterns over time. The system helps administrators understand how Hadoop clusters are being used and identify optimization opportunities.

Target Audience

Hadoop administrators and data platform engineers who need to monitor and analyze cluster utilization patterns across multiple users and jobs. Organizations running Hadoop clusters who want visibility into resource consumption.

Value Proposition

White Elephant provides a specialized solution for Hadoop log analysis with built-in aggregation and visualization, unlike generic monitoring tools. Its incremental processing capability makes it efficient for ongoing log analysis, and its Avro-based data pipeline ensures structured, queryable usage data.

Overview

Hadoop log aggregator and dashboard

Use Cases

Best For

Monitoring Hadoop cluster utilization across multiple users and teams
Analyzing resource consumption patterns in Hadoop job logs
Visualizing Hadoop cluster performance with interactive time-series charts
Building custom dashboards for Hadoop infrastructure monitoring
Processing and aggregating Hadoop logs into structured Avro format
Tracking Hadoop job execution patterns and identifying optimization opportunities

Not Ideal For

Organizations running Hadoop 2.0 or newer versions, as compatibility is limited to Hadoop 1.0.x
Teams requiring real-time cluster monitoring, since data processing is batch-oriented with hourly aggregation
Projects without existing Hadoop log infrastructure or those using alternative big data frameworks like Spark

Pros & Cons

Pros

Efficient Incremental Processing

The incremental job only processes new log data, reducing overhead for ongoing analysis, as described in the job configuration files for efficient updates.

Multi-Cluster Configuration

Supports monitoring multiple Hadoop clusters through configurable log paths in base.properties, enabling centralized management across environments.

Structured Avro Data Pipeline

Converts raw logs to Avro format, creating a queryable data cube for analytics, which simplifies integration and querying as outlined in the Hadoop jobs section.

Interactive Visualization Dashboard

Uses Rickshaw and D3.js to provide interactive charts for cluster usage insights, helping administrators visualize patterns over time with detailed graphs.

Cons

Outdated Hadoop Compatibility

Only supports Hadoop 1.0.x and explicitly does not work with Hadoop 2.0, making it unsuitable for modern Hadoop deployments as admitted in the README.

Complex Deployment Setup

Requires manual configuration of Hadoop JARs, keytabs for security, and WAR deployment to Tomcat, which can be cumbersome and error-prone for production environments.

In-Memory Database Limitations

Relies on HyperSQL in-memory database, which may not scale for large datasets and lacks built-in persistence, potentially impacting performance and availability.

Frequently Asked Questions

Related Projects

mrjob

Run MapReduce jobs on Hadoop or Amazon Web Services

Stars2,613

Forks579

Last commit5 months ago

Elasticsearch Hadoop

:elephant: Elasticsearch real-time search and analytics natively integrated with Hadoop

Stars1,971

Forks993

Last commit4 days ago

Genie

Distributed Big Data Orchestration Service

Stars1,766

Forks374