DCGM Exporter
Exposes NVIDIA GPU metrics for Prometheus monitoring using the NVIDIA Data Center GPU Manager (DCGM).
What is DCGM Exporter?
DCGM-Exporter is a Prometheus exporter that collects and exposes metrics from NVIDIA GPUs using the NVIDIA Data Center GPU Manager (DCGM). It solves the problem of monitoring GPU performance, health, and utilization in data center and cloud-native environments by providing standardized metrics that can be scraped by Prometheus and visualized in tools like Grafana.
System administrators, DevOps engineers, and data scientists who manage GPU-accelerated workloads in Kubernetes clusters, data centers, or High-Performance Computing (HPC) environments and need detailed GPU monitoring.
Developers choose DCGM-Exporter because it provides a production-ready, officially supported way to monitor NVIDIA GPUs with deep integration into the Prometheus ecosystem. Its flexibility, Kubernetes-native deployment, and support for custom metrics and HPC job mapping make it superior to basic monitoring solutions.
Overview
NVIDIA GPU metrics exporter for Prometheus leveraging DCGM
Use Cases
Best For
- Monitoring NVIDIA GPU performance in Kubernetes clusters
- Collecting GPU telemetry for Prometheus and Grafana dashboards
- Tracking GPU utilization and health in data center environments
- Integrating GPU metrics with the NVIDIA GPU Operator
- Monitoring GPU-accelerated High-Performance Computing (HPC) jobs
- Building custom GPU monitoring solutions with configurable metrics
Not Ideal For
- Monitoring non-NVIDIA GPUs (e.g., AMD or Intel) in heterogeneous hardware environments
- Teams using alternative monitoring stacks like Datadog or Zabbix without Prometheus integration
- Edge or embedded deployments where the resource overhead of DCGM and containerized exporters is prohibitive
- Projects requiring out-of-the-box GPU monitoring without Kubernetes or container orchestration setup
Pros & Cons
Pros
Exposes over 100 DCGM fields including SM clock, memory clock, and temperature as Prometheus gauges, providing deep insights into GPU performance and health directly from the NVIDIA API.
Includes a Helm chart for easy installation and integrates seamlessly with the NVIDIA GPU Operator, making it production-ready for cloud-native environments without manual configuration.
Allows users to specify which DCGM fields to collect via a custom CSV file, enabling tailored monitoring setups without code changes, as shown in the default-counters.csv example.
Can include HPC job IDs in metric labels by reading GPU-to-job mapping files from a directory, essential for tracking GPU usage in high-performance computing clusters with minimal setup.
Cons
Relies entirely on NVIDIA DCGM, so it cannot monitor GPUs from other vendors like AMD or Intel, creating vendor lock-in and limiting use in mixed-hardware setups.
Requires DCGM to be installed and compatible with GPU drivers, adding setup complexity and potential versioning issues, especially in non-containerized environments.
Official documentation is hosted on docs.nvidia.com, separate from the GitHub repo, which can make it harder to find up-to-date information and contribute, as noted in the README.
As shown in the quickstart example, some metrics like memory temperature may display unrealistic default values (e.g., 9223372036854775794), indicating potential data quality or initialization issues.
Frequently Asked Questions
Related Projects
Infiniband ExporterThe InfiniBand exporter is a specialized Prometheus exporter designed to gather metrics from InfiniBand fabrics, which are critical for high-performance computing and data center networking. It enables monitoring of switch and host channel adapter (HCA) port counters, providing visibility into fabric health and performance. ## Key Features - **Switch Port Monitoring** — Collects counters from all switch ports on the fabric connected to the host. - **HCA Port Monitoring** — Optionally gathers counters from host channel adapter ports. - **Unmanaged Switch Support** — Beta feature to collect data like power supply and fan health from unmanaged switches via ibswinfo. - **Large Fabric Optimization** — Supports run-once mode with file output for large fabrics where scrape times are prohibitive. - **Flexible Deployment** — Can run with sudo, without sudo via udev rules, in Docker, or as a systemd service. ## Philosophy The exporter is built to be reliable and efficient for production InfiniBand fabrics, with considerations for performance on large-scale deployments and flexibility in deployment methods.
GPFS ExporterThe GPFS exporter is a Prometheus exporter designed to gather extensive metrics from IBM Spectrum Scale (GPFS) filesystems. It enables system administrators and DevOps teams to monitor the health, performance, and utilization of their GPFS clusters through a standardized metrics endpoint. ## Key Features **Modular Collectors** — Supports over a dozen collectors for different GPFS aspects like state, I/O, mounts, health, quotas, snapshots, and disk pools, each configurable via command-line flags. **Flexible Deployment** — Includes optional helper exporters (`gpfs_mmdf_exporter`, `gpfs_mmlssnapshot_exporter`) for cron-based collection of heavy operations to avoid scrape timeouts. **Security Integration** — Provides detailed sudo configuration examples to securely execute GPFS commands with minimal privileges. **Production Ready** — Supports TLS and basic authentication via Prometheus exporter-toolkit and includes example Prometheus scrape configurations and a Grafana dashboard. ## Philosophy The project follows a modular and secure design, allowing users to enable only the collectors they need while providing safe integration with GPFS administrative commands through privilege separation.
Lustre ExporterPrometheus exporter for use with the Lustre parallel filesystem
Cgroup ExporterThe cgroup_exporter is a Prometheus exporter that scrapes resource utilization data from Linux cgroups, which are kernel mechanisms for limiting and monitoring resource usage for processes. It is particularly useful for monitoring jobs in high-performance computing (HPC) environments managed by schedulers like Slurm and TORQUE, as well as systemd user slices. ## Key Features - **CPU Metrics** — Exposes system, user, and total CPU seconds consumed by cgroups. - **Memory Metrics** — Reports memory usage, cache, RSS, total, and fail counts for both memory and swap. - **Cgroup Information** — Includes metadata such as user ID, username, and job ID for contextual labeling. - **Multi-Scheduler Support** — Works with cgroup paths for Slurm, TORQUE, and systemd user slices. - **Process Collection** — Optional collection of process information from procfs with appropriate capabilities. - **Docker Deployment** — Provides a Docker image for easy containerized deployment with host cgroup filesystem access. ## Philosophy The exporter is designed to be simple and focused, providing essential cgroup metrics to Prometheus without unnecessary complexity, enabling detailed resource monitoring in containerized and HPC environments.
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.