DCGM
DCGM (Data Center GPU Manager) is a GPU cluster management software framework developed by NVIDIA for data center environments. It provides system administrators and operations personnel with capabilities including GPU health monitoring, performance metrics collection, fault diagnosis, policy management, and job scheduling. DCGM supports API-based integration with mainstream cluster management and monitoring platforms such as Kubernetes and Prometheus, and is suited for unified operations management of large-scale GPU server deployments.
Core Facts
Timeline (last 90 days)
NVIDIA's NVLink, GPU drivers, and data center management software stack (such as DCGM) provide infrastructure support for closed-loop power control.
DCGM consists of the background daemon nv-hostengine and a client library; the daemon communicates with GPU hardware via NVML and CUPTI.
DCGM version 3.x introduced performance counter multiplexing capability to partially mitigate the counter mutual exclusion limitation.
DCGM's Profiling-level metrics (identified by the DCGM_FI_PROF_ prefix) require the use of GPU hardware performance counters and may be mutually exclusive with certain third-party profiling tools.
The GKE managed solution has correct attribution but misleading and non-upgradeable utilization metrics, while self-built DCGM has incorrect attribution by default but honest and configurable utilization metrics.
All Facts (8)
DCGM (Data Center GPU Manager) is a GPU management and monitoring tool provided by NVIDIA, widely used in data center environments.
70%UnverifiedDCGM Exporter能够暴露GPU温度、功耗、SM利用率、显存使用率、ECC错误计数、PCIe带宽等数十项硬件指标
90%UnverifiedFleet Intelligence的实时遥测能力建立在NVML和DCGM等底层监控技术之上
85%UnverifiedNVIDIA's NVLink, GPU drivers, and data center management software stack (such as DCGM) provide infrastructure support for closed-loop power control.
50%UnverifiedDCGM consists of the background daemon nv-hostengine and a client library; the daemon communicates with GPU hardware via NVML and CUPTI.
50%UnverifiedDCGM version 3.x introduced performance counter multiplexing capability to partially mitigate the counter mutual exclusion limitation.
50%UnverifiedThe GKE managed solution has correct attribution but misleading and non-upgradeable utilization metrics, while self-built DCGM has incorrect attribution by default but honest and configurable utilization metrics.
50%UnverifiedDCGM's Profiling-level metrics (identified by the DCGM_FI_PROF_ prefix) require the use of GPU hardware performance counters and may be mutually exclusive with certain third-party profiling tools.
50%