DCGM Exporter
DCGM Exporter is an open-source monitoring component officially provided by NVIDIA, built on top of DCGM (Data Center GPU Manager). It collects GPU device performance metrics and converts them into a format that Prometheus can scrape. The component supports exporting multi-dimensional GPU metrics including temperature, memory usage, utilization, and power consumption. It is commonly used together with Prometheus and Grafana for GPU resource monitoring in data centers and Kubernetes clusters.
Timeline (last 90 days)
NVIDIA patched a high-severity vulnerability in the GPU monitoring tool DCGM Exporter, which stemmed from unprotected Go language profiling routes.
The vulnerability allows attackers to remotely exhaust system resources by sending massive numbers of requests, causing GPU monitoring interruptions and affecting AI training and inference tasks.
DCGM Exporter exposes a Prometheus-format metrics scraping endpoint on port 9400 by default, deployed on each GPU node as a Kubernetes DaemonSet or a standalone process.
Network scanning discovered approximately 2,100 systems exposed to the public internet, distributed across approximately 300 organizations, leaking unique identifiers for more than 12,000 GPUs.
Approximately one quarter of the exposed systems had dangerous diagnostic features enabled and were in a state susceptible to attack.
The CVSS vector for this vulnerability is: attack vector network, attack complexity low, no privileges required, no user interaction required, with only availability highly impacted, and no data leakage or privilege escalation involved.
NVIDIA has released a fix and recommends that administrators upgrade to DCGM Exporter 4.8.2 or a later version.
Official guidance recommends disabling diagnostic routes such as profiling unless necessary, restricting the network exposure of monitoring endpoints, and adding authentication and rate limiting to monitoring interfaces.
When profiling routes and monitoring metrics routes share the same listening address without path-level access control, exposing the monitoring port externally is equivalent to exposing the diagnostic attack surface.
All Facts (10)
DCGM Exporter能够暴露GPU温度、功耗、SM利用率、显存使用率、ECC错误计数、PCIe带宽等数十项硬件指标
90%UnverifiedNVIDIA patched a high-severity vulnerability in the GPU monitoring tool DCGM Exporter, which stemmed from unprotected Go language profiling routes.
50%UnverifiedThe vulnerability allows attackers to remotely exhaust system resources by sending massive numbers of requests, causing GPU monitoring interruptions and affecting AI training and inference tasks.
50%UnverifiedDCGM Exporter exposes a Prometheus-format metrics scraping endpoint on port 9400 by default, deployed on each GPU node as a Kubernetes DaemonSet or a standalone process.
50%UnverifiedNetwork scanning discovered approximately 2,100 systems exposed to the public internet, distributed across approximately 300 organizations, leaking unique identifiers for more than 12,000 GPUs.
50%UnverifiedApproximately one quarter of the exposed systems had dangerous diagnostic features enabled and were in a state susceptible to attack.
50%UnverifiedThe CVSS vector for this vulnerability is: attack vector network, attack complexity low, no privileges required, no user interaction required, with only availability highly impacted, and no data leakage or privilege escalation involved.
50%UnverifiedOfficial guidance recommends disabling diagnostic routes such as profiling unless necessary, restricting the network exposure of monitoring endpoints, and adding authentication and rate limiting to monitoring interfaces.
50%UnverifiedWhen profiling routes and monitoring metrics routes share the same listening address without path-level access control, exposing the monitoring port externally is equivalent to exposing the diagnostic attack surface.
50%UnverifiedNVIDIA has released a fix and recommends that administrators upgrade to DCGM Exporter 4.8.2 or a later version.
50%