GPU optimization

StormForge can collect GPU metrics for workloads that use NVIDIA GPUs and show each workload’s own GPU usage alongside its CPU and memory usage. The Node Agent collects these metrics. See Node Agent configuration.

What StormForge shows for GPU workloads

When a workload uses GPUs, the workload details page adds the following.

The Summary and Recommendation tabs show per-pod data: they show GPU information specifically attributable to the workload’s own containers. The GPU tab is device-level: it shows total activity across whole GPU devices the workload was allocated, which other workloads may also be contributing to when devices are shared.

For more information about the workload details page, see View recommendations.

How GPU metrics are collected

The Node Agent is a DaemonSet that runs on GPU nodes. On each node it reads GPU state directly from the NVIDIA driver, reads GPU memory per process, and maps each process back to the container that owns it. This is what makes per-workload attribution possible on a GPU shared between pods. Because it reads the driver directly, you do not need to run dcgm-exporter or any other GPU exporter alongside it.

Device-level engine activity metrics, such as streaming multiprocessor and tensor core activity, depend on NVIDIA profiling counters. Where a GPU or driver does not support profiling, those metrics are absent and the rest of the GPU metrics are unaffected.

To turn collection on, see Node Agent configuration.

How GPU workloads are identified

StormForge marks a workload as using GPUs when any of its containers requests a GPU extended resource, such as nvidia.com/gpu, or when workloads request GPUs through a Dynamic Resource Allocation ResourceClaim or ResourceClaimTemplate.

Detection needs no configuration and happens whether or not the Node Agent is enabled. Collecting GPU metrics requires the Node Agent.

To list GPU workloads from the command line:

stormforge view workloads --gpu

Enabling GPU visibility

Prerequisites:

• StormForge Agent version 3.2.0 or later
• Helm version 3.14.0 or later (required for the --reset-then-reuse-values parameter)
• Nodes with NVIDIA GPUs, running the NVIDIA container runtime
• GPU nodes on the linux/amd64 architecture

The Node Agent needs node access that some managed and hardened Kubernetes platforms do not permit. It runs in the host PID namespace, holds the SYS_ADMIN capability, and mounts a host path. If your cluster restricts these, confirm that its policies admit the DaemonSet before you enable it. See Permissions and host access.

Steps

1. Enable the Node Agent. In an existing cluster, run:

   helm upgrade stormforge oci://registry.stormforge.io/library/stormforge \
     -n stormforge-system \
     --reset-then-reuse-values \
     --set nodeAgent.enabled=true
   

   By default the Node Agent is scheduled onto nodes labeled nvidia.com/gpu.present: "true" and tolerates all taints. If your GPU nodes are labeled differently, override the selector as described in Node selection. See Node Agent configuration for the full set of parameters.

2. Verify that the DaemonSet is running on your GPU nodes:

   kubectl get pods -n stormforge-system -l app.kubernetes.io/component=node-agent -o wide