Node Agent configuration

The Node Agent is an optional DaemonSet that runs on your GPU nodes and collects GPU metrics from them. It is disabled by default.

This topic covers the Node Agent configuration parameters. For what the metrics are used for, see GPU optimization.

Applying these settings

These are Helm values for the stormforge chart. Add them to a YAML values file (for example, values.yaml) and pass it to your Helm command.

At install time:

helm install stormforge oci://registry.stormforge.io/library/stormforge \
  --namespace stormforge-system \
  -f values.yaml

To apply changes after installation:

helm upgrade stormforge oci://registry.stormforge.io/library/stormforge \
  --namespace stormforge-system \
  --reset-then-reuse-values \
  -f values.yaml

Node Agent parameters:

• Enable the Node Agent
• Namespace scope
• Node selection
• Pod priority and scheduling
• Resource requests and limits
• Metrics port
• Scrape interval
• Private container registry
• Permissions and host access

Enable the Node Agent

Controls whether StormForge deploys the Node Agent.

Description

Set to true to deploy a DaemonSet named stormforge-node-agent that collects GPU metrics from the nodes it runs on. Enabling it also creates a ServiceAccount and a ClusterRole of the same name.

The Node Agent component is available in StormForge Agent version 3.2.0 or later.

Valid values

• true
• false (default)

Example

nodeAgent:
  enabled: true

Namespace scope

Limits the namespaces whose pods the Node Agent watches.

Description

The Node Agent watches pods to map each GPU process back to the container that owns it. By default it watches pods in all namespaces. Set nodeAgent.watchNamespace to a single namespace to narrow the scope, limiting GPU attribution to workloads in that namespace.

Valid values

A namespace name, or "" to watch all namespaces.

Node selection

Controls which nodes the Node Agent runs on.

Description

Because the Node Agent only needs to run on GPU nodes, it ships with its own scheduling defaults, unlike the other StormForge components. It selects nodes labeled nvidia.com/gpu.present: "true", which the NVIDIA GPU Feature Discovery component sets, and tolerates every taint so that it can be scheduled onto tainted GPU node pools.

GPU nodes are not labeled the same way in every cluster. Cloud-provider accelerator labels, Karpenter node pools, and custom labels are all common. If your GPU nodes do not carry nvidia.com/gpu.present, override nodeAgent.nodeSelector.

nodeAgent.nodeSelector and nodeAgent.tolerations behave differently when you override them:

• nodeAgent.tolerations is a list, so the value you set replaces the default.
• nodeAgent.nodeSelector is a map, so the value you set merges with the default. Any label you add applies in addition to nvidia.com/gpu.present, and a node must carry both to be selected. To remove the default, set it to null.

Valid values

For nodeSelector and tolerations format, see the Kubernetes documentation on assigning pods to nodes.

Example

To select GPU nodes by a cloud-provider accelerator label instead of the default, and tolerate only the GPU taint:

nodeAgent:
  nodeSelector:
    nvidia.com/gpu.present: null
    cloud.google.com/gke-accelerator: nvidia-tesla-t4
  tolerations:
  - key: nvidia.com/gpu
    operator: Exists
    effect: NoSchedule

To keep the default and narrow it further, add your label without the null. A node must then carry both labels:

nodeAgent:
  nodeSelector:
    cloud.google.com/gke-accelerator: nvidia-tesla-t4

Pod priority and scheduling

Enables priority configuration on StormForge Node Agent pods.

Description

The top-level priorityClassName applies to all StormForge components. Set nodeAgent.priorityClassName to override it for the Node Agent only. A non-empty per-component value replaces the top-level default.

nodeAgent.priorityClassName assigns an existing Pod PriorityClass to the stormforge-node-agent pods.

Valid values

A string that matches the .metadata.name of a PriorityClass defined in your cluster.

Example

nodeAgent:
  priorityClassName: "high-priority"

Resource requests and limits

Sets the Node Agent container’s own resource settings.

Description

The Node Agent’s memory use grows with the number of GPUs on the node. The default limit suits dense multi-GPU nodes; raise it if the Node Agent is OOMKilled.

Valid values

A Kubernetes ResourceRequirements object.

Example

nodeAgent:
  resources:
    requests:
      cpu: 50m
      memory: 128Mi
    limits:
      memory: 1Gi

Metrics port

Sets the port the Node Agent serves metrics and health checks on.

Description

The Node Agent serves its metrics at /metrics and its readiness check at /healthz on this port. The Forwarder scrapes the Node Agent on this port, and changing it updates both sides. Change it if 9401 conflicts with another workload on your GPU nodes, or if a network policy requires a specific port.

Scrape interval

Sets how often the Forwarder scrapes the Node Agent.

Description

Leave empty to use the Forwarder’s scrape interval. Set it to collect GPU metrics at a different frequency from the rest of the metrics.

Valid values

A Prometheus duration string (for example, 30s, 1m).

Private container registry

Override the Node Agent image repository for environments that use a private container registry.

Description

The Node Agent uses its own image, separate from the image the Agent, Applier, and webhook share. When using a private registry, push this image to your registry as well and set nodeAgent.image.repository accordingly.

For the full setup procedure, see If you have a private container registry.

Example

nodeAgent:
  image:
    repository: my-registry.example.com/optimize/stormforge-node-agent

Permissions and host access

The Node Agent needs more node access than the other StormForge components.

Description

Reading GPU state per process requires access to the NVIDIA driver and to process information on the node, so the DaemonSet:

• Runs in the host PID namespace. The driver reports host process IDs, and the Node Agent reads each process’s cgroup to map it to a pod. This is not configurable.
• Holds the SYS_ADMIN capability, and drops all others. On some GPU node images, including Bottlerocket’s NVIDIA variant, the NVIDIA container runtime injects the driver libraries only into containers that hold this capability. Without it, the Node Agent fails to start and logs nvml init: ERROR_LIBRARY_NOT_FOUND.
• Mounts the kubelet’s pod resources socket directory, /var/lib/kubelet/pod-resources, read-only, to learn which containers hold which GPU devices.
• Sets readOnlyRootFilesystem: true and allowPrivilegeEscalation: false.

On node images that inject the driver libraries unconditionally, such as Amazon Linux 2023, you can override nodeAgent.securityContext to drop SYS_ADMIN where your Pod Security or OPA policy forbids it. Confirm that the Node Agent starts afterward. The host PID namespace is required either way.

The Node Agent’s ClusterRole grants read-only access (get, list, and watch) on pods and on resourceslices.resource.k8s.io.

The Node Agent does not mount the container runtime socket or the host root filesystem.

Related topics

• GPU optimization
• StormForge Node Agent
• Kubernetes official documentation:
  • DaemonSet
  • Assigning Pods to Nodes