Configure by using annotations

Using resource annotations is the most efficient way to configure resources and control optimization behavior.

Annotations ensure consistency across workloads and give teams the control they need over their work - for example:

• Platform teams can set organization-wide default optimization values
• Development teams can fine-tune settings depending on their application and environment (development, staging, test, production)

Key points:

• Think of settings at the cluster, namespace, and workload levels as a hierarchy of configuration.

  • Workload-level configuration customizes the optimization behavior for individual workloads. This level has the highest precedence and cannot be overridden. You can annotate these types: Deployments, StatefulSets, and ReplicaSets.
  • Namespace-level configuration sets default values for all workloads in a namespace. Default values set at this level fill in any settings not provided by annotations on the workloads themselves.
  • Cluster-level defaults apply to all workloads in a cluster. Default values set at this level have the lowest precedence and can be overridden by values set at the namespace and workload levels.

• Best practice: Use annotations to configure workload settings. Use the UI only to view workload settings.

  A workload becomes read-only in the UI anyway when a workload is managed in any way by annotations. For example, if namespace-level defaults have been applied, you cannot use the UI to change settings for any workload in that namespace.

Topics:

• Save time: Copy a template that contains the annotations
• Configure optimization settings for a specific workload
• Configure default optimization settings for all workloads in a namespace
  • Using YAML manifests
  • Using kubectl annotate
• Configure default optimization settings for all workloads in a cluster
  • Preferred method: Using Helm
  • Using kubectl edit
• Schedule examples
• Threshold examples

Save time: Copy a template

• Choose a template
• Workload
• Namespace
• Cluster

apiVersion: apps/v1 # just an example; represents any
kind: Deployment    # supported workload resource type
metadata:
  name: example-workload-name
  annotations:
    # ---------------------------------------------
    # WORKLOAD SETTINGS
    # Specify single values for the whole workload.
    # ---------------------------------------------
    live.stormforge.io/schedule: "@daily"
    live.stormforge.io/cpu.optimization-goal: "Balanced"
    live.stormforge.io/memory.optimization-goal: "Balanced"

    # Main configuration control for auto-deploy
    live.stormforge.io/auto-deploy: "false" # Set to "true" to enable auto-deploy

    # When auto-deploy is false, the following subsettings have no effect 
    live.stormforge.io/auto-deploy.thresholds.cpu.percent: "5"    # 1/2 Paired parameters; Set both or neither
    live.stormforge.io/auto-deploy.thresholds.memory.percent: "5" # 2/2. 
    live.stormforge.io/auto-deploy.thresholds.cpu.unit: "10m"     # 1/2 Paired parameters; set both or neither
    live.stormforge.io/auto-deploy.thresholds.memory.unit: "20Mi" # 2/2.

    # The following settings have an effect only if an HPA is enabled and the HPA scales on CPU or memory.
    live.stormforge.io/hpa.cpu.target-utilization.min: "20"   
    live.stormforge.io/hpa.cpu.target-utilization.max: "80"   
    live.stormforge.io/hpa.memory.target-utilization.min: "25"  
    live.stormforge.io/hpa.memory.target-utilization.max: "75"  

    # -------------------------------------------------
    # CONTAINER SETTINGS
    # Specify defaults and/or per-container exceptions.
    # E.g. "RequestsOnly,istio-proxy=DoNotOptimize"
    # -------------------------------------------------
    live.stormforge.io/containers.cpu.optimization-policy: "RequestsOnly"
    live.stormforge.io/containers.cpu.requests.min: "20m"
    live.stormforge.io/containers.cpu.requests.max: "16000m"
    live.stormforge.io/containers.cpu.limits.min: "2000m"
    live.stormforge.io/containers.cpu.limits.max: "16000m"
    live.stormforge.io/containers.cpu.limits.limit-request-ratio: "2.0"
    live.stormforge.io/containers.memory.optimization-policy: "RequestsOnly"
    live.stormforge.io/containers.memory.requests.min: "64Mi"
    live.stormforge.io/containers.memory.requests.max: "32Gi"
    live.stormforge.io/containers.memory.limits.min: "512Mi"
    live.stormforge.io/containers.memory.limits.max: "64Gi"
    live.stormforge.io/containers.memory.limits.limit-request-ratio: "2.0"

spec:
  template:
    metadata:

apiVersion: v1
kind: Namespace
metadata:
  name: example-namespace-name
  annotations:
    # ---------------------------------------------
    # WORKLOAD SETTINGS
    # Specify single values for the whole workload.
    # ---------------------------------------------
    live.stormforge.io/schedule: "@daily"
    live.stormforge.io/cpu.optimization-goal: "Balanced"
    live.stormforge.io/memory.optimization-goal: "Balanced"

    # Main configuration control for auto-deploy
    live.stormforge.io/auto-deploy: "false" # Set to "true" to enable auto-deploy

    # When auto-deploy is false, the following subsettings have no effect 
    live.stormforge.io/auto-deploy.thresholds.cpu.percent: "5"    # 1/2 Paired parameters; set both or neither.
    live.stormforge.io/auto-deploy.thresholds.memory.percent: "5" # 2/2. 
    live.stormforge.io/auto-deploy.thresholds.cpu.unit: "10m"     # 1/2 Paired parameters; set both or neither.
    live.stormforge.io/auto-deploy.thresholds.memory.unit: "20Mi" # 2/2.

    # The following settings have an effect only if an HPA is enabled and the HPA scales on CPU or memory.
    live.stormforge.io/hpa.cpu.target-utilization.min: "20"   
    live.stormforge.io/hpa.cpu.target-utilization.max: "80"   
    live.stormforge.io/hpa.memory.target-utilization.min: "25"  
    live.stormforge.io/hpa.memory.target-utilization.max: "75"  

    # -------------------------------------------------
    # CONTAINER SETTINGS
    # Specify defaults and/or per-container exceptions.
    # E.g. "RequestsOnly,istio-proxy=DoNotOptimize"
    # -------------------------------------------------
    live.stormforge.io/containers.cpu.optimization-policy: "RequestsOnly"
    live.stormforge.io/containers.cpu.requests.min: "20m"
    live.stormforge.io/containers.cpu.requests.max: "16000m"
    live.stormforge.io/containers.cpu.limits.min: "2000m"
    live.stormforge.io/containers.cpu.limits.max: "16000m"
    live.stormforge.io/containers.cpu.limits.limit-request-ratio: "2.0"
    live.stormforge.io/containers.memory.optimization-policy: "RequestsOnly"
    live.stormforge.io/containers.memory.requests.min: "64Mi"
    live.stormforge.io/containers.memory.requests.max: "32Gi"
    live.stormforge.io/containers.memory.limits.min: "512Mi"
    live.stormforge.io/containers.memory.limits.max: "64Gi"
    live.stormforge.io/containers.memory.limits.limit-request-ratio: "2.0"

---
# Helm values.yaml file

clusterDefaultConfig:
  # All values below are optional; copy/paste only those that you need.
  # ---------------------------------------------
  # WORKLOAD SETTINGS
  # Specify single values for the whole workload.
  # ---------------------------------------------
  schedule: "P1D"
  cpuOptimizationGoal: "Balanced"
  memoryOptimizationGoal: "Balanced"

  # Main configuration control for auto-deploy
  autoDeploy: "false" # Set to "true" to enable auto-deploy

  # When auto-deploy is false, the following subsettings have no effect
  autoDeployThresholdsCpuPercent: "5"     # 1/2 Paired parameters; Set both or neither
  autoDeployThresholdsMemoryPercent: "5"  # 2/2. 
  autoDeployThresholdsCpuUnit: "10m"      # 1/2 Paired parameters; Set both or neither
  autoDeployThresholdsMemoryUnit: "20Mi"  # 2/2.

  # The following settings have an effect only if an HPA is enabled and the HPA scales on CPU or memory.
  hpaCpuTargetUtilizationMin: "20"   
  hpaCpuTargetUtilizationMax: "80"   
  hpaMemoryTargetUtilizationMin: "25"  
  hpaMemoryTargetUtilizationMax: "75" 

  # ---------------------------------------------
  # CONTAINER SETTINGS
  # Specify defaults and/or per-container values.
  # E.g. "RequestsOnly,istio-proxy=DoNotOptimize"
  # ---------------------------------------------
  containersCpuOptimizationPolicy: "RequestsOnly"
  containersCpuRequestsMin: "20m"
  containersCpuRequestsMax: "16000m"
  containersCpuLimitsMin: "2000m"
  containersCpuLimitsMax: "16000m"
  containersCpuLimitsLimitRequestRatio: "2.0"
  containersMemoryOptimizationPolicy: "RequestsOnly"
  containersMemoryRequestsMin: "64Mi"
  containersMemoryRequestsMax: "32Gi"
  containersMemoryLimitsMin: "512Mi"
  containersMemoryLimitsMax: "64Gi"
  containersMemoryLimitsLimitRequestRatio: "2.0"

When editing annotations:

• Enclose all values in double quotation marks ("") as shown in the templates.
• Best practice: When editing container-specific resources (items in the Container-specific settings section in a template, such as the container’s optimization policy), specify the value to apply to all or most containers first, followed by container-specific exceptions, as in these examples:
  • When annotating a workload or namespace:
    live.stormforge.io/containers.cpu.optimization-policy: "RequestsAndLimits,sidecar=DoNotOptimize"
  • In a cluster-default ConfigMap:
    containersCpuOptimizationPolicy: "RequestsAndLimits,sidecar=DoNotOptimize"

Configure optimization settings for a specific workload

Workload-level defaults are defined by annotations in the metadata.annotations section of a Deployment or StatefulSet resource type.

Using YAML manifests

1. Open the workload definition in your favorite editor. This is commonly done using commands like these (remember to replace DEPLOYMENT and FILENAME with your own values):

   kubectl get deployment DEPLOYMENT -o yaml > FILENAME && $EDITOR FILENAME
   

   or:

   kubectl edit deployment DEPLOYMENT
   

2. Edit the metadata.annotations section to include the annotations you need.

   • To save time, you can copy the complete list of annotations from the sample Deployment YAML excerpt on the Workload tab in the Copy a template section above.

3. Save your changes.

4. If you chose to run the kubectl get deployment DEPLOYMENT -o yaml > FILENAME ... command in step 1, apply your changes.

   kubectl apply -f FILENAME
   

5. Optional: Confirm your changes by running the following command and then reviewing the annotations in the metadata.annotations section of the output:

   kubectl get deployment DEPLOYMENT -o yaml
   

Configure default optimization settings for all workloads in a namespace

To configure settings for all workloads in a namesapce, edit the Namespace definition or run kubectl annotate.

Namespace-level defaults:

• Override equivalent cluster-level defaults
• Fill in any settings not provided by annotations on the workloads themselves

Using YAML manifests

1. Open the namespace definition in your favorite editor. This is commonly done using commands like these (remember to replace NAMESPACE and FILENAME with your own values):

   kubectl get namespace NAMESPACE -o yaml > FILENAME && $EDITOR FILENAME
   

   or:

   kubectl edit namespace NAMESPACE
   

2. Edit the metadata.annotations section to include the annotations you need.

   • To save time, you can copy the complete list of annotations from the sample Namespace YAML excerpt on the Namespace tab in the Copy a template section above.

3. Save your changes.

4. If you chose to run the kubectl get namespace NAMESPACE -o yaml > FILENAME ... command in step 1, apply your changes.

   kubectl apply -f FILENAME
   

5. Optional: Confirm your changes by running the following command and then reviewing the annotations in the metadata.annotations section of the output:

   kubectl get namespace NAMESPACE -o yaml
   

Using the kubectl annotate command

1. Run the kubectl annotate command and include the annotation:value pairs you want to set.

   • To save time, you can copy the pairs you need from the sample Namespace YAML excerpt on the Namespace tab in the Copy a template section above. Be sure to replace the : separator from the template with = as shown in the sample command below.

   As an example, a kubectl annotate might look something like this (remember to replace NAMESPACE with your appropriate value):

   kubectl annotate namespace NAMESPACE \
     live.stormforge.io/schedule="P1D" \
     live.stormforge.io/auto-deploy="false" \
     live.stormforge.io/containers.cpu.requests.min="20m" \
     live.stormforge.io/containers.cpu.requests.max="16000m"
   

2. Optional: To confirm your changes, run the following command and review the annotations in the .metadata.annotations section of the output:

   kubectl get namespace NAMESPACE -o yaml
   

Configure default optimization settings for all workloads in a cluster

Cluster-level defaults:

• Are specified in a ConfigMap named cluster-defaults in the stormforge-system namespace
• Have the lowest precedence and can be overridden by namespace-level default values and workload-specific values
• Are added and removed by either:
  • Providing updated Helm values
  • Deleting the annotation by editing the cluster-defaults ConfigMap

In the cluster-defaults.yamlfile, use annotation-name:value syntax as shown below:

apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-defaults
  namespace: stormforge-system
data:
  cluster-defaults.yaml: |
    live.stormforge.io/schedule: "P1D"
    live.stormforge.io/auto-deploy: "false"
    live.stormforge.io/containers.cpu.requests.min: "20m"
    live.stormforge.io/containers.cpu.requests.max: "16000m"
    ...    

You can create this ConfigMap yourself, but it may be easier to have Helm create and manage it for you.

Setting cluster default values using Helm

The stormforge-agent Helm chart has values you can pass to define each of the possible cluster-level default settings. Helm then creates and manages the cluster-defaults ConfigMap for you.

Preferred method: Helm chart values for cluster default settings

1. Optional: Check whether any cluster defaults are already set in the cluster-defaults ConfigMap by running:

   kubectl describe configmap cluster-defaults -n stormforge-system
   

   • If you see an Error from server (NotFound): configmaps "cluster-defaults" not found message, no cluster-level defaults are set. Proceed to the next step.
   • Otherwise, in the Data > cluster-defaults.yaml section of the output, notice which annotations are set. If you need to preserve those values, include them in the cluster-defaults.yaml file that you create in the next step.

2. Create a .yaml values file (for example, cluster-defaults.yaml) that defines the clusterDefaultConfig values you need.

   • To save time, you can copy the sample on the Cluster tab in the Copy a template section above. Edit or comment out annotations as needed.

     As an example, your cluster-defaults.yaml file might look something like this when you’re done:

   clusterDefaultConfig:
     schedule: "PT16H"
     autoDeploy: "true"
     cpuOptimizationGoal: "Savings"
     memoryOptimizationGoal: "Savings"
     containersCpuOptimizationPolicy: "RequestsAndLimits"
     containersCpuRequestsMin: "20m"
     containersCpuRequestsMax: "16000m"
     containersMemoryOptimizationPolicy: "RequestsAndLimits"
     containersMemoryRequestsMin: "64Mi"
     containersMemoryRequestsMax: "32Gi"
   

3. Run helm install or helm upgrade and include --values flag and your filename. For example (remember to replace CLUSTER_NAME and CREDENTIALS_FILE with your own values):

   helm upgrade stormforge-agent oci://registry.stormforge.io/library/stormforge-agent \
     --namespace stormforge-system \
     --set clusterName=CLUSTER_NAME \
     --values CREDENTIALS_FILE.yaml \
     --values cluster-defaults.yaml
   

   Tip: If you’re setting only a few default values, you pass them as --set arguments, as shown below. Be sure to replace the : separator from the template with = as shown in the sample command below.

   helm upgrade stormforge-agent oci://registry.stormforge.io/library/stormforge-agent \
     --namespace stormforge-system \
     --set clusterName=CLUSTER_NAME \
     --values CREDENTIALS_FILE.yaml \
     --set clusterDefaultConfig.autoDeploy=true \
     --set clusterDefaultConfig.cpuOptimizationGoal=balanced
   

   Any previous cluster-level default values are now removed, and your new defaults are in effect, unless they are overridden at the cluster or workload level.

4. Optional: Confirm your changes by running the following command and then reviewing the annotations in the Data section of the output:

   kubectl describe configmap cluster-defaults -n stormforge-system 
   

Setting cluster default values using kubectl edit

If at least one cluster-level default has been set already, you can edit the ConfigMap directly.

1. Run the folllowing command:

   kubectl edit configmap cluster-defaults -n stormforge-system
   

   If you see a Error from server (NotFound): configmaps "cluster-defaults" not found message, no cluster-level defaults are set. Use the Helm method described in the preceding section instead of continuing to the next step.

2. In the .data.cluster-defaults.yaml section, add, edit, or remove annotations as needed.

   • Using kubectl edit requires you to use the live.stormforge.io/* annotation syntax, not the clusterDefaults Helm chart values. To save time, you can copy the annotations from the sample on the Namespace tab (not the Cluster tab) in the Copy a template section above and edit them as needed.

     As an example, your annotations in the cluster-defaults ConfigMap might look something like this excerpt when you’re done:

   # Please edit the object below. Lines beginning with a '#' will be ignored,
   # and an empty file will abort the edit. If an error occurs while saving this file will be
   # reopened with the relevant failures.
   #
   apiVersion: v1
   data:
     cluster-defaults.yaml: |
       live.stormforge.io/auto-deploy: "true"
       live.stormforge.io/cpu.optimization-goal: "Reliability"
       live.stormforge.io/memory.optimization-goal: "Reliability"    
   kind: ConfigMap
   metadata:
   ...
   

3. Save your changes.

4. Restart the StormForge Agent:

   kubectl rollout restart deployment -n stormforge-system stormforge-agent-workload-controller
   

   To check the status of the restart, run:

   kubectl rollout status deployment -n stormforge-system stormforge-agent-workload-controller -w
   

   Any previous cluster-level default values are now removed, and your new defaults are in effect, unless they are overridden at the cluster or workload level.

5. Optional: Confirm your changes by running the following command and then reviewing the annotations in the Data section of the output:

   kubectl describe configmap cluster-defaults -n stormforge-system 
   

Related information

• Annotations list and descriptions

Schedule examples

Below are some additional examples of ways to set the live.stormforge.io/schedule (or schedule in a cluster-level ConfigMap) annotation using macros, ISO 8601 Duration, and cron notation:

• Once daily (default value and best practice):
  "@daily", "P1D" or "H H * * *"
• Hourly (the shortest interval you can specify):
  "@hourly", "PT1H" or "H * * * *"
• Every 12 hours:
  "PT12H" or "H H/12 * * *"
• Every morning at approximately 0800h (exact time is not guaranteed):
  "00 08 * * *"
• Every Wednesday:
  "H H * * WED"
• To stop generating recommendations:
  "@never" or "P0D"

Threshold examples

Thresholds define the minimum amount of change to requests values that make with worthwhile deploying new recommendations.

You can set a percent-change threshold, a unit-change threshold, or both.

Example: Set a percent-change threshold for either (or both) 5% CPU or 10% memory (using YAML manifests)
Suppose you want to apply only recommendations that change at least one of the following for at least one container in a workload:

• CPU requests by at least 5%
• Memory requests by at least 10%

For this example, assume you’re annotating a .yaml file. Where you add the following annotations depends on whether you’re annotating at the workload, namespace, or cluster level (as described earlier this topic):

live.stormforge.io/auto-deploy: "true"
live.stormforge.io/auto-deploy.thresholds.cpu.percent: "5"
live.stormforge.io/auto-deploy.thresholds.memory.percent: "10"

Suppose you want to remove the threshold. Use the following annotations:

live.stormforge.io/auto-deploy: "true"
live.stormforge.io/auto-deploy.thresholds.cpu.percent: "0"
live.stormforge.io/auto-deploy.thresholds.memory.percent: "0"

Example: Set percent-change and unit-change thresholds (using YAML manifests)
Suppose you want to apply only recommendations that satisfy at least one of the following conditions for at least one container in a workload:

• CPU requests change by at least 5% and 10m
• Memory requests change by at least 10% and 10Mi

For this example, assume you’re annotating a YAML manifest. Where you add the following annotations depends on whether you’re annotating at the workload, namespace, or cluster level (as described earlier this topic):

live.stormforge.io/auto-deploy: "true"
live.stormforge.io/auto-deploy.thresholds.cpu.percent: "5"
live.stormforge.io/auto-deploy.thresholds.cpu.unit: "10m"
live.stormforge.io/auto-deploy.thresholds.memory.percent: "10"
live.stormforge.io/auto-deploy.thresholds.memory.unit: "10Mi"

To remove the threshold, use the following annotations:

live.stormforge.io/auto-deploy: "true"
live.stormforge.io/auto-deploy.thresholds.cpu.percent: "0"
live.stormforge.io/auto-deploy.thresholds.memory.percent: "0"

Example: Set a unit-change threshold for 10m of CPU (using kubectl annotate)
Suppose you want to apply only recommendations that change CPU requests by at least 10m. Suppose also that memory requests don’t matter to you — you must still set a memory value greater than 0.

For this example, assume you’re using kubectl annotate as described earlier in this topic to annotate all workloads in a namespace. Your command will look something like this (replace NAMESPACE with the namespace name):

kubectl annotate namespace NAMESPACE \
  live.stormforge.io/auto-deploy="true" \
  live.stormforge.io/auto-deploy.thresholds.cpu.unit="10m" \
  live.stormforge.io/auto-deploy.thresholds.memory.unit="1Ki"

To remove the threshold, run this command (remember to replace NAMESPACE with an actual name):

kubectl annotate namespace NAMESPACE \
  live.stormforge.io/auto-deploy="true" \
  live.stormforge.io/auto-deploy.thresholds.cpu.unit="0" \
  live.stormforge.io/auto-deploy.thresholds.memory.unit="0"

Example: Set unit- and percent-change thresholds at the cluster level (using cluster default values)
Suppose you want to apply only recommendations that satisfy at least one of the following conditions for at least one container in a workload:

• CPU requests change by at least 5% and 10m
• Memory requests change by at least 10% and 10Mi

For this example, assume you’re using a cluster-defaults.yaml file to set Helm chart values via helm install or helm upgrade as described earlier in this topic to annotate all workloads in a cluster. At a minimum, this file would contain the following parameter:value pairs:

clusterDefaultConfig:
  autoDeploy: "true"
  autoDeployThresholdsCpuPercent: "5"
  autoDeployThresholdsCpuUnit: "10m"
  autoDeployThresholdsMemoryPercent: "10"
  autoDeployThresholdsMemoryUnit: "10Mi"

Suppose you want to remove only the unit-change threshold but keep the percent-change threshold. Use following parameter:value pairs (you don’t need to include the percent-change annotations):

clusterDefaultConfig:
  autoDeploy: "true"
  autoDeployThresholdsCpuUnit: "0"
  autoDeployThresholdsMemoryUnit: "0"

To remove both thresholds, use the following annotations:

clusterDefaultConfig:
  autoDeploy: "true"
  autoDeployThresholdsCpuPercent: "0"
  autoDeployThresholdsMemoryPercent: "0"
  autoDeployThresholdsCpuUnit: "0"
  autoDeployThresholdsMemoryUnit: "0"