Configure by using annotations

Define default settings at the workload, namespace, and cluster levels by using annotations

Using resource annotations is the most efficient way to configure resources and control optimization behavior.

Annotations ensure consistency across workloads and give teams the control they need over their work - for example:

  • Platform teams can set organization-wide default optimization values
  • Development teams can fine-tune settings depending on their application and environment (development, staging, test, production)

Key points:

  • Think of settings at the cluster, namespace, and workload levels as a hierarchy of configuration.

    • Workload-level configuration customizes the optimization behavior for individual workloads. This level has the highest precedence and cannot be overridden. You can annotate these types: Deployments, StatefulSets, and ReplicaSets.
    • Namespace-level configuration sets default values for all workloads in a namespace. Default values set at this level fill in any settings not provided by annotations on the workloads themselves.
    • Cluster-level defaults apply to all workloads in a cluster. Default values set at this level have the lowest precedence and can be overridden by values set at the namespace and workload levels.
  • Best practice: Use annotations to configure workload settings. Use the UI only to view workload settings.

    A workload becomes read-only in the UI anyway when a workload is managed in any way by annotations. For example, if namespace-level defaults have been applied, you cannot use the UI to change settings for any workload in that namespace.

Topics:

Save time: Copy a template

Each tab contains a template that contains the annotations from the table above.

  • Copy the Workload template to set workload-specific defaults defaults
  • Copy the Namespace template to set namespace-level defaults
  • Copy the Cluster template to set cluster-level defaults using Helm

To configure optimization settings for a workload, copy annotations as needed from the template below into the metadata.annotations section of the workload resource.

  • The template shows annotations defined on a Deployment workload. The same pattern applies for all supported workload types.
  • Annotations are shown with sample values to demonstrate what an active annotation looks like, and may not be correct for your environment. You should review and set appropriate values for each annotation you use.
  • All annotations are optional, none are required. Copy and paste the specific annotations that you need, and remove the ones you don’t.

Go to the steps for using this template.

apiVersion: apps/v1 # just an example; represents any
kind: Deployment    # supported workload resource type
metadata:
  name: example-workload-name
  annotations:
    # ---------------------------------------------
    # WORKLOAD SETTINGS
    # Specify single values for the whole workload.
    # ---------------------------------------------
    live.stormforge.io/schedule: "@daily"
    live.stormforge.io/cpu.optimization-goal: "Balanced"
    live.stormforge.io/memory.optimization-goal: "Balanced"

    # Main configuration control for auto-deploy
    live.stormforge.io/auto-deploy: "false" # Set to "true" to enable auto-deploy

    # When auto-deploy is false, the following subsettings have no effect 
    live.stormforge.io/auto-deploy.thresholds.cpu.percent: "5"    # 1/2 Paired parameters; Set both or neither
    live.stormforge.io/auto-deploy.thresholds.memory.percent: "5" # 2/2. 
    live.stormforge.io/auto-deploy.thresholds.cpu.unit: "10m"     # 1/2 Paired parameters; set both or neither
    live.stormforge.io/auto-deploy.thresholds.memory.unit: "20Mi" # 2/2.

    # The following settings have an effect only if an HPA is enabled and the HPA scales on CPU or memory.
    live.stormforge.io/hpa.cpu.target-utilization.min: "20"   
    live.stormforge.io/hpa.cpu.target-utilization.max: "80"   
    live.stormforge.io/hpa.memory.target-utilization.min: "25"  
    live.stormforge.io/hpa.memory.target-utilization.max: "75"  

    # -------------------------------------------------
    # CONTAINER SETTINGS
    # Specify defaults and/or per-container exceptions.
    # E.g. "RequestsOnly,istio-proxy=DoNotOptimize"
    # -------------------------------------------------
    live.stormforge.io/containers.cpu.optimization-policy: "RequestsOnly"
    live.stormforge.io/containers.cpu.requests.min: "20m"
    live.stormforge.io/containers.cpu.requests.max: "16000m"
    live.stormforge.io/containers.cpu.limits.min: "2000m"
    live.stormforge.io/containers.cpu.limits.max: "16000m"
    live.stormforge.io/containers.cpu.limits.limit-request-ratio: "2.0"
    live.stormforge.io/containers.memory.optimization-policy: "RequestsOnly"
    live.stormforge.io/containers.memory.requests.min: "64Mi"
    live.stormforge.io/containers.memory.requests.max: "32Gi"
    live.stormforge.io/containers.memory.limits.min: "512Mi"
    live.stormforge.io/containers.memory.limits.max: "64Gi"
    live.stormforge.io/containers.memory.limits.limit-request-ratio: "2.0"

spec:
  template:
    metadata:

To configure default optimization settings for all workloads in a namespace, copy annotations as needed from the template below into the metadata.annotations section of a namespace.

  • Annotations are shown with sample values to demonstrate what an active annotation looks like, and may not be correct for your environment. You should review and set appropriate values for each annotation you use.
  • All annotations are optional, none are required. Copy and paste the specific annotations that you need, and remove the ones you don’t.

Go to the steps for using this template.

apiVersion: v1
kind: Namespace
metadata:
  name: example-namespace-name
  annotations:
    # ---------------------------------------------
    # WORKLOAD SETTINGS
    # Specify single values for the whole workload.
    # ---------------------------------------------
    live.stormforge.io/schedule: "@daily"
    live.stormforge.io/cpu.optimization-goal: "Balanced"
    live.stormforge.io/memory.optimization-goal: "Balanced"

    # Main configuration control for auto-deploy
    live.stormforge.io/auto-deploy: "false" # Set to "true" to enable auto-deploy

    # When auto-deploy is false, the following subsettings have no effect 
    live.stormforge.io/auto-deploy.thresholds.cpu.percent: "5"    # 1/2 Paired parameters; set both or neither.
    live.stormforge.io/auto-deploy.thresholds.memory.percent: "5" # 2/2. 
    live.stormforge.io/auto-deploy.thresholds.cpu.unit: "10m"     # 1/2 Paired parameters; set both or neither.
    live.stormforge.io/auto-deploy.thresholds.memory.unit: "20Mi" # 2/2.

    # The following settings have an effect only if an HPA is enabled and the HPA scales on CPU or memory.
    live.stormforge.io/hpa.cpu.target-utilization.min: "20"   
    live.stormforge.io/hpa.cpu.target-utilization.max: "80"   
    live.stormforge.io/hpa.memory.target-utilization.min: "25"  
    live.stormforge.io/hpa.memory.target-utilization.max: "75"  

    # -------------------------------------------------
    # CONTAINER SETTINGS
    # Specify defaults and/or per-container exceptions.
    # E.g. "RequestsOnly,istio-proxy=DoNotOptimize"
    # -------------------------------------------------
    live.stormforge.io/containers.cpu.optimization-policy: "RequestsOnly"
    live.stormforge.io/containers.cpu.requests.min: "20m"
    live.stormforge.io/containers.cpu.requests.max: "16000m"
    live.stormforge.io/containers.cpu.limits.min: "2000m"
    live.stormforge.io/containers.cpu.limits.max: "16000m"
    live.stormforge.io/containers.cpu.limits.limit-request-ratio: "2.0"
    live.stormforge.io/containers.memory.optimization-policy: "RequestsOnly"
    live.stormforge.io/containers.memory.requests.min: "64Mi"
    live.stormforge.io/containers.memory.requests.max: "32Gi"
    live.stormforge.io/containers.memory.limits.min: "512Mi"
    live.stormforge.io/containers.memory.limits.max: "64Gi"
    live.stormforge.io/containers.memory.limits.limit-request-ratio: "2.0"

To configure default optimization settings for all workloads in a cluster, copy as needed from the Helm values template below into your Helm values file, then pass the file to helm install or helm upgrade using the -f flag.

  • Helm values are shown with sample settings that demonstrate what using the value looks like. The sample settings may not be correct for your environment. You should review and choose an appropriate setting for each Helm value you specify.
  • All Helm values shown are optional, none are required. Copy and paste the specific values that you need, and remove the ones you don’t.

Go to the steps for using this template.

---
# Helm values.yaml file

clusterDefaultConfig:
  # All values below are optional; copy/paste only those that you need.
  # ---------------------------------------------
  # WORKLOAD SETTINGS
  # Specify single values for the whole workload.
  # ---------------------------------------------
  schedule: "P1D"
  cpuOptimizationGoal: "Balanced"
  memoryOptimizationGoal: "Balanced"

  # Main configuration control for auto-deploy
  autoDeploy: "false" # Set to "true" to enable auto-deploy

  # When auto-deploy is false, the following subsettings have no effect
  autoDeployThresholdsCpuPercent: "5"     # 1/2 Paired parameters; Set both or neither
  autoDeployThresholdsMemoryPercent: "5"  # 2/2. 
  autoDeployThresholdsCpuUnit: "10m"      # 1/2 Paired parameters; Set both or neither
  autoDeployThresholdsMemoryUnit: "20Mi"  # 2/2.

  # The following settings have an effect only if an HPA is enabled and the HPA scales on CPU or memory.
  hpaCpuTargetUtilizationMin: "20"   
  hpaCpuTargetUtilizationMax: "80"   
  hpaMemoryTargetUtilizationMin: "25"  
  hpaMemoryTargetUtilizationMax: "75" 

  # ---------------------------------------------
  # CONTAINER SETTINGS
  # Specify defaults and/or per-container values.
  # E.g. "RequestsOnly,istio-proxy=DoNotOptimize"
  # ---------------------------------------------
  containersCpuOptimizationPolicy: "RequestsOnly"
  containersCpuRequestsMin: "20m"
  containersCpuRequestsMax: "16000m"
  containersCpuLimitsMin: "2000m"
  containersCpuLimitsMax: "16000m"
  containersCpuLimitsLimitRequestRatio: "2.0"
  containersMemoryOptimizationPolicy: "RequestsOnly"
  containersMemoryRequestsMin: "64Mi"
  containersMemoryRequestsMax: "32Gi"
  containersMemoryLimitsMin: "512Mi"
  containersMemoryLimitsMax: "64Gi"
  containersMemoryLimitsLimitRequestRatio: "2.0"

When editing annotations:

  • Enclose all values in double quotation marks ("") as shown in the templates.
  • Best practice: When editing container-specific resources (items in the Container-specific settings section in a template, such as the container’s optimization policy), specify the value to apply to all or most containers first, followed by container-specific exceptions, as in these examples:
    • When annotating a workload or namespace:
      live.stormforge.io/containers.cpu.optimization-policy: "RequestsAndLimits,sidecar=DoNotOptimize"
    • In a cluster-default ConfigMap:
      containersCpuOptimizationPolicy: "RequestsAndLimits,sidecar=DoNotOptimize"

Configure optimization settings for a specific workload

Workload-level defaults are defined by annotations in the metadata.annotations section of a Deployment or StatefulSet resource type.

Using YAML manifests

  1. Open the workload definition in your favorite editor. This is commonly done using commands like these (remember to replace DEPLOYMENT and FILENAME with your own values):

    kubectl get deployment DEPLOYMENT -o yaml > FILENAME && $EDITOR FILENAME
    

    or:

    kubectl edit deployment DEPLOYMENT
    
  2. Edit the metadata.annotations section to include the annotations you need.

    • To save time, you can copy the complete list of annotations from the sample Deployment YAML excerpt on the Workload tab in the Copy a template section above.
  3. Save your changes.

  4. If you chose to run the kubectl get deployment DEPLOYMENT -o yaml > FILENAME ... command in step 1, apply your changes.

    kubectl apply -f FILENAME
    
  5. Optional: Confirm your changes by running the following command and then reviewing the annotations in the metadata.annotations section of the output:

    kubectl get deployment DEPLOYMENT -o yaml
    

Configure default optimization settings for all workloads in a namespace

To configure settings for all workloads in a namesapce, edit the Namespace definition or run kubectl annotate.

Namespace-level defaults:

  • Override equivalent cluster-level defaults
  • Fill in any settings not provided by annotations on the workloads themselves

Using YAML manifests

  1. Open the namespace definition in your favorite editor. This is commonly done using commands like these (remember to replace NAMESPACE and FILENAME with your own values):

    kubectl get namespace NAMESPACE -o yaml > FILENAME && $EDITOR FILENAME
    

    or:

    kubectl edit namespace NAMESPACE
    
  2. Edit the metadata.annotations section to include the annotations you need.

    • To save time, you can copy the complete list of annotations from the sample Namespace YAML excerpt on the Namespace tab in the Copy a template section above.
  3. Save your changes.

  4. If you chose to run the kubectl get namespace NAMESPACE -o yaml > FILENAME ... command in step 1, apply your changes.

    kubectl apply -f FILENAME
    
  5. Optional: Confirm your changes by running the following command and then reviewing the annotations in the metadata.annotations section of the output:

    kubectl get namespace NAMESPACE -o yaml
    

Using the kubectl annotate command

  1. Run the kubectl annotate command and include the annotation:value pairs you want to set.

    • To save time, you can copy the pairs you need from the sample Namespace YAML excerpt on the Namespace tab in the Copy a template section above. Be sure to replace the : separator from the template with = as shown in the sample command below.

    As an example, a kubectl annotate might look something like this (remember to replace NAMESPACE with your appropriate value):

    kubectl annotate namespace NAMESPACE \
      live.stormforge.io/schedule="P1D" \
      live.stormforge.io/auto-deploy="false" \
      live.stormforge.io/containers.cpu.requests.min="20m" \
      live.stormforge.io/containers.cpu.requests.max="16000m"
    
  2. Restart the StormForge Agent:

    kubectl rollout restart -n stormforge-system deployment stormforge-agent
    
  3. Optional: To confirm your changes, run the following command and review the annotations in the .metadata.annotations section of the output:

    kubectl get namespace NAMESPACE -o yaml
    

Configure default optimization settings for all workloads in a cluster

Cluster-level defaults:

In the cluster-defaults.yamlfile, use annotation-name:value syntax as shown below:

apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-defaults
  namespace: stormforge-system
data:
  cluster-defaults.yaml: |
    live.stormforge.io/schedule: "P1D"
    live.stormforge.io/auto-deploy: "false"
    live.stormforge.io/containers.cpu.requests.min: "20m"
    live.stormforge.io/containers.cpu.requests.max: "16000m"
    ...    

You can create this ConfigMap yourself, but it may be easier to have Helm create and manage it for you.

Setting cluster default values using Helm

The stormforge-agent Helm chart has values you can pass to define each of the possible cluster-level default settings. Helm then creates and manages the cluster-defaults ConfigMap for you.

Preferred method: Helm chart values for cluster default settings

  1. Optional: Check whether any cluster defaults are already set in the cluster-defaults ConfigMap by running:

    kubectl describe configmap cluster-defaults -n stormforge-system
    
    • If you see an Error from server (NotFound): configmaps "cluster-defaults" not found message, no cluster-level defaults are set. Proceed to the next step.
    • Otherwise, in the Data > cluster-defaults.yaml section of the output, notice which annotations are set. If you need to preserve those values, include them in the cluster-defaults.yaml file that you create in the next step.
  2. Create a .yaml values file (for example, cluster-defaults.yaml) that defines the clusterDefaultConfig values you need.

    • To save time, you can copy the sample on the Cluster tab in the Copy a template section above. Edit or comment out annotations as needed.

      As an example, your cluster-defaults.yaml file might look something like this when you’re done:

    clusterDefaultConfig:
      schedule: "PT16H"
      autoDeploy: "true"
      cpuOptimizationGoal: "savings"
      memoryOptimizationGoal: "savings"
      containersCpuOptimizationPolicy: "RequestsAndLimits"
      containersCpuRequestsMin: "20m"
      containersCpuRequestsMax: "16000m"
      containersMemoryOptimizationPolicy: "RequestsAndLimits"
      containersMemoryRequestsMin: "64Mi"
      containersMemoryRequestsMax: "32Gi"
    
  3. Run helm install or helm upgrade and include --values flag and your filename. For example (remember to replace CLUSTER_NAME and CREDENTIALS_FILE with your own values):

    helm upgrade stormforge-agent oci://registry.stormforge.io/library/stormforge-agent \
      --namespace stormforge-system \
      --set clusterName=CLUSTER_NAME \
      --values CREDENTIALS_FILE.yaml \
      --values cluster-defaults.yaml
    

    Tip: If you’re setting only a few default values, you pass them as --set arguments, as shown below. Be sure to replace the : separator from the template with = as shown in the sample command below.

    helm upgrade stormforge-agent oci://registry.stormforge.io/library/stormforge-agent \
      --namespace stormforge-system \
      --set clusterName=CLUSTER_NAME \
      --values CREDENTIALS_FILE.yaml \
      --set clusterDefaultConfig.autoDeploy=true \
      --set clusterDefaultConfig.cpuOptimizationGoal=balanced
    

    Any previous cluster-level default values are now removed, and your new defaults are in effect, unless they are overridden at the cluster or workload level.

  4. Optional: Confirm your changes by running the following command and then reviewng the annotations in the Data section of the output:

    kubectl describe configmap cluster-defaults -n stormforge-system 
    

Setting cluster default values using kubectl edit

If at least one cluster-level default has been set already, you can edit the ConfigMap directly.

  1. Run the folllowing command:

    kubectl edit configmap cluster-defaults -n stormforge-system
    

    If you see a Error from server (NotFound): configmaps "cluster-defaults" not found message, no cluster-level defaults are set. Use the Helm method described in the preceding section instead of continuing to the next step.

  2. In the .data.cluster-defaults.yaml section, add, edit, or remove annotations as needed.

    • Using kubectl edit requires you to use the live.stormforge.io/* annotation syntax, not the clusterDefaults Helm chart values. To save time, you can copy the annotations from the sample on the Namespace tab (not the Cluster tab) in the Copy a template section above and edit them as needed.

      As an example, your annotations in the cluster-defaults ConfigMap might look something like this excerpt when you’re done:

    # Please edit the object below. Lines beginning with a '#' will be ignored,
    # and an empty file will abort the edit. If an error occurs while saving this file will be
    # reopened with the relevant failures.
    #
    apiVersion: v1
    data:
      cluster-defaults.yaml: |
        live.stormforge.io/auto-deploy: "true"
        live.stormforge.io/cpu.optimization-goal: "Reliability"
        live.stormforge.io/memory.optimization-goal: "Reliability"    
    kind: ConfigMap
    metadata:
    ...
    
  3. Save your changes.

  4. Restart the StormForge Agent:

    kubectl rollout restart -n stormforge-system deployment stormforge-agent
    

    Any previous cluster-level default values are now removed, and your new defaults are in effect, unless they are overridden at the cluster or workload level.

  5. Optional: Confirm your changes by running the following command and then reviewing the annotations in the Data section of the output:

    kubectl describe configmap cluster-defaults -n stormforge-system 
    

Schedule examples

Below are some additional examples of ways to set the live.stormforge.io/schedule (or schedule in a cluster-level ConfigMap) annotation using macros, ISO 8601 Duration, and cron notation:

  • Once daily (default value and best practice):
    "@daily", "P1D" or "H H * * *"
  • Hourly (the shortest interval you can specify):
    "@hourly", "PT1H" or "H * * * *"
  • Every 12 hours:
    "PT12H" or "H H/12 * * *"
  • Every morning at approximately 0800h (exact time is not guaranteed):
    "00 08 * * *"
  • Every Wednesday:
    "H H * * WED"

Threshold examples

Thresholds define the minimum amount of change to requests values that make with worthwhile deploying new recommendations.

You can set a percent-change threshold, a unit-change threshold, or both.

Example: Set a percent-change threshold for either (or both) 5% CPU or 10% memory (using YAML manifests)
Suppose you want to apply only recommendations that change at least one of the following for at least one container in a workload:

  • CPU requests by at least 5%
  • Memory requests by at least 10%

For this example, assume you’re annotating a .yaml file. Where you add the following annotations depends on whether you’re annotating at the workload, namespace, or cluster level (as described earlier this topic):

live.stormforge.io/auto-deploy: "true"
live.stormforge.io/auto-deploy.thresholds.cpu.percent: "5"
live.stormforge.io/auto-deploy.thresholds.memory.percent: "10"

Suppose you want to remove the threshold. Use the following annotations:

live.stormforge.io/auto-deploy: "true"
live.stormforge.io/auto-deploy.thresholds.cpu.percent: "0"
live.stormforge.io/auto-deploy.thresholds.memory.percent: "0"

Example: Set percent-change and unit-change thresholds (using YAML manifests)
Suppose you want to apply only recommendations that satisfy at least one of the following conditions for at least one container in a workload:

  • CPU requests change by at least 5% and 10m
  • Memory requests change by at least 10% and 10Mi

For this example, assume you’re annotating a YAML manifest. Where you add the following annotations depends on whether you’re annotating at the workload, namespace, or cluster level (as described earlier this topic):

live.stormforge.io/auto-deploy: "true"
live.stormforge.io/auto-deploy.thresholds.cpu.percent: "5"
live.stormforge.io/auto-deploy.thresholds.cpu.unit: "10m"
live.stormforge.io/auto-deploy.thresholds.memory.percent: "10"
live.stormforge.io/auto-deploy.thresholds.memory.unit: "10Mi"

To remove the threshold, use the following annotations:

live.stormforge.io/auto-deploy: "true"
live.stormforge.io/auto-deploy.thresholds.cpu.percent: "0"
live.stormforge.io/auto-deploy.thresholds.memory.percent: "0"

Example: Set a unit-change threshold for 10m of CPU (using kubectl annotate)
Suppose you want to apply only recommendations that change CPU requests by at least 10m. Suppose also that memory requests don’t matter to you — you must still set a memory value greater than 0.

For this example, assume you’re using kubectl annotate as described earlier in this topic to annotate all workloads in a namespace. Your command will look something like this (replace NAMESPACE with the namespace name):

kubectl annotate namespace NAMESPACE \
  live.stormforge.io/auto-deploy="true" \
  live.stormforge.io/auto-deploy.thresholds.cpu.unit="10m" \
  live.stormforge.io/auto-deploy.thresholds.memory.unit="1Ki"

To remove the threshold, run this command (remember to replace NAMESPACE with an actual name):

kubectl annotate namespace NAMESPACE \
  live.stormforge.io/auto-deploy="true" \
  live.stormforge.io/auto-deploy.thresholds.cpu.unit="0" \
  live.stormforge.io/auto-deploy.thresholds.memory.unit="0"

Example: Set unit- and percent-change thresholds at the cluster level (using cluster default values)
Suppose you want to apply only recommendations that satisfy at least one of the following conditions for at least one container in a workload:

  • CPU requests change by at least 5% and 10m
  • Memory requests change by at least 10% and 10Mi

For this example, assume you’re using a cluster-defaults.yaml file to set Helm chart values via helm install or helm upgrade as described earlier in this topic to annotate all workloads in a cluster. At a minimum, this file would contain the following parameter:value pairs:

clusterDefaultConfig:
  autoDeploy: "true"
  autoDeployThresholdsCpuPercent: "5"
  autoDeployThresholdsCpuUnit: "10m"
  autoDeployThresholdsMemoryPercent: "10"
  autoDeployThresholdsMemoryUnit: "10Mi"

Suppose you want to remove only the unit-change threshold but keep the percent-change threshold. Use following parameter:value pairs (you don’t need to include the percent-change annotations):

clusterDefaultConfig:
  autoDeploy: "true"
  autoDeployThresholdsCpuUnit: "0"
  autoDeployThresholdsMemoryUnit: "0"

To remove both thresholds, use the following annotations:

clusterDefaultConfig:
  autoDeploy: "true"
  autoDeployThresholdsCpuPercent: "0"
  autoDeployThresholdsMemoryPercent: "0"
  autoDeployThresholdsCpuUnit: "0"
  autoDeployThresholdsMemoryUnit: "0"
Last modified February 1, 2024