StormForge on StormForge Documentation

Version 2.28.1-agent

Thu, 16 Apr 2026 20:58:48 +0000

Changed

Add RBAC for OpenShift operator-owned workloads when openshift=true.
Update prometheus container version to 3.11.2
Dependency Updates:
- kubernetes library group to 0.35.4.
- prometheus/prometheus to 0.311.2.
- gramLabs/go-application to 0.0.115.
- gramLabs/stormforge-go to 4.9.1.

Version 2.13.0-applier

Mon, 13 Apr 2026 15:29:50 +0000

Added

Adds Drift Reconciliation for Horizontal Pod Autoscalers
Detects when StormForge Recommendations are rejected because of Cluster settings and triggers rollback faster.
- Failure Conditions detected: ResourceQuota violations, LimitRange validation, and Core Pod validation

Changed

Dependency Updates:
- kubernetes libraries to 0.35.3
- argoproj/argo-rollouts to 1.9.0
- rs/zerolog to 1.35.0
- gramLabs/stormforge-go to 4.9.0
- gramLabs/go-application to 0.0.114
- gramLabs/structured-logging to 0.0.7
- gramLabs/stormforge-agent/v2 to 2.28.0.

Fixed

Fixed an issue where Drift Reconciliation for Workloads was not retrieving the last applied recommendation sometimes.

Security

Updated Go to 1.26.2.

Version 2.28.0-agent

Thu, 09 Apr 2026 21:55:38 +0000

Added

Adds support for In-API Cluster Config
Adds user-override configuration for clusterProvider
- To configure custom provider string, add the following to helm values:

clusterProvider: exampleProvider

Adds support for custom cluster labels
- To configure cluster labels, add the following to helm values:

clusterLabels:
  key1: value1
  key2: value2

Changed

Updated Dependencies:
- go-jose/go-jose/v4 to 4.1.4.
- prometheus/prometheus to 0.311.1.
- aws/aws-sdk-go-v2/service/cloudwatchlogs to 1.65.0.
- aws/aws-sdk-go-v2/service/lambda to 1.88.5.
- aws/aws-sdk-go-v2/service/s3 to 1.97.3.
- google/cel-go to 0.28.0.
- gramLabs/go-application to 0.0.114.

Security

Updated Go to 1.26.2.

Version 2.27.4-agent

Wed, 01 Apr 2026 20:20:08 +0000

Added

Adds new metrics:
- container_cpu_cfs_throttled_periods_total
- container_cpu_cfs_periods_total
Adds ability to configure Auth credentials secret and env in Helm.

Changed

Strictly check file-based cluster-default config on startup and fail if invalid.
Dependency Updates:
- golang.org/x/oauth2 to 0.36.0
- k8s.io/klog/v2 to 2.140.0
- gruntwork-io/terratest to 0.56.0
- google.golang.org/grpc to 1.79.3
- argoproj/argo-rollouts to 1.9.0
- gramLabs/go-application to 0.0.113
- gramLabs/stormforge-go/v4 to 4.9.0
- gramLabs/structured-logging to 0.0.7

Version 2.12.2-applier

Fri, 13 Mar 2026 19:10:25 +0000

Changed

Dependency Updates:
- gramLabs/go-application to 0.0.112.

Fixed

Fixed a bug that causes Applier to panic when applying pod resize to ArgoRollouts.
Allows webhook cert-controller to be disabled.

Security

Updated Go to 1.26.1.
golang.org/x/oauth2 to 0.36.0.

Version 2.12.1-applier

Thu, 05 Mar 2026 20:56:07 +0000

Changed

Improved RBAC scope of the applier
Improve Webhook TLS and certificate management
Dependency Updates:
- kubernetes libs to 0.35.2.
- stormforge-agent libs to 2.27.3

Version 2.27.3-agent

Thu, 05 Mar 2026 19:56:03 +0000

Changed

Enables Optimization CRDs by default.
Updated default prometheus container image to 3.10.0.
Dependency Updates:
- filippo.io/edwards25519 to 1.1.1.
- prometheus/prometheus to 0.310.0.
- kubernetes libs to 0.35.2.

Fixed

Fixed a bug where the Agent could send duplicate StatusChanges for a workload, causing an error.
Fixed several Helm issues

Version 2.12.0-applier

Thu, 19 Feb 2026 19:18:59 +0000

Added

In-Place-Pod-Resize Support
- To enable, set the workload’s apply.method setting to WebhookImmediateRollout or WebhookHybridRollout.
  - WebhookImmediateRollout - Use a mutating webhook to modify Pods at admission. Existing pods will be resized in place when a new recommendation is applied. If Pods cannot be resized in place, a rollout/restart will be triggered.
  - WebhookHybridRollout - Use a mutating webhook to modify Pods at admission. Existing pods will be resized in place when a new recommendation is applied. If Pods cannot be resized in place, changes will be deferred until the next rollout.
HPA Reconciliation - Applier will watch, detect, and reconcile any drift in HPA TargetUtilization setting.

Version 2.27.2-agent

Thu, 19 Feb 2026 17:35:58 +0000

Fixed

Fixed a bug where the Agent would lose the pod-workload association if the only pod still in running state had an older generation.

Changed

Dependency Updates:
- golang.org/x/oauth2 to 0.35.0.
- github.com/google/cel-go to 0.27.0.
- argoproj/argo-rollouts to 1.8.4.
- gramLabs/stormforge-go/v4 to 4.8.4.
- kubernetes libraries to 0.35.1.

Security

Updated to Go 1.25.7.

Version 2.27.1-agent

Fri, 23 Jan 2026 01:26:50 +0000

Added

Added ClusterOptimizationConfiguration and OptimizationConfiguration Custom Resource Definitions (CRDs).
This release makes these CR types available, but they are not yet enabled by default. Use the Helm parameter enableOptimizationConfigurationCRDs=true to opt in to using them.

Changed

Updated Dependencies:
- gruntwork-io/terratest to 0.55.0.

Security

Updated Go to 1.25.6

Version 2.26.0-agent

Wed, 14 Jan 2026 19:35:29 +0000

Added

Helm parameter workloadResourceTypes: it defines CRD workload types for which support is not built-in to StormForge. It has subfield patchTargetTypeDefaults to define patchpaths.
Helm parameter ownerResourceTypes: it defines configuration for types which may own workloads. This parameter can be used to define default patch-paths for each owner type, to define owner types to ignore (treat as if they do not own workloads), or types at which to stop owner reference traversal. The subfields are stopTraversal, ignoreOwner and patchTargetTypeDefaults

Version 2.11.0-applier

Fri, 09 Jan 2026 22:24:31 +0000

Changed

Attention: mutation webhook is now enabled by default with webhook.enabled=true part of the helm values file. If one does not want to have the mutation webhook feature enabled, one needs to specify webhook.enabled=false
Expose applier metrics via port 8080 as part of the webhook service
Bumped K8s libraries to 1.35

Added

Helm parameter to support network policies: networkPolicies.create=true and optionally for egress network policy networkPolicies.additionalEgressRules with sample example on file

Security

Addressed CVE-2025-61729

Version 2.25.1-agent

Fri, 09 Jan 2026 22:11:05 +0000

Added

Sample egress network policy on helm values file

Changed

Bumped prometheus common library to latest version
Bumped prometheus to v3.9.1

Fixed

A config parse condition when specifying clusterDefaultsAutoDeployExemptNamespaces that cause agent not to start
in rare occasions, fixed missing “selector” field on HPA external metrics.

Security

Addressed CVE-2025-61729

Version 2.10.1-applier

Fri, 19 Dec 2025 04:00:45 +0000

Security

Addressed CVE-2025-68156

Version 2.25.0-agent

Fri, 19 Dec 2025 02:05:49 +0000

Added

Helm Parameter clusterDefaultsAutoDeployExemptNamespaces
This new Helm parameter defines a list of namespaces for which the auto-deploy setting defined in the cluster-defaults ConfigMap will be ignored.

This parameter has a default value of ["^kube-system$", "^openshift.*"]. Workloads in namespaces matching these regular expressions will not be subject to the auto-deploy setting defined in cluster-defaults. To enable auto-deploy on workloads in affected namespaces, you must use a namespace annotation, or a workload annotation.

Version 2.24.4-agent

Mon, 15 Dec 2025 20:11:41 +0000

Fixed

Fixed a Helm template ConfigMap rendering issue when using patchTargetTypes

Version 2.10.0-applier

Thu, 11 Dec 2025 21:26:14 +0000

Added

Added a Parameter manage Recommendation Ready Timeout. Defaults to 1H. This configurable parameter prevents stale recommendations from being applied at Applier start.

Changed

Dropped the de-dup logic to handle recommendation items, relying on state
Setting a recommendation item to FailedtoApply when the recommendation does not have a patch list
Dependency Updates:
- kubernetes libs to 0.34.3.
- sigs.k8s.io/controller-runtime to 0.22.4.
- golang.org/x/oauth2 to 0.34.0.
- open-policy-agent/cert-controller to 0.15.0.
- prometheus/common to 0.67.1.
- spf13/cobra to 1.10.2.
- gramLabs/go-application to 0.0.108.
- gramLabs/stormforge-agent to 2.24.3.

Fixed

Fixed scenario that applier needed to be restarted to reapply recommendations that are set to be “Ready” via sf apply. The solution was changing the de-dup logic of the applier.

Version 2.24.3-agent

Thu, 11 Dec 2025 20:47:23 +0000

Added

Added a helm parameter to dictate the setup the network policies networkPolicies.create, default is false. Upon creation, the network policies are now functional (before the selectors were misconfigured). There is no egress policies being created by default (for such, please see the next bullet)
Added helm parameters for extra network policies networkPolicies.additionalIngressRules and networkPolicies.additionalEgressRules.

Security

golang oauth2 0.34.0
golang crypto 0.46.0

Changed

KEDA 2.18.2

Version 2.24.2-agent

Fri, 05 Dec 2025 01:55:20 +0000

Added

Added collection of network metrics if enableCostMetrics=true.
- New metric name: container_network_transmit_bytes_total.

Changed

Updated prometheus container version to 3.8.0.
Dependency Updates:
- prometheus/prometheus to 0.308.0.
- spf13/cobra to 1.10.2.

Version 2.24.1-agent

Tue, 25 Nov 2025 23:15:39 +0000

Changed

When setting enableCostMetrics to true (default is false), the user must now unset workload.allowNamespaces and workload.denyNamespaces. One can add the following lines on values.yaml:

  workload:
    allowNamespaces: null
    denyNamespaces: null

Security

prometheus to version 2.7.3 to mitigate CVE-2025-54388

Version 2.24.0-agent

Thu, 20 Nov 2025 20:57:08 +0000

Added

Adds support to define custom resource types to be recognized as StormForge workloads.
Adds support to configure PatchPaths to optimize custom workload types and operators.

Changed

Dependency Updates:
- gramLabs/go-application to 0.0.107.
- sigs.k8s.io/controller-runtime to 0.22.4.
- prometheus/common to 0.67.3.
- k8s libraries to 0.34.2.

Fixed

Fixed an issue where patchable: false status condition was being applied incorrectly.

Security

golang.org/x/oauth2 to 0.33.0.
golang.org/x/crypto to 0.45.0.

Version 2.23.6-agent

Thu, 30 Oct 2025 17:02:58 +0000

Added

[EXPERIMENTAL] One time specification of patch-paths for Custom Resource Workload Types.
Added relabeling rules to Prometheus config to normalize opentelemetry JVM metrics to match client_java form.

Changed

Dependency Updates:
- gramLabs/go-application to 0.0.106.
- gramLabs/stormforge-go to 4.8.0.
- golang.org/x/oauth2 to 0.32.0.
- sigs.k8s.io/controller-runtime to 0.22.3.
- prometheus/prometheus to 0.307.3.
- prometheus/common to 0.67.2
- kedacore/keda to 2.18.1.

Fixed

Fixed an issue where deleting a WorkloadOptimizer CR does not update the ManagedBy field for a Workload.

Version 2.23.5-agent

Mon, 29 Sep 2025 18:31:09 +0000

Changed

Updated Dependencies
- prometheus/prometheus to 0.306.0.
- gramlabs/stormforge-go to 4.7.11.
- Updated prometheus container in Helm to 3.6.0.

Removed

Removed deprecated Secret helm template.
Removed deprecated autoDiscovery parameter.
Removed support for autoscaling/v2beta2 version of HPA (no longer supported by Kubernetes).
Removed unused startupProbe license check from the stormforge pod templates.

Fixed

Removed the aggregate view in the ClusterRole template

Security

Go to 1.25.1 to address CVE-2025-47910

Version 2.23.4-agent

Wed, 10 Sep 2025 17:44:30 +0000

Added

[EXPERIMENTAL] Added helm chart parameter enableCostMetrics to collect Cost Reporting metrics. Defaults to false.

Changed

Update golang to 1.24.1.
Dependencies update:
- spf13/cobra to 1.10.1.
- spf13/pflag to 1.0.10.
- stretchr/testify to 1.11.1.
- x/oauth2 to 0.31.0.
- sigs.k8s.io/controller-runtime to 0.22.1.
- prometheus/client_golang to 1.23.2.
- gramlabs/go-application to 0.0.105.
- kubernetes libraries to 0.34.1.

Version 2.9.2-applier

Fri, 29 Aug 2025 17:06:40 +0000

Fixed

Fixed an issue where new containers could be mistakenly added when using DynamicAdmissionWebhook apply method.

Version 2.23.3-agent

Fri, 22 Aug 2025 17:40:45 +0000

Fixed

Fixed a bug that resulted in the workload sync did not properly sync to the API.

Version 2.9.1-applier

Tue, 19 Aug 2025 19:19:00 +0000

Fixed

Fixed an Argo Rollout Patch Validation failure (to mitigate Argo Rollout type convention violation)

Changed

Dependency Updates:
- gramlabs/stormforge-agent to 2.23.1

Security

Update Go to 1.24.6 to address GO-2025-3849

Version 2.23.2-agent

Tue, 19 Aug 2025 19:18:26 +0000

Security

Update Go to 1.24.6 to address GO-2025-3849

Version 2.23.1-agent

Thu, 14 Aug 2025 21:52:56 +0000

Changed

Updated Prometheus Container version to 3.5.0
Dependency Updates
- prometheus/prometheus to 0.305.0.
- spf13/pflag to 1.0.7.
- kubernetes libs to 0.33.4.
- gramLabs/stormforge-go to 4.7.10.
- sigs.k8s.io/yaml to 1.6.0.
- prometheus/client_golang to 1.23.0.

Fixed

Fixes an issue where a workload will be marked as patchable despite not having RBAC permissions.
Fixes an issue where an unsupported workload or a workload without RBAC will cause a panic in the workload reconciler.

Version 2.9.0-applier

Mon, 14 Jul 2025 19:03:12 +0000

Added

[Beta] Support for InPlace Pod Resizing on annotation live.stormforge.io/apply.method=InPlacePodResizing
Added additional RBAC on Helm to support InPlace Pod Resizing

Changed

Dependency Updates:
controller-runtime 0.21.0
e2e-framework to 0.6.0
stormforge-go v4.7.9
stormforge-agent v2.23.0
google gnostic model v0.7.0

Security

Go to 1.24.5

Version 2.23.0-agent

Mon, 14 Jul 2025 18:30:20 +0000

Added

Adds support for sending metrics and configuration to optimize init-container sidecars.
Adds default requests values for Agent-installed deployments as a starting point. Users should set higher values as needed.

Changed

Dependencies
- prometheus/prometheus to 0.304.2
- argoproj/argo-rollouts to 1.8.3.
- gramLabs/go-application to 0.0.102.
- kedacore/keda to `2.17.1-0.20250521173204-e23ef462abb5.
- sigs.k8s.io/controller-runtime to 0.21.0.
- gramLabs/go-application to 0.0.102.
- gramLabs/stormforge-go to 4.7.9.

Fixed

Fixes an issue where config for patchPath annotations did not support Ephemeral Storage.
- Patch Paths can now be defined in quoted comma-delimited strings such as:
  - live.stormforge.io/containers.cpu.limits.patch-path: "/path1,/additional/path"
Fixes a panic when unable to retrieve a pod from the Spark.

Security

Go to 1.24.5.

Agent version 2.22.5

Wed, 02 Jul 2025 19:30:00 +0000

Fixed

Fixed a WorkloadOptimizer resource issue that was preventing configuration from being applied correctly.

Version 2.8.3-applier

Thu, 26 Jun 2025 19:57:07 +0000

Added

Add Helm parameters to set commonMetaLabels for StormForge deployed resources.
- Example values.yaml configuration to set commonMetaLabels:
```
commonMetaLabels:
  example.com/app: stormforge
```

Changed

Dependency Updates:
- kubernetes libraries to 0.33.2.
- argorproj/argo-rollouts to 1.8.3.
- sigs.k8s.io/yaml to 1.5.0.

Agent version 2.22.4

Mon, 23 Jun 2025 13:00:00 +0000

Added

Added helm parameter to specify securityContext for StormForge deployed containers.
- Example values.yaml configuration to set securityContext for containers in the metrics-forwarder and workload-controller pods.
```
commonContainerSecurityContext:
  allowPrivilegeEscalation: false
  capabilities:
    drop:
      - ALL
  readOnlyRootFilesystem: true
  seccompProfile:
    type: RuntimeDefault
```

Version 2.8.2-applier

Wed, 04 Jun 2025 20:20:02 +0000

Added

OLV2-2431 - Adding default requests on helm chart to set workload with Burstable QoS

Fixed

OLV2-2426 - Applier in continuous reconciliation mode skips certain workloads

Applier version 2.8.1

Thu, 29 May 2025 19:30:00 +0000

Changed

Dependencies
- gramlabs/stormforge-go to 4.7.7
- gramlabs/go-application to 0.0.101
- gramlabs/stormforge-agent to 2.22.3

Security

golang to 1.24.3 to address GO-2025-3563

Agent version 2.22.3

Thu, 29 May 2025 18:00:00 +0000

Changed

Dependencies
- go-logr/logr to 1.4.3

Security

Upgraded Prometheus container dependency to 3.4.0 to address CVE-2025-22871.

Agent version 2.22.2

Wed, 28 May 2025 15:30:00 +0000

Added

Allows specifying multiple paths for patchPath setting:
- Example: live.stormforge.io/containers.cpu.requests.patch-path: /path/to/target,containerA="/new/path,/alternative/path"
Version and build SHA on log during initialization

Changed

A new way to run controllers: a generic workload controller instead of multiple controllers used to support replica 0 workloads. Expected faster warm up time and more data integrity on annotations on replica 0 workloads.
Dependencies
- keda/v2 to 2.17.1
- prometheus/prometheus to 0.304.0
- prometheus/common to 0.64.0
- kubernetes libraries to 0.33.1
- gramLabs/go-application to 0.0.100
- gramLabs/stormforge-go to 4.7.7

Fixed

Potential problem of missing memory metrics when ephemeral_storage is set.

Security

x/oauth2 to 0.30.0
golang to 1.24.3 to address GO-2025-3563

Applier version 2.8.0

Wed, 30 Apr 2025 09:30:00 +0000

Added

[EXPERIMENTAL] Configuration for Alerts for Recommendation Apply Events

Fixed

An issue where applying a patch to an unhealthy workload continued to rollback the patch.

Changed

Dependency updates:
- gramLabs/stormforge-agent to 2.22.1
- kubernetes/api to 0.33.0
- Masterminds/sprig/v3 to 3.3.0

Agent version 2.22.1

Mon, 21 Apr 2025 09:30:00 +0000

Added

Helm parameters to support auth configuration for JVM Scrape jobs

Fixed

We fixed an issue where the Agent would fail if API Resource discovery partially fails.
We fixed an issue with potential deadlock in the Node Cache.

Applier version 2.7.2

Mon, 21 Apr 2025 09:30:00 +0000

Added

Add a flag enableFallbackToClientSideApply to work around an issue when improper configuration on a patch target prevents us from being able to apply a recommendation using Server-Side Apply. Setting this flag to true will result in handling errors by falling back to Client-Side Apply, which will disable automatic rollback in the event of healthcheck failures.

Changed

Dependency updates:
- x/oauth2 to 0.29.0
- golang/jwt/v4 to 4.5.2
- rs/zerolog to 1.34.0
- structured-merge-diff/v4 to 4.7.0
- argoproj/argo-rollouts to 1.8.2
- prometheus/client_golang to 1.22.0
- controller-runtime to 0.20.4
- gramLabs/stormforge-go/v4 to 4.7.6
- gramLabs/stormforge-agent/v2 to 2.22.0

Agent version 2.22.0

Wed, 02 Apr 2025 19:30:00 +0000

Changed

Internal improvements
Engineering work to support future feature additions

Added

Resource Request Mirroring
Enable recommendation mirroring, to support DR use cases or workloads owned by progressive delivery operators such as Flagger. Mirroring works by creating recommendations for a mirror workload based on usage metric observations of another. To configure mirroring, set the metrics-from annotation on the mirror workload.

Agent version 2.21.2

Mon, 31 Mar 2025 14:30:00 +0000

Fixed

Fixes an issue with hpa and pod reconciles could conflict and result in incomplete workload config.

Changed

Updated Dependencies
- golang-jwt/jwt/v5 to v5.2.2
- argoproj/argo-rollouts to v1.8.2
- rs/zerolog to v1.34.0
- gramlabs/go-application to v0.0.99
- gramlabs/stormforge-go to v4.7.6

Agent version 2.21.1

Thu, 20 Mar 2025 08:30:00 +0000

Added

Recommendations for Java workloads
You can now configure the Agent to generate Java maximum heap size recommendations for each Java container in a workload, in addition to requests and limits.

The following settings have been added in support of this feature:
- Support for the runtime:java label selector when setting annotations.
  
  A best practice is to use the RequestsAndLimits optimization policy for Java containers, which ensures that Optimize Live can implement the desired heap size. Example:

Applier version 2.7.1

Thu, 20 Mar 2025 08:30:00 +0000

Added

New optional Helm parameter disableAutomaticRollback: Default value is false. Typically, automatic rollback is disabled only upon guidance from StormForge Support or when workloads frequently become unhealthy.
This parameter replaces the setting of STORMFORGE_SKIP_ROLLBACK=true as an additionalEnvVar.
RBAC permissions are now granted by default for Argo Rollouts, eliminating the need for additonal manual configuration.
New optional Helm parameter additionalEnvVars replaces extraEnvVars, which is now deprecated. Existing extraEnvVars settings are respected and no changes are required.
In the Agent, this parameter was deprecated in version 2.20.2.
New Helm parameters enable you to specify additional volumes and mounts to add to pods (for example, to mount custom CA bundles as part of using a transparent proxy):
- additionalVolumes configures additional volumes that will be added to all pods
- additionalVolumeMounts configures additional volume mounts that will be added to all containers

Changed

Dependency updates:
- evanphx/json-patch/v5 to v5.9.11
- gomodules.xyz/jsonpatch/v2 to v2.5.0
- go-jose/v4 to v4.0.5
- kubernetes libraries to v0.32.3
- controller-runtime to v0.20.3
- x/net to v0.36.0
- structured-merge-diff to v4.6.0
- gramLabs/stormforge-agent to v2.21.1
- gramLabs/go-application to v0.0.98
- gramLabs/stormforge-go to v4.7.5

Agent version 2.20.2

Wed, 05 Mar 2025 08:30:00 +0000

Added

New optional Helm parameter additionalEnvVars replaces extraEnvVars, which is now deprecated. See Deprecated below.
New sf_node_labels metric and improved handling of node resource usage metrics.

For the complete list metrics that StormForge collects, see the Security FAQ.

Changed

Updated dependencies:
- prometheus to v3.2.1
- prometheus libraries to v0.302.0
- kubernetes libraries to v0.32.2
- stormforge-go to v4.7.4
- go-application to v0.0.97
- cobra to v1.9.1
- client_golang to v1.21.1

Fixed

Node controller metrics are now garbage collected.
Fixed a typo in the Helm template.

Security

Updated dependencies
- go-jose/v4 to v4.0.5
- x/oauth2 to v0.28.0

Deprecated

extraEnvVars Helm parameter. Use additionalEnvVars (see Added above). Existing extraEnvVars settings are respected and no changes are required.

Agent version 2.20.1

Wed, 19 Feb 2025 08:30:00 +0000

Changed

All workload metrics are now enabled by default. For a complete list of metrics, see the Security FAQ.
Several noisy INFO-log entries are now to DEBUG-level log entries.
Updated dependencies:
- kubernetes to v0.34.2

Fixed

We fixed the issue of missing metrics when workloads are configured with limits only and no requests.

Agent version 2.20.0

Tue, 11 Feb 2025 08:30:00 +0000

Added

Node metrics collection is now enabled by default.
New custom node metrics:
- sf_node_capacity_resources: The total amount of resources that a node has.
- node_cpu_usage_seconds_total: Used in calculating the average cluster CPU utilization
- node_memory_working_set_byes: Used in calculating the average cluster memory utilization
For a complete list of metrics, see the Security FAQ.

Changed

Added namespaces to useViewRole in RBAC configuration.
Updated dependencies:
- spf13/pflag to v1.0.6
- argo-rollouts to v1.8.0
- stormforge-go to v4.7.2
- x/oauth2 to v0.26.0
- terratest to v0.48.2
- go-application to v0.0.95

Fixed

We fixed the synchronization of workload configurations and HPA deletions.
We fixed an issue with how DaemonSets were reconciled.
We fixed an issue that prevented memory bump-ups from being applied immediately when configured to do so.

Agent version 2.19.1

Tue, 28 Jan 2025 08:30:00 +0000

Changed

Updated dependencies:

prometheus library to v0.301.0
prometheus/common to v0.62.0
stormforge-go to v4.7.1
go-application to v0.0.94

Deprecated

Installing Prometheus 2.x via Helm is no longer supported.

Fixed

We fixed how the Agent handles jobs that have Jobs Completions set to null.

Security

Updated Prometheus container to v3.1.0 to fix CVE-2024-45337 in x/crypto.

Agent version 2.19.0

Thu, 23 Jan 2025 08:30:00 +0000

Added

A new Helm parameter priorityClassName enables scheduling and priority configurations to add to all pods.

Fixed

Better HPA detection and deletion now prevent the Agent from attempting to patch a workload if its HPA has been deleted.
Rate limiting now prevents the Agent from making more than one recommendation request in response to a workload OOM event.
We fixed an issue with the Helm schema validation for OptimizationPolicy.
We updated how the Agent handles invalid credentials.

Applier version 2.7.0

Wed, 22 Jan 2025 08:30:00 +0000

Added

New Helm parameter priorityClassName enables scheduling and priority configurations to add to all pods

Fixed

Log message improvement and cleanup

Changed

Safer recommendation rollback
The rollback operation now reverts only the changes made to fields managed by stormforge at the time the patch was applied. Previously, in a rollback, all changes made between when the recommendation was applied and when the rollback happens were lost.

Security

Dependency updates:

Agent version 2.18.0

Wed, 08 Jan 2025 08:30:00 +0000

Added

Support for adding volumes and mounts to pods
The following new Helm parameters enable you to specify additional volumes and mounts to add to pods (for example, to mount custom CA bundles as part of using a transparent proxy):
- additionalVolumes configures additional volumes that will be added to all pods
- additionalVolumeMounts configures additional volume mounts that will be added to all containers

Changed

Dependency updates:
- stormforge-go to v4.6.1
- go-application to v0.0.93
- keda/v2 to v2.16.1
- x/oauth2 to v0.25.0
- terratest to v0.48.1
- controller-runtime to v0.19.4

Agent version 2.17.0

Thu, 12 Dec 2024 08:30:00 +0000

Added

Improved support for Argo Rollouts
Support for Argo Rollouts and built-in Kubernetes types is now equivalent.

Changed

Dependency updates:
- controller-runtime to v0.19.3
- kube-api libraries to v0.32.0
- deckarep/golang-set to v2.7.0
- prometheus/common to v0.61.0
- crypto to v0.31.0
- stormforge-go to v4.6.0
- go-application to v0.0.90

Fixed

Boolean types are now parsed correctly during workload autodiscovery.

Agent version 2.16.1

Wed, 20 Nov 2024 08:30:00 +0000

Added

Support for applying recommendations by using a mutating admission webhook
You can now configure Optimize Live to apply recommendations via a mutating admission webhook instead of the default patch method by setting the following annotation: live.stormforge.io/apply.method: DynamicAdmissionWebhook

You can define the apply method at the cluster, namesapce, or workload level. For details, see Apply method in the Apply settings topic.

Note: A cluster administrator must enable this feature via a Helm parameter when installing the Applier. Feature-specific namespace allow/deny lists can be configured to restrict webhook usage. For details, see Enabling the DynamicAdmissionWebhook in Applier configuration help topic.

Applier version 2.6.0

Wed, 20 Nov 2024 08:30:00 +0000

Added

Support for optional mutating admission webhook

Fixed

Better handling of JSON type patches

Changed

Upgraded dependencies to the following versions:

kubernetes dependencies to v0.31.1
go-application to v0.0.89
stormforge-go to v4.5.1
stormforge-agent to v2.16.1

Security

Upgraded packages include:

go to v1.23.3
prometheus to v0.60.1
prometheus go client to v1.20.5
golang.org/x/oauth2 to v0.24.0

Agent version 2.15.3

Thu, 31 Oct 2024 08:30:00 +0000

Changed

Updated prometheus/common to v0.60.1
Updated controller-runtime to 0.19.1
Reduced the wait timeout for RBAC (to minimize pod blocking operation) from 20s to 5s

Fixed

We fixed a problem that was causing the sf_workload_pod_container_resource_requests metric to drop intermittently, resulting in No containers metrics observed for this workload in the last 10 minutes errors and incomplete graph lines.

Agent version 2.15.2

Wed, 23 Oct 2024 08:30:00 +0000

Changed

Upgraded Prometheus container to v2.55.0

Fixed

We now correctly detect HPAs on clusters with open telemetry operator configured with autoscaling.

UI enhancements

Tue, 15 Oct 2024 11:04:44 +0000

Added

New graph: OOM Events
On the Reports page, a new OOM Events graph shows the count of OOMKills over time across the estate. Use the filters at the top of the page to filter by cluster and time period.

Agent version 2.15.1

Tue, 01 Oct 2024 08:30:00 +0000

Changed

Dependency updates:

kubernetes libraries to 0.31.1
stormforge-go to 4.3.3
go-application to 0.0.86
prometheus/client_golang to 1.20.4
prometheus/common to 0.60.0
terratest to 0.47.2

Fixed

Fixed the issue of the Agent not cleaning up stale metrics for deleted containers.

UI enhancements

Thu, 12 Sep 2024 08:04:44 +0000

Added

Learning period shown on Schedule page
You can now view a workload’s learning period on the Settings > Schedule section of workload’s details page. The learning period defines how long Optimize Live observes a new workload before providing recommendations that can be auto-deployed.

To adjust the learning period, you can set a cluster, namespace, or workload default by setting the live.stormforge.io/learning-period annotation.

Agent version 2.15.0

Tue, 10 Sep 2024 08:30:00 +0000

Added

Configurable learning period
You can now configure the duration of the learning period during which Optimize Live observes a new workload before providing recommendations that can be auto-deployed (default value is P7D, 7 days). For example, a shorter learning period is helpful for branch-based deploys.

To adjust the learning period, you can set a cluster, namespace, or workload default by setting the live.stormforge.io/learning-period annotation. You can view this setting on the Settings > Schedule section of workload’s details page.

UI enhancements

Thu, 29 Aug 2024 08:04:44 +0000

Added

Deployment breakdown graph
A new graph on the Workloads Summary page shows the following workload and deployment information:
- The proportion of workloads that have auto-deploy enabled vs. disabled
- Deployment outcome counts

Agent version 2.14.0

Thu, 22 Aug 2024 08:30:00 +0000

Added

CronJob controller now reconciles inactive Cronjob workload types
Support for CronJob, ScheduledSparkApplication, and GitHubARC types.
To enable discovery and recommendations for these types, set the corresponding featureGate to true either via Helm or a values.yaml file as shown in this excerpt:
```
featureGates:
    optimizeCronJobs: true
    optimizeSparkOperator: true
    optimizeGithubARC: true
```

Changed

Zerolog is the standard logging mechanism. Logging arguments are now --log-level=debug and --log-level=trace, replacing --v=<level>.
Added a name to the workload-agent metrics port
Updated go-application to v0.0.82
Updated keda to v2.15.1
Updated structured-logging to v0.0.5
Updated stormforge-go to v4.2.0
Updated controller-runtime to v0.19.0

Removed

Log arguments --v=<level>. See Changed above for the new arguments.
The workload.workloadResources Helm parameter used for enabling the workloadResource types has been replaced as described in the Added section above.

Fixed

CronJobs: Completed pods are now garbage collected on Optimize Live metrics

Security

Updated grpc to v1.64.1
Updated golang.org/x/oauth2 to v0.22.0
Upgraded the Prometheus agent to v2.54.0 to address low- and medium-severity CVEs

Applier version 2.5.0

Fri, 02 Aug 2024 08:30:00 +0000

Changed

Validation rollout of patches and persisting the state of the recommendation is now more reliable:
- After patches are applied, the recommendation status is either ApplierSuccess or ApplierFailed
- When validating the rollout and checking workload health, the validation status is either RolloutValidationSuccess, RolloutValidationTimeout or RolloutValidationFailure
Changed the rollout validation timeout to 5m (5 minutes) from 30m

Security

Multiple packages upgraded, including go package upgraded to 1.22.5
GHSA-m5vv-6r4h-3vj9
GHSA-v6v8-xj6m-xwqh
GHSA-xr7q-jx4m-x55m

Agent version 2.13.0

Tue, 23 Jul 2024 08:30:00 +0000

Added

Adds 10 minute “Learning Period” to the Agent sidecar in the Metrics-forwarder pod for faster metrics scraping after init.
Adds a workload metric sf_workload_terminated_total that tracks the count of workload termination events. Initially will only track OOMKilled events.

Changed

Added a default value to authorization.issuer to the Helm chart.

Removed

Cadvisor metric container_oom_events_total no longer collected.

Fixed

Workload container order was not being preserved.
Potential contention between DeletionPredicate and Reconcile.

UI enhancements

Thu, 18 Jul 2024 08:04:44 +0000

Added

Graphs showing overview data over time
On the new Reports page, you can now view graphs that show recommendation impact data for your entire estate over a time range of 2 weeks, 1 month, or 3 months:
- Current and recommended requests totals for both CPU and memory
- Optimization progress showing the number of overprovisioned/underprovisioned workloads
- Estimated monthly resource costs
- Workload count
  
  Example:

Changed

Improved recommendation details view
On the Recommendation tab on the workload details page, recommended container values are now stacked along the left side of the pane, making it easier to view recommended values. Clicking a container name still reveals the Average CPU/Memory Usage, Requests, and Limits graphs.

Agent version 2.12.2

Mon, 15 Jul 2024 08:30:00 +0000

Security

Go upgraded from 1.22.4 to 1.22.5 to address CVE GO-2024-2963
Addressed several CVEs related to the Prometheus container image (upgraded from 2.52.0 to 2.53.1)

Agent version 2.12.1

Tue, 02 Jul 2024 08:30:00 +0000

Added

The strings "Enabled" and "Disabled" are now accepted for the live.stormforge.io/auto-deploy annotation, in addition to the existing values ("true" and "false").

Changed

HPAs owned by Flagger are now supported (patchTarget is a KEDA ScaledObject).
If the HPA patchTarget is not detected, patchTarget is set to the HPA itself.

Fixed

HPA settings are no longer silently ignored when an HPA is owned by a ScaledObject other than KEDA.
HPA objects with .spec changes now update HPA values.
ReplicaSets that have 0 pods are now reported correctly.

UI enhancements

Wed, 26 Jun 2024 08:04:44 +0000

Changed

Get Started improvements
You no longer have to save the generated Yaml values file when you install.
Simply provide a cluster name, run the generated helm install command, and click Check Connection.
Optimization settings improved layout
On a workload’s Config tab, settings are now read-only and grouped as shown in the image below. To change these settings, use annotations as shown in the examples provided.

Agent version 2.11.0

Thu, 13 Jun 2024 08:30:00 +0000

Added

New default container optimization policy
We added a RequestsRaiseLimitsIfNeeded container optimization policy: Requests are always optimized; limits are optimized only if they already exist. If an existing limit is lower than what Optimize Live recommends, the limit is raised to Optimize Live’s recommended limit value.

With this policy, limits are not lowered if they already exist, nor are they added if they don’t exist.

Note: The default optimization policy is still RequestsOnly.

Agent version 2.10.8

Thu, 23 May 2024 08:30:00 +0000

Fixed

The StormForge Agent no longer crashes when it cannot find deleted custom resources from an installed package.

Agent version 2.10.7

Fri, 17 May 2024 08:30:00 +0000

Fixed

We fixed the regex validation of hashed cron expressions (such as "H H(0-5) * * *"), which are used to set recommendation schedules. For more schedule examples, see the Recommendation schedule topic in the product docs.

Agent version 2.10.6

Mon, 06 May 2024 08:30:00 +0000

Changed

Updated prometheus dependency to 0.53.0

Fixed

patchPath and patchFormat on the WorkloadOptimizer CRD are now processed accordingly.
Updated example command lines on Helm README.

Applier version 2.4.0

Fri, 03 May 2024 08:30:00 +0000

Added

Continuous reconciliation
You can now configure the StormForge Applier to continuously reconcile drift, ensuring that the latest recommended settings are always maintained on a workload. See the new Helm configuration options below.
- reconciliation.mode: EventBased (default value) maintains the existing Applier behavior of patching workloads only when new recommendations are ready. Continuous enables continuous reconciliation.
- reconciliation.conflictingManagers: [value1,value2]: Used in conjunction with mode: Continuous. The Applier cannot reconcile the desired state for fields owned by conflicting managers.

Agent version 2.10.5

Fri, 19 Apr 2024 08:30:00 +0000

Added

Added network policies to ensure StormForge pods can communicate with each other now that the workload-agent and prometheus containers have been separated into their own pods.

Changed

Updated controller-runtime to 0.17.3
Updated kubernetes dependencies to 0.17.3
Updated stormforge-go to 3.9.2
Updated prometheus dependency to 0.51.3
Updated uber/zap to 1.27.0
Updated stretchr/testify to 1.9.0
Updated kubernetes group to 0.29.4
Updated jackc/pgx/v5 to 5.5.4
Removed kedacore 2.13.0

Fixed

We added code to ensure that the cluster_name label in the ENV secrets file is the source of truth for the cluster name.
We fixed the problem of ReplicaSets being incorrectly created as workloads when owned by Argo rollouts.
We fixed the bug that caused the Failed to watch *v1.Secret error, which occurred when the Agent requested access to secrets in order to parse certain KEDA ScaledObjects, depending on the configuration.

Security

Updated Prometheus container image to v2.51.0
Updated go-jose/v3 to 3.0.3

UI enhancements

Wed, 03 Apr 2024 08:04:44 +0000

Changed

Improved recommendation impact charts
- Shading for Usage now includes the 95th and 75th percentile total usage in addition to average total usage.
- Shading for Net impact more clearly explains the “gap” between current and recommended requests settings.
- Chart tooltips on the Impact tab now include Waste (overprovisioning) or Performance risk (underprovisioning) details, as shown below.

UI enhancements

Tue, 19 Mar 2024 11:04:44 +0000

Added

Easier to understand recommendation impact information
Redesigned page headers on the Overview, Cluster, and Namespace pages make it easier to understand projected savings at each level.

The new Optimization Score measures how well the current request settings align with Optimize Live’s recommendations (100 indicates perfect alignment). This score replaces the Overall Efficiency score.

In the page header, click More Information to reveal the recommended CPU and memory adjustments and their estimated cost impact.

Applier version 2.3.0

Sat, 09 Mar 2024 08:30:00 +0000

Added

Continuously enforce the latest recommendation after workload updates
You can now configure Optimize Live to detect changes to workload requests or limits and then automatically reapply the latest recommendation. This ensures the workload remains optimized.

By default, Optimize Live does not reapply recommendations after such changes.

To continuously enforce the latest recommendation, do one of the following when you install the Applier:
- In your Helm values.yaml file, set enforce: true.
- In the helm install command, include --set enforce=true.

Define which CD tools’ changes do not trigger reapplying a recommendation
If you set enforce to true as described above, you can also define which continuous delivery tools’ changes (for example, changes by Argo CD, Flux) do not trigger reapplying a recommendation. If an exempt manager changes a workload’s requests or limits, Optimize Live will not reapply the latest recommendation.

Agent version 2.10.4

Thu, 15 Feb 2024 08:30:00 +0000

Changed

To improve performance, the workload-agent and prometheus containers now run in their own pods.

Fixed

We fixed a bug in how we reconcile workload annotations. You’ll now see your updated configurations in the UI.
We removed cluster_name actual values from the prometheus ConfigMap and now set the cluster name from the secret to prevent a cluster name mismatch.
We fixed the validation of CPU and memory optimization goals to allow capitalized values (Savings, Balanced, Reliability).

Dependencies

Prometheus agent upgraded to version 2.49.1
prometheus/common upgraded to 0.47.0
controller-runtime upgraded to 0.17.1
stormforge-go upgraded to 3.8.6
Kubernetes API dependencies upgraded to 0.29.2

Security

The workload-agent container now addresses CVE-2020-8911 and CVE-2020-8912 with the upgrade of prometheus Go module to version 0.49.1.

Agent version 2.10.3

Thu, 18 Jan 2024 08:30:00 +0000

Added

Support for workloads with names that exceed 47 characters
Optimize Live can now optimize workloads that have names that exceed 47 characters. Previously for such workloads, Optimize Live showed this error: "No container metrics observed for this workload in the last 10 minutes."

Fixed

Schedule annotations are now created consistently
We fixed a regression in which workloads with annotations were created without the schedule value.

Agent version 2.10.2

Tue, 16 Jan 2024 08:30:00 +0000

This release consists of internal improvements that build on version 2.10.1 of the Agent.

Agent version 2.10.1

Thu, 11 Jan 2024 08:30:00 +0000

Added

Added HPA Helm parameters
To set cluster-level HPA target utilization defaults, you can now use the following Helm chart values:
```
hpaCpuTargetUtilizationMin
hpaCpuTargetUtilizationMax
hpaMemoryTargetUtilizationMin
hpaMemoryTargetUtilizationMax
```

Fixed

HPA annotations released in version 2.10.0
The new HPA annotations added in 2.10.0 did not work as documented, and have been been fixed.

Security

Upgraded cloudflare/circl from version 1.3.3 to version 1.3.7 to address high-severity security issue CIRCL's Kyber: timing side-channel (kyberslash2)
Upgraded golang.org/x/crypto to v0.17.0

Agent version 2.10.0

Tue, 09 Jan 2024 08:30:00 +0000

Added

Annotations for HPA CPU and memory target utilization
You can now configure HPA target utilization ranges (min and max values) for both CPU and memory by using the following annotations:

live.stormforge.io/hpa.cpu.target-utilization.min
live.stormforge.io/hpa.cpu.target-utilization.max
live.stormforge.io/hpa.memory.target-utilization.min
live.stormforge.io/hpa.memory.target-utilization.max

Deprecated

Pod template (PodTemplateSpec) annotations on workloads
We no longer support PodTemplateSpec (.metadata.spec.template.metadata.annotations) annotations for annotating workloads.

Instead, workload-level annotations must now be added to the workload object directly in the .metadata.annotations section, as shown in this example:

Applier version 2.2.0

Tue, 09 Jan 2024 08:30:00 +0000

Security

Updated golang net/http packages to address CVE-2023-44487.
Updated golang.org/x/crypto from version 0.14.0 to version 0.17.0.
Updated go-application to version 0.0.64.

UI enhancements

Wed, 06 Dec 2023 08:04:44 +0000

New: Workloads Summary page
You can now view workload totals categorized by workload status, errors, and warnings.

In the left navigation menu, click Workloads.

Agent version 2.9.1

Wed, 29 Nov 2023 08:30:00 +0000

Added

Additional metrics scraping
Optimize Live now scrapes the following cAdvisor metrics:
container_cpu_cfs_throttled_seconds_total, container_memory_max_usage_bytes and container_oom_events_total.

Agent version 2.9.0

Wed, 08 Nov 2023 08:30:00 +0000

Added

Support for setting deployment thresholds (via UI, CLU, annotations)
To help reduce pod churn, you can now define minimum change thresholds to prevent Optimize Live from automatically applying new recommendations.

You can define a percent change, unit amount change, or both, by using the following methods:
- UI: On the Config tab of workload details page, as part of the Automatic Deployment setting.
- Annotations: live.stormforge.io/auto-deploy.thresholds.* annotations. See the examples in the Auto-deploy topic.
- CLI: --auto-deploy-thresholds-min* arguments as part of the stormforge edit workloads handler command.
For details, see Auto-deploy thresholds in the Auto-deploy settings topic.

UI enhancements

Thu, 26 Oct 2023 08:04:44 +0000

UI enhancements when working with workloads:

Impact Overview graphs: Choose a time range
On the Impact tab of a workload details page, you can now zoom in or out to view different data points and the potential recommendation impact across different time intervals: 1 day, 1 week, 2 weeks, or 1 month.
Configure page: Enhanced validation of requests values
We now do more checks on the requests values that you enter. Most notably, we let you know if a CPU or memory value seems unexpectedly high or low.

Agent version 2.8.0

Fri, 20 Oct 2023 08:30:00 +0000

Added

Easy troubleshooting
You can now do a health check and get troubleshooting info (for example, if the Agent doesn’t install as expected) more easily by running:
```
helm test stormforge-agent -n stormforge-system --logs
```
UI: Add Cluster button now available on the Clusters page; improved cluster management workflow
You can now add clusters via the Optimize Live UI: In the left navigation, click Clusters, and on the Clusters page, click Add Cluster.

Agent version 2.7.1

Mon, 25 Sep 2023 08:30:00 +0000

Added

Recommendations for workloads that scale down to 0 replicas
Optimize Live now provides recommendations for workloads that scale down to zero replicas for short periods of time within a 7-day period.
- If a workload is scaled down to 0 replicas for 25% of the time over 7 days, a recommendation is still provided, and you’ll see a warning in UI about the amount of time scaled down.

Agent version 2.7.0

Thu, 14 Sep 2023 08:30:00 +0000

Added

OpenShift support
You can now optimize workloads on clusters managed by OpenShift. When you install the StormForge Agent on a cluster, include the --set openshift=true argument. For details, see Install on Red Hat OpenShift Container Platform in the product docs.
Support for automatic rightsizing of workloads scaled by a KEDA-owned HPA
Optimize Live can now provide recommendations and patch workloads that have an HPA that is managed by KEDA. Previously, Optimize Live could only make recommendations for these workloads.

Applier version 2.1.0

Thu, 14 Sep 2023 08:30:00 +0000

Added

Improved patch rollback and workload health checks
The Applier now performs additional health checks to ensure that a workload is always in a healthy state before applying patches. If a patch fails, the Applier can now roll back all patches to reach the previous healthy state.

We also improved the Applier logging, making it easier to understand what a patch is doing, patch application progress, and the health of a workload after applying patches.

Agent version 2.6.0

Fri, 18 Aug 2023 10:30:00 -0500

Added

Optimize Live 30-day free trial
If you’re not already using Optimize Live, sign up at app.stormforge.io/signup and within minutes, you’ll have Optimize Live running on the Kubernetes cluster you specify.

Need to see it to believe it? This 3-min Getting Started video walks you through setup (which takes less than 2 minutes) and gives you a quick overview of the insights you’ll get in just 1 hour after installation.

Applier version 2.0.6

Thu, 03 Aug 2023 15:00:00 -0500

Changed

Applier Helm chart updates
We updated the Applier’s Helm chart to use new Agent secret names and values. No action is required unless you have changed Agent secret values.

Agent version 2.5.1

Thu, 03 Aug 2023 10:30:00 -0500

Fixed

We fixed an encoding problem with the manageAuthSecret feature released in version 2.5.0.

Agent version 2.5.0

Fri, 28 Jul 2023 18:30:00 -0500

Changed

Helm installation: --set stormforge.clusterName=CLUSTER_NAME must be changed to --set clusterName=CLUSTER_NAME.
We now consume the prometheus image from quay.io instead from Docker Hub. To fallback to Docker Hub, pass the following parameter as part of the Helm installation: --set prom.image.repository=prom/prometheus.
To manage or rotate authorization credentials outside of Helm, set the Helm value manageAuthSecret to false. If you set this value to false, make sure that the stormforge-agent-auth secret exists before installing or upgrading.

Known Issues

In certain environments, changing namespace-level StormForge annotations might not trigger reconciliation. For these scenarios, the annotations will be granted only on the following Agent restart.

Agent version 2.4.1

Fri, 21 Jul 2023 16:00:00 -0500

Fixed

The controller no longer crashes when reconciling WorkloadOptimizer custom resources.

Agent version 2.4.0

Mon, 17 Jul 2023 16:00:00 -0500

Added

Support for setting default workload values at the cluster level
You can now use a configuration file or command line arguments as part of the helm install command to set default values for all workloads in a cluster. See the examples below.

Configuration file: Create a .yaml file and set the workload- and container-level values.
```
clusterDefaultConfig:
    schedule: P1D
    containersCpuRequestsMin: 100m,istio-proxy=50m
```
Command line arguments:
```
--set clusterDefaultConfig.schedule=P1D \
--set clusterDefaultConfig.containersCpuRequestsMin=100m,istio-proxy=50m
```
Support for setting default workload values at the namespace level

Applier version 2.0.5

Thu, 06 Jul 2023 15:00:00 -0500

This release contained internal enhancements only. No action is required.

Agent version 2.3.0

Thu, 29 Jun 2023 16:00:00 -0500

Added

Support for ingesting workload labels
By default, Optimize Live now ingests labels on Kubernetes workload objects, ensuring that StormForge workload labels match your workload labels, making it easier for you to search for workloads.

To change this default behavior, setting collectLabels to false in a helm install or helm upgrade command: --set collectLabels=false
Support for setting container default values by using annotations
You can now set default values for all containers in a workload by using annotations in pod objects.

Applier version 2.0.4

Thu, 29 Jun 2023 15:00:00 -0500

Additional custom Prometheus metrics
You can now get more Applier performance details with the following Prometheus metrics:
- The following new metrics are vectored and have workload_namespace and workload_resource labels:
  - sf_patches_processed_total
  - sf_patches_failed_total
  - sf_patches_rolled_back_total
  - sf_patches_rollback_failures_total
- The following new metric does not have the workload_namespace and workload_resource labels - it applies only to the Applier, not a specific workload:
  - sf_applier_api_disconnections_total

Agent version 2.2.0

Thu, 15 Jun 2023 16:00:00 -0500

Workload garbage collection
We now do garbage collection (by default, every hour) to ensure that when workloads are removed from the cluster, they’re also removed from Optimize Live. To change this reconciliation interval, edit the workload.workloadSyncInterval in the workload’s values.yaml file.
Regex support
We added regex support for the optional allowNamespaces and denyNamespaces parameters that you can provide when you install the StormForge agent.

Applier version 2.0.3

Thu, 15 Jun 2023 15:00:00 -0500

Patch rollback support
We added rollback logic: If an error occurs when applying a patch for a recommendation, successful patches are rolled back.

Version 2.0

Thu, 13 Apr 2023 00:00:00 +0000

Be sure to check out our press release!

Added

New install method: Helm chart for stormforge-agent
You no longer need to download the StormForge CLI to install. Get up and running within minutes, and download the CLI later to run StormForge commands to manage your cluster.
Optional Applier installation using a single Helm command
We separated the Agent and the Applier to simplify the permissions required at install.
If you plan to apply configurations on demand (outside of a regular schedule) or automatically outside of a CI/CD workflow, be sure to install the Applier component.

Version 0.7.8

Mon, 20 Mar 2023 00:00:00 +0000

Fixed

Permissions issue during upgrade

This release fixes a permissions issue that sometimes caused the TSDB to crash when upgrading an existing Optimize Live installation.

Version 0.7.7

Thu, 09 Mar 2023 00:00:00 +0000

Added

All components now run as non-root

Individual components (TSDB, Applier, Recommender, Grafana) now run with runAsNonRoot: true set in their PodSecurityContext. The Controller continues to run as non-root by default. This feature is helpful if you deploy Optimize Live in clusters that have security policies that require all containers to run as non-root.
Improved handling of Datadog rate limit errors

The TSDB now gracefully handles HTTP 429 responses from the Datadog API. If Datadog is your metrics provider, you’ll see better performance when the Datadog rate limit is reached.

Version 0.7.6

Thu, 02 Mar 2023 00:00:00 +0000

Controller

Added

Support for DaemonSet optimization

Optimize Live can now optimize DaemonSets in workloads, resulting in even more resource savings.

Fixed

You can now specify any Grafana image or version

The Controller can now install Grafana using the image repository and tag that you specify in the Helm chart values.yaml file. Previously, the Controller installed the latest version of Grafana from the official registry only.

Version 0.7.5

Thu, 16 Feb 2023 00:00:00 +0000

Recommender

Added

Support for workloads that scale based on custom metrics in the HorizontalPodAutoscaler

Optimize Live now produces a recommendation to size the workload to best align with the currently configured HorizontalPodAutoscaler custom metric. Previously, CPU utilization metrics were the only supported HorizontalPodAutoscaler metric.

Version 0.7.4

Thu, 02 Feb 2023 00:00:00 +0000

Recommender and Controller

Updated

Show recommendations even if some workloads in an application fail

Optimize Live now, by default, shows recommendations even if it couldn’t generate recommendations for all discovered workloads (for example, when workloads crash or fail, or when new workloads don’t yet have enough metrics data).

Previously, recommendations were shown only if they were computed for all discovered workloads. To preserve this behavior, set FF_ONLY_COMPLETE_RECOMMENDATIONS=true in the extraEnvVars section of the Helm chart.

UI enhancements

Wed, 01 Feb 2023 11:04:44 +0000

Launch from the left navigation

Launch or switch between Optimize Live and Optimize Pro from the left navigation rather than from the tabs within an application. This update takes you to your applications and recommendations faster.

Version 0.7.3

Thu, 26 Jan 2023 00:00:00 +0000

Controller

Added

Deleting a Live object now deletes the corresponding application

When you delete a Live object from your cluster, Optimize Live now also deletes the application from the UI and the API. To restore the original behavior (in which the application isn’t deleted from the UI and API), label the Live object by running this command:

kubectl label -n stormforge-system live/my-applive.optimize.stormforge.io/skipSync=skip
Grafana cleanup when uninstalling Optimize Live

Version 0.7.2

Wed, 11 Jan 2023 00:00:00 +0000

TSDB

Added

Expose recommendation count, recommendation tx/timestamp metrics

The following optimize live metrics are available via /metrics endpoint:
- optimize_live_recommendation_count, which displays a count of the most recent number of recommendations received
- optimize_live_recommendation_timestamp, which displays a timestamp of when the last set of recommendations were made
- optimize_live_tsdb_series_timestamp, which displays a timestamp for each top level metric we ingest (limits, requests, usage, etc.)

Fixed

Limit Datadog query length when querying HPA metrics

Version 0.7.1

Thu, 15 Dec 2022 11:03:52 -0500

Controller and Applier

Added

Support for reducing the resources used by a cluster

If you have many applications (for example, upwards of a couple hundred) and apply recommendations conservatively (for example, every few days), you can set FF_PATCHER="true" in the extraEnvVar section of your Helm chart. This consolidates and simplifies the cluster component stack and does not negatively affect cluster performance.
Support for persisting patches to ConfigMaps

Version 0.7.0

Fri, 09 Dec 2022 15:28:51 -0500

Updated Helm chart value: DEBUG=false, and set DEBUG to Boolean in the values schema
Grafana updates simplify the information you see on dashboards

UI enhancements

Tue, 06 Dec 2022 11:04:44 +0000

On the Configure Recommendations page:

In the Optional Settings section, you can now specify the CPU target utilization of the HPA recommendation.
In the Advanced Settings section, you can choose either of the following:
- Enable Guaranteed Quality of Service.
- Exclude Memory Limits, CPU limits, or both from recommendations.

To access the Advanced Settings, contact your StormForge sales rep.

Version 0.6.0

Tue, 18 Oct 2022 18:38:48 -0400

Added support for HPA constraints for min and max target CPU utilization
Added support for collecting min and max replica metrics to provide better recommendations

Version 0.5.0 (HPA support)

Tue, 27 Sep 2022 14:30:50 -0400

Applier

Added

Support for jsonpath custom patches
Support for generating HPA patches

Controller

Added

Support for bidimensional autoscaling
Support for providing target utilization recommendations alongside CPU and memory
Enabled HPA lookup by default
Added recommendation labels to the dashboard to better filter results

Fixed

We now correctly look up existing Live resources when syncing from the API

UI enhancements

Mon, 26 Sep 2022 11:04:44 +0000

Added a progress bar that shows the progress of the TSDB backfill
Added support for maximum CPU and memory limits
Added a clusters list page

Version 0.4.0

Mon, 19 Sep 2022 17:33:54 -0400

Applier

Added

Support for custom patches:
- You can now create a Live custom resource definition (CRD) to provision and configure a new Optimize Live instance
- You can now apply recommendations via a Live object
Support for pods with multiple containers
Support for arm64 architecture
Updated Grafana dashboards:
- You can choose low, medium, or high risk tolerance for both CPU and memory when viewing recommendation summaries
- You can now see HPA-related data
The Recommender now provides recommendations that honor the maximum limits that you specify
You can now specify the following values in a Live object:
- Maximum bound for CPU and memory requests
- Minimum and maximum CPU and memory limits

TSDB

Added

Significantly reduced TSDB ConfigMap size, allowing now up to 700+ targets per Live object (from previously only 100+ targets). For testing and troubleshooting, you can still add raw queries to the Controller’s configuration file, but you must add them manually.

Version 0.3.0

Tue, 13 Sep 2022 18:13:48 -0400

UI enhancements

You can now set CPU and memory minimum limits when you configure recommendations
Deleting an application in the UI now also deletes the application from the cluster
A progress bar now displays data backfilling progress
New search capability helps you to find your applications faster

Version 0.2.2

Tue, 17 May 2022 17:27:52 +0000

Qualify Datadog metrics with cluster name
Suppress log messages during backfill of data
Fixed bug that could cause the recommender to stall

Version 0.2.1

Fri, 29 Apr 2022 16:55:46 +0000

Added support for non-standard replicaset owners (e.g., rollouts)
The Grafana dashboard has been updated to highlight the containers’ maximum usage
The recommender now supports varying number of replicas
The TSDB allows for customization of the persistent volume
Added DEBUG log level for all the components
The Controller supports proxies
Beta support for Datadog as a metrics provider

Version 0.1.6

Wed, 23 Feb 2022 05:18:59 +0000

Controller deploys the TSDB, the recommender, the applier and Grafana deployment
Support for metrics stores in Prometheus