# Previous StormForge Agent and Applier releases --- For the latest release notes, see [StormForge Agent releases]. [StormForge Agent releases]: /docs/releases/stormforge/ --- ## Version 2.15.5-applier _October 26, 2026_ ### Changed * Dependency updates ### Security * **Dependency updates**: Updated dependencies to address [CVE-2026-113470](https://nvd.nist.gov/vuln/detail/cve-2026-113470). --- ## Version 2.28.7-agent _October 22, 2026_ ### Fixed * **Node provider ID updates**: The Agent now detects node updates that change only `spec.providerID`, which happens when the cloud controller sets the provider ID after the Agent first sees the node. The Agent refreshes `sf_node_labels` and updates the cluster record in StormForge with the new provider ID instead of keeping an empty or outdated one. * **CPU pressure recommendations**: When `nodeCPUPressureDetection` is enabled, recommendations created by the node CPU pressure analyzer now use the `CPUPressureDetected` reason. They also include a message that names the node and the number of containers in the workload that exceeded their CPU requests. ### Security * **Dependency updates**: Updated dependencies to address [CVE-2026-113470](https://nvd.nist.gov/vuln/detail/cve-2026-113470). --- ## Version 2.15.4-applier _October 22, 2026_ ### Deprecated * **v2 Applier in maintenance mode**: The v2 Applier receives only security fixes going forward. To get new features, move to the unified `stormforge` Helm chart; see [Migrate from v2 to v3](https://docs.stormforge.io/docs/installation/migration/). ### Changed * **Kubernetes libraries updated**: Upgraded the Kubernetes client libraries to v0.37.1 and controller-runtime to v0.25.2. ### Fixed * **Applier events missing when the edit role is disabled**: The Helm chart's ClusterRole now allows the Applier to `create` and `patch` events in the core and `events.k8s.io` API groups. Previously, the Applier could only read events, so installs with `rbac.useEditRole=false` lost every Applier event (`RecommendationApplied`, `InPlaceResizeSucceeded`, `WorkloadResourcesChanged`, and the drift and rollback events), logging only a "Server rejected event" message. --- ## Version 2.28.6-agent _September 4, 2026_ ### Changed * Dependency Updates ### Fixed * **Pod-scoped metrics retention:** Fixed an issue where pod-scoped metrics could leak after pod deletion * **Metrics store name collision:** Fixed metrics collision issue for pods with the same name, which could result in overwriting or deleting metrics --- ## Version 2.28.5-agent _July 17, 2026_ ### Fixed * Fixes an issue where webhook-injected sidecars (e.g. istio-proxy, linkerd-proxy, etc) could be omitted from right-sizing. --- ## Version 2.15.3-applier _July 15, 2026_ ### Changed * Dependency Updates ### Fixed * Fix suspend mode Kubernetes event emission on start-up --- ## Version 2.28.4-agent _July 15, 2026_ ### Changed * Dependency Updates ### Fixed * **Stale container requests/limits pruned**: request/limit metric series for containers removed from a workload's spec no longer keep being emitted indefinitely during stuck rollouts or after agent restarts mid-rollout. * **StatefulSet revision handling**: pod revision checks now handle StatefulSets (controller-revision-hash label) and requeue with backoff on lookup failure instead of risking a wrong-revision pod's spec being reported as the workload's observed settings. * **ScheduledSparkApplication pods detected again**: executor pods of Spark apps launched by a ScheduledSparkApplication use a non-controller ownerRef, so the owner walk stopped early; the agent now uses the `sparkoperator.k8s.io/scheduled-app-name` label to attribute them, restoring metrics for these workloads. --- ## Version 2.15.2-applier _June 10, 2026_ ### Security - **Go 1.26.4 toolchain and `x/crypto` update**: Bumping `github.com/gramLabs/go-application` to `v0.0.118` transitively updates the Go toolchain to `1.26.4` and `golang.org/x/crypto` to `v0.52.0`, picking up all security fixes included in those releases. --- ## Version 2.28.2-agent _June 4, 2026_ ### Changed - Dependency updates --- ## Version 2.15.1-applier _June 4, 2026_ ### Changed * Dependency Updates ### Fixed * Fixed an issue where `WebhookImmediateRollout` apply method while handling fallback on In-Place-Pod-Resize failure, might not deploy the correct recommendation across pod admission. --- ## Version 2.15.0-applier _May 20, 2026_ ### Added - **Suspend capability**: New `suspend` Helm parameter disables applier behavior at runtime without scaling down the Deployment or removing the webhook server. Allowed values: `None` (default), `NewRecommendations`, `All`. Invalid values are rejected at startup. When `All`, the webhook returns `Allowed` immediately after decode so the `MutatingWebhookConfiguration` stays responsive. ### Changed - **Event reason renames** `(breaking)`: All applier event reason strings have been renamed to use StormForge domain language. Dashboards, alerts, or tooling that filters on old reasons must be updated: - `PatchApplied` `RecommendationApplied` - `PatchFailed` `RecommendationApplyFailed` - `PatchRolledBack` `RecommendationRolledBack` - `RollbackFailed` `RecommendationRollbackFailed` - `DriftDetected` `RecommendationDriftDetected` - `DriftReconciled` `RecommendationDriftReconciled` - `ValidationFailed` (unrecoverable) `RolloutValidationAbandoned` - `ValidationTimedOut` `RolloutValidationTimedOut` --- ## Version 2.14.0-applier _May 7, 2026_ ### Added * Centralized Kubernetes Event Emission ### Changed * Dependency Updates --- ## Version 2.28.1-agent _April 16, 2026_ ### Changed * Add RBAC for OpenShift operator-owned workloads when `openshift=true`. * Update prometheus container version to 3.11.2 * Dependency Updates: * kubernetes library group to 0.35.4. * `prometheus/prometheus` to 0.311.2. * `gramLabs/go-application` to 0.0.115. * `gramLabs/stormforge-go` to 4.9.1. --- ## Version 2.13.0-applier _April 13, 2026_ ### Added * Adds Drift Reconciliation for Horizontal Pod Autoscalers * Detects when StormForge Recommendations are rejected because of Cluster settings and triggers rollback faster. * Failure Conditions detected: ResourceQuota violations, LimitRange validation, and Core Pod validation ### Changed * Dependency Updates: * kubernetes libraries to 0.35.3 * `argoproj/argo-rollouts` to 1.9.0 * `rs/zerolog` to 1.35.0 * `gramLabs/stormforge-go` to 4.9.0 * `gramLabs/go-application` to 0.0.114 * `gramLabs/structured-logging` to 0.0.7 * `gramLabs/stormforge-agent/v2` to 2.28.0. ### Fixed * Fixed an issue where Drift Reconciliation for Workloads was not retrieving the last applied recommendation sometimes. ### Security * Updated Go to 1.26.2. --- ## Version 2.28.0-agent _April 9, 2026_ ### Added * Adds support for In-API Cluster Config * Adds user-override configuration for `clusterProvider` * To configure custom provider string, add the following to helm values: ``` clusterProvider: exampleProvider ``` * Adds support for custom cluster labels * To configure cluster labels, add the following to helm values: ``` clusterLabels: key1: value1 key2: value2 ``` ### Changed * Updated Dependencies: * `go-jose/go-jose/v4` to 4.1.4. * `prometheus/prometheus` to 0.311.1. * `aws/aws-sdk-go-v2/service/cloudwatchlogs` to 1.65.0. * `aws/aws-sdk-go-v2/service/lambda` to 1.88.5. * `aws/aws-sdk-go-v2/service/s3` to 1.97.3. * `google/cel-go` to 0.28.0. * `gramLabs/go-application` to 0.0.114. ### Security * Updated Go to 1.26.2. --- ## Version 2.27.4-agent _April 1, 2026_ ### Added * Adds new metrics: * container_cpu_cfs_throttled_periods_total * container_cpu_cfs_periods_total * Adds ability to configure Auth credentials secret and env in Helm. ### Changed * Strictly check file-based cluster-default config on startup and fail if invalid. * Dependency Updates: * `golang.org/x/oauth2` to 0.36.0 * `k8s.io/klog/v2` to 2.140.0 * `gruntwork-io/terratest` to 0.56.0 * `google.golang.org/grpc` to 1.79.3 * `argoproj/argo-rollouts` to 1.9.0 * `gramLabs/go-application` to 0.0.113 * `gramLabs/stormforge-go/v4` to 4.9.0 * `gramLabs/structured-logging` to 0.0.7 --- ## Version 2.12.2-applier _March 13, 2026_ ### Changed * Dependency Updates: * `gramLabs/go-application` to 0.0.112. ### Fixed * Fixed a bug that causes Applier to panic when applying pod resize to ArgoRollouts. * Allows webhook cert-controller to be disabled. ### Security * Updated Go to 1.26.1. * `golang.org/x/oauth2` to 0.36.0. --- ## Version 2.12.1-applier _March 5, 2026_ ### Changed * Improved RBAC scope of the applier * Improve Webhook TLS and certificate management * Dependency Updates: * kubernetes libs to `0.35.2`. * stormforge-agent libs to 2.27.3 --- ## Version 2.27.3-agent _March 5, 2026_ ### Changed * Enables Optimization CRDs by default. * Updated default prometheus container image to 3.10.0. * Dependency Updates: * `filippo.io/edwards25519` to 1.1.1. * `prometheus/prometheus` to 0.310.0. * kubernetes libs to 0.35.2. ### Fixed * Fixed a bug where the Agent could send duplicate StatusChanges for a workload, causing an error. * Fixed several Helm issues --- ## Version 2.12.0-applier _February 19, 2026_ ### Added * In-Place-Pod-Resize Support * To enable, set the workload's `apply.method` setting to `WebhookImmediateRollout` or `WebhookHybridRollout`. * `WebhookImmediateRollout` - Use a mutating webhook to modify Pods at admission. Existing pods will be resized in place when a new recommendation is applied. If Pods cannot be resized in place, a rollout/restart will be triggered. * `WebhookHybridRollout` - Use a mutating webhook to modify Pods at admission. Existing pods will be resized in place when a new recommendation is applied. If Pods cannot be resized in place, changes will be deferred until the next rollout. * HPA Reconciliation - Applier will watch, detect, and reconcile any drift in HPA `TargetUtilization` setting. * To enable this via helm: ``` reconciliation: mode: EventBased controllers: - resource: hpa enabled: true ``` - Restart of the Applier when the ConfigMap changes. ### Changed * Updated Dependencies: * kubernetes libraries to 0.35.1. * `golang.org/x/oauth2` to 0.35.0. * `sigs.k8s.io/structured-merge-diff/v6` to 6.3.2. * `argoproj/argo-rollouts` to 1.8.4. * `gramLabs/stormforge-agent/v2` to 2.27.2. * `gramLabs/go-application` to 0.0.111. * `gramLabs/stormforge-go/v4` to 4.8.4. ### Fixed * Fixed a typo in `priorityClassName` in the helm default values file. * Invalidates recommendations that have HPA metric changes that are not metrics that StormForge optimizes (resource and container resource metrics). ### Security * Update Go to 1.25.7 --- ## Version 2.27.2-agent _February 19, 2026_ ### Fixed * Fixed a bug where the Agent would lose the pod-workload association if the only pod still in running state had an older generation. ### Changed * Dependency Updates: * `golang.org/x/oauth2` to 0.35.0. * `github.com/google/cel-go` to 0.27.0. * `argoproj/argo-rollouts` to 1.8.4. * `gramLabs/stormforge-go/v4` to 4.8.4. * kubernetes libraries to 0.35.1. ### Security * Updated to Go 1.25.7. --- ## Version 2.27.1-agent _January 23, 2026_ ### Added * Added `ClusterOptimizationConfiguration` and `OptimizationConfiguration` Custom Resource Definitions (CRDs). _This release makes these CR types available, but they are not yet enabled by default. Use the Helm parameter `enableOptimizationConfigurationCRDs=true` to opt in to using them._ ### Changed * Updated Dependencies: * `gruntwork-io/terratest` to 0.55.0. ### Security * Updated Go to 1.25.6 --- ## Version 2.26.0-agent _January 14, 2026_ ### Added * Helm parameter `workloadResourceTypes`: it defines CRD workload types for which support is not built-in to StormForge. It has subfield `patchTargetTypeDefaults` to define patchpaths. * Helm parameter `ownerResourceTypes`: it defines configuration for types which may own workloads. This parameter can be used to define default patch-paths for each owner type, to define owner types to ignore (treat as if they do not own workloads), or types at which to stop owner reference traversal. The subfields are `stopTraversal`, `ignoreOwner` and `patchTargetTypeDefaults` ### Deprecated * Helm value `patchTargetTypes`. Use `workloadResourceTypes` instead. --- ## Version 2.11.0-applier _January 9, 2026_ ### Changed * Attention: mutation webhook is now enabled by default with `webhook.enabled=true` part of the helm values file. If one does not want to have the mutation webhook feature enabled, one needs to specify `webhook.enabled=false` * Expose applier metrics via port 8080 as part of the webhook service * Bumped K8s libraries to 1.35 ### Added * Helm parameter to support network policies: `networkPolicies.create=true` and optionally for egress network policy `networkPolicies.additionalEgressRules` with sample example on file ### Security * Addressed CVE-2025-61729 --- ## Version 2.25.1-agent _January 9, 2026_ ### Added * Sample egress network policy on helm values file ### Changed * Bumped prometheus common library to latest version * Bumped prometheus to `v3.9.1` ### Fixed * A config parse condition when specifying `clusterDefaultsAutoDeployExemptNamespaces` that cause agent not to start * in rare occasions, fixed missing "selector" field on HPA external metrics. ### Security * Addressed CVE-2025-61729 --- ## Version 2.10.1-applier _December 19, 2025_ ### Security * Addressed CVE-2025-68156 --- ## Version 2.25.0-agent _December 19, 2025_ ### Added * **Helm Parameter `clusterDefaultsAutoDeployExemptNamespaces`** This new Helm parameter defines a list of namespaces for which the `auto-deploy` setting defined in the cluster-defaults ConfigMap will be ignored. This parameter has a default value of `["^kube-system$", "^openshift.*"]`. Workloads in namespaces matching these regular expressions will not be subject to the `auto-deploy` setting defined in cluster-defaults. To enable `auto-deploy` on workloads in affected namespaces, you must use a namespace annotation, or a workload annotation. To modify this behavior, set the parameter to `null` or to a list of regular expressions that you define. ### Changed * **Helm Parameter `workload.denyNamespaces`** The default value for this parameter is now `null`. StormForge will observe and create recommendations for workloads in all namespaces by default. Previously, this parameter had a default value of `["kube-system", "openshift.*"]`. ### Security * Addressed CVE-2025-68156 --- ## Version 2.24.4-agent _December 15, 2025_ Fixed * Fixed a Helm template ConfigMap rendering issue when using `patchTargetTypes` --- ## Version 2.10.0-applier _December 11, 2025_ ### Added * Added a Parameter manage Recommendation Ready Timeout. Defaults to `1H`. This configurable parameter prevents stale recommendations from being applied at Applier start. ### Changed * Dropped the de-dup logic to handle recommendation items, relying on state * Setting a recommendation item to `FailedtoApply` when the recommendation does not have a patch list * Dependency Updates: * kubernetes libs to 0.34.3. * `sigs.k8s.io/controller-runtime` to 0.22.4. * `golang.org/x/oauth2` to 0.34.0. * `open-policy-agent/cert-controller` to 0.15.0. * `prometheus/common` to 0.67.1. * `spf13/cobra` to 1.10.2. * `gramLabs/go-application` to 0.0.108. * `gramLabs/stormforge-agent` to 2.24.3. ### Fixed * Fixed scenario that applier needed to be restarted to reapply recommendations that are set to be "Ready" via `sf apply`. The solution was changing the de-dup logic of the applier. --- ## Version 2.24.3-agent _December 11, 2025_ ### Added * Added a helm parameter to dictate the setup the network policies `networkPolicies.create`, default is false. Upon creation, the network policies are now functional (before the selectors were misconfigured). There is no egress policies being created by default (for such, please see the next bullet) * Added helm parameters for extra network policies `networkPolicies.additionalIngressRules` and `networkPolicies.additionalEgressRules`. ### Security * golang oauth2 `0.34.0` * golang crypto `0.46.0` ### Changed * KEDA 2.18.2 --- ## Version 2.24.2-agent _December 5, 2025_ ### Added * Added collection of network metrics if `enableCostMetrics=true`. * New metric name: `container_network_transmit_bytes_total`. ### Changed * Updated prometheus container version to 3.8.0. * Dependency Updates: * `prometheus/prometheus` to 0.308.0. * `spf13/cobra` to 1.10.2. --- ## Version 2.24.1-agent _November 25, 2025_ ### Changed * When setting `enableCostMetrics` to true (default is false), the user must now unset `workload.allowNamespaces` and `workload.denyNamespaces`. One can add the following lines on `values.yaml`: ``` workload: allowNamespaces: null denyNamespaces: null ``` ### Security * prometheus to version `2.7.3` to mitigate CVE-2025-54388 --- ## Version 2.24.0-agent _November 20, 2025_ ### Added * Adds support to define custom resource types to be recognized as StormForge workloads. * Adds support to configure PatchPaths to optimize custom workload types and operators. ### Changed * Dependency Updates: * `gramLabs/go-application` to 0.0.107. * `sigs.k8s.io/controller-runtime` to 0.22.4. * `prometheus/common` to 0.67.3. * k8s libraries to 0.34.2. ### Fixed * Fixed an issue where `patchable: false` status condition was being applied incorrectly. ### Security * `golang.org/x/oauth2` to 0.33.0. * `golang.org/x/crypto` to 0.45.0. --- ## Version 2.23.6-agent _October 30, 2025_ ### Added * [EXPERIMENTAL] One time specification of patch-paths for Custom Resource Workload Types. * Added relabeling rules to Prometheus config to normalize opentelemetry JVM metrics to match client_java form. ### Changed * Dependency Updates: * `gramLabs/go-application` to 0.0.106. * `gramLabs/stormforge-go` to 4.8.0. * `golang.org/x/oauth2` to 0.32.0. * `sigs.k8s.io/controller-runtime` to 0.22.3. * `prometheus/prometheus` to 0.307.3. * `prometheus/common` to 0.67.2 * `kedacore/keda` to 2.18.1. ### Fixed * Fixed an issue where deleting a WorkloadOptimizer CR does not update the `ManagedBy` field for a Workload. --- ## Version 2.23.5-agent _September 29, 2025_ ### Changed * Updated Dependencies * `prometheus/prometheus` to 0.306.0. * `gramlabs/stormforge-go` to 4.7.11. * Updated `prometheus` container in Helm to 3.6.0. ### Removed * Removed deprecated `Secret` helm template. * Removed deprecated `autoDiscovery` parameter. * Removed support for `autoscaling/v2beta2` version of HPA (no longer supported by Kubernetes). * Removed unused startupProbe license check from the stormforge pod templates. ### Fixed * Removed the aggregate view in the ClusterRole template ### Security * Go to `1.25.1` to address CVE-2025-47910 --- ## Version 2.23.4-agent _September 10, 2025_ ### Added * [EXPERIMENTAL] Added helm chart parameter `enableCostMetrics` to collect Cost Reporting metrics. Defaults to `false`. ### Changed * Update golang to 1.24.1. * Dependencies update: * `spf13/cobra` to 1.10.1. * `spf13/pflag` to 1.0.10. * `stretchr/testify` to 1.11.1. * `x/oauth2` to 0.31.0. * `sigs.k8s.io/controller-runtime` to 0.22.1. * `prometheus/client_golang` to 1.23.2. * `gramlabs/go-application` to 0.0.105. * kubernetes libraries to 0.34.1. --- ## Version 2.9.2-applier _August 29, 2025_ ### Fixed * Fixed an issue where new containers could be mistakenly added when using `DynamicAdmissionWebhook` apply method. --- ## Version 2.23.3-agent _August 22, 2025_ ### Fixed * Fixed a bug that resulted in the workload sync did not properly sync to the API. --- ## Version 2.9.1-applier _August 19, 2025_ ### Fixed * Fixed an Argo Rollout Patch Validation failure (to mitigate Argo Rollout [type convention violation](https://github.com/argoproj/argo-rollouts/issues/3402)) ### Changed * Dependency Updates: * `gramlabs/stormforge-agent` to 2.23.1 ### Security * Update Go to 1.24.6 to address [GO-2025-3849](https://pkg.go.dev/vuln/GO-2025-3849) --- ## Version 2.23.2-agent _August 19, 2025_ ### Security * Update Go to 1.24.6 to address [GO-2025-3849](https://pkg.go.dev/vuln/GO-2025-3849) --- ## Version 2.23.1-agent _August 14, 2025_ ### Changed * Updated Prometheus Container version to 3.5.0 * Dependency Updates * `prometheus/prometheus` to 0.305.0. * `spf13/pflag` to 1.0.7. * kubernetes libs to 0.33.4. * `gramLabs/stormforge-go` to 4.7.10. * `sigs.k8s.io/yaml` to 1.6.0. * `prometheus/client_golang` to 1.23.0. ### Fixed * Fixes an issue where a workload will be marked as patchable despite not having RBAC permissions. * Fixes an issue where an unsupported workload or a workload without RBAC will cause a panic in the workload reconciler. --- ## Version 2.9.0-applier _July 14, 2025_ ### Added - [Beta] Support for InPlace Pod Resizing on annotation `live.stormforge.io/apply.method=InPlacePodResizing` - Added additional RBAC on Helm to support InPlace Pod Resizing ### Changed - Dependency Updates: - controller-runtime 0.21.0 - e2e-framework to 0.6.0 - stormforge-go v4.7.9 - stormforge-agent v2.23.0 - google gnostic model v0.7.0 ### Security * Go to 1.24.5 --- ## Version 2.23.0-agent _July 14, 2025_ ### Added * Adds support for sending metrics and configuration to optimize init-container sidecars. * Adds default requests values for Agent-installed deployments as a starting point. Users should set higher values as needed. ### Changed * Dependencies * `prometheus/prometheus` to 0.304.2 * `argoproj/argo-rollouts` to 1.8.3. * `gramLabs/go-application` to 0.0.102. * `kedacore/keda` to `2.17.1-0.20250521173204-e23ef462abb5. * `sigs.k8s.io/controller-runtime` to 0.21.0. * `gramLabs/go-application` to 0.0.102. * `gramLabs/stormforge-go` to 4.7.9. ### Fixed * Fixes an issue where config for `patchPath` annotations did not support Ephemeral Storage. * Patch Paths can now be defined in quoted comma-delimited strings such as: * `live.stormforge.io/containers.cpu.limits.patch-path: "/path1,/additional/path"` * Fixes a panic when unable to retrieve a pod from the Spark. ### Security * Go to 1.24.5. --- ## Agent version 2.22.5 _July 2, 2025_ ### Fixed {#v2.22.5-agent-fixed} - Fixed a WorkloadOptimizer resource issue that was preventing configuration from being applied correctly. --- ## Version 2.8.3-applier _June 26, 2025_ ### Added - Add Helm parameters to set `commonMetaLabels` for StormForge deployed resources. - Example `values.yaml` configuration to set `commonMetaLabels`: ``` commonMetaLabels: example.com/app: stormforge ``` ### Changed - Dependency Updates: - kubernetes libraries to 0.33.2. - `argorproj/argo-rollouts` to 1.8.3. - `sigs.k8s.io/yaml` to 1.5.0. --- ## Agent version 2.22.4 _June 23, 2025_ ### Added {#v2.22.4-agent-added} - Added helm parameter to specify `securityContext` for StormForge deployed containers. - Example `values.yaml` configuration to set `securityContext` for containers in the `metrics-forwarder` and `workload-controller` pods. ``` commonContainerSecurityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL readOnlyRootFilesystem: true seccompProfile: type: RuntimeDefault ``` --- ## Version 2.8.2-applier _June 4, 2025_ ### Added * OLV2-2431 - Adding default requests on helm chart to set workload with Burstable QoS ### Fixed * OLV2-2426 - Applier in continuous reconciliation mode skips certain workloads --- ## Applier version 2.8.1 _May 29, 2025_ ### Changed {#v2.8.1-applier-changed} - Dependencies - `gramlabs/stormforge-go` to 4.7.7 - `gramlabs/go-application` to 0.0.101 - `gramlabs/stormforge-agent` to 2.22.3 ### Security {#v2.8.1-applier-security} - `golang` to 1.24.3 to address `GO-2025-3563` --- ## Agent version 2.22.3 _May 29, 2025_ ### Changed {#v2.22.3-agent-changed} - Dependencies - `go-logr/logr` to 1.4.3 ### Security {#v2.22.3-agent-security} - Upgraded Prometheus container dependency to 3.4.0 to address `CVE-2025-22871`. --- ## Agent version 2.22.2 _May 28, 2025_ ### Added {#v2.22.2-agent-added} - Allows specifying multiple paths for patchPath setting: - Example: `live.stormforge.io/containers.cpu.requests.patch-path: /path/to/target,containerA="/new/path,/alternative/path"` - Version and build SHA on log during initialization ### Changed {#v2.22.2-agent-changed} - A new way to run controllers: a generic workload controller instead of multiple controllers used to support replica 0 workloads. Expected faster warm up time and more data integrity on annotations on replica 0 workloads. - Dependencies - `keda/v2` to 2.17.1 - `prometheus/prometheus` to 0.304.0 - `prometheus/common` to 0.64.0 - `kubernetes libraries` to 0.33.1 - `gramLabs/go-application` to 0.0.100 - `gramLabs/stormforge-go` to 4.7.7 ### Fixed {#v2.22.2-agent-fixed} - Potential problem of missing memory metrics when `ephemeral_storage` is set. ### Security {#v2.22.2-agent-security} - `x/oauth2` to 0.30.0 - `golang` to 1.24.3 to address `GO-2025-3563` --- ## Applier version 2.8.0 _April 30, 2025_ ### Added {#v2.8.0-applier-added} * [EXPERIMENTAL] Configuration for Alerts for Recommendation Apply Events ### Fixed {#v2.8.0-applier-fixed} * An issue where applying a patch to an unhealthy workload continued to rollback the patch. ### Changed {#v2.8.0-applier-changed} - Dependency updates: - `gramLabs/stormforge-agent` to 2.22.1 - `kubernetes/api` to 0.33.0 - `Masterminds/sprig/v3` to 3.3.0 --- ## Agent version 2.22.1 _April 21, 2025_ ### Added {#v2.22.1-agent-added} - Helm parameters to support auth configuration for JVM Scrape jobs ### Fixed {#v2.22.1-agent-fixed} - We fixed an issue where the Agent would fail if API Resource discovery partially fails. - We fixed an issue with potential deadlock in the Node Cache. --- ## Applier version 2.7.2 _April 21, 2025_ ### Added {#v2.7.2-applier-added} - Add a flag `enableFallbackToClientSideApply` to work around an issue when improper configuration on a patch target prevents us from being able to apply a recommendation using Server-Side Apply. Setting this flag to `true` will result in handling errors by falling back to Client-Side Apply, **which will disable automatic rollback in the event of healthcheck failures.** ### Changed {#v2.7.2-applier-changed} - Dependency updates: - `x/oauth2` to 0.29.0 - `golang/jwt/v4` to 4.5.2 - `rs/zerolog` to 1.34.0 - `structured-merge-diff/v4` to 4.7.0 - `argoproj/argo-rollouts` to 1.8.2 - `prometheus/client_golang` to 1.22.0 - `controller-runtime` to 0.20.4 - `gramLabs/stormforge-go/v4` to 4.7.6 - `gramLabs/stormforge-agent/v2` to 2.22.0 --- ## Agent version 2.22.0 _April 2, 2025_ ### Changed {#v2.22.0-agent-changed} - **Internal improvements** Engineering work to support future feature additions ### Added {#v2.22.0-agent-added} - **Resource Request Mirroring** Enable recommendation mirroring, to support DR use cases or workloads owned by progressive delivery operators such as Flagger. Mirroring works by creating recommendations for a mirror workload based on usage metric observations of another. To configure mirroring, set the `metrics-from` annotation on the mirror workload. --- ## Agent version 2.21.2 _March 31, 2025_ ### Fixed {#2.21.2-agent-fixed} - Fixes an issue with hpa and pod reconciles could conflict and result in incomplete workload config. ### Changed {#2.21.2-agent-changed} - Updated Dependencies - `golang-jwt/jwt/v5` to v5.2.2 - `argoproj/argo-rollouts` to v1.8.2 - `rs/zerolog` to v1.34.0 - `gramlabs/go-application` to v0.0.99 - `gramlabs/stormforge-go` to v4.7.6 --- ## Agent version 2.21.1 _March 20, 2025_ ### Added {#v2.21.1-agent-added} - **Recommendations for Java workloads**\ You can now configure the Agent to generate Java maximum heap size recommendations for each Java container in a workload, in addition to requests and limits. The following settings have been added in support of this feature: - Support for the `runtime:java` label selector when setting annotations. A best practice is to use the `RequestsAndLimits` optimization policy for Java containers, which ensures that Optimize Live can implement the desired heap size. Example: `live.stormforge.io/containers.memory.optimization-policy: "balanced,runtime:java=reliability,container-a=savings"` - Annotations for defining heap size range and patch path details: ```yaml live.stormforge.io/containers.java.max-heap.min live.stormforge.io/containers.java.max-heap.max live.stormforge.io/containers.java.max-heap.patch-path live.stormforge.io/containers.java.max-heap.patch-format ``` ### Changed {#v2.21.1-agent-changed} - Node optimization and reporting is now enabled by default. To disable this feature, set the `clusterData` Helm parameter to `false`. - Updated dependencies: - `x/net` to v0.36.0 - `prometheus/common` to v0.63.0 - `gramLabs/stormforge-go` to v4.7.5 - `gramLabs/go-application` to v0.0.98 - `kubernetes` libraries to v0.32.3 ### Fixed {#v2.21.1-agent-fixed} - We made several changes to increase the Agent's efficiency at startup. --- ## Applier version 2.7.1 _March 20, 2025_ ### Added {#v2.7.1-applier-added} - New optional Helm parameter `disableAutomaticRollback`: Default value is `false`. Typically, automatic rollback is disabled only upon guidance from StormForge Support or when workloads frequently become unhealthy.\ This parameter replaces the setting of `STORMFORGE_SKIP_ROLLBACK=true` as an `additionalEnvVar`. - RBAC permissions are now granted by default for Argo Rollouts, eliminating the need for additonal manual configuration. - New optional Helm parameter `additionalEnvVars` replaces `extraEnvVars`, which is now deprecated. Existing extraEnvVars settings are respected and no changes are required.\ In the Agent, this parameter was deprecated in version [2.20.2][2-20-2]. - New Helm parameters enable you to specify additional volumes and mounts to add to pods (for example, to mount custom CA bundles as part of using a transparent proxy): - `additionalVolumes` configures additional volumes that will be added to all pods - `additionalVolumeMounts` configures additional volume mounts that will be added to all containers ### Changed {#v2.7.1-applier-changed} - Dependency updates: - `evanphx/json-patch/v5` to v5.9.11 - [`gomodules.xyz/jsonpatch/v2`][jsonpatch] to v2.5.0 - `go-jose/v4` to v4.0.5 - `kubernetes` libraries to v0.32.3 - `controller-runtime` to v0.20.3 - `x/net` to v0.36.0 - `structured-merge-diff` to v4.6.0 - `gramLabs/stormforge-agent` to v2.21.1 - `gramLabs/go-application` to v0.0.98 - `gramLabs/stormforge-go` to v4.7.5 [2-20-2]: /docs/releases/stormforge/#v2.20.2-agent-added [Troubleshooting]: /docs/reference/troubleshooting/#disable-rollback [jsonpatch]: https://pkg.go.dev/gomodules.xyz/jsonpatch/v2 --- ## Agent version 2.20.2 _March 5, 2025_ ### Added {#v2.20.2-agent-added} - New optional Helm parameter `additionalEnvVars` replaces `extraEnvVars`, which is now deprecated. See [Deprecated] below. - New `sf_node_labels` metric and improved handling of node resource usage metrics. For the complete list metrics that StormForge collects, see the [Security FAQ][metrics]. ### Changed {#v2.20.2-agent-changed} - Updated dependencies: - `prometheus` to v3.2.1 - `prometheus` libraries to v0.302.0 - `kubernetes` libraries to v0.32.2 - `stormforge-go` to v4.7.4 - `go-application` to v0.0.97 - `cobra` to v1.9.1 - `client_golang` to v1.21.1 ### Fixed {#v2.20.2-agent-fixed} - Node controller metrics are now garbage collected. - Fixed a typo in the Helm template. ### Security {#v2.20.2-agent-security} - Updated dependencies - `go-jose/v4` to v4.0.5 - `x/oauth2` to v0.28.0 ### Deprecated {#v2.20.2-agent-deprecated} - `extraEnvVars` Helm parameter. Use `additionalEnvVars` (see [Added] above). Existing `extraEnvVars` settings are respected and no changes are required. [metrics]: /docs/reference/faq-security/#metrics [Added]: #v2.20.2-agent-added [Deprecated]: #v2.20.2-agent-deprecated --- ## Agent version 2.20.1 _February 19, 2025_ ### Changed {#v2.20.1-agent-changed} - All workload metrics are now enabled by default. For a complete list of metrics, see the [Security FAQ][metrics]. - Several noisy `INFO`-log entries are now to `DEBUG`-level log entries. - Updated dependencies: - `kubernetes` to v0.34.2 ### Fixed {##v2.20.1-agent-fixed} - We fixed the issue of missing metrics when workloads are configured with limits only and no requests. [metrics]: /docs/reference/faq-security/#metrics --- ## Agent version 2.20.0 _February 11, 2025_ ### Added {#v2.20.0-agent-added} - Node metrics collection is now enabled by default. - New custom node metrics: - `sf_node_capacity_resources`: The total amount of resources that a node has. - `node_cpu_usage_seconds_total`: Used in calculating the average cluster CPU utilization - `node_memory_working_set_byes`: Used in calculating the average cluster memory utilization For a complete list of metrics, see the [Security FAQ][metrics]. ### Changed {#v2.20.0-agent-changed} - Added `namespaces` to `useViewRole` in RBAC configuration. - Updated dependencies: - `spf13/pflag` to v1.0.6 - `argo-rollouts` to v1.8.0 - `stormforge-go` to v4.7.2 - `x/oauth2` to v0.26.0 - `terratest` to v0.48.2 - `go-application` to v0.0.95 ### Fixed {##v2.20.0-agent-fixed} - We fixed the synchronization of workload configurations and HPA deletions. - We fixed an issue with how DaemonSets were reconciled. - We fixed an issue that prevented memory bump-ups from being applied immediately when configured to do so. [metrics]: /docs/reference/faq-security/#metrics --- ## Agent version 2.19.1 _January 28, 2025_ ### Changed {#v2.19.1-agent-changed} Updated dependencies: - `prometheus` library to v0.301.0 - `prometheus/common` to v0.62.0 - `stormforge-go` to v4.7.1 - `go-application` to v0.0.94 ### Deprecated {#v2.19.1-agent-deprecated} - Installing Prometheus 2.x via Helm is no longer supported. ### Fixed {#v2.19.1-agent-fixed} - We fixed how the Agent handles jobs that have Jobs Completions set to `null`. ### Security {#v2.19.1-agent-security} - Updated Prometheus container to v3.1.0 to fix [CVE-2024-45337][cve] in `x/crypto`. [cve]: https://github.com/advisories/GHSA-v778-237x-gjrc/ --- ## Agent version 2.19.0 _January 23, 2025_ ### Added {#v2.19.0-agent-added} - A new Helm parameter `priorityClassName` enables scheduling and priority configurations to add to all pods. ### Fixed {#v2.19.0-agent-fixed} - Better HPA detection and deletion now prevent the Agent from attempting to patch a workload if its HPA has been deleted. - Rate limiting now prevents the Agent from making more than one recommendation request in response to a workload OOM event. - We fixed an issue with the Helm schema validation for OptimizationPolicy. - We updated how the Agent handles invalid credentials. --- ## Applier version 2.7.0 _January 22, 2025_ ### Added {#v2.7.0-applier-added} - New Helm parameter `priorityClassName` enables scheduling and priority configurations to add to all pods ### Fixed {#v2.7.0-applier-fixed} - Log message improvement and cleanup ### Changed {#v2.7.0-applier-changed} - **Safer recommendation rollback**\ The rollback operation now reverts only the changes made to fields managed by `stormforge` at the time the patch was applied. Previously, in a rollback, *all* changes made between when the recommendation was applied and when the rollback happens were lost. ### Security {#v2.7.0-applier-security} Dependency updates: - `kubernetes` dependencies to v0.32.1 - `go-application` to v0.0.93 - `stormforge-go` to v4.7.0 - `stormforge-agent` to v2.18.1 - `e2e` to v0.6.0 --- ## Agent version 2.18.0 _January 8, 2025_ ### Added {#v2.18.0-agent-added} - **Support for adding volumes and mounts to pods**\ The following new Helm parameters enable you to specify additional volumes and mounts to add to pods (for example, to mount custom CA bundles as part of using a transparent proxy): - `additionalVolumes` configures additional volumes that will be added to all pods - `additionalVolumeMounts` configures additional volume mounts that will be added to all containers ### Changed {#v2.18.0-agent-changed} - Dependency updates: - `stormforge-go` to v4.6.1 - `go-application` to v0.0.93 - `keda/v2` to v2.16.1 - `x/oauth2` to v0.25.0 - `terratest` to v0.48.1 - `controller-runtime` to v0.19.4 --- ## Agent version 2.17.0 _December 12, 2024_ ### Added {#v2.17.0-agent-added} - **Improved support for Argo Rollouts**\ Support for Argo Rollouts and built-in Kubernetes types is now equivalent. ### Changed {#v2.17.0-agent-changed} - Dependency updates: - `controller-runtime` to v0.19.3 - `kube-api` libraries to v0.32.0 - `deckarep/golang-set` to v2.7.0 - `prometheus/common` to v0.61.0 - `crypto` to v0.31.0 - `stormforge-go` to v4.6.0 - `go-application` to v0.0.90 ### Fixed {#v2.17.0-agent-fixed} - Boolean types are now parsed correctly during workload autodiscovery. --- ## Agent version 2.16.1 _November 20, 2024_ ### Added {#v2.16.1-agent-added} - **Support for applying recommendations by using a mutating admission webhook**\ You can now configure Optimize Live to apply recommendations via a mutating admission webhook instead of the default patch method by setting the following annotation: `live.stormforge.io/apply.method: DynamicAdmissionWebhook` You can define the apply method at the cluster, namesapce, or workload level. For details, see [Apply method][apply-method] in the Apply settings topic. **Note**: A cluster administrator must enable this feature via a Helm parameter when installing the Applier. Feature-specific namespace allow/deny lists can be configured to restrict webhook usage. For details, see [Enabling the DynamicAdmissionWebhook][applier-config] in Applier configuration help topic. - **Support for memory bump-up in response to OOM Events**\ You can now configure Optimize Live to respond to OOM events with a temporary memory bump-up. You can define the min and max memory bump-up amount, as well as when to apply the bump-up. You can view OOM events on the OOM Events graph on the Reports page ([example graph][oom-graph]) Memory bump-ups can be configured at the resourcetype level, as shown below: ```yaml live.stormforge.io/reliability.oom.memory-bump-up.apply-immediately: "Always,resource:daemonsets=Never" live.stormforge.io/reliability.oom.memory-bump-up.max: "300Mi" live.stormforge.io/reliability.oom.memory-bump-up.min: "50Mi,resource:deployments=40Mi,resource:daemonsets=0" live.stormforge.io/reliability.oom.memory-bump-up.percent: "15,resource:deployments=25,resource:daemonsets=0" live.stormforge.io/reliability.oom.memory-bump-up.period: "P2D,resource:deployments=P1D" ``` For details and examples, see the [Reliability response][reliability] topic. ### Changed {#v2.16.1-agent-changed} - Dependency updates: - `go` to v1.23.3 - `go-application` to v0.0.89 - `stormforge-go` to v4.5.2 - `prometheus` library to v0.55.1 - `oauth2` to v0.24.0 - `keda` to v2.16.0 ### Fixed {#v2.16.1-agent-fixed} - We fixed the issue of unmanaged workloads not deleting HPA from the workload object when HPA was deleted. - We fixed the issue of sidecar containers using the same resource settings as the main containers. [applier-config]: /docs/installation/applier-config/#webhook [oom-graph]: /docs/releases/stormforge/#UI-15Oct2024-added [apply-method]: /docs/configure/settings/apply-settings/#apply-method [reliability]: /docs/configure/settings/reliability/ --- ## Applier version 2.6.0 _November 20, 2024_ ### Added {#v2.6.0-applier-added} - Support for optional mutating admission webhook ### Fixed {#v2.6.0-applier-fixed} - Better handling of JSON type patches ### Changed {#v2.6.0-applier-changed} Upgraded dependencies to the following versions: - `kubernetes` dependencies to v0.31.1 - `go-application` to v0.0.89 - `stormforge-go` to v4.5.1 - `stormforge-agent` to v2.16.1 ### Security {#v2.6.0-applier-security} Upgraded packages include: - `go` to v1.23.3 - `prometheus` to v0.60.1 - `prometheus go` client to v1.20.5 - `golang.org/x/oauth2` to v0.24.0 --- ## Agent version 2.15.3 _October 31, 2024_ ### Changed {#v2.15.3-agent-changed} - Updated `prometheus/common` to v0.60.1 - Updated `controller-runtime` to 0.19.1 - Reduced the wait timeout for RBAC (to minimize pod blocking operation) from 20s to 5s ### Fixed {#v2.15.3-agent-fixed} - We fixed a problem that was causing the `sf_workload_pod_container_resource_requests` metric to drop intermittently, resulting in `No containers metrics observed for this workload in the last 10 minutes` errors and incomplete graph lines. --- ## Agent version 2.15.2 _October 23, 2024_ ### Changed {#v2.15.2-agent-changed} - Upgraded Prometheus container to v2.55.0 ### Fixed {#v2.15.2-agent-fixed} - We now correctly detect HPAs on clusters with [open telemetry operator][open-telemetry] configured with autoscaling. [open-telemetry]: https://opentelemetry.io/docs/kubernetes/operator/ --- ## Agent version 2.15.1 _October 1, 2024_ ### Changed {#v2.15.1-agent-changed} Dependency updates: - `kubernetes` libraries to 0.31.1 - `stormforge-go` to 4.3.3 - `go-application` to 0.0.86 - `prometheus/client_golang` to 1.20.4 - `prometheus/common` to 0.60.0 - `terratest` to 0.47.2 ### Fixed {#v2.15.1-agent-fixed} - Fixed the issue of the Agent not cleaning up stale metrics for deleted containers. --- ## Agent version 2.15.0 _September 10, 2024_ ### Added {#v2.15.0-agent-added} - **Configurable learning period**\ You can now configure the duration of the _learning period_ during which Optimize Live observes a new workload before providing recommendations that can be auto-deployed (default value is `P7D`, 7 days). For example, a shorter learning period is helpful for branch-based deploys. To adjust the learning period, you can set a [cluster], [namespace], or [workload] default by setting the `live.stormforge.io/learning-period` annotation. You can view this setting on the **Settings** > **Schedule** section of workload's details page. The value for this setting must be [ISO-8601][Duration] compliant. Examples: - `P1D`: 1 day (minimum valid value) - `PT12H`: 12 Hours ### Changed {#v2.15.0-agent-changed} Updated the following images and libraries: - Prometheus container image to v2.54.1 - `prometheus` library to v0.54.1 - `prometheus/common` to v0.59.1 - `prometheus/client_golang` to v1.20.3 - `oauth2` to v0.23.0 - `go-application` to v0.0.83 - `stormforge-go` to v4.3.0 - Golang to 1.23.1 ### Fixed {#v2.15.0-agent-fixed} - Improved error message when a Network Connection or Proxy error occurs - Noisy error log messages are now debug messages [cluster]: /docs/configure/cluster-defaults/ [workload]: /docs/configure/annotate-workloads/ [namespace]: /docs/configure/annotate-namespaces/ [duration]: https://en.wikipedia.org/wiki/ISO_8601#Durations --- ## Agent version 2.14.0 _August 22, 2024_ ### Added {#v2.14.0-agent-added} - CronJob controller now reconciles inactive Cronjob workload types - Support for CronJob, ScheduledSparkApplication, and GitHubARC types.\ To enable discovery and recommendations for these types, set the corresponding featureGate to `true` either via Helm or a `values.yaml` file as shown in this excerpt: ```yaml featureGates: optimizeCronJobs: true optimizeSparkOperator: true optimizeGithubARC: true ``` ### Changed {#v2.14.0-agent-changed} - Zerolog is the standard logging mechanism. Logging arguments are now `--log-level=debug` and `--log-level=trace`, replacing `--v=`. - Added a name to the `workload-agent` metrics port - Updated `go-application` to v0.0.82 - Updated `keda` to v2.15.1 - Updated `structured-logging` to v0.0.5 - Updated `stormforge-go` to v4.2.0 - Updated `controller-runtime` to v0.19.0 ### Removed {#2.14.0-agent-removed} - Log arguments `--v=`. See [Changed](#v2.14.0-agent-changed) above for the new arguments. - The `workload.workloadResources` Helm parameter used for enabling the workloadResource types has been replaced as described in the [Added](#v2.14.0-agent-added) section above. ### Fixed {#v2.14.0-agent-fixed} - CronJobs: Completed pods are now garbage collected on Optimize Live metrics ### Security {#v2.14.0-agent-security} - Updated `grpc` to v1.64.1 - Updated `golang.org/x/oauth2` to v0.22.0 - Upgraded the Prometheus agent to v2.54.0 to address low- and medium-severity CVEs --- ## Applier version 2.5.0 _August 2, 2024_ ### Changed {#v2.5.0-applier-changed} - Validation rollout of patches and persisting the state of the recommendation is now more reliable: - After patches are applied, the recommendation status is either `ApplierSuccess` or `ApplierFailed` - When validating the rollout and checking workload health, the validation status is either `RolloutValidationSuccess`, `RolloutValidationTimeout` or `RolloutValidationFailure` - Changed the rollout validation timeout to `5m` (5 minutes) from `30m` ### Security {#v2.5.0-applier-security} - Multiple packages upgraded, including `go` package upgraded to 1.22.5 - [GHSA-m5vv-6r4h-3vj9](https://github.com/advisories/GHSA-m5vv-6r4h-3vj9) - [GHSA-v6v8-xj6m-xwqh](https://github.com/advisories/GHSA-v6v8-xj6m-xwqh) - [GHSA-xr7q-jx4m-x55m](https://github.com/advisories/GHSA-xr7q-jx4m-x55m) --- ## Agent version 2.13.0 _July 23, 2024_ ### Added {#v2.13.0-agent-added} - Adds 10 minute "Learning Period" to the Agent sidecar in the Metrics-forwarder pod for faster metrics scraping after init. - Adds a workload metric `sf_workload_terminated_total` that tracks the count of workload termination events. Initially will only track `OOMKilled` events. ### Changed {#v2.13.0-agent-changed} - Added a default value to authorization.issuer to the Helm chart. ### Removed {#v2.13.0-agent-removed} - Cadvisor metric `container_oom_events_total` no longer collected. ### Fixed {#v2.13.0-agent-fixed} - Workload container order was not being preserved. - Potential contention between DeletionPredicate and Reconcile. --- ## Agent version 2.12.2 _July 15, 2024_ ### Security {#v2.12.2-agent-security} - Go upgraded from 1.22.4 to 1.22.5 to address [CVE GO-2024-2963](https://pkg.go.dev/vuln/GO-2024-2963) - Addressed several CVEs related to the Prometheus container image (upgraded from 2.52.0 to 2.53.1) --- ## Agent version 2.12.1 _July 2, 2024_ ### Added {#v2.12.1-agent-added} - The strings `"Enabled"` and `"Disabled"` are now accepted for the `live.stormforge.io/auto-deploy` annotation, in addition to the existing values (`"true"` and `"false"`). ### Changed {#v2.12.1-agent-changed} - HPAs owned by Flagger are now supported (`patchTarget` is a KEDA ScaledObject). - If the HPA `patchTarget` is not detected, `patchTarget` is set to the HPA itself. ### Fixed {#v2.12.1-agent-fixed} - HPA settings are no longer silently ignored when an HPA is owned by a ScaledObject other than KEDA. - HPA objects with `.spec` changes now update HPA values. - ReplicaSets that have 0 pods are now reported correctly. --- ## Agent version 2.11.0 _June 13, 2024_ ### Added {#v2.11.0-agent-added} - **New default container optimization policy**\ We added a `RequestsRaiseLimitsIfNeeded` container optimization policy: Requests are always optimized; limits are optimized only if they already exist. If an existing limit is lower than what Optimize Live recommends, the limit is raised to Optimize Live’s recommended limit value. With this policy, limits are not lowered if they already exist, nor are they added if they don't exist. Note: The default optimization policy is still `RequestsOnly`. Use the following annotation values (for details, see the [product docs](/docs/configure/settings/cpu-and-memory/#opt-policy)): ``` live.stormforge.io/containers.cpu.optimization-policy: "RequestsRaiseLimitsIfNeeded" live.stormforge.io/containers.memory.optimization-policy: "RequestsRaiseLimitsIfNeeded" ``` - **PatchPath annotations**\ You can now use the following annotations to map a container resource value from a recommendation to where it belongs in the `patchTarget`: ``` live.stormforge.io/containers.memory.limits.patch-path live.stormforge.io/containers.memory.requests.patch-path live.stormforge.io/containers.cpu.limits.patch-path live.stormforge.io/containers.cpu.requests.patch-path ``` ### Fixed {#v2.11.0-agent-fixed} - We fixed a condition that caused an Agent panic with ownerless pods on clusters running older versions of spark-operator. ### Security {#v2.11.0-agent-security} - Upgraded the Prometheus agent to version 2.52.0 ### Known conditions {#v2.11.0-agent-known-conditions} - When setting requests and limits `patch-path` annotations along with the `DoNotOptimize` policy, you will see values on patchpath fields (previously the fields were set with a `-`). Regardless of the patchpath, the recommendations will respect the policy. The same is true for limits `patch-path` annotations when using the `RequestsOnly` policy. --- ## Agent version 2.10.8 _May 23, 2024_ ### Fixed {#v2.10.8-agent-fixed} - The StormForge Agent no longer crashes when it cannot find deleted custom resources from an installed package. --- ## Agent version 2.10.7 _May 17, 2024_ ### Fixed {#v2.10.7-agent-fixed} - We fixed the regex validation of hashed cron expressions (such as `"H H(0-5) * * *"`), which are used to set recommendation schedules. For more schedule examples, see the [Recommendation schedule](/docs/configure/settings/schedule/) topic in the product docs. --- ## Agent version 2.10.6 _May 6, 2024_ ### Changed {#v2.10.6-agent-changed} - Updated `prometheus` dependency to 0.53.0 ### Fixed {#v2.10.6-agent-fixed} - `patchPath` and `patchFormat` on the WorkloadOptimizer CRD are now processed accordingly. - Updated example command lines on Helm README. --- ## Applier version 2.4.0 _May 3, 2024_ ### Added {#v2.4.0-applier-added} - **Continuous reconciliation**\ You can now configure the StormForge Applier to continuously reconcile drift, ensuring that the latest recommended settings are always maintained on a workload. See the new Helm configuration options below. - `reconciliation.mode`: `EventBased` (default value) maintains the existing Applier behavior of patching workloads only when new recommendations are ready. `Continuous` enables continuous reconciliation. - `reconciliation.conflictingManagers: [value1,value2]`: Used in conjunction with `mode: Continuous`. The Applier cannot reconcile the desired state for fields owned by conflicting managers. Use the following Helm values in a YAML file when you install or update the Applier: ```yaml reconciliation: mode: Continuous # default is `EventBased` # List the field managers that conflict with the Applier; # Fields managed by a conflicting field manager cannot be reconciled. conflictingManagers: [] ``` ### Changed {#v2.4.0-applier-changed} - Updated stormforge-go to 3.9.2 - Updated go-application to 0.0.70 - Updated stormforge-agent to 2.10.5 - Updated controller-runtime to 0.17.3 - Updated oauth2 to 0.19.0 - Updated kubernetes api to 0.29.3 --- ## Agent version 2.10.5 _April 19, 2024_ ### Added {#v2.10.5-agent-added} - Added network policies to ensure StormForge pods can communicate with each other now that the `workload-agent` and `prometheus` containers have been separated into their own pods. ### Changed {#v2.10.5-agent-changed} - Updated `controller-runtime` to 0.17.3 - Updated `kubernetes` dependencies to 0.17.3 - Updated `stormforge-go` to 3.9.2 - Updated `prometheus` dependency to 0.51.3 - Updated `uber/zap` to 1.27.0 - Updated `stretchr/testify` to 1.9.0 - Updated `kubernetes` group to 0.29.4 - Updated `jackc/pgx/v5` to 5.5.4 - Removed `kedacore` 2.13.0 ### Fixed {#v2.10.5-agent-fixed} - We added code to ensure that the `cluster_name` label in the ENV secrets file is the source of truth for the cluster name. - We fixed the problem of ReplicaSets being incorrectly created as workloads when owned by Argo rollouts. - We fixed the bug that caused the `Failed to watch *v1.Secret` error, which occurred when the Agent requested access to secrets in order to parse certain KEDA ScaledObjects, depending on the configuration. ### Security {#v2.10.5-agent-security} - Updated Prometheus container image to v2.51.0 - Updated `go-jose/v3` to 3.0.3 --- ## Applier version 2.3.0 _March 9, 2024_ ### Added {#v2.3.0-applier-added} - **Continuously enforce the latest recommendation after workload updates**\ You can now configure Optimize Live to detect changes to workload requests or limits and then automatically reapply the latest recommendation. This ensures the workload remains optimized. By default, Optimize Live does **not** reapply recommendations after such changes. To continuously enforce the latest recommendation, do one of the following when you install the Applier: - In your Helm `values.yaml` file, set `enforce: true`. - In the `helm install` command, include `--set enforce=true`. - **Define which CD tools' changes do not trigger reapplying a recommendation**\ If you set `enforce` to `true` as described above, you can also define which continuous delivery tools' changes (for example, changes by Argo CD, Flux) do not trigger reapplying a recommendation. If an exempt manager changes a workload's requests or limits, Optimize Live will **not** reapply the latest recommendation. To define the list of exempt managers, do one of the following when you install the Applier: - In your Helm `values.yaml` file, set the `exemptManagers` key.\ Example: `exemptManagers: [value1,value2]` - In the `helm install` command, include `--set exemptManagers={value1,value2}`. ### Changed {#v2.3.0-applier-changed} - We increased the default health timeout from 5 minutes to 30 minutes. If pods don’t restart within 30 minutes after applying a recommendation, the recommendation is rolled back. - Updated kubernetes to 1.29.x - Updated controller-runtime to 0.17.2 - Updated gramLabs/go-application to 0.0.68 - Updated argo-rollouts to 1.6.5 - Updated zerolog to 1.32.0 - Updated oauth2 to 0.17.0 - Updated stormforge-go to 3.8.7 ### Security {#v2.3.0-applier-security} - Updated go-jose to 4.0.1 to address DoS vulnerability --- ## Agent version 2.10.4 _February 15, 2024_ ### Changed {#v2.10.4-agent-changed} - To improve performance, the `workload-agent` and `prometheus` containers now run in their own pods. ### Fixed {#v2.10.4-agent-fixed} - We fixed a bug in how we reconcile workload annotations. You’ll now see your updated configurations in the UI. - We removed `cluster_name` actual values from the `prometheus` ConfigMap and now set the cluster name from the secret to prevent a cluster name mismatch. - We fixed the validation of CPU and memory optimization goals to allow capitalized values (`Savings`, `Balanced`, `Reliability`). ### Dependencies {#v2.10.4-agent-dependencies} - Prometheus agent upgraded to version 2.49.1 - `prometheus/common` upgraded to 0.47.0 - `controller-runtime` upgraded to 0.17.1 - `stormforge-go` upgraded to 3.8.6 - Kubernetes API dependencies upgraded to 0.29.2 ### Security {#v2.10.4-agent-security} - The `workload-agent` container now addresses [CVE-2020-8911](https://nvd.nist.gov/vuln/detail/cve-2020-8911) and [CVE-2020-8912](https://nvd.nist.gov/vuln/detail/cve-2020-8912) with the upgrade of `prometheus` Go module to version 0.49.1. --- ## Agent version 2.10.3 _January 18, 2024_ ### Added {#v2.10.3-agent-added} - **Support for workloads with names that exceed 47 characters**\ Optimize Live can now optimize workloads that have names that exceed 47 characters. Previously for such workloads, Optimize Live showed this error: `"No container metrics observed for this workload in the last 10 minutes."` ### Fixed {#v2.10.3-agent-fixed} - **Schedule annotations are now created consistently**\ We fixed a regression in which workloads with annotations were created without the `schedule` value. --- ## Agent version 2.10.2 _January 16, 2024_ This release consists of internal improvements that build on version 2.10.1 of the Agent. --- ## Agent version 2.10.1 _January 11, 2024_ ### Added {#v2.10.1-agent-added} - **Added HPA Helm parameters**\ To set cluster-level HPA target utilization defaults, you can now use the following Helm chart values: ``` hpaCpuTargetUtilizationMin hpaCpuTargetUtilizationMax hpaMemoryTargetUtilizationMin hpaMemoryTargetUtilizationMax ``` ### Fixed {#v2.10.1-agent-fixed} - **HPA annotations released in version 2.10.0**\ The new HPA annotations added in 2.10.0 did not work as documented, and have been been fixed. ### Security {#v2.10.1-agent-security} - Upgraded `cloudflare/circl` from version 1.3.3 to version 1.3.7 to address high-severity security issue `CIRCL's Kyber: timing side-channel (kyberslash2)` - Upgraded `golang.org/x/crypto` to v0.17.0 --- ## Agent version 2.10.0 _January 9, 2024_ ### Added {#v2.10.0-agent-added} - **Annotations for HPA CPU and memory target utilization**\ You can now configure HPA target utilization ranges (min and max values) for both CPU and memory by using the following annotations: ``` live.stormforge.io/hpa.cpu.target-utilization.min live.stormforge.io/hpa.cpu.target-utilization.max live.stormforge.io/hpa.memory.target-utilization.min live.stormforge.io/hpa.memory.target-utilization.max ``` ### Deprecated {#v2.10.0-agent-deprecated} - **Pod template (PodTemplateSpec) annotations on workloads**\ We no longer support PodTemplateSpec (`.metadata.spec.template.metadata.annotations`) annotations for annotating workloads. Instead, workload-level annotations must now be added to the workload object directly in the **`.metadata.annotations`** section, as shown in this example: ``` apiVersion: apps/v1 kind: Deployment metadata: name: example namespace: default annotations: live.stormforge.io/auto-deploy: "false" live.stormforge.io/containers.cpu.limits.limit-request-ratio: "1.3" ... ``` ### Security {#v2.10.0-agent-security} - Upgraded go-application to version 0.0.64 - Upgraded Prometheus dependency to version 2.48.1 for security fixes --- ## Applier version 2.2.0 _January 9, 2024_ ### Security {#v2.2.0-applier-security} - Updated `golang net/http` packages to address [CVE-2023-44487](https://nvd.nist.gov/vuln/detail/CVE-2023-44487). - Updated [`golang.org/x/crypto`](https://pkg.go.dev/golang.org/x/crypto) from version 0.14.0 to version 0.17.0. - Updated `go-application` to version 0.0.64. --- ## Agent version 2.9.1 _November 29, 2023_ ### Added {#v2.9.1-agent-added} - **Additional metrics scraping** Optimize Live now scrapes the following cAdvisor metrics:\ `container_cpu_cfs_throttled_seconds_total`, `container_memory_max_usage_bytes` and `container_oom_events_total`. --- ## Agent version 2.9.0 _November 8, 2023_ ### Added {#v2.9.0-agent-added} - **Support for setting deployment thresholds (via UI, CLU, annotations)** To help reduce pod churn, you can now define minimum change thresholds to prevent Optimize Live from automatically applying new recommendations. You can define a percent change, unit amount change, or both, by using the following methods: - **UI**: On the **Config** tab of workload details page, as part of the **Automatic Deployment** setting. - **Annotations**: `live.stormforge.io/auto-deploy.thresholds.*` annotations. See the examples in the [Auto-deploy](/docs/configure/settings/apply-settings/#thresholds) topic. - **CLI:** `--auto-deploy-thresholds-min*` arguments as part of the [`stormforge edit workloads handler`](/docs/stormforge-cli/reference/#stormforge-edit-workloads-handlers) command. For details, see [Auto-deploy thresholds](/docs/configure/settings/apply-settings/#thresholds) in the Auto-deploy settings topic. - **Changes to the default optimization policy and limit-to-request ratio**\ The default optimization policy is now RequestsOnly. Previously, it was RequestsAndLimits. When the optimization policy is set to RequestsAndLimits, the new default limit-to-request ratio is 2.0 (increased from 1.2). Optimize Live will not update containers using a 1.2 ratio to the new default 2.0 ratio. In the UI, a 1.2 limit-to-request ratio is now shown as **Custom**. ### Security {#v2.9.0-agent-security} - To address [CVE-2023-44487](https://github.com/advisories/GHSA-qppj-fm5r-hxr3), we upgraded grpc-go to version 1.56.3. --- ## Agent version 2.8.0 _October 20, 2023_ #### Added {#v2.8.0-agent-added} - **Easy troubleshooting** You can now do a health check and get troubleshooting info (for example, if the Agent doesn't install as expected) more easily by running: ```sh helm test stormforge-agent -n stormforge-system --logs ``` - **UI: Add Cluster button now available on the Clusters page; improved cluster management workflow**\ You can now add clusters via the Optimize Live UI: In the left navigation, click **Clusters**, and on the Clusters page, click **Add Cluster**. **Note:** If you're using the Optimize Live free trial license and want to add another cluster, click **Add Cluster** and on the next page that's shown, click **Add a cluster to my trial**. - **Annotations: Support for additional case types (optimization goals and policies only)**\ UpperCamelCase values align with Kubernetes conventions, but you also can use lowercase values. - Valid values for the `live.stormforge.io/cpu.optimization-goal` and `live.stormforge.io/memory.optimization-goal` annotations: `"Balanced"` (default), `"balanced"`, `"Reliability"`, `"reliability"`, `"Savings"`, `"savings"` - Valid values for the `live.stormforge.io/containers.cpu.optimization-policy` and `live.stormforge.io/containers.memory.optimization-policy` annotations: `"RequestsAndLimits"` (default), `"requestsandlimits"`, `"RequestsOnly"`, `"requestsonly"`, `"DoNotOptimize"`, `"donotoptimize"` ### Changed {#v2.8.0-agent-changed} - **Helm parameter: Use `debug` instead of `workload.debug`**\ To enable or disable debug mode, use the `--set debug=` (`true` or `false`) Helm argument. We have deprecated `workload.debug`. - **Helm `lookup` function removed from workload initialization**\ We changed the workload initialization routine to ensure compliance with AWS EKS scans. ### Fixed {#v2.8.0-agent-fixed} - **The namespace-level `auto-deploy=true` annotation now has precedence**\ This release fixes a bug that prevented the namespace-level `live.stormforge.io/auto-deploy=false` annotation from overriding the cluster-defaults ConfigMap `live.stormforge.io/auto-deploy=true` annotation. ### Deprecated {#v2.8.0-agent-deprecated} - We deprecated the Helm `workload.debug` parameter. Use `--set debug=true` or `--set debug=false` instead. ### Security {#v2.8.0-agent-security} - To address [CVE-2023-44487](https://nvd.nist.gov/vuln/detail/CVE-2023-44487), we updated the Golang `net/http` package versions. --- ## Agent version 2.7.1 _September 25, 2023_ ### Added {#v2.7.1-agent-added} - **Recommendations for workloads that scale down to 0 replicas**\ Optimize Live now provides recommendations for workloads that scale down to zero replicas for short periods of time within a 7-day period. - If a workload is scaled down to 0 replicas for 25% of the time over 7 days, a recommendation is still provided, and you'll see a warning in UI about the amount of time scaled down. - If a workload is at 0 replicas for more than 75% of the time over 7 days, no recommendation is provided, and you'll see an error in the UI. ### Fixed {#v2.7.1-agent-fixed} - **Helm value `clusterDefaultConfig.schedule` permits day specifiers and commas**\ This release fixes a Helm validation error that prevented users from installing a cluster default schedule that included multiple weekdays or used day names (MON,TUE,WED and so on). Examples: - As an argument in a `helm install` command:\ `--set clusterDefaultConfig.schedule="H 20 * * WED,THU,FRI"` --- ## Agent version 2.7.0 _September 14, 2023_ ### Added {#v2.7.0-agent-added} - **OpenShift support**\ You can now optimize workloads on clusters managed by OpenShift. When you install the StormForge Agent on a cluster, include the `--set openshift=true` argument. For details, see [Install on Red Hat OpenShift Container Platform](/docs/installation/install-openshift/) in the product docs. - **Support for automatic rightsizing of workloads scaled by a KEDA-owned HPA**\ Optimize Live can now provide recommendations and patch workloads that have an HPA that is managed by KEDA. Previously, Optimize Live could only make recommendations for these workloads. --- ## Applier version 2.1.0 _September 14, 2023_ ### Added {#v2.1.0-applier-added} - **Improved patch rollback and workload health checks**\ The Applier now performs additional health checks to ensure that a workload is always in a healthy state before applying patches. If a patch fails, the Applier can now roll back all patches to reach the previous healthy state. We also improved the Applier logging, making it easier to understand what a patch is doing, patch application progress, and the health of a workload after applying patches. - **Support for automatic rightsizing of workloads scaled by a KEDA-owned HPA**\ Although this is a new _Agent_ (version 2.7.0) feature, you must grant the StormForge _Applier_ additional RBAC permissions. --- ## Agent version 2.6.0 _August 18, 2023_ ### Added {#v2.6.0-agent-added} - **Optimize Live 30-day free trial**\ If you’re not already using Optimize Live, sign up at **[app.stormforge.io/signup](https://app.stormforge.io/signup)** and within minutes, you’ll have Optimize Live running on the Kubernetes cluster you specify. Need to see it to believe it? This 3-min **[Getting Started video](https://youtu.be/CkpEqJssNfo)** walks you through setup (which takes less than 2 minutes) and gives you a quick overview of the insights you'll get in just 1 hour after installation. - **New StormForge Agent installation wizard**\ If you’re just starting out with Optimize Live, you can now install the StormForge Agent by using the Get Started wizard. Log in to [app.stormforge.io](https://app.stormforge.io) with your StormForge login, and in the left navigation, click **Overview**. - **Recommendation schedules now support for Cron format**\ When you configure workloads using Kubernetes annotations, you can now set a schedule using [Cron][cron] format. Previously, only macros and [ISO 8601 Duration][duration] strings were supported. Examples: - Once daily (default value and best practice):\ `live.stormforge.io/schedule: "H H * * *"` - Every morning at approximately 0800h (exact time is not guaranteed): `live.stormforge.io/schedule: "00 08 * * *"` For details, see the [Recommendation schedule](/docs/configure/settings/schedule/) topic. [cron]: https://en.wikipedia.org/wiki/Cron [duration]: https://en.wikipedia.org/wiki/ISO_8601#Durations --- ## Applier version 2.0.6 _August 3, 2023_ ### Changed {#v2.0.6-applier-changed} - **Applier Helm chart updates**\ We updated the Applier’s Helm chart to use new Agent secret names and values. No action is required unless you have changed Agent secret values. --- ## Agent version 2.5.1 _August 3, 2023_ ### Fixed {#v2.5.1-agent-fixed} - We fixed an encoding problem with the manageAuthSecret feature released in version 2.5.0. --- ## Agent version 2.5.0 _July 28, 2023_ ### Changed {#v2.5.0-agent-changed} - Helm installation: `--set stormforge.clusterName=CLUSTER_NAME` must be changed to `--set clusterName=CLUSTER_NAME`. - We now consume the `prometheus` image from quay.io instead from Docker Hub. To fallback to Docker Hub, pass the following parameter as part of the Helm installation: `--set prom.image.repository=prom/prometheus`. - To manage or rotate authorization credentials outside of Helm, set the Helm value manageAuthSecret to `false`. If you set this value to `false`, make sure that the stormforge-agent-auth secret exists before installing or upgrading. ### Known Issues {#v2.5.0-agent-known-issues} - In certain environments, changing namespace-level StormForge annotations might not trigger reconciliation. For these scenarios, the annotations will be granted only on the following Agent restart. --- ## Agent version 2.4.1 _July 21, 2023_ ### Fixed {#v2.4.1-agent-fixed} - The controller no longer crashes when reconciling WorkloadOptimizer custom resources. --- ## Agent version 2.4.0 _July 17, 2023_ ### Added {#v2.4.0-agent-added} - **Support for setting default workload values at the cluster level**\ You can now use a configuration file or command line arguments as part of the `helm install` command to set default values for all workloads in a cluster. See the examples below. **Configuration file:** Create a .yaml file and set the workload- and container-level values. ``` clusterDefaultConfig: schedule: P1D containersCpuRequestsMin: 100m,istio-proxy=50m ``` **Command line arguments:** ``` --set clusterDefaultConfig.schedule=P1D \ --set clusterDefaultConfig.containersCpuRequestsMin=100m,istio-proxy=50m ``` - **Support for setting default workload values at the namespace level** You can now use annotations to set default workload values at the namespace level. Namespace-level values override cluster-level values. Add the annotations to the `metadata.annotations` section of the namespace values file. ```yaml apiVersion: v1 kind: Namespace metadata: annotations: live.stormforge.io/schedule: "P1D" live.stormforge.io/containers.cpu.requests.min: "100m,istio-proxy=50m" creationTimestamp: "2023-07-20T17:28:42Z" labels: kubernetes.io/metadata.name: kube-public name: kube-public resourceVersion: "9" uid: spec: ... ``` ### Changed {#v2.4.0-agent-changed} - **Updated the Helm installation to simplify proxy configuration** - You can now add `--set proxyUrl=http://proxy.example.com` to the `helm install` command to specify a proxy server. - The corresponding `no_proxy` variable is now seeded as follows with several RFC 1918 addresses: `no_proxy:127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16` If these seeded values meet your organizaton's needs, you no longer have to create a separate proxy.yaml file as described in this [advanced installation scenario](/docs/installation/install-v2-adv/#if-you-have-a-proxy-server) in the product docs. - **Updated the Helm installation to expose Prometheus scrape interval**\ In typical installations, you don't need to change this interval — by default, the Prometheus scrapes for metrics every 15s. If StormForge Support requests that you change this interval, you can change it by adding `--set prom.scrapeInterval=INTERVAL` to the `helm install` command, replacing INTERVAL with the suggested frequency. ### Fixed {#v2.4.0-agent-fixed} - We now handle workload updates more gracefully, correcting an issue that caused controller panic. --- ## Applier version 2.0.5 _July 6, 2023_ This release contained internal enhancements only. No action is required. --- ## Agent version 2.3.0 _June 29, 2023_ ### Added {#v2.3.0-agent-added} - **Support for ingesting workload labels**\ By default, Optimize Live now ingests labels on Kubernetes workload objects, ensuring that StormForge workload labels match your workload labels, making it easier for you to search for workloads. To change this default behavior, setting `collectLabels` to `false` in a `helm install` or `helm upgrade` command: `--set collectLabels=false` - **Support for setting container default values by using annotations**\ You can now set default values for _all_ containers in a workload by using annotations in pod objects. In the Pod template metadata (`spec.template.metadata.annotations`) of a Deployment object, use the following syntax: `live.stormforge.io/containers.CONTAINER_PARAMETER: "DEFAULT_VALUE"` Example: To set `cpu.requests.min` to `20m` for all containers in a workload: `live.stormforge.io/containers.cpu.requests.min: “20m”` To learn more about setting default values by using annotations, see the topics under [Configure optimization settings]( /docs/configure/ ). ### Changed {#v2.3.0-agent-changed} - Prometheus agent version bumped to version 2.45 (Prometheus Long-Term Support release) --- ## Applier version 2.0.4 _June 29, 2023_ - **Additional custom Prometheus metrics**\ You can now get more Applier performance details with the following Prometheus metrics: - The following new metrics are vectored and have `workload_namespace` and `workload_resource` labels: - sf_patches_processed_total - sf_patches_failed_total - sf_patches_rolled_back_total - sf_patches_rollback_failures_total - The following new metric does not have the `workload_namespace` and `workload_resource` labels - it applies only to the Applier, not a specific workload: - sf_applier_api_disconnections_total --- ## Agent version 2.2.0 _June 15, 2023_ - **Workload garbage collection**\ We now do garbage collection (by default, every hour) to ensure that when workloads are removed from the cluster, they’re also removed from Optimize Live. To change this reconciliation interval, edit the `workload.workloadSyncInterval` in the workload’s `values.yaml` file. - **Regex support**\ We added regex support for the optional allowNamespaces and denyNamespaces parameters that you can provide when you install the StormForge agent. --- ## Applier version 2.0.3 _June 15, 2023_ - **Patch rollback support**\ We added rollback logic: If an error occurs when applying a patch for a recommendation, successful patches are rolled back. --- ## Version 2.0 _April 13, 2023_ Be sure to check out our [press release](https://www.stormforge.io/press-releases/stormforge-launches-v2-optimize-live-kubernetes-resource-optimization/)! ##### Added {#v2.0-added} - **New install method: Helm chart for stormforge-agent**\ You no longer need to download the StormForge CLI to install. Get up and running within minutes, and download the CLI later to run StormForge commands to manage your cluster. - **Optional Applier installation using a single Helm command**\ We separated the Agent and the Applier to simplify the permissions required at install.\ If you plan to apply configurations on demand (outside of a regular schedule) or automatically outside of a CI/CD workflow, be sure to install the Applier component. - **Control workload metrics collection with allow and deny lists**\ By default, Optimize Live collects metrics on all the workloads in the cluster.\ To restrict metrics collection to specific namespaces, you can provide a namespace allow list or deny list to when you install the agent. - **New UI workflows**\ **Workload List View:** See all workloads and the recommendations for those workloads, view cost estimates, and search against workload name, namespace, or cluster.\ **Workload Detail View**: View the impact of the recommendations on the workload, drill into the recommendation details, and get container-specific details. Export or download the patch from the UI. ##### Removed {#v2.0-removed} - **In-cluster components**\ We no longer install the recommender or time-series database (TSDB).