Optimize job-type workloads

If you run CronJobs, Jobs, or Spark applications, StormForge doesn’t recognize them as workloads until you declare them in workloadResourceKinds. The Agent doesn’t watch these kinds by default, because job-type workloads have requirements beyond those of long-running workloads. You can declare four kinds: Kubernetes CronJobs and Jobs, and the Spark operator’s ScheduledSparkApplications and SparkApplications.

StormForge generates recommendations for all four kinds, but the Applier applies them only to the scheduled kinds: CronJobs and ScheduledSparkApplications. For the one-off kinds, Jobs and SparkApplications, you apply the recommended values yourself.

Declare the kinds

The Agent can only see the kinds you declare. For the full field reference, see Custom workload kinds.

CronJob

workloadResourceKinds:
- apiGroup: batch
  kind: CronJob
  resource: cronjobs

Also declare Job, as described under Job. The Agent reaches a CronJob through the Jobs it creates, so without a Job entry it can’t find the CronJob.

ScheduledSparkApplication

workloadResourceKinds:
- apiGroup: sparkoperator.k8s.io
  kind: ScheduledSparkApplication
  resource: scheduledsparkapplications

As with CronJobs, also declare the kind it creates: SparkApplication, as described under SparkApplication.

Job

workloadResourceKinds:
- apiGroup: batch
  kind: Job
  resource: jobs
  role: conditionalWorkload

conditionalWorkload makes a Job act as a workload when it stands alone, and defer to its owner when a CronJob created it.

To have recommendations applied to Jobs, run them from a CronJob, or declare the longer-lived object that creates them, as described under Apply the pattern to other job-like kinds. For standalone Jobs, see Group standalone runs.

SparkApplication

workloadResourceKinds:
- apiGroup: sparkoperator.k8s.io
  kind: SparkApplication
  resource: sparkapplications
  role: conditionalWorkload

The same pattern as Job: a workload when submitted directly, deferring to its owner when a ScheduledSparkApplication created it. To have recommendations applied to Spark runs, schedule them through a ScheduledSparkApplication. For SparkApplications submitted directly, see Group standalone runs.

Optimize scheduled jobs

A CronJob keeps one name, and each run is created from its job template, so its usage accumulates across runs and it stays available as a patch target between them. Keep the default apply method: the patch lands on the job template and takes effect on the next run. StormForge knows where a CronJob holds its Pod template, so no patchTargetDefaults is needed.

A ScheduledSparkApplication behaves the same way: it keeps a stable name, is patched between runs, and uses the default apply method.

StormForge optimizes only the executor (the spark-kubernetes-executor container) and leaves the driver and any sidecars unchanged. It sets the executor’s coreRequest, coreLimit, and memory, but not a memory limit, because the SparkApplication executor spec has no memory limit field.

Enable auto-deploy

Enable auto-deploy to have the Applier apply recommendations to CronJobs and ScheduledSparkApplications automatically:

live.stormforge.io/auto-deploy: "Enabled"

Set it at the cluster, namespace, or workload level. Automatic deployment begins once the workload’s learning period is complete. For the full setting reference, see Auto-deploy.

Run requirements

StormForge treats only CronJobs and ScheduledSparkApplications as recurring batch jobs. These kinds produce a recommendation only when each run is observed for long enough, and when successive runs accumulate into a single history.

Runs shorter than 10 minutes often don’t produce enough usage data to work from. There is no exact cutoff, and a single run of 10 minutes may not be enough on its own, because usage accumulates across runs.

When runs aren’t observed for long enough, the recommendation fails to generate with the reason ShortBatchJobs, and the workload is reported as Unsupported.

The time between runs isn’t held against these kinds. Usage is collected from the Pods a workload runs, so periods when it has no Pods contribute nothing either way. A job that runs for fifteen minutes once an hour is measured on those fifteen minutes. What has to add up is observed running time, not elapsed wall-clock time.

Temporary memory bump-ups after an OOM event aren’t available for CronJobs or ScheduledSparkApplications, as described in Reliability.

Group standalone runs

StormForge can’t apply recommendations to standalone Jobs or SparkApplications. Instead, group their runs and then apply the values yourself.

Each standalone Job or SparkApplication runs its Pods exactly once. A recommendation can only be produced after StormForge has observed that run, by which point the Pods have already started and the object that created them will never create any more. A later run is a separate workload under a new name.

A Job is immutable once created, so StormForge resolves no patch path for batch/v1 Jobs and marks the recommendation unpatchable.

How standalone runs are measured

The guidance under Run requirements applies only to CronJobs and ScheduledSparkApplications. StormForge treats standalone Jobs and SparkApplications the same way as every other workload kind, such as Deployments, so they never show ShortBatchJobs. Each run still needs to last long enough for StormForge to collect usage data.

Size later runs

Without a workload group, a recommendation reflects a single run: each run creates a workload with a new name and no history, so nothing accumulates. A group pools the Pods from every matching run, so the recommendation reflects usage across many runs rather than one.

1. Create a workload group that matches the runs. Groups can match on labels, on name patterns, or on explicit workload identity. Workload groups covers the full set of options, and Ephemeral workloads explains how to configure runs that get a new name each time.
2. After a run completes, check the group’s recommendation.
3. Copy the recommended requests and limits into the manifest, Helm chart, or pipeline that defines the Jobs’ Pod template.

Troubleshoot job workloads

To see how StormForge has resolved a workload, and which workloads its group contains:

stormforge describe workload <workload>

Apply the pattern to other job-like kinds

The same scheduled vs. one-off split applies to any workload that runs to completion. Ask whether the kind owns Pods directly, or is templated by a longer-lived object that outlives each run:

• A scheduler or template object keeps a stable identity and accumulates usage. Declare it workload and keep the default apply method.
• A per-run object exists only while the run does, and creates its Pods once. Declare it conditionalWorkload and group its runs. Expect these to be observable only, for the reasons described under Group standalone runs.

The measurement rules don’t carry over. StormForge applies the recurring batch job rules to CronJob and ScheduledSparkApplication only, so any other kind is measured like a long-running workload whichever role you give it. See Run requirements.