Trial Template

5 minute read

A trial is a single run of an experiment. This includes a given set of parameters that will be measured against the metrics defined in the experiment.

Each trial consists of two components: the trial job spec and an optional set of setup tasks.

Trial Spec

The trial spec is composed of a list setup tasks and a trial job. While the trial spec itself is optional, it is strongly recommended to make use of the trial job.

Additional details can be found in the trial API reference.

Trial Job

A trial job controls the duration of the parameters that are being tested. This is commonly represented as a load testing tool such as StormForge, Locust, or pgbench. This can also be a simple script that waits for another resource to complete.

A trial job is launched after all patches have been applied and all setup tasks are complete.

If no trial job is specified, a default trial job will be launched that will sleep for a period of seconds.

The following example will run a pgbench pod for each trial run.

spec:
  trialTemplate:
    spec:
      jobTemplate:
        spec:
          template:
            spec:
              containers:
              - image: crunchydata/crunchy-pgbench:centos7-11.4-2.4.1
                name: pgbench

Setup Tasks

Setup tasks allow you to perform additional work that is needed for each trial run, such as deploying a Helm chart, launching a Prometheus instance, or deploying an operator.

Setup task pods will also have additional environment variables presented for each trial run. These environment variables contain all of the current parameter values and the current mode (create or delete). Parameter environment variables are presented as uppercase names of their respective parameters. For example, if there is a parameter named memory, it will be presented to the pod as MEMORY.

Helm

A Helm setup task allows you to deploy a Helm chart as part of each trial run.

The Helm setup task should include the chart name, a repository, and either a set of Helm values inline or supplied through a configmap. When Helm values are supplied inline, the value field is evaluated as a Go template which allows you to consume the supplied parameters through the .Values struct.

An example of a Helm setup task can be seen in our elasticsearch example:

spec:
  trialTemplate:
    spec:
      setupTasks:
      - name: elasticsearch
        helmChart: elasticsearch
        helmChartVersion: 7.9.2
        helmRepository: https://helm.elastic.co
        helmValues:
        - name: cluster.name
          value: rally-demo
        - name: replicas
          value: "{{ .Values.replicas }}"
        - name: resources.limits.cpu
          value: "{{ .Values.cpu }}m"
        - name: resources.limits.memory
          value: "{{ .Values.memory }}Mi"
        - name: resources.requests.cpu
          value: "{{ .Values.cpu }}m"
        - name: resources.requests.memory
          value: "{{ .Values.memory }}Mi"
        - name: esJavaOpts
          value: "-Djava.net.preferIPv4Stack=true -Xms{{ percent .Values.memory .Values.heap_percent }}m -Xmx{{ percent .Values.memory .Values.heap_percent }}m"
        - name: persistence.enabled
          value: "false"
        - name: antiAffinity
          value: soft

Prometheus

The Prometheus setup task enables you to automatically deploy a Prometheus instance for each trial run. This includes a Prometheus server, kube state metrics, push-gateway, and configmap reloader.

The bundled Prometheus instance scrapes a very limited set of targets: nodes, Kube State Metrics, and the Pushgateway. The included configuration is enough to provide the ability to capture resource requests and limits as well as CPU and memory utilization measurements.

spec:
  trialTemplate:
    spec:
      setupTasks:
      - name: prometheus
        helmChart: oci://registry.stormforge.io/optimize-pro/prometheus

If your experiment requires a Prometheus instance, take note of the following requirements for this setup task:

Deploying a bundled (simplified) Prometheus instance is completed via a Helm chart available at oci://registry.stormforge.io/optimize-pro/prometheus. If your environment is air-gapped, you must download the chart to your own chart repository and set helmChart to your repo.
The Prometheus setup task requires additional RBAC permissions to function.

To generate the RBAC manifests, run:
```
stormforge rbac --namespace NAMESPACE EXPERIMENT_NAME.yaml > EXPERIMENT_NAME-rbac.yaml
```
Then, make sure you create the required RBAC resources prior to creating the experiment itself:
```
kubectl apply -f EXPERIMENT_NAME-rbac.yaml -f EXPERIMENT_NAME.yaml
```

Note

If you have copied the prometheus helm chart into your own registry, you need to set the OPTIMIZE_PRO_PROMETHEUS_CHART environment variable to your custom helmChart value. This enables stormforge rbac to generate the correct RBAC rules.

When using the setupTask to your experiment, any metric with type: prometheus will default the url field to the temporary prometheus instance provisioned for this trial.

Readiness Gates

Readiness gates provide the ability to wait for additional conditions to be present before creating the trial job.

spec:
  trialTemplate:
    spec:
      readinessGates:
      - kind: Deployment
        apiVersion: apps/v1
        name: postgres
        conditionTypes:
        - Available
        periodSeconds: 5
        failureThreshold: 10

Readiness Gate Example

You’ll use the included Locust load tester from the Online Boutique example.

The example includes a Readiness Gate to ensure that the load tester is available before starting the trial.

The trial job consists of a small shell script that resets the load tester stats, kicks off a new load test, waits 3 minutes, and then stops the load test.

spec:
  trialTemplate:
    spec:
      readinessGates:
      - kind: Deployment
        apiVersion: apps/v1
        name: loadgenerator
        conditionTypes:
        - stormforge.io/app-ready
        InitialDelaySeconds: 5
        periodSeconds: 5
        failureThreshold: 30
      jobTemplate:
        spec:
          template:
            spec:
              containers:
              - name: main
                image: ghcr.io/thestormforge/setuptools:edge
                command:
                - /bin/sh
                args:
                - -c
                - |
                  curl http://locust/stats/reset && \
                  sleep 5 && \
                  curl -X POST -F 'user_count=20' -F 'hatch_rate=10' http://locust/swarm && \
                  sleep 180 && \
                  curl http://locust/stop && \
                  curl http://locust/stats/requests

Last modified June 9, 2023