Trial Template
5 minute read
A trial is a single run of an experiment. This includes a given set of parameters that will be measured against the metrics defined in the experiment.
Each trial consists of two components: the trial job spec and an optional set of setup tasks.
Trial Spec
The trial spec is composed of a list setup tasks and a trial job. While the trial spec itself is optional, it is strongly recommended to make use of the trial job.
Additional details can be found in the trial API reference.
Trial Job
A trial job controls the duration of the parameters that are being tested. This is commonly represented as a load testing tool such as StormForge, Locust, or pgbench. This can also be a simple script that waits for another resource to complete.
A trial job is launched after all patches have been applied and all setup tasks are complete.
If no trial job is specified, a default trial job will be launched that will sleep for a period of seconds.
The following example will run a pgbench pod for each trial run.
spec:
trialTemplate:
spec:
jobTemplate:
spec:
template:
spec:
containers:
- image: crunchydata/crunchy-pgbench:centos7-11.4-2.4.1
name: pgbench
Setup Tasks
Setup tasks allow you to perform additional work that is needed for each trial run, such as deploying a Helm chart, launching a Prometheus instance, or deploying an operator.
Setup task pods will also have additional environment variables presented for each trial run.
These environment variables contain all of the current parameter values and the current mode (create or delete).
Parameter environment variables are presented as uppercase names of their respective parameters.
For example, if there is a parameter named memory
, it will be presented to the pod as MEMORY
.
Helm
A Helm setup task allows you to deploy a Helm chart as part of each trial run.
The Helm setup task should include the chart name, a repository, and either a set of Helm values inline or supplied through a configmap.
When Helm values are supplied inline, the value
field is evaluated as a Go template which allows you to consume the supplied parameters through the .Values
struct.
An example of a Helm setup task can be seen in our elasticsearch example:
spec:
trialTemplate:
spec:
setupTasks:
- name: elasticsearch
helmChart: elasticsearch
helmChartVersion: 7.9.2
helmRepository: https://helm.elastic.co
helmValues:
- name: cluster.name
value: rally-demo
- name: replicas
value: "{{ .Values.replicas }}"
- name: resources.limits.cpu
value: "{{ .Values.cpu }}m"
- name: resources.limits.memory
value: "{{ .Values.memory }}Mi"
- name: resources.requests.cpu
value: "{{ .Values.cpu }}m"
- name: resources.requests.memory
value: "{{ .Values.memory }}Mi"
- name: esJavaOpts
value: "-Djava.net.preferIPv4Stack=true -Xms{{ percent .Values.memory .Values.heap_percent }}m -Xmx{{ percent .Values.memory .Values.heap_percent }}m"
- name: persistence.enabled
value: "false"
- name: antiAffinity
value: soft
Prometheus
The Prometheus setup task enables you to automatically deploy a Prometheus instance for each trial run. This includes a Prometheus server, kube state metrics, push-gateway, and configmap reloader.
The bundled Prometheus instance scrapes a very limited set of targets: nodes, Kube State Metrics, and the Pushgateway. The included configuration is enough to provide the ability to capture resource requests and limits as well as CPU and memory utilization measurements.
spec:
trialTemplate:
spec:
setupTasks:
- name: prometheus
helmChart: oci://registry.stormforge.io/optimize-pro/prometheus
If your experiment requires a Prometheus instance, take note of the following requirements for this setup task:
-
Deploying a bundled (simplified) Prometheus instance is completed via a Helm chart available at
oci://registry.stormforge.io/optimize-pro/prometheus
. If your environment is air-gapped, you must download the chart to your own chart repository and set helmChart to your repo. -
The Prometheus setup task requires additional RBAC permissions to function.
To generate the RBAC manifests, run:
stormforge rbac --namespace NAMESPACE EXPERIMENT_NAME.yaml > EXPERIMENT_NAME-rbac.yaml
Then, make sure you create the required RBAC resources prior to creating the experiment itself:
kubectl apply -f EXPERIMENT_NAME-rbac.yaml -f EXPERIMENT_NAME.yaml
Note
If you have copied theprometheus
helm chart into your own registry, you need to set the OPTIMIZE_PRO_PROMETHEUS_CHART
environment variable to your custom helmChart
value. This enables stormforge rbac
to generate the correct RBAC rules.
When using the setupTask
to your experiment, any metric with type: prometheus
will default the url
field to
the temporary prometheus instance provisioned for this trial.
Readiness Gates
Readiness gates provide the ability to wait for additional conditions to be present before creating the trial job.
spec:
trialTemplate:
spec:
readinessGates:
- kind: Deployment
apiVersion: apps/v1
name: postgres
conditionTypes:
- Available
periodSeconds: 5
failureThreshold: 10
Readiness Gate Example
You’ll use the included Locust load tester from the Online Boutique example.
The example includes a Readiness Gate to ensure that the load tester is available before starting the trial.
The trial job consists of a small shell script that resets the load tester stats, kicks off a new load test, waits 3 minutes, and then stops the load test.
spec:
trialTemplate:
spec:
readinessGates:
- kind: Deployment
apiVersion: apps/v1
name: loadgenerator
conditionTypes:
- stormforge.io/app-ready
InitialDelaySeconds: 5
periodSeconds: 5
failureThreshold: 30
jobTemplate:
spec:
template:
spec:
containers:
- name: main
image: ghcr.io/thestormforge/setuptools:edge
command:
- /bin/sh
args:
- -c
- |
curl http://locust/stats/reset && \
sleep 5 && \
curl -X POST -F 'user_count=20' -F 'hatch_rate=10' http://locust/swarm && \
sleep 180 && \
curl http://locust/stop && \
curl http://locust/stats/requests