What is an Experiment
An Experiment is the basic unit of organization in StormForge Optimize. The purpose of an experiment is to try different configurations of an application’s parameters and measure their impact. An experiment is composed of three primary concepts: Parameters, Metrics, and Trials.
Parameters are the input variables to an experiment. They represent what you want to tune in your application (such as cost, memory, or number of replicas).
A parameter has a name, minimum value, maximum value, and optionally a baseline value. Taken together, an experiment’s parameters define the search space for the experiment: the set of all possible configurations the machine learning can explore. For details, see Parameters.
Metrics are the output variables in an experiment. They represent how you measure the performance of your application.
Metrics are taken at the end of every trial to measure the result of using a particular set of parameters. A metric is a numeric value that an experiment attempts to minimize (such as cost in dollars) or maximize (such as throughput) by adjusting the values of parameters. For details, see Metrics.
A Trial is a single run of an experiment, with specific values assigned to every parameter and results queried for every metric.
An experiment typically consists of many trials. For details, see Trials.
- Create an experiment to deploy and test an application.
- Make applications manifests tunable via parameters.
- Run trials with specific values assigned to each parameter.
- Assess the outcome of each trial by measuring one or more metrics.
In the following sections, you can explore these concepts in more detail and walk through the process of creating an experiment.