Experiments

What is an Experiment

An Experiment is the basic unit of organization in StormForge Optimize. The purpose of an experiment is to try different configurations of an application’s parameters and measure their impact. An experiment is composed of three primary concepts - Parameters, Trials, and Metrics.

Parameters

Parameters are the input variables to an experiment, i.e. what you want to tune in your application (for example cost, memory ,or number of replicas).

A parameter has a name, minimum value, maximum value, and optionally a baseline value. Taken together, an experiment’s parameters define the search space for the experiment: the set of all possible configurations the machine learning can explore. For in-depth explanation, see the parameters page.

Trials

A Trial is a single run of an experiment, with values assigned to every parameter.

An experiment typically consists of many trials. For in-depth explanation, see the trials page.

Metrics

A Metric is the output or outcome of a trial.

Metrics are used to measure the result of a particular set of parameters. A metric is a numeric value that an experiment attempts to minimize (like cost in dollars) or maximize (like throughput) by adjusting the values of parameters. For in-depth explanation, see the metrics page.

Review

  • Create an experiment to deploy and test an application
  • Make applications manifests tunable via parameters
  • Run trials with specific values assigned to each parameter
  • Assess the outcome of each trial by measuring one or more metrics

In the following sections we will explore these concepts in greater detail and guide you through the process of creating an experiment.

Last modified January 27, 2021