Frequently Asked Questions

Reports

What is Apdex?

Apdex (Application Performance Index) is an index that summarizes the performance of a system under test.

What is median?

Median is a statistical value, describing the measurement where 50% of all measured values lie below and 50% above. A median response time of 100 ms means that 50% of all requests have been completed within 100 ms and 50% took longer then 100 ms.

What is percentile?

A percentile (or a centile) is a measure used in statistics indicating the value below which a given percentage of observations in a group of observations fall. – https://en.wikipedia.org/wiki/Percentile

Example: Your test showed that your 95th percentile is 42 ms and your 99th percentile is 150 ms. This means that 95% of your requests were completed within 42 ms (and 5% took longer), and 99% of your requests took 150 ms or less (and only 1% of all requests took longer).

A word on Mean and Standard Deviation

Only looking at mean values for performance measurements is dangerous because you have no indication of how your actual response times are spread around the mean value.

In statistics, the standard deviation […] is a measure that is used to quantify the amount of variation or dispersion of a set of data values. A standard deviation close to 0 indicates that the data points tend to be very close to the mean (also called the expected value) of the set, while a high standard deviation indicates that the data points are spread out over a wider range of values. – https://en.wikipedia.org/wiki/Standard_deviation

Standard Deviation gives an indication how close the measured response times are to the mean. This gives you some idea how far spread your response times are.

We highly recommend to not use mean as your primary performance indicator. You should take a look at median, 95th and 99th percentile instead.

Making Requests

Which HTTP Headers are always sent during a test run?

By default we always send three headers: User-Agent, X-Stormforger-User and X-Request-Id. See the request tracing section for more details and how to customize these.

Which HTTP Headers can not be overwritten?

You cannot set headers beginning with X-StormForger-.

Why is the number of requests not exactly as defined?

There are several reasons for this.

Most likely the test duration is over before a client has made all its requests. It also appears if a client waits to long for a response and thus can not make the following requests.

Also take a look at Why did my test end early?

Are there any details to the requests defined in a test case?

Yes there are. On the TestCase detail page all defined sessions are listed in the sessions box below the test case summary. If you click on a session a more detailed view will expand. This view contains all defined requests as curl requests which you can and should validate by copying and pasting them in the shell of your choice.

Why are the defined requests not arriving at once in my system?

Although you define a rate in terms of new clients per second in the test case definition, the performance testing application uses a poisson distribution to launch the defined clients.

This is due to the goal of creating a more realistic behavior.

For more details check out the arrival phases reference.

Can I specify which session should use which target?

No you can’t. Each session uses all specified targets in the setTargets definition in a round robin approach. If you need to define different targets in specific requests you have to hardcode an absolute URL in the request like so:

session.get("https://api2.mytarget.com/login")

Miscellaneous

Where will the traffic/requests come from?

We currently only utilize resources from the Amazon AWS cloud. If you need to know what IP ranges will be used, consult the AWS documents on AWS IP Address Ranges. Test traffic will only originate from AWS IP addresses assigned to the EC2 service. Note that each test run is executed from a single region which is selectable by the user.

How can I check that traffic is coming from StormForge Performance Testing?

If you want to authenticate StormForge when accessing your testing system, we recommend one of the following:

1. Use basic auth (or client certificates)
2. Set a custom defined header
3. Filter on the standard headers we send

Note that we do not recommend restricting access to your test system only via the AWS IP ranges, as these IP ranges are quite larger and shared with all customers of AWS.

I am getting an URL is invalid error

StormForge Performance Testing is a tool to easily load test any HTTP API. If you get an error that a URL is invalid, but it looks correct, it might be because we maintain a block list for certain well-known, public services. To ensure you are only testing your own targets, we block known targets like Salesforce, AWS, Microsoft etc. by default. If you have explicit permission to load test these endpoints and services, customers can get in touch with our support to allow testing those domains.

Why did my test end early?

In general, your test will end if it reaches the defined duration (sum of all phases). But it might end before the defined duration.

Your test will end early (before the configured duration) if:

1. no more clients should be launched (max_clients has been reached in the last arrival phase) and
2. all previously launched clients have completed their session

A common reason for irritation comes from max_clients value in your arrival phase. If you launch 10 clients per second, but only allow 300 max_clients your test might end within ~30 seconds (if your session is short).

Let’s say you have a very short session (only one request, taking ~1sec). The following test will run for ~5 minutes and 30 seconds:

definition.setArrivalPhases([

  // Phase 1 will run for 5 minutes, launching 10 clients per second. It
  // will launch ~3000 clients.
  {
    duration: 5 * 60,
    rate: 10,
  },

  // Phase 2 will launch up to 300 clients (max_clients) with 10 clients per second.
  // After 30 seconds all 300 clients have been launched. If they complete
  // their requests, the test will end.
  {
    duration: 5 * 60,
    rate: 10,
    max_clients: 300
  },

]);

definition.session("very-quick", function(session) {
  session.get("/");
});

Why do I see more requests in my logs compared to your report?

We only analyze completed requests currently. So if your test ends before all requests have been completed you might see slightly more requests hitting your target compared to our reporting.

What is Preflight Mode?

Preflight mode means that you will get a fully functional test, but only very limited resources will be allocated to your test. Measurements and reports have to be taken with a grain of salt.

In preflight mode we limit the maximum amount of clients to 300 in your test case.

If you need more load you can send a request via the Request more power button.