Performance Testing

Performance Testing release history

2021 Q4

  • Added: When launching a Test Run via the Web UI users can select a custom Pool of Servers (if configured and eligibile).
  • Improved/Editor: Users may now select & copy text from the Test Case Editor in the Web UI.
  • Improved/Reports: Added CSV Export for Statistics by Request Tag on the Test Run Report.
  • Added: The Test Run Cockpit view display the Cluster Memory Utilization instead of individual Free Memory per Node.
  • Added: The Test Run Cockpit view displays an Overview of Arrival Phases (for past, current, future Phases) if more than 1 Arrival Phase is defined.
  • Changed: Owners may leave their own Organisation once there are no other members.
  • Added: The Organisation Manage Page displays configured custom Pools with detailed information (region, total/in-use IPs).
  • Improved/HAR Converter: Added smarter filtering for “Accept”-headers, so that intentional “Accept”-Headers are less likely to be auto-removed.
  • Added/TCDSL: Distributing clients via withProbability now supports dynamic user input.

2021 Q3

  • Added/TCDSL: Added MultiPartBuilder which adds support for Multipart Requests. Useful to to construct multipart (RFC 2388) payloads for requests.
  • Improved: A test case can now have multiple actively running test runs.
  • Improved/Docs: Added information with technical details about the concept of Arrival Rate in Arrival Phases.
  • Improved/Docs: Extended CLI guide with details on Building a Test Case with ECMAScript Modules.
  • Added/Docs: Added new guide with recommendations regarding Secret Management in Test Runs.
  • Improved/Reports: Added details to common Request/Response-related errors on the Test Run Report.
  • Improved/TCDSL: Added Picking Modes for Structured Data Sources (CSV files).
  • Improved/Traffic Dump: Request Tags are now accessible in the Traffic dump.
  • Improved/Docs: Added details about Test Run Report terms like Active Connections, New Connection Rate and more. See Test Run Details.
  • Added/Docs: Added docs for the TCDSL functions hmac_sha1 and hmac_sha256. See Functions: HMAC (SHA1, SHA256).
  • Added/CLI: New sub-commands to allow archiving/unarchiving Test Cases and Test Runs.
    • forge test-run archive --help
    • forge test-run unarchive --help
    • forge test-case archive --help
    • forge test-case unarchive --help
  • Added/TCDSL: Added the doWhile construct to support periodical conditional requests.
  • Improved/HAR Converter: Added option to filter “User-Agent Client Hints”-headers (sec-ch-ua).
  • Improved/TCDSL: Cookie Domain Mapping and DNS Mapping can now be used with dynamic values.

2021 Q2

  • Added/CLI: Added new command to create and list Service Accounts via the CLI. See forge serviceaccount --help.
  • Added/Docs: Added an explainer about the TCDSLs JavaScript execution, runtime and environment. See JavaScript Definition.
  • Changed: Validation runs do not count against daily Test Run limits anymore.
  • Added/TCDSL: To mimic clients who are using older versions of TLS, a Test Run may configure a specific TLS Version.
  • Added/CLI: A new sub-command which allows to abort all active/running Test Runs per Organisation. See forge test-run abort-all --help.
  • Added/CLI: A Test Run now accepts multiple --label-flags when launching. The labels values are accessible on the Test Run Report.
  • Fixed: Binary request bodies are now no longer included in traffic dumps.

2021 Q1

  • Added: Request traffic (egress) can now be rate limited.
  • Added: uuid4 function added.
  • Changed: Validation runs are now request rate limited.
  • Changed/CLI: Launching Test Runs with the --validate-flag automatically enables Traffic Dumping.
  • Improved/TCDSL: Better validation for response Content Extraction.
  • Improved: Significantly reduced time to validate and start very complex test cases.
  • Improved: Traffic dumps now include the dialed hostname in addition to the target IP.
  • Fixed: In rare cases the test run report failed to be generated automatically. No report was lost, but manual repair was required.
  • Fixed: In certain cases with preflight or tiny clusters the CPU utilization was not displayed correctly for running tests.
  • Fixed: Better validation for duplicate HTTP request headers (ignoring capitalization).
  • Fixed: Service Accounts are now only manageable by managers or owners of an organisation.
  • Fixed/JSONPath: Support boolean literals (true/false) for filter expressions.
  • Fixed/JSONPath: Support unequal operator (!=) for filter expressions.
  • Added/JSONPath: Added random() and length() functions.

2020 Q4

  • Added: Managers of an organisation can invite users to their organisation via E-Mail.
  • Added: Dedicated “Validate” Test Case Action:
    • Validating a Test Case launches a Test Run with a Preflight Cluster and enables Session Validation Mode and Traffic Dumping.
    • Traffic Dumping and Session Validation Mode are now options exclusively available to Validating a Test Case
  • Added: DNS Mapping to remap host names to different ones, see our documentation for more information.
  • Added: Arrival Phases can now have a name field which is displayed e.g. on reports.
  • Added: format_timestamp_as_iso_date allows converting a unix timestamp (seconds since 1970-01-01 to a ISO 8601/RFC 3339 date strings.
  • Improved: There is now a dedicated option to run a validation of a given test case. Traffic dumping is enabled by default for all validation runs.
  • Improved/TCDSL: session.waitUniform() now supports floats values.
  • Improved/TCDSL: JSONPath extractions now better support selecting keys of objects, see here for more details.
  • Improved: Reports now list used data sources directly.
  • Fixed/TCDSL: “random_numbers” are now allowed to have the lower and upper bound to be equal.
  • Fixed: In some cases, uploaded CSV files were not processed correctly in case they contained double quotes.

2020 Q3

  • Added: Users of your organisation can now be promoted to managers to allow multiple people to add and remove members.
  • Added: We now lock test cases for edits in the Web UI if it was lastly updated via our CLI. This was added to prevent accidential edits.
  • Added: You now can create Service Accounts to use for authentication using our CLI.
  • Added/Reports: You can now attach labels to test runs.
  • Added/TCDSL: Added several functions, e.g. to URL or Query-encode variables. See our documentation for more information.
  • Improved: For running tests, you can now see in which arrival phase the test is.
  • Improved: For running tests, you can now directly used configured integration links.
  • Improved: We updated various parts of our documentation.
  • Fixed: We removed language in our code bases around “{black,white}list”.
  • Fixed/TCDSL: setVar("foo", undefined) caused test runs to fail.
  • Fixed/Reports: “Statistics by Arrival Phase” is no longer shown for preflight tests.

2020 Q2

In case you are curious about our engine transition, you can check out our article on Load Testing Engine Evolution. Note that the jsonpath extraction behaves slightly different in some cases. This and other differences can be found in the engine reference.

  • Added/TCDSL: Using the new regexpheader you can extract data from response headers
  • Added/TCDSL: Cookie Domain Mapping
  • Added: We added graphs on “active connections” and “new connection rate” to the test run report
  • Added: A table with “other errors and events” is now shown in the report
  • Fixed: regexp results are no longer trimmed for whitespace
  • Fixed: The test case outline no longer incorrectly adds a --http1.1 option to the curl example
  • Improved/TCDSL: Validation of request headers is now more strict
  • Improved/TCDSL: cors: false is now supported as an request option to disable CORS (this is the default)
  • Improved: Validation Runs are now allowed in parallel to one normal test run per test case
  • Improved: Test Runs can now be aborted while they are still launching
  • Improved: Abnormally terminated test runs are now collected in a new list
  • Improved: Traffic Dumps filter out binary responses by replacing them with an omission marker
  • Improved: Graphs in test run reports are now faster to use (navigate and zoom)
  • Improved: Durations in the test report are now “human readable” and split into minutes, seconds and milliseconds (e.g. “1m 23s”) in case they exceed a certain threshold
  • Improved: “Other errors” are now available for filtering in the latency distribution section of the report
  • Improved/CLI: NFR validation and traffic dumping are no longer mutually exclusive
  • Improved/CLI: Using test-case create --update <file> will either create a new test case or update an existing one
  • Improved/CLI: There is now some more documentation on Getting Started with the forge CLI.
  • Improved/CLI: test-case launch now also allows to update the test case with a new JavaScript file in one operation by passing --test-case-file=<file>.

2020 Q1

  • Added: New session.redirectTarget() helper function for easier following of redirects (only available in the new engine)
  • Removed/TCDSL: Relative request paths are no longer allowed if multiple targets are defined. Before, a random target was chosen which lead to unexpected behaviour
  • Removed: Legacy data sources are no longer supported
  • Improved/HAR Converter:
    • improved scaffolded test case
    • added filter options for hosts and assets
    • allow to ignore global filter lists
    • more debug options
  • Improved: Various checks and timeouts have been added to ensure that test runs don’t get stuck (which rarely happened anyway). We also improved the time-to-test start significantly
  • Improved/TCDSL: Added more validation for CORS shorthand
  • Improved/TCDSL: Improved validation of incorrect JSONPath expressions
  • Improved/TCDSL: Improved validation for setTestOptions when configuring cluster types
  • Fixed: In certain cases the test run call log could not be downloaded
  • Fixed: Some validation errors were hidden uploading fixtures / data sources using the web UI
  • Fixed: Validation runs and preflight tests did not work in eu-north-1 (AWS Stockholm)

2019 Q4

  • Removed/TCDSL: Test Cases with more than one Target could use request paths to load balance traffic between these targets. Instead define multiple sessions (e.g. per target) with absolute URLs to imitate such load balancing.
  • Added: Allow variables for session.wait() and session.waitExp(), e.g. session.wait(session.getVar("retry_in"))
  • Added: Function timestamp to get the current time as a unix timestamp [docs]
  • Improved/Extractions: Better error messages for invalid JSONPath expressions for $[0]_id
  • Improved/Editor: ECMAScript 6 is now supported
  • Fixed: Allow single character tag, variable and check names
  • Fixed/TCDSL: Fixed validation for session.addTarget()
  • Fixed/SSL: Clients can now change the client certificate multiple times, e.g. to check resigning flows
  • Improved/Editor: Use CodeMirror for our web editor
  • Improved/TrafficDump: Traffic Dumping a session-validation test run now records all requests correctly
  • Improved/TCDSL: Improved validation around max_clients in the arrival-phases to alert on non-integers
  • Improved/TCDSL: Improve support for multi-line error messages
  • Fixed/Test-Run Dashboard: Do not show null requests during test run dashboard

2019 Q3

  • Added/TCSDL: Added “pinned TLS version” support (definition.setTestOptions({ special: { pinned_tls_version: "tlsv1.1" } });)
  • Improved/HAR Converter: Generated test cases contain more details about skipped requests, if blocked due to the HTTP method
  • Improved/HAR Converter: UI now links to the corresponding guide in the docs
  • Improved/Test-Cases: test case outline now shows the Request-ID header [docs]
  • Fixed/Test-Cases: auto terminate test run for rare cases when cluster failed to start
  • Fixed/Test-Cases: test cases can now be differed with the first version
  • Fixed/TCDSL: Fixed setAuthentication typo
  • Fixed/Test-Cases: Region selection contained wrong US state names for Oregon / Ohio

2019 Q2

  • Added/TCDSL: Added session.addTarget() [Docs]
  • Added/Data Sources: Show uploaded file history
  • Improved/Requests: additional_cookies structure validation has been improved
  • Improved/HAR Converter: OPTIONS requests are supported and no longer filtered
  • Improved/Test-CaseLaunch: Region selector now shows the AWS region ID first, then the name
  • Improved/Test-CaseUpload: Deny uploads with a zero file size
  • Improved/Test-Runs: Made handling of empty variables more robust
  • Fixed/Test-Case Launch: Skip disabled sessions in validation mode
  • Fixed/Latency Distribution: Fixed shown data being truncated when certain filters where applied
  • Fixed/Latency Distribution: Fixed error when not enough data is available
  • Fixed/Test-Cases: Fixed internal server error when showing aborted test runs with enabled traffic dumps
  • Fixed/HAR Converter: Generate valid variable names for csrf_token instead of illegal csrf-token

2019 Q1

  • Added: Configurable latency distribution analysis. You can filter requests based on arrival phase, tag and status and get insights on the response time distributions.
  • Added: A X-Request-ID header is now send with all requests. This can be configured (or disabled). The value is guaranteed to be unique per request.
  • Added: cookie extraction to extract the value of a Set-Cookie response header
  • Added: JavaScript definition can be downloaded for completed test runs
  • HAR Converter/Added: The conversion can now be performed in context of an organisation to allow for black and white-listed targets etc.
  • Added: session.withProbability() to execute steps with a given probability
  • Added: Test reports now contain free memory on load generators
  • Improved: Integration Links now support more timestamp variants
  • Improved: Test Case error stack traces are now truncated by default to hide internal frames
  • Improved: session.setVar(name, value) now allows for dynamic values
  • Improved: The test case outliner now includes data source pick, check and assert operations
  • Improved: Time to start a test run is now below 60 seconds for 98% of all tests (time from trigger to first client launched); 50% (median) start time is 39 seconds.
  • HAR Converter/Improved: Detection of XMLHTTPRequests (XHR) has been improved
  • HAR Converter/Improved: Better support for non-browser HAR recordings like Charles Proxy
  • Fixed: Path substitutions were not applied for CORS request
  • Fixed: In certain situations checks based on session.lastHttpStatus() would not fail even if the request failed
  • Fixed: Allow request payload for DELETE requests, see RFC7231 section 4.3.5
  • Fixed: session.chooseBy{Probability,Weight}() now have improved validations
  • Fixed: In certain situations request options passed to session.{get,post,…}() could be mutated leading to unindented side-effects.
  • HAR Converter/Fixed: CONNECT requests are now filtered correctly

2018 Q4

  • CLI/Added: Add //#include comment to allow simple inclusion of test case snippets (https://github.com/stormforger/cli/pull/58)
  • CLI/Fixed: Improved error messages for test case evaluation errors (https://github.com/stormforger/cli/pull/57)
  • Fixed: Sorting of test case list was broken
  • Fixed: In certain situations, the test case overview showed the same target multiple times
  • Fixed: For test runs started in Session Validation Mode, the “Archive all Preflight” action did not work
  • Fixed: For CORS request, URL params were not properly substituted
  • Fixed: Validations were added for session.if expressions to make sure the variable name is correct

2018 Q3

  • Added new launch option for test runs: You can now overwrite configuration options like cluster sizing and region as well as other, mostly debugging related options. This is available when launching tests via the web UI as well as the command line tool.
  • Added session validation mode: This will ignore arrival phases and cluster configuration and instead launch a single user per defined scenario one after another. This can be used to verify that a test case works properly.
  • CLI: Added various new options around launching test runs and listing resources. See README or use --help for more details.
  • Added Traffic Dump option for test runs: You can now request a traffic dump for a preflight test for debugging purposes
  • Added: Organisations may now be renamed
  • Fixed HEAD requests. Under certain conditions session.head(…) requests caused clients to hang for up to 60s. This has now been resolved.
  • Fixed: JSONPath expressions may now contain dynamic parts
  • Fixed: session.abort() did not always set an abort reason properly
  • Fixed: The value in session.if(…) may now be dynamic. It had to be a static value before.

2018 Q2

  • NOTE We changed the URL schema for many resources. Previous links should continue to work. The new URLs are more consistent and are easier to share.
  • Added session.check(), session.abort() and session.assert(): Session Checks (OK/KO criteria) documentation for more details
  • Added non-functional requirements (NFR) check feature, see our Non-functional Requirements documentation
  • Added visualiation of test cluster utilization to reports
  • Enabled ability to (soft) delete file fixtures
  • CLI: Refactored all commands to work with organisation/test-case/test-run-id as references, see https://github.com/stormforger/cli#usage for usage information

2018 Q1

  • Added converter feature for legacy file fixtures to help with migration to our new data source API, see File Fixture Migration Guide
  • Added average session duration to report (only clients that finished)
  • Added sharable reports: You can now create a read-only share link for a test run report
  • Added scheduled execution of test runs
  • Added session.lastHttpStatus() to get the last HTTP status code
  • Deprecated session.matchedValue() (use session.getVar() now)
  • Various releases of our forge CLI (including support to manage test cases, start/stop/monitor test runs), check out https://github.com/stormforger/cli/releases for details

2017 Q4

  • Reworked test case data source API and management of file fixtures, see /reference/data-sources/ for details and /reference/data-sources/#migration for information on how to migrate your existing test cases
  • Various releases of our forge CLI (including support to manage test cases, start/stop/monitor test runs), check out https://github.com/stormforger/cli/releases for details
  • HAR Converter: Improved handling of HTTP2 pseudo headers (they get filtered out now)
  • Test Run Reports: Improved loading speed of data for charting
  • Fixed an issue that could cause a test run analysis to take longer then necessary
  • Fixed an issue with definition.setSessionWeights: In case you define weights, sessions not explicitly listed now get a weight of 0 instead of 1
  • Fixed an issue with the test case outliner: Random strings were truncated after 33 characters. This had no impact on test executions.

2017 Q3

  • Added session.defaults to set request option defaults: /reference/requests/#setting-defaults
  • Added additional_cookies request option to set additional cookies: /reference/requests/#cookies
  • Added session.waitExp() as an alias to session.wait({ random: true }): /reference/sessions/#waitexp
  • Added Markdown support for test case and test run descriptions
  • Added an indication in the editor when the test case has been changed
  • Fixed an issue which prevented the usage of whitespace in JSONPath expressions
  • Improved asset request detection in HAR converter

2017 Q2

  • Added more AWS regions from which you can run tests: /reference/test-options/#cluster-region
  • Fixed issue with wait steps in loops
  • Fixed validation for data source names (currently no dashes allowed)
  • Fixed a rare case where the organisation management page would cause an error
  • Improved detection of problems for newly launched test runs
  • Improved automatic tag generation for HAR converter

2017 Q1

2016 Q4

  • Added support for bandwidth rate limiting, see /reference/sessions/#rate-limiting
  • Added support for authentication via GitHub (sign up and sign in)
  • Added support for dynamic field names for application/x-www-form-urlencoded payloads
  • Added support for uniformly distributed wait times via waitUniform(min, max), see /reference/sessions/#waituniform
  • Added error/warning event, when if is used to compare to an undefined value
  • Improved performance of test run index page when there are many test runs

2016 Q3

  • Added organisation management to manage your memberships and teams
  • Added realtime telemetry on response code distribution, CPU & memory usage of the test cluster during test runs
  • Added validation for JSONPath expressions
  • Added request batch option to test case definitions
  • Added branching by probability for test case definitions
  • Added support for negative random numbers
  • BREAKING Added option to use weights instead of percent probabilities for client sessions, /reference/sessions/#session-likelihood
  • Improved HTTP response header extraction, /reference/content-extraction/#http-response-header
  • Improved Slack notification during test runs
  • Updated inline documentation

2016 Q2

  • Significantly speed up Test Run analysis
  • Improved API and made Closed-Beta™ version available
  • new and highly improved search for documentation
  • added basic support for TLS client certificates

2016 Q1

  • Added a bulk of new content to our documentation at https://docs.stormforge.io/perftest
  • Added limit of concurrent tests per target host (can be raised on request)
  • Test cases and runs can now be archived
  • Improved UI for launching and running test runs
  • Various table views throughout the app can now be sorted and filtered
  • Significantly improved charting performance for longer test runs
  • Fixed issue with URL encoding of payloads containing data sources

2015 Q4

  • Fixed issue with test runs getting sometimes stuck in “analyzing” state
  • Improved feedback on various sessions and file fixture related errors
  • Fixed error reporting of incorrect probabilities of sessions
  • Introducing our first, consolidated documentation at https://docs.stormforge.io/perftest
  • Improved test cluster resource allocation
  • Improved test case editor shortcuts
  • Reworked life validation and error messages for test case editing
  • Fixed issue with payloads getting ignored for PUT and PATCH requests
  • Reworked UX and UI around active test runs and test cases
  • Fixed an issue concerning: under certain conditions traffic metrics were not calculated correctly
  • Open sourced a rough version of an HAR to JavaScript DSL Test Case Definition converter.

2015 Q3

  • Improved file fixture management
  • Added a simple counter for matching response contents (useful for debugging)
  • Added reporting of non-HTTP errors and events
  • Added our current Terms of Service
  • Added Intercom as our Chat and Support solution to assist customers even better
  • Improved test run report stacked graphs

2015 Q2

  • Major UI rework and improvements
  • Fixed an issue with data source usage when using control structures
  • Introducing “Preflight Mode” to test and debug test cases
  • Starting with Open Beta! WOOHOOO \o/