Software Testing Social Network

In this second article of our three-part series on testing for performance we look at building our test assets and the work required to support that effort. As a reminder, the Testing for Performance series is broken into the following parts:

      * Assess the problem space: Understand your content, the system and figure out where to start

    * Build out the test assets: Stage the environments, identify data, building out the scripts and calibrate your tests

    * Provide information: Run your tests, analyze results, make them meaningful to the team and work through the  tuning process

  As we look at the various environments, tools, data and scripts we use to build our tests, we will try to tie those artifacts back to the work that we did in the first article when we assessed the problem space. Often, as we build out the test assets, our understanding of the problem space changes, becoming more (or sometimes less) detailed, and often we are required to reconsider some of the decisions we made before we got down into the details.

Understand and manage your tools and test environments

The first thing we need to look at, normally before starting to build out any test assets, is the environment and tooling. It is seen that very talented performance testers present results that weren't useful because they tested using the wrong version of the application. It is also also seen that project teams identify a performance problem only to discover that they had no way to isolate and debug it. And, unfortunately, there are times, where you couldn't execute your tests on time because your performance tool environment couldn't go more than a day without throwing errors or losing results.

In each of those instances, the team let one of three key environment factors get away from them. They didn't correctly identify their needs up front, didn't build or buy the right tools for their problem, miss-configured something, or failed to coordinate their testing. The three key areas where I've seen teams fall apart include managing the system/application being tested, managing the tools to support monitoring and debugging, and managing the tools to support performance test execution. For each of those areas, you must identify what you need, install the required software and hardware, configure it to fit your project and then coordinate how it gets used going forward.

In the following table, one way of thinking about how to manage your tools and test environments is shown. It's only an illustration of some of the questions you might ask.

Depending on what you are performance-testing and how you're testing it, you may have a much larger list. Building a matrix like this up front may help you and your team members think about everything you're going to need to be successful. Some of these questions will be answered with purchases, others with processes, and others may be left unanswered (and that may be OK).