| Article Index |
|---|
| Performance Testing: Fundamentals and Best Practices |
| Page 2 |
| Page 3 |
| All Pages |
Performance testing
in general terms is testing that is performed, from one perspective, to determine how fast some aspect of a system performs under a particular workload. It can also serve to validate and verify other quality attributes of the system, such as scalability, reliability and resource usage. Performance testing is a subset of Performance Engineering, an emerging computer science practice that strives to build performance into the design and architecture of a system, prior to the onset of actual coding effort.Performance testing can serve different purposes. It can demonstrate that the system meets performance criteria. It can compare two systems to find which performs better. Or it can measure what parts of the system or workload cause the system to perform badly. In the diagnostic case, software engineers use tools such as profilers to measure what parts of a device or software contribute most to the poor performance or to establish throughput levels (and thresholds) for maintained acceptable response time. It is critical to the cost performance of a new system, which performance test efforts begin at the inception of the development project and extend through to deployment. The later a performance defect is detected, the higher the cost of remediation. This is true in the case of functional testing, but even more so with performance testing, due to the end-to-end nature of its scope.
In performance testing, it is often crucial (and often difficult to arrange) for the test conditions to be similar to the expected actual use. This is, however, not entirely possible in actual practice. The reason is that production systems have a random nature of the workload and while the test workloads do their best to mimic what may happen in the production environment, it is impossible to exactly replicate this workload variability - except in the simplest system.
Loosely coupled architectural implementations (e.g.: SOA) have created additional complexities with performance testing. Enterprise services or assets (that share common infrastructure or platform) require coordinated performance testing (with all consumers creating production-like transaction volumes and load on shared infrastructures or platforms) to truly replicate production-like states. Due to the complexity and financial and time requirements around this activity, some organizations now employ tools that can monitor and create production-like conditions (also referred as "noise") in their performance testing environments (PTE) to understand capacity and resource requirements and verify / validate quality attributes.
To sum up above discussion, Performance testing is a type of testing intended to determine the responsiveness, throughput, reliability, and/or scalability of a system under a given workload. Performance testing is commonly conducted to accomplish the following:
* Assess production readiness
* Evaluate against performance criteria
* Compare performance characteristics of multiple systems or system configurations
* Find the source of performance problems
* Support system tuning
* Find throughput levels
Performance Testing: its Core Activities
Performance testing is typically done to help identify bottlenecks in a system, establish a baseline for future testing, support a performance tuning effort, determine compliance with performance goals and requirements, and/or collect other performance-related data to help stakeholders make informed decisions related to the overall quality of the application being tested. In addition, the results from performance testing and analysis can help you to estimate the hardware configuration required to support the application(s) when you "go live" to production operation.
The performance testing approach consists of the following activities:
1. Activity 1. Identify the Test Environment. Identify the physical test environment and the production environment as well as the tools and resources available to the test team. The physical environment includes hardware, software, and network configurations. Having a thorough understanding of the entire test environment at the outset enables more efficient test design and planning and helps you identify testing challenges early in the project. In some situations, this process must be revisited periodically throughout the projectÙs life cycle.
2. Activity 2. Identify Performance Acceptance Criteria.
Identify the response time, throughput, and resource utilization goals and constraints. In general, response time is a user concern, throughput is a business concern, and resource utilization is a system concern. Additionally, identify project success criteria that may not be captured by those goals and constraints; for example, using performance tests to evaluate what combination of configuration settings will result in the most desirable performance characteristics.3. Activity 3. Plan and Design Tests. Identify key scenarios, determine variability among representative users and how to simulate that variability, define test data, and establish metrics to be collected. Consolidate this information into one or more models of system usage to be implemented, executed, and analyzed.
4
. Activity 4. Configure the Test Environment. Prepare the test environment, tools, and resources necessary to execute each strategy as features and components become available for test. Ensure that the test environment is instrumented for resource monitoring as necessary.5. Activity 5. Implement the Test Design. Develop the performance tests in accordance with the test design.
6. Activity 6. Execute the Test. Run and monitor your tests. Validate the tests, test data, and results collection. Execute validated tests for analysis while monitoring the test and the test environment.
7. Activity 7. Analyze Results, Report, and Retest.
Consolidate and share results data. Analyze the data both individually and as a cross-functional team. Reprioritize the remaining tests and re-execute them as needed. When all of the metric values are within accepted limits, none of the set thresholds have been violated, and all of the desired information has been collected, you have finished testing that particular scenario on that particular configuration.The Significance of Performance Testing
At the highest level, performance testing is almost always conducted to address one or more risks related to expense, opportunity costs, continuity, and/or corporate reputation. Some more specific reasons for conducting performance testing include:
* Assessing release readiness by:
o Enabling you to predict or estimate the performance characteristics of an application in production and evaluate whether or not to address performance concerns based on those predictions. These predictions are also valuable to the stakeholders who make decisions about whether an application is ready for release or capable of handling future growth, or whether it requires a performance improvement/hardware upgrade prior to release.
o Providing data indicating the likelihood of user dissatisfaction with the performance characteristics of the system.
o Providing data to aid in the prediction of revenue losses or damaged brand credibility due to scalability or stability issues, or due to users being dissatisfied with application response time.
* Assessing infrastructure adequacy by:
o Evaluating the adequacy of current capacity. o Determining the acceptability of stability.
o Determining the capacity of the applicationÙs infrastructure, as well as determining the future resources required to deliver acceptable application performance.
o Comparing different system configurations to determine which works best for both the application and the business.
o Verifying that the application exhibits the desired performance characteristics, within budgeted resource utilization constraints.
* Assessing adequacy of developed software performance by:
o Determining the applicationÙs desired performance characteristics before and after changes to the software.
o Providing comparisons between the applicationsÙs current and desired performance characteristics.
* Improving the efficiency of performance tuning by:
o Analyzing the behavior of the application at various load levels.
o Identifying bottlenecks in the application.
o Providing information related to the speed, scalability, and stability of a product prior to production release, thus enabling you to make informed decisions about whether and when to tune the system.
Parameters for a successful Performance Testing Project
For a performance-testing project to be successful, both the approach to testing performance and the testing itself must be relevant to the context of the project. Without an understanding of the project context, performance testing is bound to focus on only those items that the performance tester or test team assumes to be important, as opposed to those that truly are important, frequently leading to wasted time, frustration, and conflicts.
The project context is nothing more than those things that are, or may become, relevant to achieving project success. This may include, but is not limited to:
* The overall vision or intent of the project
* Performance testing objectives
* Performance success criteria
* The development life cycle
* The project schedule
* The project budget
* Available tools and environments
* The skill set of the performance tester and the team
* The priority of detected performance concerns
* The business impact of deploying an application that performs poorly
Some examples of items that may be relevant to the performance-testing effort in your project context include:
* Project vision. Before beginning performance testing, ensure that you understand the current project vision. The project vision is the foundation for determining what performance testing is necessary and valuable. Revisit the vision regularly, as it has the potential to change as well.
* Purpose of the system. Understand the purpose of the application or system you are testing. This will help you identify the highest-priority performance characteristics on which you should focus your testing. You will need to know the systemÙs intent, the actual hardware and software architecture deployed, and the characteristics of the typical end user.
* Customer or user expectations. Keep customer or user expectations in mind when planning performance testing. Remember that customer or user satisfaction is based on expectations, not simply compliance with explicitly stated requirements.
* Business drivers. Understand the business drivers – such as business needs or opportunities – that are constrained to some degree by budget, schedule, and/or resources. It is important to meet your business requirements on time and within the available budget.
* Reasons for testing performance. Understand the reasons for conducting performance testing very early in the project. Failing to do so might lead to ineffective performance testing. These reasons often go beyond a list of performance acceptance criteria and are bound to change or shift priority as the project progresses, so revisit them regularly as you and your team learn more about the application, its performance, and the customer or user.
* Value that performance testing brings to the project. Understand the value that performance testing is expected to bring to the project by translating the project- and business-level objectives into specific, identifiable, and manageable performance testing activities. Coordinate and prioritize these activities to determine which performance testing activities are likely to add value.
* Project management and staffing. Understand the teamÙs organization, operation, and communication techniques in order to conduct performance testing effectively.
* Process. Understand your teamÙs process and interpret how that process applies to performance testing. If the teamÙs process documentation does not address performance testing directly, extrapolate the document to include performance testing to the best of your ability, and then get the revised document approved by the project manager and/or process engineer.
* Compliance criteria. Understand the regulatory requirements related to your project. Obtain compliance documents to ensure that you have the specific language and context of any statement related to testing, as this information is critical to determining compliance tests and ensuring a compliant product. Also understand that the nature of performance testing makes it virtually impossible to follow the same processes that have been developed for functional testing.
* Project schedule. Be aware of the project start and end dates, the hardware and environment availability dates, the flow of builds and releases, and any checkpoints and milestones in the project schedule.
| < Prev | Next > |
|---|
