Software Testing Risks
As project manager identifies and manages potential problems which can occur and affect the success of project such as – cost overruns, attrition of key resources etc. similarly Test manager should also identify key risks related to testing process. Test manager should also assess the exposure and probability for the risk factor and report the results of that analysis. These results help in identifying mitigation and contingency plan.
The testing process is fundamentally a risk-based activity. Starting from writing test cases to executing them in each cycle, test manager cannot afford to cover 100% due to restricted budget with aspect to cost and resources. Many techniques and best practices have been defined which help in minimizing the risks related to same such as different best practices to follow while writing test cases and creating different packs of test cases for smoke testing, regression testing, User acceptance testing etc.
The range of potential points of failure for a typical application development and testing effort are:
>> Failure to meet deadlines and budgets
>> Failure to meet business goals
>> Failure of the application logic
>> Failure to keep up with business change
>> Failure to deliver an application that meets the business needs
>> Failure to understand where your projects stand—exactly how reliable is application and what’s been tested?
When Test manager documents risk, following information will be stored for each risk:
• Risk ID
• Risk Exposure
• First Indicator that risk is becoming a problem
• Mitigation Approaches
• Date Due
• Contingency Plan
• Contingency Plan Trigger
Some of the common test project risks are:
• Lack of trained resources and low test competency
• Shortage of test resources required to execute as per plan
• Limited time assigned for testing
• Frequent changes to the test object definition and the scope
• Lack of coordination between development team and testing team
• Lack of experience in using new technology, tools, programming languages or methods;
Early in the testing process, the test team needs to develop the relevant risk response plans and contingency plans to deal with the test project risks. The test team can use some measures to manage risk.
The following are some measures which can be used:
1) Early intervention of testing team in the project
The testing team should be involved early in the software development life cycle, thus ensuring that testing team should be able to complete the preparation of the test earlier. This ensures that the testing team can begin testing immediately after delivery of the test object. This helps in conducting following activities as part of test inception –
» Define business drivers
» Define test plan and strategy
» Define testing team structure- roles and responsibilities
» Document and agree on SLA’s
» Setup engagement processes and tools
» Setup metrics and status reporting tools
» Documented inter-component/supplier dependencies
» Test data creation / usage guidelines
» Define testing scope
» Define basic build acceptance test criteria
In addition, early intervention helps in reducing cost of quality by finding defects much early in development stages. Major number of defects detected during system testing and acceptance testing can lead to a serious risk of project delay. Therefore, early intervention and active defect prevention activities early in the software development life cycle can effectively reduce the project risks.
2) Test preparation
Before the official start of the test execution, the testing team should perform test preparation. Following activities should be carried out:
» Environment shakeout
» Execute Business, Technology and Process Trainings / Ramp-up
3) Define test entry and exit criteria
The test team should define stringent entry criteria to avoid the project extension due to the testing team. The aim of defining the criteria is to help development teams recognize the importance of delivering high quality software.
4) Testability requirements
The testing team should carefully review the specifications testability. For example instead of using text box for entering date and then validating the same by testing , development team can directly use calendar control. This will reduce testing effort drastically and will ensure that such items are not missed out during testing.
5) Defect estimation and monitoring
Defect estimation in a project is performed by Test Manager. It involves estimating the defect injection at various stages and defect detection by various quality control activities. The basis for estimating the defects injected/detected in a project is by using historical data. Either data on similar projects from the process database, or data from process capability baseline, can be used. The procedure for estimating the defects is same whatever be the basis chosen by the project.
The defects injected at various stages in the project like requirements, design and build, are first estimated, after considering the improvements due to defect prevention activities. The defects to be detected at various quality control activities are then estimated.
As for a project, project manager performs milestone analysis to determine its status with reference to the set goals and take corrective actions or to reset goals if necessary. Similarly, it will be necessary to review the quality requirements of the work products at the beginning of a phase & plan for these requirements. Generally, the quality goals for each life cycle stage are set during the project planning stage. However, these may be required to be revised based on the actual performance of the previous phases or based on revisions to effort, size, scope changes etc. Similarly, if there are conflicting requirements for quality between work products these may need to be reconciled. If there are SLA requirements with reference to quality & there are conflicts these may also be required to be reconciled in consultation with the customer.