There are various knowledge base and forums available which talk about software development processes and best practices related to project management. However, nothing much is elaborated on test management. This article provides general recommendations to improve the management of software testing.
If not managed properly, test management could lead to problem areas both at the strategic level for application quality as well for day-to-day execution. Both of them impact or slow down the progress to achieve business objectives.
The strategic level challenges could be –
» Testing not driven by Business Risks / Cost of failure. The deemed priority area by testing team may not be the critical area for business users. The root cause of this problem is due to lack of collaboration between business users and testing team.
» Testing approach is tuned to testing the technical solution and is not tailored based on Domain / Business Processes. The root cause of this problem is due to lack of domain/business process knowledge by testing team. Further, this can be mitigated by creating collaboration between business users and testers.
» Mindset of Defect Detection (reactive, at the end of development) vs Defect Prevention (Proactive). This issue occurs when test team not is involved in a release life cycle right from requirements stage.
» Resourcing is not done on the scientific basis. It is purely based on gut feeling. Due to lack of well-defined and well evolving scope, absence of a standardized, consistent and scientific estimation approach it is difficult to estimate the effort and perform capacity planning.
» Lack of Quantitative Approach to Quality improvement. The root cause of this problem is due to lack of historical data analysis for release-over-release improvement.
The above problems force the Testing team to be in a fire-fighting mode, rather than the model where we propel success. These strategic issues could lead to various challenges such as detection of critical bugs towards end of testing cycles increasing the risk to a timely release of program due to lack of business coverage.
The execution problems can be categorized into two broad categories viz. Test Management and Processes
Execution problems which are related to Test Management are –
• Lack of robust collaborative test strategy.
• Test schedules are not flexible enough to align with the constantly changing release scope and plan
• No consistent estimation/planning approach for all components
• Scope of testing is not defined at the start of a project
» Requirements Management:
• Requirement specifications are not consistently defined
• Requirements are not driven by business scenarios
• No change management function for testing
• No process / mechanism to understand test coverage through requirements traceability
• Clear expectations are not set with third party integrations for testing needs leading to limited support and datasets from third party vendors
• Ineffective communication plan with limited involvement from various stakeholders to manage dependencies, issues, risks and defect prioritization
• Not effective Reporting / Metrics Analysis framework to provide:
» Defect Analysis/Defect Detection rate/Defect Trend Chart
» Application stability analysis to drive project/deployment/business decisions related to the release/Go-live
» Knowledge Management:
• Mechanism to learn from the lessons of past releases like Best Practices, Lessons Learnt for continuous quality improvement is not in place
• No mechanism for share and manage Business, Technical and Process knowledge between Testing team, Development team and Business users teams
» Defect Management :
• No common understanding and adoption of severity/priority definitions, defect life cycles and resolution process
Execution problems which are related to processes are –
» Test Data:
• Absence of standard strategy for Test Data creation and management.
» Test Scenario Creation :
• Lack of intelligent approach resulting in large number of Test scripts with substandard quality.
• Business scenarios/use cases are not in place thus testers cannot visualize requirements.
• Identification of application scenarios mapping with business scenarios/requirements, Domain angle approach, commonly done mistakes and all such knowledge is missing.
» Test Script Execution:
• Ripple effect of release delays, no prioritized execution.
• Regression Testing
» Creation: Strategy for creating regression test pack is not in place. Scope of regression test pack is not defined.
» Execution: No structured approach to keep regression test scripts up-to-date with releases. No approach to provide adequate regression coverage based on application stability
• Automated Testing: Automated Test scripts are not evaluated and analyzed for cross release testing. Effectiveness of automated test scripts is a question – there are still defects after the automation suite is run prior to manual test execution.
» Development Testing:
• Testing at development levels is not effective. No standard test cases are created for Unit Testing, Integration Testing
» Tools :
• Efficient usage of tools and techniques across all stakeholders is missing (Defect management tool, Collaborative configuration management tool, Coverage report)
» Deployment Readiness :
• Application Quality certification processes and Quality Gates are not set
• Testing team not in a decision making capacity about go/no-go based on application quality
Based on our experience we recommend a risk-based, defect prevention and measurement based approach for testing and test management. This is the most cost effective way to ensure the quality of software applications.
The first step of Test Management is to perform Structured Process Definition and Adoption drive. This helps in-
» Intelligent creation of test scripts by identifying test scenarios and getting them reviewed by business users.
» Creation of generic test scripts called master test scripts which would be called in other test scripts like workflows, usability test steps.
» Prioritization of critical business test scripts and execution accordingly. Further creation of common regression pack.
» Effectively maintaining the regression pack in identified Test Tools and re-visiting the same regularly to evaluate its effectiveness.
» An agreed upon Build Acceptance/Rejection Criteria.
» Risk-based testing approach:
The focus of testing should be on business objectives, business complexity, and the cost of failure. The technical complexities should not dictate the testing focus. Risk based testing approach provides a critical level of objectivity in determining what things to test by using a combination of business and technical requirements to prioritize the testing process. This ensures that the most important functions of an application are always tested first, regardless of how limited the testing time frame or scope might be.
In a perfect world, all applications would be completely tested, in all possible scenarios and environments, before the application is released into production (regardless of whether it takes weeks or months to complete 100 percent testing). However, with the limited time allotted for testing, usually testing group focus on testing the technical areas of an application that are most likely to fail, without considering how those areas match up with the most critical business requirements of that application.
By using reports that provide insight into the business needs for certain application functionality and test plans that are generated from user defined business requirements, Test Management effectively plans, organizes and validates testing requirements using a risk-based testing approach.
Test Management team works closely with the business stakeholders to understand business goals, business priorities and cost of failure considering the development and release schedules from the Development team. Further, risk based tiered regression testing post releases is performed to enable right level of regression testing. This balanced approach ensures that the application delivered meets the critical business needs.
With a risk-based approach to testing, Test Management Team works with the Business so it can understand exactly what requirements are most critical from a business and technical perspective to test, yielding a balanced set of testing priorities. This helps prioritize what is required to be tested initially, to deliver the application on time, thereby reducing the costly testing process while mitigating the risk that the application will fail.
» Defect prevention approach:
Preventing defects should be the goal of testing team rather than detecting defects after they have occurred. This is much more cost effective. Historical data is used to improve application quality from release to release.
The out-of-phase defect multiplier is two. Defects removed one phase away from insertion are twice as expensive, two phases away are four times as expensive, etc.
Testing team can provide suggestions and insights through quantitative reporting on Cycle to cycle improvements/regressions Analysis, Problematic area identification, Performance and Stability trend analysis, Test Coverage Report etc.
» Measurement based testing:
There should be measurable factor defined for testing team and should make test management accountable for the quality of software application.
Measurable Business Results should be identified and should help in achieving business goals through software application Quality improvement. These goals should help in-
• Reduction in Operational Cost – This can be achieved through Testing process efficiency, Automated Regression Testing, Reduction in Testing Support/site maintenance cost
• Increase in Customer Satisfaction- This can be achieved by improvement in Solution Quality, reliability and availability, reduction in solution downtime and reduction in Production Defects
• Improvement in Time-to-market – This can be achieved by reducing test cycles and hence release cycles.
» Traceability of requirements:
Requirement traceability ensures activity/deliverables at every level of testing are driven by business requirements and are fulfilling business objective. Requirements tracing helps the project team to understand which parts of the design and code implement the user’s requirements, and which tests are necessary to verify that the user’s requirements have been implemented correctly. This helps in ensuring –
» The system you build has the necessary functionality to meet your customers’ and users’ needs and expectations.
» Appropriate test plans and test cases are written and executed to verify that all of the system requirements were implemented. Mapping test cases to the requirements helps make sure that you don’t miss a major defect in the system.
» All system components affected by a change request are modified and nothing is overlooked. Changed / new business requirements mean change in Test strategy and Test cases. Traceability helps in managing the same. This also helps you evaluate the impact of a change request. A seemingly simple request might involve changes to several parts of the system, and it’s better to understand how much work will be needed before agreeing to make the change.
Requirements Traceability results in –
• Better Coverage Analysis – All the business requirements envisioned are implemented in the solution and everything is tested.
• Better Change Management – Entire trace is reviewed for the impact on the overall solution.
» Early collaboration:
The testing and development teams collaborate from the start of the engagement. Testing team can help prevent defects by working side by side with the development team. This helps in achieving continuous improvement in Quality and Development practices and Testing Tools.
» Tiered Regression approach:
In the tiered regression, test scenarios are categorized as 3 tiered test suites (Tier 1, 2 and 3). Tier 1 – contains core functionalities having direct impact core functional areas. Tier 2 – contains functionalities having linkages to core functionalities. Tier 3 – contains ALL the functionality in the regression pack. Usually Tier 1 constitutes 20%, Tier 2 40% and Tier 3 40% of existing test scenarios. Test Execution moves in the flow of Tier 1 -> Tier 2 and finally Tier 3. All 3 tiers executed initially for first couple of system test cycles, subsequently reducing leading to elimination of scenarios in next cycles – assuming test scenarios of lower levels passes in earlier cycles.
This enables execution of relevant scenarios and reduced length of test cycles. This can result in frequent builds to testing team which enables an earlier go-live of application.
Benefits of Tiered Regression
• Early application stability
• Reduced resourcing constraints
• Last line of defence assuring quality of releases
• Extensive End to End testing of release
» Incremental Testing model:
Incremental testing helps in testing subsystems in isolation. The testing team continues to test the software application as development team integrates more and more subsystems. Choosing Incremental testing strategy helps to see a working version of software application as soon as possible.
Types of incremental testing are:
– Bottom up testing: This is one of the most popular approaches of incremental testing. In this approach, each module at the lowest level of the system hierarchy is tested individually first, next modules to be tested are the ones that call the previously tested ones.
– Top down testing: In the approach, one controlling module is tested by itself, next all modules called by the tested module are tested as a larger unit.
– Big bang testing: In this approach all modules are tested in isolation and they are mixed together as a final system to see if they work. This is also known as “Shake ‘n Bake” method.
– Sandwich testing: This approach combines top-down with bottom-up. In this system is viewed as three layers just like a sandwich.
The other important aspect of Test management is to perform estimation and capacity planning
Below are the suggested guidelines for estimation:
» Estimate execution time for Regression pack created from Baseline scripts. Regression pack will have two components- Regression of new functionality, Regression of complete site
» In the subsequent system test cycles only regression scripts related to defect fixing should be executed
» Number of system test cycles will be function of Entry criteria –Based on Unit testing results, which defines quality of application developed.
» The test scripts will be linked to easily identify and relate them to one defect. Further analysis can be done on basis -If x defect found and number 10 script failed , total y scripts will get affected and will gate further testing
» Root cause of failure analysis and SLA for fixing the defects
» Smoke testing/ shake out testing which will test stability of build
For Automation aspect, Test management should follow three step approach – Planning, Execution and Evolution.
The analyses phase includes evaluation of application suitability and level of automation which can be performed. Then test automation goals and objectives are identified along with tool to be used for the same.
In the execution phase automation framework is created and test scripts are defined. During evolution phase testing team ensures that continuous updates and enhancements are done to test procedures to keep pace with application evolutions.