In the process of creation of a successful software product, there is an inevitable problem of finding a balance between the quality and the release date of the software product. The testing allows of obtaining a product that satisfies all requirements. But the covering of each product risk with various test cases and compiling them take too long. The correctly prepared testing process should provide a required quality level without exceeding project time and budget . If the time for testing was estimated wrongly, it can lead you either to the late product delivery, or to the decrease of its quality and competitiveness . The estimation of time resources in software testing is a rather complicated and volumetric process but its significance for the creation of the successful project shouldn’t be underestimated.
This article contains recommendations that can help you to obtain more realistic and functional time estimates for the testing of a new project.
QA Specialist of Driver Testing Team
1. Decomposition of Testing Tasks 1.1. Testing Process Planning 1.2. Test Plan and Test Case Development 1.3. Test Environment Configuration 1.4. Execution of Test Cases 2. Debugging of Test Cases After the First Run or Product Change 2.1. Regression Testing Conclusion
The number one attribute of any good prediction of the future is, of course, whether it comes true.
According to the statistics, testing of a single-component console application takes about 20% of its development time. Testing of a two-component console application takes 20-30% of its development time, an application with GUI – 30-35%, a distributed application with GUI – 35-50%. But each project and each team are unique that is why this time estimate is rather rough and does not include some risks.
To make the estimate of testing time more accurate and realistic, you should use the method of decomposition, i.e. you should divide the process of testing into several parts and estimate the time for each of them.
As a rule, testing process of a new product can be divided into 6 main stages:
- Process planning .
- Test plan and test case development .
- Test environment configuration .
- Performing of test cases.
- Debugging of test cases after the first run or after product change .
- Regression testing.
This stage, in its turn, consists of two sub-stages: researching of the project and working out the testing strategy. Researching of the project can be performed in different ways in different companies and commands: reading and analyzing the project documentation, initial meeting, discussi ng the project details with managers, etc. On the average, we recommend to spend one-two days on this task. It is not necessary that this task takes the whole working day but 1-2 day estimation will allow you to have enough time in case of new ideas and questions. Additional time is required in the cases:
- The project is rather large-scale, has a lot of documentation .
- The work with such kind of project is performed for the first time and additional time for research is required.
- Specialists who take part in the project discussion have different operating schedules and are geographically dispersed. In such cases, additional time for the meetings organization is required.
Working out the testing strategy implies defining a number of important nuances of the future work:
- What types of testing will be performed.
- What kind of specialists is required for the development of test plans and test cases.
- How many specialists will perform testing of the project and what skills they should have.
In our practice, to define the required kinds and types of testing, we base on the ready pattern, which includes all kinds of testing that can be topical.
Depending on the project type, its quality requirements and other factors, the pattern is corrected . Such fine-tuning takes about one hour. Attention: On this stage, defining of specialists for the work on the project will help to estimate the time more precisely as we can take into account the their skills and experience. It will also help to include into the testing schedule their vacations and days off .
The development of a test plan and test cases is a rather laborious process and requires considerable time expenditure. When planning it, you should base on the testing strategy developed on a previous stage . The testing strategy shows what kinds of test cases should be created and it also helps to set their priority.
In our practice, we base on the rule that one requirement to the product must be covered by at least five test cases. With the help of this rule, we can approximately calculate the number of required test cases and time for their creation.
T ime for the development of test case depends on the complexity of the test plan but on the average, developing of one test case takes 10 minutes. In the general case, the developing of a test plan without test cases and its review can take two-three days. Correspondingly, if the project requires test cases, you should estimate additional time for their developing.
Some peculiarities of planning of this stage:
- If a specialist prepares a test plan and test cases for the first time, you should estimate more time than for the more experienced specialist.
- If the project uses technologies that are new for the team, you should take into account time for its research . Depending on the technology complexity and also on the qualification of the specialists who will perform this task, additional time for the research can be from one day to several weeks. You should not neglect it by any means. If the test plan is prepared without taking into account project specifics, it will lead, at the best, to the increase of time for making changes in it on the testing stage and at the worst, to the decrease of the product quality.
- Depending on the types of testing and the project, you may need time for creation of the test data . It should be also taken into account on this stage.
- The order of the developing of test cases should correspond to the order of their running. There can be two variants for the test priority:
- First, you should run the tests that cover the first-priority modules and features of the project.
- First, you should run the tests for the project modules ready for testing.
Depending on the order of the test running, you should prepare the order of creation of the test plan and test cases. It can also influence the final estimation.
- If the company has an experience in testing of similar projects, then the use of old test plans and test cases as the basis will cut the estimated time.
- Time for reviewing of prepared test plans and test cases should be allotted both for the newly created and altered old ones. When planning this time, you should take into account the workload and the personal schedules of the specialists who will perform review .
- The more experienced the specialist who creates the test plan and the test cases is, the less time for reviewing and further changes is required.
The time, required for this stage of work, depends on the following factors:
- Is there a necessity to buy some equipment or not? When it is necessary to expand the testing laboratory, the time should be planned with taking into account specific and possibilities of the company. If the equipment is on sale and the company can buy it at once, the time will be considerably cut comparing with the case when the specific equipment should be delivered by the foreign customer or when the company cannot afford it at the moment.
- Time for installing and configuring of the test environment depends on the qualification of the specialist and on the experience of working with similar test environments. If the company has a special department for such tasks, less time will be required comparing with the case when QA specialists will perform this work.
The time for installing and configuring of the test environment directly depends on the complexity of the last. Usually, it takes from one hour to one day to configure the test environment for the mid-size project that works with popular operating systems and does not require complicated system solutions. In other case, additional time estimation is required depending on the task peculiarities.
In our practice, we use the rule that the execution of one test case takes the QA specialist about 5 minutes. The test plan contains test cases of various complexity and scale: some test cases can be executed in 1 minute, others – in 10 minutes. As a result, the average duration of the test case is 5 minutes.
It is recommended to increase the time for one test case up to 10 minutes if the testing is performed by the junior QA specialist. Thus we can take into account the risks that exist if there is no required work experience.
One of the main difficulties of planning of this stage of work is that you cannot predict precisely the number of bugs that will be found while testing and also the complexity of their reproducing. On the average, writing a report on one bug takes 10-15 minutes. The more bugs are found, the more time for the reports on each of them is required . If the bug is too complicated, it is not enough even several hours to find out its exact location. There are methods that allow you to estimate the number of possible bugs in every subsequent product version. It is difficult to exactly define the number of bugs in the new version though.
To not let the testing exceed the planned time, you should include the time for various risks in the schedule. Additional time for preparing a number of reports on found bugs and also the time for the locating of the most complicated bugs will be included into the common testing risks. We recommend that you add about 20-25% of time for such cases to your final estimate.
Anyway, you should follow your common sense. If testing of the product version is estimated for 10 hours and the localization of one found bug takes more than 2-3 hours, it is logical that you postpone it till the end of the testing phase (if the bug is not too critical or blocking). After the end of the testing phase, if there is some time left you can get back to work on this bug. In this case, you provide the presence of all required testing results without putting the project out of the schedule.
This stage of work takes about 10-15% of time that was required for the creation of the test plan and test cases.
Also the required time directly depends on the time that was allotted for reviewing of the test plan and test cases just after their preparing. If the reviewing was thorough enough, the time for the further remaking after the run can decrease. But the time for reviewing should be taken into account adequately.
When estimating time for the regression testing of the new product versions, you should take into account the following nuances:
- What type of testing is required on this stage? Is it necessary to perform full testing again or just smoke testing for the intermediate product version?
- The number of test cases in the test plan can be changed after its correction.
When the testing is performed by one and the same specialist, the time for the tests running can decrease as the tests will not be new for him/her.
If the project supports several operating systems, there is a necessity of testing its work on all operating systems required by the specification. When the product has a client – server architecture, it is also important to run the tests on different system combinations. It is often not possible to test all combinations because great time resources are required and it is not commensurable with the total development time.
In such cases, it is important to cover the matrix of system combinations as much as possible. For example, if an application supports five different systems for the server side and five systems for the client side, you should choose the combinations that cover all these systems without repetitions. This means that you should avoid the variant when the XP SP3 client is tested with all possible systems for the server side or vice versa. You just try to cover each system in the testing combinations.
If the product version is intermediate, the full testing should be performed on several systems that are the most critical for users. Smoke testing should be performed on the rest of the systems. Taking into account the system priorities for your product, you can fill the matrix as efficient as possible and perform the testing in a limited period of time.
When the matrix of the system combinations actual for this stage of development is prepared, it is easy to estimate the total time for the version testing:
T = (T(A)*K(A) + T(F)*K(F) + T(S)*K(S))*1.25,
where T( ) – time for passing of the test plan on one configuration;
K( ) – number of configurations for passing;
A – acceptance testing;
S – smoke testing;
F – full testing.
Testing is a complicated process with a high risk and there is always some deviation in all its estimates. That is why it is efficient to combine different methods and approaches taking into account the specifics of the project and the testing team with understanding of the factors that influence the costs, time, and resources.
The method that we propose you will help you to make the basic estimate for the testing of your product; the accounting of all its specifics will help to increase the accuracy of the estimate and to make it as realistic as possible.
So, the general formula for the testing estimate is the following:
T = T (studying of specifications + test strategy) + T (preparing of the testing documentation) + T(preparing of the test environment) + T (first run + updating of the testing documentation) + T(regression testing).