“Test Automation” is much more than using a record/playback tool – no matter how sophisticated it is. If your testing only goes through the UI(s) provided, you can’t test until the UI is available, and you only test what the UI exposes. There are some products and product features that are ONLY testable via automation.
When you design a test, whether it is manual or automated, you define a series of steps with verification points interspersed, and how pass/fail will be determined. If a test procedure is not specific enough to be run the same way every time, then it is a poor test procedure. If your test automater is translating a manual test with less verification than the manual test, you should fire him. So, let’s assume that you have equally well-defined manual and automated tests – whether you run the automated test again, or run the manual test again, they both should find the same bugs. How can you claim that the manual test is going to find more than the automated test?
A manual tester doesn’t limit themselves to just what the test procedure says to do. They will either change the way things are executed, or they will be on the lookout for side effects, or other collateral issues. This is my main concern with automated tests – they typically are written to do things the same way every time, and only look at the same specific things to determine success or failure. But, that is a minor issue compared to how little time and human effort are required to run the automated tests. So these “extra” bugs that the tester found while running the manual test were NOT found by the test – they were found by a good manual tester.
In my experience, automated tests do not require more effort to maintain than do manual tests. If the GUI changes, and features are re-arranged, and the way, you get to them is different – every manual test script will have to be updated to have the new steps. Maybe you think modifying text is less effort than modifying automation/code?
Advantages of Test automation are clear:
– One of the most important benefits of test automation is a rapid response. The cost of fixing a bug prior to QA entry is roughly five times lower than a bug found in QA. Even if if the scope of automated tests is identical to preexisting manual test coverage, ability to run automated tests in CI environment will drastically reduce the number of bugs found at the time manual QA kicks in, thus saving valuable test cycles.
– Cost of ownership is another important concept. Unit tests have the lowest cost of ownership over the long haul. Then, non-UI functional tests. Automated GUI tests are valuable, but they are infinitely more expensive than unit tests and non-UI functional tests. That’s why it is very important to pick your spots when identifying the scope of test automation.
So yes, Automation packs will likely find fewer bugs, but that is because of the way they are designed to be used rather than any failure in the testing process.
What Defects test automation cannot find:
Automation test will not cover the instant defects includes alternative behavior that are not part of requirement, but those can be bound to happen during testing. Some of the defects are.
1. ” Web page will browse to the previous page if “Back'” button on the page has been selected. This can be covered in automation test. There is an alternate procedure to browse to the previous page by clicking the browser back button. Usually this cannot be covered in automation tests because it will not be part of requirements. However, any user can have a chance to browse to the previous page using the browser back button. This can be covered in manual testing, when the test engineer thinks this scenario as a naive user.”
2. Automation test will test for the existence of object in a page, but it cannot always test that whether all the objects are placed in that particular pixels during page load. Those tests are executed only on mandatory conditions. However, in manual testing, those things can be taken care automatically on awareness of a test engineer.
As many people have mentioned, automated test cases might be helpful in uncovering defects. However, If you look at it from management’s perspective, automation of an application under development is challenging and time consuming. When you invest so much of time and money in automation, one must be conscious of the ROI part. I personally believe automation is at its best in a regression phase of a stable application. Your ROI increases each time you successfully run your automated test case. Value for Money Factor.