In any software testing project, we do broadly two types of testing – functional testing and non functional testing.
Functional testing is done by using the functional specifications provided by the client or by using the design specifications like use cases provided by the design team. On contrary the non functional testing is quite complex and often subjective as opposed to functional testing. Here the client tries to document what they expect from the system in terms of how fast, how efficiently, how safely, etc.
Now in this article we will check some of the common challenges of non functional testing:
1. Requirements are too subjective: In many cases, the non functional requirements are too generic and subject to interpretation. Like people may say, “application must response quickly when the user makes a search” or “the application must be secure” etc. Now while doing the testing, this kind of requirement is a big challenge, and we need to quantify that. For example, the first requirement can be restated in this way: the application must response in three seconds while 100 concurrent users make a search into the system. Like this all the other requirements needs to be quantified before we start doing any testing.
2. Infrastructure overhead: Second challenge is simulating a production like environment. If in production, you are having 200 machines running, then what should be the ideal load of the test environment. Should we have 100 machines configured, which would give a realistic performance results during performance testing. So in this 100 machine test bed if we get a result of six seconds for a user’s search, then can we say in production with 200 machines the response time would be three seconds as per the requirement.
3. How many concurrent users: Again as we mentioned in the point #1, it is not enough to say I want the application to return a user’s search result within three seconds. If 1 user does that search at that point, then three seconds is fine. However, how about 1000 users are going to perform the same operation say login to the system at the start of the day. Of course, that time the memory and CPU utilization would be much higher and consequently, the response time would be much lower. So while designing non function test cases you always need to consider the number of concurrent users for a particular transaction.
4. Test Data: This is one of the biggest challenges. Two things you need to determine while running performance, load and stress this kind of non functional testing. First, the data volume you need to load in your test environment. If the production had 100 terabytes of data, how much data do we need in the test system? More data will have more impact on performance. Because the database insert would take more time. Second, the data variation is also important. If you use a data where the update, insert, delete operations are not happening, then definitely it would take less time. Furthermore, if you use the same type of data or same data again and again, then the system would simple cache it and in a next run, the data will be fetched from the cache, and the response time would be much lower. So you need to have a production like data volume based on your test infrastructure alone with a good data variation.