In today’s world, many organizations are moving their data from the old system to the new system for faster access and better business decision taking. In simple words transferring data between storage types, formats, or computer systems is referred to as data migration. Data migration is usually performed programmatically to achieve an automated migration. It is required when organizations change computer systems or upgrade to new systems. In one of the projects, we had a client with millions of data into their old system, and the scope of the data migration project was to move it to a new system where it would work normally.
In this article, we will see some common approach you need to take to do the data migration testing:
1. Understand the data structure: first of all to successfully test the data migration QA team needs to understand the data structure, database structure and data type of the old legacy system as well as the new future system. Until you have a very clear understanding of both systems there is no way you can perform the migration testing. In most of the projects, the lead architect prepares a spreadsheet which provided you that end-to-end migration view. Other than that the design documents play a vital role as a source of requirements for this kind of project.
2. Overall reconcile balances and counts of records: Data reconciliation is a mandatory testing in most of the migration project. In data reconciliation, we compare records counts with legacy system(s), in every single table, which has been migrated. And ensure that there’s no difference. At some level, we need to include business users again for this data reconciliation activity and get confidence and confirmation of logic behind population of every table.
3. Item level reconciliation testing: Item level reconciliation – that’s where you count items of interest. E.g. when you are reading a batch of invoice lines instead of only (or even) counting invoice lines, you might count the number and balance of the invoices. Whereby this case the invoice is the Item. There are many areas where this can be helpful, especially where you’re changing the granularity, e, g when say invoice lines might be migrated to a series of conditions – e.g. You derive new data from the individual invoice lines.
4. Testing the mapping rules: this is the next big area of testing. In most of the migration project, you have some conversion involved. The mapping rules should be captured in excel and tested. Traceability matrices are primarily done to confirm that the developed code performs the calculation and the transformations precisely. Additionally, benefit is you have not overlooked any mapping rules (we have more than 400 rules. 30 are really very complex), and implementation of one mapping has not messed another functionality/calculations Each rule is picked and parsed with a data and to verify the desired result. Ticking each rule can act as acceptance criteria from one phase to another. In many projects, it has been observed the fields for dates (different date formats will cause you considerable trouble in data validation so be careful with MM-DD-YYYY formats).
5. Pilot testing: This is the final testing stage where the old system is still intact, but people are testing a copy of the new system, to see if it is satisfactory yet, or if more work is needed for the conversion. Then next week it is a replacement pilot with a lot of stuff fixed, and people do their testing again until the system is fully stable and from that point we can plan to phase out the old legacy systems.