Unlocking the Code to a Complex Data Migration

A federal agency was in desperate need of a new electronic records application (ERA) to handle the sending and receiving of reams of sensitive data from other agencies. They had paid a vendor to build them a new system in the AWS cloud but there was an issue. The new platform could not "speak" the data language of any of the legacy apps that they used on a daily basis, rendering the system useless. It was as if they had purchased a new smartphone, but since their apps, SIM card and contact numbers had not been migrated to the new phone, it remained unusable.

When CDW Government was hired as a rescue team to assess the situation, we faced a unique problem: how could the legacy data fit into the new system so that the agency’s new platform could finally be used the way it was intended? While the agency’s technical team kept the software up and running, no one knew what source code or technologies had been used to build the old system. The good news was that the new system did have sufficient documentation. Armed with that information alone, CDW Government began a step-by-step process to "unlock" the agency’s new platform and make it a functioning solution.

Our unique AWS cloud expertise coupled with our experience helping other federal agencies made us the ideal choice for this federal agency.

From a workflow perspective, CDW Government prides itself on using small teams with strong leaders who wear multiple hats. That meant lower overhead for the agency along with one point of contact, making communication seamless during the project.

Working hand-in-hand with the agency, we tackled this complex migration in three phases.

CDW Government began studying both the new and old systems over a three-month period of discovery. During this phase, the team examined each system's business functionalities and unique features at a high level to determine which business logic they used and why they weren't "talking" to one another. This analysis required moving field-by-field to perform a painstaking, in-depth analysis. Once that analysis was complete, we provided the government agency with a high-level list of which applications should remain and how we planned to convert one format to the other.

After unraveling the logic behind each app and determining why the various business objects were not communicating with one another, the next step was figuring out the transformation loading process. To do this, CDW Government’s team built several different use cases, or POCs. For example, one scheduled business object required a manual conversion. The old format consisted of 10-digit numbers but the new required 12 digits. The team built a use case demonstrating how this transformation could be accomplished.

The fact that there were multiple complex business objects made it necessary to build multiple POCs, since what worked for one would not necessarily work for all. The team had to map out how all the different data processes worked. This step was a key to success, as they knew that the more time

they spent mapping the POCs, the easier and more streamlined the implementation phase would be — a critical piece since the implementation phase needed to function correctly the first time.

Next, the team mapped out an architecture to show how they would extract data from the old system, transform it so it was readable by the new system. They presented the plan to the agency’s architect team and introduced it to the security and product teams, ensuring stakeholder buy-in before moving to implementation.

The implementation was modularized using the extraction, transformation, and loading (ETL) process:

• Extraction: Extraction involved retrieving the data from the legacy system, telling the agency the number of files to be extracted, what the business objects were, how they'd be built into the new system, and which validations were used to ensure everything was on track. During this step, CDW Government took the zip files from the old system, unzipped them and put them into an S3 bucket in the AWS cloud.
  
  
• Transformation: Once the team extracted the data, we applied the relevant business rules and stored the data so it could be uploaded. We looked at each XML element in the unzipped files, applied transformation rules, and stored the output in another database in the AWS cloud.
  
  
• Loading: During loading, the team created new versions of the application programming interfaces (APIs) to handle the new business rules. This involved loading data from the database and calling the APIs to insert the data. With the data in the new system, we could then observe and validate the system.

As the project moved into the production phase, the team built, tested and executed into production. Once the migration was complete, the government agency and the other agencies that interact with it will be trained on how to use the new system — a milestone that once seemed unreachable. Not only will the federal agency be able to use their new system at long last, but other agencies will be able to leverage it to improve their own operations. With CDW Government’s help, the new system has been successfully "unlocked," and the digital records system is finally ready for modernization.