Automatización de la Revisión de Datos Procesados por una ETL para el Área de Pruebas del Banco de Occidente

Abstract

The purpose of this document is to highlight the process of creating an automated tool for reviewing and ensuring the quality of data processed by an ETL (Extract, Transform, and Load) system, which is utilized by Banco de Occidente to migrate information from multiple databases to a central database.

During this migration process, the data tables are exposed to the risk of corruption due to potential errors in the ETL development. Given the substantial amount of data involved, manually reviewing these tables becomes impractical as they may contain millions of records. In response to this need, the bank's testing team identified the urgency of having a tool to verify the correctness of files before loading them into the destination database.

As a solution, a tool was developed using the Python programming language. This tool takes files in .csv format extracted from databases and compares them with data stored in Excel spreadsheets (.xlsx), providing indicators of similarity between both sources. This ensures the integrity of the files once processed. Additionally, the tool offers the option to download a .xlsx file with a boolean table detailing which data has been modified, which is valuable for Quality Control Analysts (QA) when reporting faults in supplied ETLs.

In conclusion, a user-friendly interface was successfully developed that meets the established objectives for ensuring the quality of data processed by the ETL. The tool also has significant potential for implementation in other software testing and data migration teams, offering an efficient and reliable solution to ensure data accuracy in similar processes.