US 12,353,428 B2
System and method for expanding a data transfer framework
Upal Sayeed Hossain, Toronto (CA); Paul Michael Montag, London (CA); Peter George McInnis, Etobicoke (CA); and Robert Lawrence Goodman, Wilmington, DE (US)
Filed by The Toronto-Dominion Bank, Toronto (CA)
Filed on Aug. 8, 2022, as Appl. No. 17/882,993.
Prior Publication US 2024/0045881 A1, Feb. 8, 2024
Int. Cl. G06F 16/25 (2019.01); G06F 16/28 (2019.01)
CPC G06F 16/254 (2019.01) [G06F 16/258 (2019.01); G06F 16/284 (2019.01)] 16 Claims
OG exemplary drawing
 
1. An apparatus configured for processing instructions associated with one or more data transfers, the apparatus comprising:
a non-transient computer-readable storage medium having executable instructions embodied thereon; and
one or more hardware processors configured to execute the instructions to:
provide an ETL (extract, transform, load) framework comprising a plurality of ETL modules and comprising code, wherein each ETL module is associated with a respective portion of the code of the ETL framework, the respective portion of the code associated with each ELT module includes one or more variables;
obtain a configuration file, the configuration file comprising a selection of one or more of the plurality of ETL modules and one or more data values to be used in place of the one or more variables of the portion of the code associated with each of the one or more selected ETL modules when executing the portion of the code associated with each of the one or more selected ETL modules, the configuration file further comprising external command data configured to execute a new data transformation external to and absent from the ETL framework, the external command data comprising a reference to an external module generated in relation to an external interface;
for each of the selected one or more ETL modules, replace the one or more variables in the respective portion of the code associated with each of the selected one or more ETL modules with the corresponding one or more data values from the configuration file, and execute the portions of the code associated with the selected one or more ETL modules, based on the one or more data values in the configuration file, and based on the external command data;
generate, as part of the executing the one or more selected ETL modules, data in a target table based on source data from a source table, the configuration file defining the source table and the target table;
read the generated data from the target table;
determine a count of records read from the target table; and
halt execution of the one or more selected ETL modules when the count of records read from the target table is zero wherein the external command data specifies an external location of a Spark SQL (structured query language) statement expressing a data transformation using standard SQL statements; and
wherein the external command data specifies an external location of a Spark SQL (structured query language) statement expressing a data transformation using standard SQL statement.