CPC G06F 16/258 (2019.01) [G06F 9/4881 (2013.01); G06F 11/0766 (2013.01); G06F 16/2386 (2019.01)] | 18 Claims |
1. A method for ingesting data, the method comprising:
receiving a request to import data into a database environment;
analyzing a characteristic of importing the data to determine whether the request is associated with a first type of data load or a second type of data load;
when the request is associated with the first type of data load, invoking a first handler for the first type of data load for performing:
performing validation operations on the data;
importing records of the data that is validated into the database environment;
writing validation data to an error file, wherein the validation data is deduplicated while writing so that identical errors are written to the error file once;
during validation, tracking a number of records that have validation errors based on the validation operations; and
terminating the importing of the data and generating a user notification when the number of records having validation errors being tracked meets a threshold and has reached a selected percentage of total records failed; and
when the request is associated with a second type of data load, importing the data into the database environment using a second handler for the second type of data load without performing the validation operations on the data.
|