| CPC G06F 40/186 (2020.01) | 9 Claims |
|
1. A method for processing complex data, comprising:
obtaining a target processing template pair set by a user, wherein the target processing template pair comprises a target collection template and a target screening template; and the target collection template and the target screening template are a collection template and a screening template generated by the user according to a current data processing task, or a collection template and a screening template selected by the user from a collection template library and a screening template library according to the current data processing task;
determining whether execution parameters of the target processing template pair need to be modified;
if not, generating a query instruction set based on the execution parameters of the target screening template;
transmitting the query instruction set to a data processing engine, so that the data processing engine obtains data from each target data source based on the query instruction set to acquire an initial data set, wherein the query instruction set comprises at least one query instruction, and the target data source is one or more data sources specified among pre-connected data sources;
standardizing the data in the initial data set through a data processing model to acquire a standard data set; and
importing the data in the standard data set into the target collection template to acquire a target data set,
wherein a process of generating the target collection template and the target screening template by the user according to the current data processing task comprises:
generating the target collection template and the target screening template for the current data processing task based on a target data dictionary and the execution parameters input by the user, wherein the target data dictionary is an industry data dictionary imported by the user or a data dictionary generated by the pre-trained data processing model through data organization of unstructured data in a data set imported by the user; and
a process of obtaining data from each target data source by the data processing engine based on the query instruction set comprises:
determining whether each piece of data in each target data source satisfies a current query instruction; and if the data in the target data source satisfies the current query instruction, adding the data to a pre-selected data set;
determining whether the current query instruction is the last query instruction;
if the current query instruction is not the last query instruction, taking a next query instruction as a new current query instruction and determining whether each piece of data in the pre-selected data set satisfies the current query instruction, and deleting the remaining data in the pre-selected data set that does not satisfy the current query instruction to acquire an updated pre-selected data set, and returning to execute the step of determining whether the current query instruction is the last query instruction; and
if the current query instruction is the last query instruction, determining whether there is a connection relationship among each piece of data in the pre-selected data set; if there is a connection relationship, taking the data in the pre-selected data set and associated data having a connection relationship with the pre-selected data set as the data in the initial data set; and if there is no connection relationship, taking the pre-selected data set as the initial data set.
|