US 12,242,892 B1
Implementation of a data processing pipeline using assignable resources and pre-configured resources
Ricky Burnett, Davis, CA (US)
Assigned to Splunk Inc., San Francisco, CA (US)
Filed by Splunk Inc., San Francisco, CA (US)
Filed on Jul. 30, 2021, as Appl. No. 17/444,157.
Claims priority of provisional application 63/182,699, filed on Apr. 30, 2021.
Int. Cl. G06F 9/50 (2006.01); G06F 9/38 (2018.01); G06F 9/455 (2018.01); G06F 9/48 (2006.01)
CPC G06F 9/5005 (2013.01) [G06F 9/3869 (2013.01); G06F 9/455 (2013.01); G06F 9/45533 (2013.01); G06F 9/45558 (2013.01); G06F 9/48 (2013.01); G06F 9/4806 (2013.01); G06F 9/4843 (2013.01); G06F 9/485 (2013.01); G06F 9/4881 (2013.01); G06F 9/50 (2013.01); G06F 9/5061 (2013.01); G06F 9/5077 (2013.01)] 19 Claims
OG exemplary drawing
 
1. A method implemented by a streaming data processing system, the method comprising:
identifying, from a pool of pre-configured resources shared by a plurality of users and pre-configured to support preview jobs requested by the plurality of users, a pre-configured resource, wherein each pre-configured resource of the pool of pre-configured resources comprises a first distributed streaming data processing engine configured to process data obtained from a data stream;
obtaining a first request for a first deployment of a data processing pipeline, the data processing pipeline specifying a series of nodes and interconnections between individual nodes within the series, wherein the first deployment corresponds to a preview deployment type, wherein the nodes designate a transformation of data items within the data processing pipeline, and wherein the interconnections designate a routing of messages through the data processing pipeline;
implementing the first deployment using the pre-configured resource based on the first deployment corresponding to the preview deployment type;
obtaining a second request for a second deployment of the data processing pipeline, wherein the second deployment corresponds to a full deployment type; and
implementing the second deployment using a dedicated resource from a pool of dedicated resources configured to be assigned to a particular user of the plurality of users based on the second deployment corresponding to the full deployment type, wherein each dedicated resource of the pool of dedicated resources comprises a second distributed streaming data processing engine configured to process data obtained from a data stream,
wherein implementing the second deployment comprises:
provisioning the dedicated resource for the second deployment, and
deploying the data processing pipeline using the dedicated resource.