US 11,989,592 B1
Workload coordinator for providing state credentials to processing tasks of a data processing pipeline
Sanjeev Kulkarni, Los Altos, CA (US); Zhimin Liang, West Vancouver (CA); Boyang Peng, Mountain View, CA (US); and Dragoljub Profirovic, Coquitlam (CA)
Assigned to Splunk Inc., San Francisco, CA (US)
Filed by Splunk Inc., San Francisco, CA (US)
Filed on Jul. 30, 2021, as Appl. No. 17/444,170.
Int. Cl. G06F 9/50 (2006.01); H04L 9/40 (2022.01)
CPC G06F 9/505 (2013.01) [G06F 9/5061 (2013.01); H04L 63/0853 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A method implemented by a streaming data processing system, the method comprising:
obtaining a specification of a data processing pipeline, the data processing pipeline specifying a series of nodes and interconnections between individual nodes within the series, wherein the nodes designate a transformation of data items within the data processing pipeline and wherein the interconnections designate a routing of messages through the data processing pipeline; and
implementing the data processing pipeline on the streaming data processing system by execution of a plurality of processing tasks, wherein the data processing pipeline is associated with state information, wherein implementing the data processing pipeline on the streaming data processing system comprises:
identifying an update to the state information;
identifying a data source from which to obtain the update to the state information for processing by the data processing pipeline,
using a workload coordinator, dynamically assigning the data source to individual processing tasks of the plurality of processing tasks, and
executing the plurality of processing tasks to cause individual processing tasks to retrieve the update to the state information from data source and perform one or more options based on the retrieved update to the state information.