US 11,748,160 B1
Load balancing computing resources in an observability pipeline system
Dritan Bitincka, Edgewater, NJ (US); Ledion Bitincka, San Francisco, CA (US); Nicholas Robert Romito, Chicago, IL (US); and Clint Sharp, Oakland, CA (US)
Assigned to Cribl, Inc., San Francisco, CA (US)
Filed by Cribl, Inc., San Francisco, CA (US)
Filed on Jun. 14, 2021, as Appl. No. 17/346,881.
Int. Cl. G06F 9/48 (2006.01); G06F 9/50 (2006.01); G06F 9/38 (2018.01)
CPC G06F 9/4887 (2013.01) [G06F 9/3842 (2013.01); G06F 9/505 (2013.01); G06F 9/5072 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A method of load balancing across computing resources in an observability pipeline system, the method comprising:
in an observability pipeline system deployed on one or more computer nodes operating in a leader role and a plurality of worker roles, identifying a plurality of computing jobs, each of the computing jobs comprising a plurality of computing tasks, each of the computing tasks associated with event data to be processed by the observability pipeline system;
by operation of the leader role, dispatching the plurality of computing tasks to the plurality of worker roles according to a least in-flight task dispatch criterion, wherein dispatching the plurality of computing tasks according to the least in-flight task dispatch criterion comprises, iteratively:
identifying one of the plurality of worker roles as an available worker role;
identifying, from the plurality of computing jobs, one or more incomplete computing jobs comprising one or more computing tasks that have not been executed in the observability pipeline system;
selecting, from the one or more incomplete computing jobs, a computing job that has the least number of in-flight computing tasks currently being executed in the observability pipeline system;
identifying a next computing task from the selected computing job; and
dispatching the next computing task to the available worker role;
by operation of the worker roles in the observability pipeline system, executing the computing tasks dispatched to the respective worker roles, wherein executing each respective computing task comprises applying an observability pipeline process to the event data associated with the respective computing task.