CPC G06F 9/4887 (2013.01) [G06F 9/3842 (2013.01); G06F 9/505 (2013.01); G06F 9/5072 (2013.01)] | 20 Claims |
1. A method of load balancing across computing resources in an observability pipeline system, the method comprising:
in an observability pipeline system deployed on one or more computer nodes operating in a leader role and a plurality of worker roles, identifying a plurality of computing jobs, each of the computing jobs comprising a plurality of computing tasks;
by operation of the leader role, dispatching the plurality of computing tasks to the plurality of worker roles, wherein dispatching the plurality of computing tasks comprises, iteratively:
identifying one of the plurality of worker roles as an available worker role;
identifying, from the plurality of computing jobs, one or more incomplete computing jobs comprising one or more computing tasks that have not been executed in the observability pipeline system;
selecting, from the one or more incomplete computing jobs, a computing job that has the least number of in-flight computing tasks currently being executed in the observability pipeline system;
identifying a next computing task from the selected computing job; and
dispatching the next computing task to the available worker role;
by operation of the worker roles in the observability pipeline system, executing the computing tasks dispatched to the respective worker roles.
|