US 11,915,041 B1
Method and system for sequencing artificial intelligence (AI) jobs for execution at AI accelerators
Moshe Tanach, Bet Herut (IL); and Yossi Kasus, Haifa (IL)
Assigned to NEUREALITY LTD., Bet Herut (IL)
Filed by NeuReality LTD., Bet Herut (IL)
Filed on Sep. 11, 2020, as Appl. No. 17/018,628.
Claims priority of provisional application 62/899,428, filed on Sep. 12, 2019.
Int. Cl. G06F 9/48 (2006.01); G06F 9/38 (2018.01); G06F 9/50 (2006.01); G06F 9/54 (2006.01); G06N 3/08 (2023.01); G06N 3/063 (2023.01)
CPC G06F 9/4881 (2013.01) [G06F 9/3856 (2023.08); G06F 9/5027 (2013.01); G06F 9/544 (2013.01); G06N 3/063 (2013.01); G06N 3/08 (2013.01)] 20 Claims
OG exemplary drawing
 
1. An artificial intelligence (AI) sequencer adapted to be coupled to a plurality of AI accelerators, the AI sequencer, comprising:
a queue manager comprising logic configured to manage a plurality of queues for maintaining data of a plurality of different AI jobs for each of which a request to employ at least one of the AI accelerators is received, wherein an AI job includes at least one or more AI functions;
a scheduler configured to schedule execution of the AI jobs associated with the data maintained by the plurality of queues;
a plurality of job processing units (JPUs), wherein each of the plurality of JPUs is configured to generate an execution sequence for each respective one of the AI jobs based on respective received AI job descriptors; and
a plurality of dispatchers connected to the plurality of AI accelerators, wherein each of the plurality of dispatchers is configured to dispatch at least an AI function of one of the AI jobs from the plurality of queues to one of the AI accelerators, wherein each AI function is dispatched to one of the AI accelerators in an order determined by the execution sequence created for each respective one of the AI jobs;
wherein each AI accelerator is a dedicated processor configured to process a specific AI function of the AI job;
wherein the AI sequencer is at least partially circuitry implemented and is coupled to each of the AI accelerators via electronic connection;
wherein the AI sequencer operates to dispatch each AI function of the AI job to minimize any possible idle time of the AI accelerators; and
wherein, the AI sequencer operates to dispatch each of the AI functions of the AI jobs so that at any point in time, different ones of the AI accelerators may each be executing an AI function of different ones of the AI jobs.