CPC G06F 9/4881 (2013.01) [G06F 9/485 (2013.01)] | 30 Claims |
1. An apparatus comprising at least one processor and a storage to store instructions that, when executed by the at least one processor, cause the at least one processor to perform operations comprising:
receive, at the at least one processor, and from a requesting device via a network, a request to perform a job flow, wherein:
the job flow is defined in a job flow definition that specifies a set of tasks to be performed via execution of a corresponding set of task routines during a performance of the job flow, and that specifies data dependencies among the set of task routines; and
the set of task routines are executed within a set of task containers instantiated within at least one virtual machine (VM);
store, within a job queue, a job performance request message comprising an indication of receiving the request to perform the job flow;
within a performance container, execute instructions of a performance routine to cause the at least one processor to, in response to the storage of the job performance request message within the job queue, perform operations comprising:
based on the data dependencies among the set of tasks specified in the job flow definition, derive an order of performance of the set of tasks that specifies a subset of tasks of the set of tasks that are able to be performed in parallel, and derive a quantity of task containers within the set of task containers that enables parallel execution of a corresponding subset of task routines to cause the parallel performance of the subset of tasks;
based on at least the derived quantity of task containers within the set of task containers, derive a quantity of at least one other VM that, in combination with the at least one VM, provides resources to enable instantiation of the derived quantity of task containers to enable the parallel execution of the subset of task routines;
provide, to a VM allocation routine executed by the at least one processor, an indication of a need for provision of the at least one other VM, wherein execution of the VM allocation routine causes the at least one processor to allocate the at least one VM; and
store, within a task queue, multiple task routine execution request messages that each comprise an identifier associated with a task of the subset of tasks to enable the parallel execution of the subset of task routines within the set of task containers; and
within the performance container, in response to at least storage of multiple execution completion messages within the task queue that are indicative of completion of execution of the subset of task routines, perform operations comprising:
provide, to the VM allocation routine, an indication of cessation of the need for provision of the at least one other VM.
|