US 12,346,721 B2
Methods and electronic devices for batching requests for processing by a processing unit
Dmitry Viktorovich Emelyanenko, d. Goluboye (RU)
Assigned to Y.E. Hub Armenia LLC, Yerevan (AM)
Filed by YANDEX EUROPE AG, Lucerne (CH)
Filed on May 3, 2022, as Appl. No. 17/735,256.
Claims priority of application No. RU2021130744 (RU), filed on Oct. 21, 2021.
Prior Publication US 2023/0127306 A1, Apr. 27, 2023
Int. Cl. G06F 9/46 (2006.01); G06F 9/48 (2006.01)
CPC G06F 9/4806 (2013.01) 20 Claims
OG exemplary drawing
 
1. A method of batching requests for processing by a processing unit, the method being executable by an electronic device, the method comprising:
acquiring, by the electronic device, a queue of requests comprising a plurality of requests to be processed by the processing unit,
each request of the plurality of requests having a respective number of tokens, a given number of tokens defining a length of a given request, the number of tokens in the given request to be processed as a whole by the processing unit, and
executing, by the electronic device, a current iteration for a given batch of requests comprising a subset of the plurality of requests, having been previously added to the given batch, each request in the given batch having a same length, the executing the current iteration including:
determining a first cost value of adding a first candidate request from the queue to the given batch,
the first cost value being indicative of a first number of padding tokens that are to be added to either the subset of the plurality of requests or the first candidate request for ensuring that all requests in the given batch have the same length if the first candidate request is added to the given batch;
determining a second cost value of adding a second candidate request from the queue to the given batch,
the second cost value being indicative of a second number of padding tokens that are to be added to either the subset of the plurality of requests or the second candidate request for ensuring that all requests in the given batch have the same length if the second candidate request is added to the given batch; and
selectively adding, by the electronic device, a target request amongst the first candidate request and the second candidate request to the given batch,
the target request being associated with a lowest one of the first cost value and the second cost value.