US 12,229,118 B2
Method, apparatus and device for data shuffling, computer-readable storage medium and product
Haiyang Shi, Los Angeles, CA (US); and Hao Wang, Los Angeles, CA (US)
Assigned to Beijing Volcano Engine Technology Co., Ltd., Beijing (CN)
Filed by Beijing Volcano Engine Technology Co., Ltd., Beijing (CN)
Filed on Jun. 13, 2024, as Appl. No. 18/742,351.
Claims priority of application No. 202310879264.8 (CN), filed on Jul. 17, 2023.
Prior Publication US 2025/0028705 A1, Jan. 23, 2025
Int. Cl. G06F 16/23 (2019.01); G06F 7/14 (2006.01)
CPC G06F 16/2365 (2019.01) [G06F 7/14 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A method for data shuffling, comprising:
acquiring a data shuffling request for distributing at least one data to be shuffled pre-stored in a host memory to at least one target server, wherein the data to be shuffled comprises at least one data segment to be sent to different target servers;
acquiring a shuffling request parameter linked list associated with the at least one data to be shuffled based on the data shuffling request, wherein the shuffling request parameter linked list comprises at least one shuffling request parameter corresponding to the at least one data segment, and the shuffling request parameter comprises associated information of a target server to which the data segment is to be sent and memory buffer information corresponding to the data segment;
performing a merging operation on shuffling request parameters in the shuffling request parameter linked list according to the data amount of the data segment corresponding to the shuffling request parameter and the memory buffer information to obtain at least one target request parameter composed of at least one shuffling request parameter; and caching the data to be shuffled corresponding to the at least one target request parameter to a predetermined remote direct memory access network card; and
distributing respectively data segments associated with at least one data to be shuffled cached in the remote direct memory access network card to the target server indicated in the shuffling request parameter associated with the data segment.