US 11,941,532 B2
Method for adapting deep learning framework to hardware device based on unified backend engine
Hongsheng Wang, Hangzhou (CN); Wei Hua, Hangzhou (CN); Hujun Bao, Hangzhou (CN); and Fei Yang, Hangzhou (CN)
Assigned to ZHEJIANG LAB, Hangzhou (CN)
Filed by ZHEJIANG LAB, Zhejiang (CN)
Filed on Apr. 22, 2022, as Appl. No. 17/726,563.
Claims priority of application No. 202111408005.4 (CN), filed on Nov. 25, 2021.
Prior Publication US 2023/0162048 A1, May 25, 2023
Int. Cl. G06N 3/10 (2006.01); G06N 3/04 (2023.01)
CPC G06N 3/10 (2013.01) [G06N 3/04 (2013.01)] 9 Claims
OG exemplary drawing
 
1. A method for adapting a deep learning framework to a hardware device based on a unified backend engine, comprising the following steps:
S1: adding the unified backend engine to the deep learning framework;
S2: adding the unified backend engine to the hardware device;
S3: converting a computational graph, wherein the computational graph compiled and generated by the deep learning framework is converted into an intermediate representation of the unified backend engine, which comprises the following substeps:
S31: creating a graph launcher of the unified backend engine, and adding the graph launcher of the unified backend engine to the deep learning framework, wherein
the graph launcher inherits from operators of the computational graph of the framework and realizes a forward propagation interface, when the graph computation enters a run-time stage, the framework selects a route compiled and run by the unified backend engine when starting to run a kernel function of each operator in the computational graph by a runner;
S32: registering the graph launcher of the unified backend engine, and using the graph launcher to receive the computational graph compiled and generated by the framework, which comprises:
S321: creating a global static dictionary, wherein a key value is an element of an enumeration type, listing all optional graph launcher variables, wherein a value value is to implement the graph launcher;
S322: adding enumeration members of the graph launcher of the unified backend engine to a key value list of the enumeration type;
S323: transmitting a key value of the graph launcher of the unified backend engine to a registry by means of a front end of the framework by using the unified backend engine, and a graph executor of the computational graph of the framework choosing to use a corresponding value value of the graph launcher of the unified backend engine to start a graph computation process;
S33: converting the computational graph into the intermediate representation of the unified backend engine, which comprises the following substeps:
S331: the graph executor of the framework loading a computational graph of the framework into the backend engine through the graph launcher of the unified backend engine and executing the forward propagation interface when the graph computation enters a running period;
S332: creating a computational graph conversion interface in the forward propagation interface, wherein the computational graph conversion interface is responsible for converting the computational graph of the framework into the intermediate representation of the unified backend engine;
S333: the computational graph conversion interface first traversing all nodes of the computational graph according to a topological order of the computational graph of the framework, then creating the corresponding intermediate representation of the unified backend engine for operators of each node, and finally, performing the computational graph conversion of the kernel function of each operator to generate the intermediate representation of the unified backend engine;
S4: compiling the intermediate representation, wherein the unified backend engine compiles the intermediate representation on the hardware device to generate an executable object;
S5: running the executable object, wherein the deep learning framework runs the executable object on the hardware device;
S6: managing memory of the unified backend engine.