| CPC G06N 3/096 (2023.01) [G06V 10/764 (2022.01); G06V 10/774 (2022.01); G06V 10/82 (2022.01)] | 18 Claims |
|
1. A method comprising:
receiving a first image data set and a second sensor data set, the first image data set being based on images of a first environment, the second data set being measurements taken by sensors of the first environment, the first image data set having a first number of first channels, the second sensor data set having a second number of second channels, the second number of second channels being greater than the first number of first channels;
training a first image model using the first image data set, the first image model including a plurality of layers, a first layer of the plurality of layers receiving predetermined dimensions and the first number of first channels of the first image data set, subsequent layers of the plurality of layers receiving the output of previous layers, weights of the layers being determined through training the first image model;
pre-training a second sensor model using the second sensor data set and the weights of the layers that were determined through training the first image model, the pre-training the second sensor model comprising:
dividing an output of a convolution of predetermined dimensions of the second data set and the second number of second channels to generate different divided outputs to provide to a second plurality of branches;
for each of the second plurality of branches:
selecting a third number of the second number of channels, the third number of the second channels being equal to the first number of first channels,
applying each layer of the plurality of layers using the weights of the layers that were determined through training the first model to the particular divided output of that particular branch,
determining a particular weight for the particular branch to apply to the output of a last layer of the plurality of layers to generate a first particular branch output;
fine-tune training the second sensor model using the second sensor data set using the particular weights of the particular branches and defining weights for the plurality of layers, the fine-tune training of the second sensor model comprising:
dividing output of the convolution of predetermined dimensions of the second data set and the second number of second channels to generate different divided outputs to provide to the second plurality of branches,
for each of the second plurality of branches:
applying a first layer of the plurality of layers to the particular divided output of that particular branch,
concatenating output of each particular branch and applying a second layer of the plurality of layers to at least a portion of the concatenated output,
determining a particular weight for each layer of each branch based on the training of the second model,
determining a particular weight for the particular branch to apply to the output of a last layer of the plurality of layers to generate a second particular branch output; and
generating the second model for similar sensors of the second data set, the second model including weights determined in the fine-tune training.
|