US 12,205,324 B2
Learning to fuse geometrical and CNN relative camera pose via uncertainty
Bingbing Zhuang, San Jose, CA (US); and Manmohan Chandraker, Santa Clara, CA (US)
Assigned to NEC Corporation, Tokyo (JP)
Filed by NEC Laboratories America, Inc., Princeton, NJ (US)
Filed on Nov. 5, 2021, as Appl. No. 17/519,894.
Claims priority of provisional application 63/113,961, filed on Nov. 15, 2020.
Claims priority of provisional application 63/111,274, filed on Nov. 9, 2020.
Prior Publication US 2022/0148220 A1, May 12, 2022
Int. Cl. G06T 17/00 (2006.01); G06T 1/00 (2006.01); G06T 5/50 (2006.01); G06T 7/73 (2017.01); G06T 7/77 (2017.01)
CPC G06T 7/75 (2017.01) [G06T 1/0014 (2013.01); G06T 7/77 (2017.01); G06T 2207/20076 (2013.01); G06T 2207/20081 (2013.01); G06T 2207/20084 (2013.01); G06T 2207/20221 (2013.01); G06T 2207/30244 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A computer-implemented method for fusing geometrical and Convolutional Neural Network (CNN) relative camera pose, comprising:
receiving two images having different camera poses;
inputting the two images into a geometric solver branch to return, as a first solution, an estimated camera pose and an associated pose uncertainty value determined from a Jacobian of a reproduction error function;
inputting the two images into a CNN branch, having a pose branch multi-layer perceptron (MLP) and an uncertainty branch MLP, to return, as a second solution, a predicted camera pose by the pose branch MLP and an associated pose uncertainty value predicted by the uncertainty branch MLP by extracting appearance features with a feature extractor and geometric features with an attentional graph neural network and a geometric feature MLP; and
fusing, by a processor device, the first solution and the second solution in a probabilistic manner based on the uncertainty predictions using Bayes' rule to obtain a fused pose.