	US 12,008,779 B2
	Disparity estimation optimization method based on upsampling and exact rematching
Wei Zhong, Liaoning (CN); Hong Zhang, Liaoning (CN); Haojie Li, Liaoning (CN); Zhihui Wang, Liaoning (CN); Risheng Liu, Liaoning (CN); Xin Fan, Liaoning (CN); Zhongxuan Luo, Liaoning (CN); and Shengquan Li, Liaoning (CN)
Assigned to DALIAN UNIVERSITY OF TECHNOLOGY, Liaoning (CN); and PENG CHENG LABORATORY, Guangdong (CN)
Appl. No. 17/604,588
Filed by DALIAN UNIVERSITY OF TECHNOLOGY, Liaoning (CN); and PENG CHENG LABORATORY, Guangdong (CN)
PCT Filed Mar. 5, 2020, PCT No. PCT/CN2020/077961 § 371(c)(1), (2) Date Oct. 18, 2021, PCT Pub. No. WO2021/138992, PCT Pub. Date Jul. 15, 2021.
Claims priority of application No. 202010028308.2 (CN), filed on Jan. 10, 2020.
Prior Publication US 2022/0198694 A1, Jun. 23, 2022
Int. Cl. G06T 7/593 (2017.01); G06T 3/4076 (2024.01); G06V 10/40 (2022.01); G06V 10/44 (2022.01)

CPC G06T 7/593 (2017.01) [G06T 3/4076 (2013.01); G06V 10/40 (2022.01); G06V 10/443 (2022.01); G06T 2207/20084 (2013.01)]

3 Claims

1. A disparity estimation optimization method based on upsampling and exact rematching, comprising the following steps:

step 1: extracting discriminable features;

step 2: conducting initial cost matching and cost map optimization to obtain an initial disparity map with low resolution;

step 3: obtaining a disparity map with one resolution higher from the initial disparity map with low resolution by a propagation upsampling method and an exact rematching method, and repeating the process until the original resolution is restored;

3.1 the propagation upsampling method

the initial disparity map D_n+1with minimum resolution is first subjected to interpolation and upsampling to obtain a coarsely matched disparity map D′_n, the disparity map obtained at this moment is only obtained by numerical interpolation without reference to any structural information of an original image, a left view is reestablished with an original right view I_raccording to the coarsely matched disparity map D′_nand denoted as Ĩ_l, and then the error between the reestablished left view Ĩ_land a real left view I_lis calculated to obtain a confidence map M_c:

M_c=1−normalization(I_l−Ĩ_l) (2)

normalization(⋅) is normalized operation, the difference is normalized to (0,1), and the probability value at each point on the confidence map M_crepresents the confidence of the disparity value of the pixel; and the confidence map is reproduced and translated to become a confidence map group which is denoted as M_cg,

M_cg=f_c(M_c,k,s) (3)

wherein f_c(⋅) represents the operation of reproduction and translation to resize, k represents the size of a neighboring window, and s represents the void content of a sampling window; and the receptive field is (2s+1)², and a confidence vector of k*k is obtained at each position, which represents the confidence of a pixel in a k*k neighboring window around the pixel;

a relative relation network module is proposed, a left feature map with the corresponding resolution is input into the module, and a weight vector is worked out at each position, which indicates the relative relation of the neighboring pixel and the center pixel, i.e., the larger the weight is, the greater the effect of a neighboring pixel on the pixel is; and the weight is donated as W_relative;

W_relative= custom character

_relative(F_n^l,k) (4)

wherein k represents the size of a neighboring window, and custom character

_relativerepresents the relative relation network module;

the coarsely matched disparity map D′_n, the confidence map M_cgand the relative relation weight W_relativeare used for propagation to obtain a propagated disparity map, and the propagation calculation process is as follows:

D_n^p=<f_c(D′_n,k,s),softmax(W_relative*M_cg)> (5)

wherein D_n^prepresents the propagated disparity map, <, > represents dot product operation, f_c(⋅) represents the operation of reproduction and translation to resize, and softmax(W_relative*M_cg) represents the support strength of the surrounding pixel to the center pixel during propagation and is obtained by multiplying the confidence of the surrounding pixel and the relative relation weight;

then the void content of the window is used for repeating the propagation process so that the optimized disparity map can be propagated in different receptive fields; and at this point, the propagation upsampling process from D_n+1to D_n^pis completed;

3.2 the exact rematching method

first, a left feature map is reestablished with a right feature map F_n^rwith the corresponding resolution in a feature list custom character

according to D_n^pand donated as custom character

, and

=f_w(F_n^r, D_n^p); and rematching is conducted once with the reestablished left feature map custom character

and the original left feature map F_n^lwithin a small range of the disparity d=[−d₀, d₀] to obtain a cost map, then the cost map is optimized through an hourglass network, the disparity is regressed to obtain a bias map Δ which represents an offset from D_n^p, and the two maps are added to obtain a final disparity map D_nof an optimized network;

D_n=D_n^p+Δ (6)

the processes of 3.1 and 3.2 are iterated repeatedly until the original resolution is restored to obtain a final high-precision disparity map.