US 11,657,524 B2
Image capturing and depth alignment method
Wen-Hsiung Lin, New Taipei (TW); Chih-Yuan Chu, New Taipei (TW); and Kai-Pin Tung, New Taipei (TW)
Assigned to JORJIN TECHNOLOGIES INC., New Taipei (TW)
Filed by JORJIN TECHNOLOGIES INC., New Taipei (TW)
Filed on Jan. 6, 2021, as Appl. No. 17/142,256.
Claims priority of application No. 109124569 (TW), filed on Jul. 21, 2020.
Prior Publication US 2022/0028097 A1, Jan. 27, 2022
Int. Cl. G06T 7/521 (2017.01); G06T 7/33 (2017.01); H04N 13/122 (2018.01); H04N 13/167 (2018.01)
CPC G06T 7/521 (2017.01) [G06T 7/33 (2017.01); H04N 13/122 (2018.05); H04N 13/167 (2018.05)] 1 Claim
OG exemplary drawing
 
1. An image capturing and depth alignment method, comprising:
Radar scanning step: assigning an absolute coordinate system to a scene and producing a steam of data points constituting a 3D point cloud map of the scene through at least a millimeter-wave radar, where each millimeter-wave radar is assigned with a first relative coordinate system, and the stream of data points is associated with stereoscopic information;
Image capturing step: producing a steam of image points constituting a planar image of the scene through at least an optical camera, where each optical camera is assigned with a second relative coordinate system, and the stream of image points provides optical information to the scene;
Translation and synchronization step: translating the steam of data points and the stream of image points by superimposing the first and second relative coordinate systems on the absolute coordinate system, and synchronizing the steam of data points and the stream of image points so that they are synchronized for each part of the scene;
Alignment step: aligning the stream of data points and the stream of image points after they are processed by the translation and synchronization step and storing the aligned stream of data points and the aligned stream of image points in a back-end server, where the back-end server therefore obtains surveillance information with both image and depth, the stereoscopic information comprises at least one of length, height, depth, and distance, and the surveillance information comprises information about a hazardous zone;
Client detection step: detecting the movement of a smart glasses by its wearer from an Inertial Measurement Unit (IMU) inside the smart glasses, and transmitting various parameters from the IMU to the back-end server;
Client positioning step: obtaining a coordinate and an angle of the smart glasses within the scene by the back-end server through superimposing the various parameters in the various coordinate systems;
Scene map construction step: based on the coordinate and angle of the smart glasses within the scene, obtaining a scene map by the back-end server corresponding to what is covered by the viewing range of the smart glasses from matching the planar image of the scene to the coordinate and angle of the smart glasses;
View image transmitting step: transmitting a view image perceived by the smart glasses to the back-end server;
View image processing step: spatially aligning the view image and the scene map and uniformizing the coordinate systems by the back-end server, where the uniformization of coordinate systems involves finding center points respectively within 3D point cloud map and the scene, calculating the distances between various reference points of 3D point cloud map and the scene relative to the center points, keeping the ones of the smallest distances as the key reference points, obtaining and transmitting a spatial correspondence information between the 3D point cloud map and the scene to the smart glasses; and
Virtual object placement step: determining the coordinate of a virtual object in the scene based on the spatial correspondence information between the 3D point cloud map and the scene and thereby placing the virtual object in the scene.