US 11,741,631 B2
Real-time alignment of multiple point clouds to video capture
Arturas Serackis, Vilnius (LT); Dalius Navakauskas, Vilnius (LT); Dalius Matuzevicius, Vilnius (LT); Darius Plonis, Vilnius (LT); Andrius Katkevicius, Vilnius (LT); Tomyslav Sledevic, Vilnius (LT); and Vytautas Abromavicius, Vilnius (LT)
Assigned to Vilnius Gediminas Technical University, Vilnius (LT)
Filed by Vilnius Gediminas Technical University, Vilnius (LT)
Filed on Jul. 15, 2021, as Appl. No. 17/376,168.
Prior Publication US 2023/0015751 A1, Jan. 19, 2023
Int. Cl. G06T 7/73 (2017.01); G06T 7/246 (2017.01); G06V 20/40 (2022.01); G06F 18/22 (2023.01); G06V 10/75 (2022.01)
CPC G06T 7/75 (2017.01) [G06F 18/22 (2023.01); G06T 7/251 (2017.01); G06V 20/46 (2022.01); G06V 20/48 (2022.01); G06V 10/759 (2022.01)] 4 Claims
OG exemplary drawing
 
1. A method for real-time indoor or outdoor environment analysis and continuous measurement of a captured object's position, comprising:
capturing a video stream continuously on a mobile device;
capturing motion sensor data continuously from motion sensors which are integrated with the mobile device, wherein the motion sensors are accelerometer, gyroscope, and magnetometer;
synchronizing in time the video stream captured on the mobile device with the motion sensor data;
transferring the video stream and the motion sensor data to a video stream processing service in parallel using the same or separate data streaming protocols;
computing a moving point cloud from the video stream and motion sensor data using video and motion data fusion algorithms;
estimating an initial location of the mobile device by identifying a reference object in the moving point cloud using an object detection algorithm and then querying for similar objects in a higher-resolution pre-captured point cloud of a point cloud database using semantic identification algorithms by artificial intelligence means;
wherein, the following steps are repeated while the video stream is being captured:
extracting a set of point cloud segments-candidates from the moving point cloud and from the pre-captured point cloud corresponding to the initial location estimate or a previously determined location by parametrization of the moving point clouds using a chain of filters with coefficients, wherein the coefficients are estimated using a mathematical iterative algorithm;
matching the moving point cloud to the set of pre-captured point cloud segments-candidates, wherein object features and euclidean distance measurements are used to find the intersections of the moving and pre-captured point cloud segments;
producing a 3D representation of the environment surface that is captured on the mobile device;
determining the location of the mobile device in the 3D representation using the initial location estimate or known previous location;
tracking the location of the mobile device in the 3D representation; and
wherein, after capturing of video stream has been stopped, the following steps are executed:
finding and gathering differences in the 3D scene representation to the moving point cloud; and
marking disappearance or presence of new objects in the 3D scene representation.