US 12,356,095 B2
Visual-inertial tracking using rolling shutter cameras
Matthias Kalkgruber, Vienna (AT); Erick Mendez Mendez, Vienna (AT); Daniel Wagner, Vienna (AT); Daniel Wolf, Modling (AT); and Kai Zhou, Wr Neudorf (AT)
Assigned to Snap Inc., Santa Monica, CA (US)
Filed by Snap Inc., Santa Monica, CA (US)
Filed on May 29, 2024, as Appl. No. 18/677,296.
Application 18/677,296 is a continuation of application No. 18/098,939, filed on Jan. 19, 2023, granted, now 12,028,626.
Application 18/098,939 is a continuation of application No. 17/161,937, filed on Jan. 29, 2021, granted, now 11,582,409, issued on Feb. 14, 2023.
Claims priority of provisional application 63/045,568, filed on Jun. 29, 2020.
Prior Publication US 2024/0314461 A1, Sep. 19, 2024
Int. Cl. H04N 25/531 (2023.01); G02B 27/01 (2006.01); H04N 23/51 (2023.01)
CPC H04N 25/531 (2023.01) [G02B 27/0172 (2013.01); H04N 23/51 (2023.01); G02B 2027/0138 (2013.01); G02B 2027/014 (2013.01); G02B 2027/0178 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A method for visual-inertial tracking with a mobile device having a motion detector and a camera system including a first rolling shutter camera, the method comprising:
obtaining an initial pose for the first rolling shutter camera;
capturing, during a capture period, an image of an environment using the camera system, the image including feature points, each feature point captured at a capture time during the capture period;
sensing, during the capture period, movement of the mobile device with the motion detector;
computing a number of poses for the first rolling shutter camera based on the initial pose and the sensed movement, each computed pose corresponding to a computed time within the capture period, wherein the number of computed poses is responsive to the sensed movement of the mobile device;
storing each of the computed number of poses in a lookup table;
selecting, for each feature point in the image, one of the number of computed poses by matching the capture time for the feature point to the computed time for the computed pose, wherein the selecting includes retrieving, for feature points in the image, the one of the number of computed poses from the lookup table with the corresponding capture time that is prior to the corresponding computed time; and
determining a position of the mobile device within the environment using the feature points and the selected computed poses for the feature points.