US 12,003,806 B2
Augmented reality system for viewing an event with multiple coordinate systems and automatically generated model
Sankar Jayaram, Los Gatos, CA (US); Wayne O. Cochran, Ridgefield, WA (US); John Harrison, Hillsboro, OR (US); Timothy P. Heidmann, Los Altos, CA (US); and John Buddy Scott, Lawrenceville, GA (US)
Assigned to Quintar, Inc., Santa Clara, CA (US)
Filed by Quintar, Inc., Santa Clara, CA (US)
Filed on Apr. 27, 2021, as Appl. No. 17/242,265.
Claims priority of provisional application 63/159,870, filed on Mar. 11, 2021.
Prior Publication US 2022/0295139 A1, Sep. 15, 2022
Int. Cl. H04N 21/431 (2011.01); G06T 19/00 (2011.01); H04N 13/117 (2018.01); H04N 13/178 (2018.01); H04N 13/183 (2018.01); H04N 13/194 (2018.01); H04N 21/2187 (2011.01)
CPC H04N 21/4316 (2013.01) [G06T 19/006 (2013.01); H04N 13/117 (2018.05); H04N 13/178 (2018.05); H04N 13/183 (2018.05); H04N 13/194 (2018.05); H04N 21/2187 (2013.01)] 18 Claims
OG exemplary drawing
 
12. A system, comprising:
a registration server configured to access data from one or more databases, to receive data, including image data and corresponding image metadata, from and transmit data to a plurality of mobile devices at a venue, each of the mobile devices including a camera configured to generate the image data of the venue and a display configured to display a live view of the venue for an event and to display requested graphics over and aligned with the live view of the venue by use of a corresponding coordinate transformation between a corresponding internal coordinate system of the mobile device and a real word coordinate system of the venue, and each of the mobile devices independently maintaining the corresponding internal coordinate system and the corresponding image metadata including information specifying a location and an orientation of the corresponding mobile device within the venue for the event in the internal coordinate system of the corresponding mobile device when capturing the image data, and the registration server configured to:
retrieve point features of the venue determined in a first coordinate system;
retrieve locations of a set of fiducials features for the venue in a real world coordinate system from the one or more databases, the location of each of the set of fiducial features being at corresponding point of the venue in the real world coordinate system as determined in a survey of the venue;
determine, from the set of fiducials features for the venue in a real world coordinate system and the point features of the venue in the first coordinate system, the point features of the venue in the real world coordinate system;
receive, from each of the mobile devices, the image data and the corresponding image metadata;
from the point features of the venue and the locations of the set of fiducials features in the real world coordinate system, build a model of the venue in the real world coordinate system, the model comprising a reference map including location data of a set reference features in the real world coordinate system;
from the image data and corresponding image metadata from each of the mobile devices and the model of the venue in the real world coordinate system, generate for each of the mobile devices the corresponding transformation between the mobile device's corresponding internal coordinate system and the real world coordinate system; and
transmit to each of the mobile devices the corresponding coordinate transformation between the mobile device's corresponding internal coordinate system and the real world coordinate system; and
a content server configured to receive data from and transmit data to the plurality of mobile devices, the content server configured to:
for each of the mobile devices, receive corresponding requests for graphics to be displayed by the mobile device over a view of the venue as specified by location and orientation in the real world coordinate system; and
transmit to each of the mobile devices the corresponding requested graphics.