US 12,309,449 B2
Augmented reality system for viewing an event with multiple coordinate systems and automatically generated model
Sankar Jayaram, Los Gatos, CA (US); Wayne O. Cochran, Ridgefield, WA (US); John Harrison, Hillsboro, OR (US); Timothy P. Heidmann, Los Altos, CA (US); and John Buddy Scott, Lawrenceville, GA (US)
Assigned to Quintar, Inc., Santa Clara, CA (US)
Filed by Quintar, Inc., Santa Clara, CA (US)
Filed on Apr. 22, 2024, as Appl. No. 18/641,702.
Application 18/641,702 is a continuation of application No. 17/242,265, filed on Apr. 27, 2021, granted, now 12,003,806.
Claims priority of provisional application 63/159,870, filed on Mar. 11, 2021.
Prior Publication US 2024/0276056 A1, Aug. 15, 2024
Int. Cl. H04N 21/431 (2011.01); G06T 19/00 (2011.01); H04N 13/117 (2018.01); H04N 13/178 (2018.01); H04N 13/183 (2018.01); H04N 13/194 (2018.01); H04N 21/2187 (2011.01)
CPC H04N 21/4316 (2013.01) [G06T 19/006 (2013.01); H04N 13/117 (2018.05); H04N 13/178 (2018.05); H04N 13/183 (2018.05); H04N 13/194 (2018.05); H04N 21/2187 (2013.01)] 20 Claims
OG exemplary drawing
 
12. A system, comprising:
a plurality of mobile devices, each comprising:
a camera configured to generate image data, including a live view of a venue of an event;
a display configured to display the generated image data;
one or more sensors configured to generate, in a coordinate system for the mobile device, image metadata for the image data specifying a location and an orientation of the mobile device within the venue for the event in a corresponding internal coordinate system of the mobile device when capturing the image data;
one or more interfaces configured to exchange data with one or more servers; and
one or more processing circuits configured to:
independently maintain the corresponding internal coordinate system;
transmit the image data and the image metadata from the mobile device to the one or more servers;
transmit a request for graphics to be displayed by the display over a view of a venue to the one or more servers;
receive from the one or more servers a corresponding transformation between the corresponding internal coordinate system the mobile device and a real world coordinate system; and
receiving from the one or more servers the requested graphics;
convert the requested graphics into the mobile device's coordinate system based on the corresponding transformation; and
present the converted requested graphics on the display aligned with the live view of the venue on the display; and
a server system comprising the one or more servers, configured to:
retrieve point features of a venue for an event in a first coordinate system and locations of a set of fiducials features for the venue in a real world coordinate system from one or more databases;
receive, from each of the mobile devices, the image data and the corresponding image metadata;
from the point features of the venue and the locations of the set of fiducials features in the world coordinate system, build a model of the venue in the real world coordinate system;
from the image data and corresponding image metadata from each of the mobile devices and the model of the venue in the real world coordinate system, generate for each of the mobile devices the corresponding transformation between the mobile device's corresponding internal coordinate system and the real world coordinate system;
transmit to each of the mobile devices the corresponding coordinate transformation between the mobile device's corresponding internal coordinate system and the real world coordinate system;
for each of the mobile devices, receive corresponding requests for graphics to be displayed by the mobile device over a view of the venue as specified by location and orientation in the real world coordinate system; and
transmit to each of the mobile devices the corresponding requested graphics.