US 11,948,260 B1
Streaming mixed-reality environments between multiple devices
Roger Cordes, San Francisco, CA (US); and David Brickhill, San Francisco, CA (US)
Assigned to Lucasfilm Entertainment Company Ltd., San Francisco, CA (US)
Filed by Lucasfilm Entertainment Company Ltd., San Francisco, CA (US)
Filed on Nov. 18, 2022, as Appl. No. 17/990,042.
Application 17/990,042 is a continuation of application No. 17/494,890, filed on Oct. 6, 2021, granted, now 11,514,653.
Application 17/494,890 is a continuation of application No. 16/130,269, filed on Sep. 13, 2018, granted, now 11,145,125, issued on Oct. 12, 2021.
Claims priority of provisional application 62/558,249, filed on Sep. 13, 2017.
This patent is subject to a terminal disclaimer.
Int. Cl. G06T 19/00 (2011.01); A63F 13/213 (2014.01); A63F 13/35 (2014.01); A63F 13/60 (2014.01); A63F 13/65 (2014.01); G02B 27/01 (2006.01); G06F 3/01 (2006.01); G06T 7/246 (2017.01); G06T 7/593 (2017.01); G06T 7/73 (2017.01); G06T 13/40 (2011.01); G06T 15/08 (2011.01); G06T 19/20 (2011.01); G06V 40/20 (2022.01); H04L 65/611 (2022.01); G06V 40/10 (2022.01)
CPC G06T 19/006 (2013.01) [A63F 13/213 (2014.09); A63F 13/35 (2014.09); A63F 13/60 (2014.09); A63F 13/65 (2014.09); G02B 27/017 (2013.01); G06F 3/011 (2013.01); G06T 7/246 (2017.01); G06T 7/596 (2017.01); G06T 7/73 (2017.01); G06T 13/40 (2013.01); G06T 15/08 (2013.01); G06T 19/20 (2013.01); G06V 40/20 (2022.01); G06V 40/23 (2022.01); H04L 65/611 (2022.05); G06V 40/107 (2022.01)] 20 Claims
OG exemplary drawing
 
1. A method comprising:
receiving a motion or position of a first real-world performer in a first real-world environment;
moving or positioning a virtual representation of the first real-world performer in a 3-D virtual environment based on the motion or position of the first real-world performer;
rendering a 2-D video stream of the virtual representation of the first real-world performer in the 3-D virtual environment;
adding or altering a virtual asset in a 3-D virtual environment in response to the motion or position of the first real-world performer;
determining a mathematical 3-D transform to be applied to the virtual asset based on identifying the motion or position;
preparing a stream of real-time communication packets, wherein each communication packet in the stream of real-time communication packets comprises a field configured to represent the mathematical 3-D transform to be applied to the virtual asset in the 3-D virtual environment; and
transmitting the stream of real-time communication packets to a plurality of display devices distributed within a second real-world environment; and
causing a real-time video stream to be visible on a display device of a handheld or wearable device in the plurality of display devices within the second real-world environment, wherein:
the handheld or wearable device captures the real-time view of a second real-world environment comprising view of a second real-world performer;
the real-time video stream comprises the 2-D video stream of the virtual representation of the first real-world performer composited with the real-time view of the second real-world environment comprising a view of the second real-world performer; and
the motion or position of the first real-world performer in the first real-world environment moves or positions the virtual representation of the first real-world performer in the 3-D virtual environment in real time as the 2-D video stream is composited with the real-time view of the second real-world environment such that the second real-world performer and the virtual representation of the first real-world performer appear to interact in real time with each other when viewed on the display device of the handheld or wearable device.