US 12,217,365 B1
Multiplexing video streams in an aggregate stream for a three-dimensional virtual environment
Petr Polyakov, Tampa, FL (US)
Assigned to Katmai Tech Inc., New York, NY (US)
Filed by Katmai Tech Inc., New York, NY (US)
Filed on Jul. 31, 2023, as Appl. No. 18/228,418.
Int. Cl. G06T 19/00 (2011.01); G06T 3/40 (2006.01); G06T 15/04 (2011.01); H04N 19/136 (2014.01); H04N 19/156 (2014.01); H04N 19/172 (2014.01); H04N 19/42 (2014.01)
CPC G06T 19/00 (2013.01) [G06T 3/40 (2013.01); G06T 15/04 (2013.01); H04N 19/136 (2014.11); H04N 19/156 (2014.11); H04N 19/172 (2014.11); H04N 19/42 (2014.11)] 18 Claims
OG exemplary drawing
 
1. A method for transferring video for generating avatars in a three-dimensional virtual environment, comprising:
(a) receiving a first video stream captured from a first camera on a first device of a first user, the first camera positioned to capture photographic images of the first user;
(b) receiving a second video stream captured from a second camera on a second device of a second user, the second camera positioned to capture photographic images of the second user;
(c) generating a third video stream incorporating the first and second video streams and metadata specifying how the first and second video streams are incorporated into the third video stream, wherein the generating comprises:
identifying a first area of frames from the third video stream;
identifying a second area of the frames from the third video stream;
for the respective frames of the third video stream:
identifying a first current frame from the first video stream;
identifying a second current frame from the second video stream;
mapping the first current frame onto the first area of the respective frame; and
mapping the second current frame onto the second area of the respective frame, wherein the metadata specifies the first and second areas and the mapping of the first and second areas to the respective first and second frames; and
(d) transmitting the third video stream and the metadata to a third device of a third user such that the third device is configured to:
(i) decode the first and second video streams from the third video stream based on the metadata,
(ii) generate, based on the decoded first video stream, a first avatar in the three-dimensional virtual environment at a first position of the first avatar controlled by the first user,
(iii) generate, based on the decoded second video stream, a second avatar in the three-dimensional virtual environment at a second position of the second avatar controlled by the second user, and
(iv) render, for display to the third user and from a perspective of a virtual camera controlled by the third user, the three-dimensional virtual environment including the first and second avatars.