US 11,676,348 B2
Dynamic mixed reality content in virtual reality
Sarah Tanner Simpson, Mountain View, CA (US); Gregory Smith, San Francisco, CA (US); Jeffrey Witthuhn, Oakland, CA (US); Ying-Chieh Huang, Fremont, CA (US); Shuang Li, San Jose, CA (US); Wenliang Zhao, Belmont, CA (US); Peter Koch, Los Altos, CA (US); Meghana Reddy Guduru, Mountain View, CA (US); Ioannis Pavlidis, Newark, CA (US); Xiang Wei, Fremont, CA (US); Kevin Xiao, San Carlos, CA (US); Kevin Joseph Sheridan, Redwood City, CA (US); Bodhi Keanu Donselaar, London (GB); and Federico Adrian Camposeco Paulsen, Redwood City, CA (US)
Assigned to Meta Platforms Technologies, LLC, Menlo Park, CA (US)
Filed by Meta Platforms Technologies, LLC, Menlo Park, CA (US)
Filed on Jun. 2, 2021, as Appl. No. 17/336,776.
Prior Publication US 2022/0392169 A1, Dec. 8, 2022
Int. Cl. G06T 19/00 (2011.01); G06T 7/11 (2017.01); G06K 9/00 (2022.01); G06T 7/70 (2017.01); G06V 20/20 (2022.01); G06V 40/10 (2022.01)
CPC G06T 19/006 (2013.01) [G06T 7/11 (2017.01); G06T 7/70 (2017.01); G06V 20/20 (2022.01); G06V 40/10 (2022.01); G06T 2207/30196 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A method comprising, by a mobile computing device:
capturing, using one or more cameras of the mobile computing device, one or more images of a first user wearing a virtual reality (VR) display device in a real-world environment, wherein the VR display device is displaying a first VR rendering of a VR environment, and wherein the first VR rendering is from the perspective of the VR display device;
transmitting, from the mobile computing device, a pose of the mobile computing device with respect to the VR display device to a VR system of the VR display device;
receiving, at the mobile computing device and from the VR system, a second VR rendering of the VR environment, wherein the second VR rendering is from the perspective of the mobile computing device with respect to the VR display device;
segmenting the first user from the one or more images; and
generating, at the mobile computing device in real-time responsive to capturing the one or more images, a mixed reality (MR) rendering of the first user in the VR environment, wherein the MR rendering of the first user is based on a compositing of the segmented one or more images of the first user and the second VR rendering.