US 11,989,838 B2
Mixed reality display device and mixed reality display method
Osamu Kawamae, Kyoto (JP); and Masuo Oku, Kyoto (JP)
Assigned to MAXELL, LTD., Kyoto (JP)
Appl. No. 17/428,646
Filed by Maxell, Ltd., Kyoto (JP)
PCT Filed Feb. 6, 2019, PCT No. PCT/JP2019/004191
§ 371(c)(1), (2) Date Aug. 5, 2021,
PCT Pub. No. WO2020/161816, PCT Pub. Date Aug. 13, 2020.
Prior Publication US 2022/0114792 A1, Apr. 14, 2022
Int. Cl. G06T 7/194 (2017.01); G06F 3/01 (2006.01); G06F 3/14 (2006.01); G06T 7/50 (2017.01); G06T 15/40 (2011.01); G06T 19/00 (2011.01); G06T 19/20 (2011.01); G06V 20/20 (2022.01); H04L 67/131 (2022.01); H04N 13/363 (2018.01)
CPC G06T 19/006 (2013.01) [G06F 3/011 (2013.01); G06F 3/1454 (2013.01); G06T 7/194 (2017.01); G06T 7/50 (2017.01); G06T 15/40 (2013.01); G06T 19/20 (2013.01); G06V 20/20 (2022.01); H04L 67/131 (2022.05); H04N 13/363 (2018.05); G06T 2219/024 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A mixed reality display device that is worn by an experiencing person on a head and displays a video of a virtual object while being superimposed on a reality video visually recognized by the experiencing person, wherein
the mixed reality display device is configured to be connected to other mixed reality display devices worn by other experiencing persons and to cause the experiencing persons to share the reality video and the video of the virtual object with each other,
the mixed reality display device comprises:
a camera configured to image a reality video,
a distance measurement sensor configured to measure a distance to a reality object in the reality video,
a position and direction sensor configured to detect a position of the camera and an imaging direction,
a controller configured to recognize the reality object from the reality video imaged by the camera and link a predetermined virtual object to the recognized reality object based on an operation of the experiencing person,
a display configured to display a video of the linked virtual object,
a memory configured to store data of the virtual object linked to the reality object, and
a communicator configured to connect to the other mixed reality display devices and transmit and receive data to and from each other,
the controller is configured to:
acquire a recognition result of a reality object generated by the other mixed reality display device and data of the linked virtual object, via the communicator,
synthesize the acquired data and the data stored in the memory,
display a video obtained by the synthesis on the display, and
allow an edit of the virtual object linked to the reality object on the video displayed on the display, based on an operation of the experiencing person,
a reference point for the position of the camera and the imaging direction detected by the position and direction sensor is set to coincide with reference points of positions of cameras and imaging directions in the other mixed reality display devices,
when the data acquired from the other mixed reality display device and the data stored in the memory are synthesized, the controller viewpoint-transforms the video in accordance with the position of each of the cameras and the imaging direction and then performs synthesis, and
the controller updates the video to a new video based on a new position of the camera and a new imaging direction every time the experiencing person moves.