US 12,271,514 B2
Mixed reality interaction with eye-tracking techniques
Justin Strawn, San Francisco, CA (US); and Nathan Aschenbach, Seattle, WA (US)
Assigned to Meta Platforms Technologies, LLC, Menlo Park, CA (US)
Filed by Meta Platforms Technologies, LLC, Menlo Park, CA (US)
Filed on Jan. 24, 2023, as Appl. No. 18/100,707.
Prior Publication US 2024/0248527 A1, Jul. 25, 2024
Int. Cl. H04N 13/383 (2018.01); G02B 27/00 (2006.01); G02B 27/01 (2006.01); G06F 3/01 (2006.01); G06T 19/00 (2011.01)
CPC G06F 3/011 (2013.01) [G02B 27/0093 (2013.01); G02B 27/0172 (2013.01); G06T 19/006 (2013.01); H04N 13/383 (2018.05)] 15 Claims
OG exemplary drawing
 
1. A head mounted display (HMD) device system for presenting mixed reality (MR) interactive content, comprising:
a display system to project a three-dimensional (3D) interactive environment to a user wearing the HMD device;
an eye tracking system to determine a gaze of the user and information regarding at least one fixation and at least one saccade of an eye of the user;
a sensor system to sense a body movement of the user corresponding to an action to be taken within the projected 3D interactive environment; and
a controller communicatively coupled to the display system, the eye tracking system, and the sensor system, the controller to:
identify a 3D point of interest (POI) of the user within the projected 3D interactive environment based on the determined gaze of the user and information regarding at least one fixation and at least one saccade of the eye of the user;
identify the action to be taken on the identified 3D POI within the projected 3D interactive environment based on the sensed body movement and the identified 3D POI; and
control the display system to adjust the projected 3D interactive environment to show the action being taken on the identified 3D POI.