US 11,999,356 B2
Cognitive heat map: a model for driver situational awareness
Guy Rosman, Newton, MA (US); Simon A. I. Stent, Cambridge, MA (US); Luke Fletcher, Cambridge, MA (US); John Leonard, Newton, MA (US); Deepak Gopinath, Chicago, IL (US); and Katsuya Terahata, Nisshin (JP)
Assigned to Toyota Research Institute, Inc., Los Altos, CA (US)
Filed by TOYOTA RESEARCH INSTITUTE, INC., Los Altos, CA (US)
Filed on Jun. 18, 2021, as Appl. No. 17/351,611.
Claims priority of provisional application 63/113,454, filed on Nov. 13, 2020.
Prior Publication US 2022/0153278 A1, May 19, 2022
Int. Cl. B60W 40/08 (2012.01); G06N 3/045 (2023.01); G06V 20/59 (2022.01); G06V 40/19 (2022.01)
CPC B60W 40/08 (2013.01) [G06N 3/045 (2023.01); G06V 20/597 (2022.01); G06V 40/19 (2022.01); B60W 2420/403 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A system, comprising:
a camera configured to capture image data of an environment;
a monitoring system configured to generate a gaze sequences of a subject; and
a computing device communicatively coupled to the camera and the monitoring system, the computing device configured to:
receive the image data from the camera and the gaze sequences from the monitoring system,
implement a machine learning model comprising a convolutional encoder-decoder neural network configured to process the image data and a side-channel configured to inject the gaze sequences into a decoder stage of the convolutional encoder-decoder neural network,
generate, with the machine learning model, a gaze probability density heat map, and
generate, with the machine learning model, an attended awareness heat map.