US 12,115,995 B2
In-vehicle multimodal violence detection for shared vehicles and self-driving shuttles
Alaa M. Khamis, Courtice (CA); Wei Tong, Troy, MI (US); Sai Vishnu Aluru, Commerce Township, MI (US); Upali P. Mudalige, Rochester Hills, MI (US); and Sravan Daruri, Walled Lake, MI (US)
Assigned to GM GLOBAL TECHNOLOGY OPERATIONS LLC, Detroit, MI (US)
Filed by GM Global Technology Operations LLC, Detroit, MI (US)
Filed on Jun. 1, 2022, as Appl. No. 17/829,643.
Prior Publication US 2023/0391344 A1, Dec. 7, 2023
Int. Cl. B60W 40/08 (2012.01); A61B 5/00 (2006.01); A61B 5/0205 (2006.01); A61B 5/16 (2006.01); B60W 30/18 (2012.01); G06V 10/12 (2022.01); G06V 20/59 (2022.01); G06V 20/70 (2022.01); G10L 15/22 (2006.01); G10L 25/63 (2013.01)
CPC B60W 40/08 (2013.01) [B60W 30/18109 (2013.01); G06V 10/12 (2022.01); G06V 20/59 (2022.01); G06V 20/70 (2022.01); G10L 15/22 (2013.01); G10L 25/63 (2013.01); A61B 5/0205 (2013.01); A61B 5/165 (2013.01); A61B 5/6893 (2013.01); B60W 2040/089 (2013.01)] 19 Claims
OG exemplary drawing
 
1. An in-vehicle multimodal violence detection system, comprising:
a speech and non-speech audio event recognition module capturing threat words and non-speech audio events of occupants of an automobile vehicle;
multiple in-vehicle accelerometers generating in-vehicle accelerometer data analyzed in a shaking movement recognition module;
one or more in-vehicle radar devices detecting heart rates and breathing rates of vehicle occupants, the system directing signals from the in-vehicle radar devices to a heart rate and breathing rate detection module, the heart rate and breathing detection module measuring physiological changes in heart rates and breathing rhythms of the occupants and analyzing sudden changes in heart rates and breathing rhythms;
one or more in-vehicle cameras capturing occupant positions and movements over time, the system directing signals from the one or more in-vehicle cameras to an in-vehicle semantic scene recognition module that captures and analyzes non-verbal interactions between the occupants;
one or more occupant threat indicators including an audible threat indicator generated by the speech and non-speech audio event recognition module, a visual threat indicator generated by the in-vehicle semantic scene recognition module, a physiological threat indicator generated by the heart rate and breathing rate detection module and a vibration-based threat indicator generated by the shaking movement recognition module;
a threat level determination unit initiated by a time interval time-out signal if a predetermined time interval is exceeded prior to receiving a response from any one of the occupants;
an active threat signal generated and forwarded to the threat level determination unit if any one of the occupants confirm that an active threat is present;
the threat level determination unit identifying if the active threat should be categorized as one of a “low threat”, a “medium threat” or a “high threat”, wherein:
identification of the “low threat” activates a vehicle horn and a vehicle warning flasher;
identification of the “medium threat” activates the vehicle horn and the vehicle warning flasher and activates a vehicle brake assist system and notification of the medium threat to an outside source; and
identification of the “high threat” activates the vehicle horn and the vehicle warning flasher, activates the vehicle brake assist system and forwards a request to the outside source to initiate an emergency assistance request.