US 12,379,722 B1
System and methods with a multimodal interface for command and control (inputs) and monitoring (outputs) of unmanned systems
Kyle Jeffrey Nehman, Upper St Clair, PA (US); Dennis Alan Underwood, Jr., Oakmont, PA (US); and Jeremy Brett Whitsitt, New York, NY (US)
Assigned to Squire Solutions, Inc., New York, NY (US)
Filed by Squire Solutions, Inc., New York, NY (US)
Filed on Aug. 30, 2021, as Appl. No. 17/460,395.
Application 17/460,395 is a continuation in part of application No. 17/021,403, filed on Sep. 15, 2020, granted, now 11,783,820.
Claims priority of provisional application 63/073,082, filed on Sep. 1, 2020.
Claims priority of provisional application 62/901,259, filed on Sep. 16, 2019.
Int. Cl. G06F 3/16 (2006.01); B64C 39/02 (2023.01); G05D 1/00 (2006.01); G06F 3/01 (2006.01); G06F 3/04883 (2022.01); G08G 5/55 (2025.01); G08G 5/57 (2025.01)
CPC G05D 1/005 (2013.01) [B64C 39/024 (2013.01); G05D 1/0016 (2013.01); G06F 3/016 (2013.01); G06F 3/04883 (2013.01); G06F 3/16 (2013.01); G08G 5/55 (2025.01); G08G 5/57 (2025.01); B64U 2201/20 (2023.01)] 22 Claims
OG exemplary drawing
 
1. A multi-modal system for controlling an unmanned vehicle comprising:
a human-to-machine interface including a plurality of selectable input modes, including:
(i) at least one of a voice input, a hand-held controller, and a touch screen input, and
(ii) at least one selectable human feedback device;
an interface process responsive to data indicative of instructions to the unmanned vehicle input on selected ones of the input modes and operable to map the inputs to a data model that includes data indicating the instructions, the unmanned vehicle, and the selected input mode corresponding to the input data indicative of the instructions; and
a ground control effects process operable to process the data model including a process to extract from the data model input context data including the data indicative of the instructions to the unmanned vehicle and data indicative of the selected input mode corresponding to the input data indicative of the instructions and to translate the instructions to control command data executable by the unmanned vehicle;
wherein the unmanned vehicle is operable to respond to instructions provided in a pre-determined machine language translated from data indicative of the instructions provided via an input mode native to the unmanned vehicle, and
wherein the selectable input modes include the input mode native to the unmanned vehicle and at least one non-native input mode, the ground control effects process operable to normalize the extracted input context data to out of context command data and to translate the out of context command data to the pre-determined machine language.