US 11,948,348 B2
Operator behavior monitoring system
Jaco Cronje, Zwavelpoort (SA); Johann Ephraim Hough, Brooklyn (SA); Willie Moller, Monument Park (SA); Riaan Van Rensburg, Halfway Gardens (SA); and Paul Olckers, Faerie Glen X6 (SA)
Assigned to 5DT, Inc., Orlando, FL (US)
Appl. No. 17/257,550
Filed by 5DT, Inc., Orlando, FL (US)
PCT Filed Oct. 23, 2019, PCT No. PCT/IB2019/059065
§ 371(c)(1), (2) Date Dec. 31, 2020,
PCT Pub. No. WO2020/084518, PCT Pub. Date Apr. 30, 2020.
Claims priority of provisional application 62/749,190, filed on Oct. 23, 2018.
Prior Publication US 2021/0287539 A1, Sep. 16, 2021
Int. Cl. G06V 10/82 (2022.01); G06F 18/214 (2023.01); G06F 18/2415 (2023.01); G06V 10/94 (2022.01); G06V 20/59 (2022.01); G08G 1/13 (2006.01); H04L 67/12 (2022.01); G06V 10/20 (2022.01); G06V 10/80 (2022.01); G06V 40/10 (2022.01); G06V 40/16 (2022.01)
CPC G06V 10/82 (2022.01) [G06F 18/214 (2023.01); G06F 18/2415 (2023.01); G06V 10/95 (2022.01); G06V 20/597 (2022.01); G08G 1/13 (2013.01); H04L 67/12 (2013.01); G06V 10/255 (2022.01); G06V 10/809 (2022.01); G06V 40/11 (2022.01); G06V 40/161 (2022.01)] 20 Claims
OG exemplary drawing
 
1. An operator behavior monitoring system comprising:
an operator behavior recognition system comprising hardware including at least one processor, a data storage facility in communication with the processor and input/output interfaces in communication with the processor, the system being configured to implement a set of convolutional neural networks (CNNs) including:
an object detection group into which at least one image is received from an image source for detecting at least one object in the image and to delineate the object from the image for further processing;
a feature extraction group which extracts features of the at least one object detected by the object detection group, the features including key points, orientation, descriptors and other features of the at least one object;
a classifier group which assesses the features extracted by the feature extraction group and classifies the features into a predefined number of events, and which is operable to report the events to a remote computer,
wherein the classifier group includes one or both of:
a single image CNN of the at least one object; and/or
a single image CNN of the at least one object in combination with a long-term-short-term memory (LSTM) recurrent network, which keeps a memory of a series of previous images of the at least one object; and
wherein the classifier group includes an ensemble function to ensemble the outputs of the classifiers together with the output of the single image CNN of the at least one object together with the combination of the single image CNN and the LSTM recurrent network by a weighted sum of the classifiers where the weights are determined by optimizing the weights on the training dataset;
a server operable to communicate with the operator behavior recognition system for receiving predefined events detected by the operator behavior recognition system;
a database, in communication with the server, operable to store and retrieve detected operator incidents; and
a web front-end, in communication with the server, for interfacing with the server.