US 12,340,335 B2
Automated collaboration skills assessment
Svati Dhamija, Sunnyvale, CA (US); Amir Tamrakar, Princeton, NJ (US); Nonye M. Alozie, Redwood City, CA (US); Elizabeth McBride, Oakland, CA (US); Ajay Divakaran, Monmouth Junction, NJ (US); Anirudh Som, Plainsboro, NJ (US); Sujeong Kim, Hyattsville, MD (US); and Bladimir Lopez-Prado, Union City, CA (US)
Assigned to SRI International, Menlo Park, CA (US)
Filed by SRI International, Menlo Park, CA (US)
Filed on Jun. 15, 2021, as Appl. No. 17/348,588.
Claims priority of provisional application 63/039,074, filed on Jun. 15, 2020.
Prior Publication US 2021/0390492 A1, Dec. 16, 2021
Int. Cl. G06Q 10/0639 (2023.01); G06N 3/045 (2023.01); G06N 3/049 (2023.01); G06N 20/00 (2019.01); G06Q 10/10 (2023.01); G06V 20/40 (2022.01)
CPC G06Q 10/06398 (2013.01) [G06N 3/045 (2023.01); G06N 3/049 (2013.01); G06N 20/00 (2019.01); G06Q 10/103 (2013.01); G06V 20/41 (2022.01)] 20 Claims
OG exemplary drawing
 
1. A collaboration assessment system to evaluate a collaboration effort of two or more individuals, the collaboration assessment system comprising:
an output device;
an image capture device configured to obtain video data;
a computation engine comprising processing circuitry configured to execute a machine learning system comprising a hierarchical collaboration assessment model having a plurality of machine learning models organized according to a hierarchy, the hierarchy defining a plurality of collaboration assessment levels, wherein each of the plurality of machine learning models comprises one or more neural networks, wherein the processing circuitry executes the machine learning system to:
identify a subset of sample video data included in training data, the subset of sample video data being labeled with an underrepresented classification code of a plurality of classification codes associated with assessing collaboration effort,
generate mixup training data to at least include the training data and pseudo-data created based on the subset of sample video data,
train the one or more neural network of each of the plurality of machine learning models using the mixup training data,
receive temporal data comprising the video data,
apply a first model of the plurality of machine learning models of the hierarchical collaboration assessment model to identify, using the neural network of the first model, at a first collaboration assessment level of the plurality of collaboration assessment levels and based at least on low-level feature tracking of actions of each of the two or more individuals depicted in the video data, first behaviors comprising at least one of bodily movements or facial expressions, and
apply, to the outputs of the first model, a second model of the plurality of machine learning models of the hierarchical collaboration assessment model to identify, using the neural network of the second model and based at least on outputs of the first model and the actions of each of the two or more individuals depicted in the video data, second behaviors at a second collaboration assessment level of the plurality of collaboration assessment levels,
generate, based at least on the first behaviors at the first collaboration assessment level and the second behaviors at the second collaboration assessment level, an indication of at least one of: an assessment of a collaboration effort of the two or more individuals or respective assessments of individual contributions of the two or more individuals to the collaboration effort, wherein the assessment or the respective assessments include a classification code of the plurality of classification codes, and
output the indication via the output device.