US 11,657,288 B2
Convolutional computing using multilayered analysis engine
Panu James Turcot, Pacifica, CA (US); Rana el Kaliouby, Milton, MA (US); and Daniel McDuff, Cambridge, MA (US)
Assigned to Affectiva, Inc., Boston, MA (US)
Filed by Affectiva, Inc., Boston, MA (US)
Filed on Jun. 8, 2020, as Appl. No. 16/895,071.
Application 16/895,071 is a continuation in part of application No. 15/910,385, filed on Mar. 2, 2018, granted, now 11,017,250.
Application 15/910,385 is a continuation in part of application No. 15/886,275, filed on Feb. 1, 2018, granted, now 10,592,757, issued on Mar. 17, 2020.
Application 15/886,275 is a continuation in part of application No. 15/875,644, filed on Jan. 19, 2018, granted, now 10,627,817, issued on Apr. 21, 2020.
Application 15/875,644 is a continuation in part of application No. 15/273,765, filed on Sep. 23, 2016, abandoned.
Application 15/273,765 is a continuation in part of application No. 14/796,419, filed on Jul. 10, 2015, abandoned.
Application 14/796,419 is a continuation in part of application No. 14/460,915, filed on Aug. 15, 2014, abandoned.
Application 14/460,915 is a continuation in part of application No. 13/153,745, filed on Jun. 6, 2011, abandoned.
Application 15/886,275 is a continuation in part of application No. 14/144,413, filed on Dec. 30, 2013, granted, now 9,934,425, issued on Apr. 3, 2018.
Application 14/144,413 is a continuation in part of application No. 14/064,136, filed on Oct. 26, 2013, granted, now 9,204,836, issued on Dec. 8, 2015.
Application 14/064,136 is a continuation in part of application No. 13/153,745, filed on Jun. 6, 2011, abandoned.
Application 15/910,385 is a continuation in part of application No. 15/666,048, filed on Aug. 1, 2017, abandoned.
Application 15/666,048 is a continuation in part of application No. 15/395,750, filed on Dec. 30, 2016.
Application 15/395,750 is a continuation in part of application No. 15/262,197, filed on Sep. 12, 2016, abandoned.
Application 15/262,197 is a continuation in part of application No. 14/796,419, filed on Jul. 10, 2015, abandoned.
Claims priority of provisional application 62/955,493, filed on Dec. 31, 2019.
Claims priority of provisional application 62/954,819, filed on Dec. 30, 2019.
Claims priority of provisional application 62/954,833, filed on Dec. 30, 2019.
Claims priority of provisional application 62/925,990, filed on Oct. 25, 2019.
Claims priority of provisional application 62/926,009, filed on Oct. 25, 2019.
Claims priority of provisional application 62/893,298, filed on Aug. 29, 2019.
Claims priority of provisional application 62/611,780, filed on Dec. 29, 2017.
Claims priority of provisional application 62/593,440, filed on Dec. 1, 2017.
Claims priority of provisional application 62/593,449, filed on Dec. 1, 2017.
Claims priority of provisional application 62/557,460, filed on Sep. 12, 2017.
Claims priority of provisional application 62/541,847, filed on Aug. 7, 2017.
Claims priority of provisional application 62/524,606, filed on Jun. 25, 2017.
Claims priority of provisional application 62/503,485, filed on May 9, 2017.
Claims priority of provisional application 62/469,591, filed on Mar. 10, 2017.
Claims priority of provisional application 62/448,448, filed on Jan. 20, 2017.
Claims priority of provisional application 62/370,421, filed on Aug. 3, 2016.
Claims priority of provisional application 62/301,558, filed on Feb. 29, 2016.
Claims priority of provisional application 62/273,896, filed on Dec. 31, 2015.
Claims priority of provisional application 62/265,937, filed on Dec. 10, 2015.
Claims priority of provisional application 62/222,518, filed on Sep. 23, 2015.
Claims priority of provisional application 62/128,974, filed on Mar. 5, 2015.
Claims priority of provisional application 62/082,579, filed on Nov. 20, 2014.
Claims priority of provisional application 62/047,508, filed on Sep. 8, 2014.
Claims priority of provisional application 62/023,800, filed on Jul. 11, 2014.
Claims priority of provisional application 61/972,314, filed on Mar. 30, 2014.
Claims priority of provisional application 61/953,878, filed on Mar. 16, 2014.
Claims priority of provisional application 61/927,481, filed on Jan. 15, 2014.
Claims priority of provisional application 61/924,252, filed on Jan. 7, 2014.
Claims priority of provisional application 61/916,190, filed on Dec. 14, 2013.
Claims priority of provisional application 61/867,007, filed on Aug. 16, 2013.
Claims priority of provisional application 61/467,209, filed on Mar. 24, 2011.
Claims priority of provisional application 61/447,464, filed on Feb. 28, 2011.
Claims priority of provisional application 61/447,089, filed on Feb. 27, 2011.
Claims priority of provisional application 61/439,913, filed on Feb. 6, 2011.
Claims priority of provisional application 61/414,451, filed on Nov. 17, 2010.
Claims priority of provisional application 61/388,002, filed on Sep. 30, 2010.
Claims priority of provisional application 61/352,166, filed on Jun. 7, 2010.
Claims priority of provisional application 61/844,478, filed on Jul. 10, 2013.
Claims priority of provisional application 61/789,038, filed on Mar. 15, 2013.
Claims priority of provisional application 61/790,461, filed on Mar. 15, 2013.
Claims priority of provisional application 61/793,761, filed on Mar. 15, 2013.
Claims priority of provisional application 61/798,731, filed on Mar. 15, 2013.
Claims priority of provisional application 61/747,651, filed on Dec. 31, 2012.
Claims priority of provisional application 61/747,810, filed on Dec. 31, 2012.
Claims priority of provisional application 61/719,383, filed on Oct. 27, 2012.
Claims priority of provisional application 62/625,274, filed on Feb. 1, 2018.
Claims priority of provisional application 62/524,606, filed on Jun. 25, 2017.
Claims priority of provisional application 62/503,485, filed on May 9, 2017.
Claims priority of provisional application 62/469,591, filed on Mar. 10, 2017.
Claims priority of provisional application 62/448,448, filed on Jan. 20, 2017.
Claims priority of provisional application 62/442,325, filed on Jan. 4, 2017.
Claims priority of provisional application 62/442,291, filed on Jan. 4, 2017.
Claims priority of provisional application 62/370,421, filed on Aug. 3, 2016.
Claims priority of provisional application 62/301,558, filed on Feb. 29, 2016.
Claims priority of provisional application 62/273,896, filed on Dec. 31, 2015.
Claims priority of provisional application 62/265,937, filed on Dec. 10, 2015.
Claims priority of provisional application 62/222,518, filed on Sep. 23, 2015.
Claims priority of provisional application 62/217,872, filed on Sep. 12, 2015.
Claims priority of provisional application 62/439,928, filed on Dec. 29, 2016.
Prior Publication US 2020/0302235 A1, Sep. 24, 2020
Int. Cl. G06F 18/214 (2023.01); G06N 3/08 (2023.01); G06N 3/04 (2023.01); G06V 40/16 (2022.01); G06V 10/82 (2022.01)
CPC G06F 18/2148 (2023.01) [G06F 18/2155 (2023.01); G06N 3/04 (2013.01); G06N 3/08 (2013.01); G06V 10/82 (2022.01); G06V 40/171 (2022.01); G06V 40/174 (2022.01)] 27 Claims
OG exemplary drawing
 
1. A computer-implemented method for convolutional processing comprising:
initializing a computer for convolutional neural network processing;
obtaining, using an imaging device, a plurality of images for processing on the computer for convolutional processing;
training, on the computer, a multilayered analysis engine, using the plurality of images, wherein the multilayered analysis engine comprises multiple layers that include one or more convolutional layers, one or more hidden layers, and at least one output layer;
evaluating a further image, using the multilayered analysis engine, wherein the evaluating includes identifying a facial portion and identifying a facial expression based on the facial portion wherein the evaluating includes calculating a facial expression metric as part of the evaluating, wherein the facial expression metric includes an intensity of the facial expression;
generating a cumulative cognitive state metric as part of the evaluating, based on the facial expression metric, wherein the cumulative cognitive state metric includes an engagement level based on the facial portion, and wherein the cumulative cognitive state metric is computed by summing a plurality of engagement levels and dividing by a number of engagement levels from the plurality of engagement levels that is below a predetermined threshold; and
outputting a convolutional neural network analysis from the at least one output layer.