US 11,875,559 B2
Systems and methodologies for automated classification of images of stool in diapers
Brian Alexander Cohn, Manhattan Beach, CA (US); Austin Richard Napana Lamb, Manhattan Beach, CA (US); Jamie Chen, Los Altos, CA (US); Melissa Trieu, San Jose, CA (US); Nikki Jamshidbaigi, Salt Lake City, UT (US); Pavle Joshua Medvidovic, Manhattan Beach, CA (US); and Craig Gravina, Farmington, CT (US)
Assigned to OBVIO HEALTH USA, INC., New York, NY (US)
Filed by OBVIO HEALTH USA, INC., New York, NY (US)
Filed on Jul. 12, 2021, as Appl. No. 17/372,852.
Prior Publication US 2023/0012236 A1, Jan. 12, 2023
Int. Cl. G06N 3/084 (2023.01); G06T 3/40 (2006.01); G06N 3/045 (2023.01); G06V 10/94 (2022.01); G06V 10/25 (2022.01); G06V 10/22 (2022.01)
CPC G06V 10/95 (2022.01) [G06N 3/045 (2023.01); G06N 3/084 (2013.01); G06T 3/4046 (2013.01); G06V 10/235 (2022.01); G06V 10/25 (2022.01)] 20 Claims
OG exemplary drawing
 
1. A method involving use of multiple convolutional neural networks and multiple segmentation masks to programmatically generate a rating for one or more test images, the method comprising:
(a) training, using a first plurality of digital images of stool, a first convolutional neural network to identify one or more areas of a digital image that correspond to stool, such training comprising, for each respective batch of digital images of the first plurality of digital images, for each of a plurality of iterations,
(i) calculating, by the first convolutional neural network for each respective digital image of the respective batch of digital images, a class probability value for each pixel of the respective digital image, each class probability value being calculated based on one or more parameters associated with one or more layers of the first convolutional neural network, the one or more parameters including one or more weight parameters and one or more bias parameters,
(ii) calculating, by the first convolutional neural network based on a loss function, a respective loss value for the respective batch, such calculating comprising
(A) comparing, for each respective pixel of each respective digital image of the respective batch, the calculated respective class probability value for the respective pixel to a respective encoded truth mask representing an indication of pixels of the respective digital image that were manually identified by a person as corresponding to stool, and
(B) determining the respective loss value for the respective batch based on summing up loss values determined based on the comparisons of the calculated class probability values for the pixels of the respective digital images with the encoded truth masks,
(iii) updating one or more parameters of the first convolutional neural network, such updating comprising
(A) calculating a gradient of a matrix of the calculated class probability values,
(B) starting from this calculated gradient of the matrix of the calculated class probability values, backpropagating through layers of the first convolutional neural network and calculating gradients for parameters associated with these layers, including weight parameters and bias parameters associated with these layers,
(C) performing, for each respective parameter of a set of one or more parameters of the first convolutional neural network, a parameter update based on a corresponding calculated gradient for that respective parameter and a step size value;
(b) training, using a second plurality of digital images of diapers, a second convolutional neural network to identify an area of a digital image that corresponds to a diaper, such training comprising, for each respective batch of digital images of the second plurality of digital images, for each of a plurality of iterations,
(i) calculating, by the second convolutional neural network for each respective digital image of the respective batch of digital images, a class probability value for each pixel of the respective digital image, each class probability value being calculated based on one or more parameters associated with one or more layers of the second convolutional neural network, the one or more parameters including one or more weight parameters and one or more bias parameters,
(ii) calculating, by the second convolutional neural network based on a loss function, a respective loss value for the respective batch, such calculating comprising
(A) comparing, for each respective pixel of each respective digital image of the respective batch, the calculated respective class probability value for the respective pixel to a respective encoded truth mask representing an indication of pixels of the respective digital image that were manually identified by a person as corresponding to a diaper, and
(B) determining the respective loss value for the respective batch based on summing up loss values determined based on the comparisons of the calculated class probability values for the pixels of the respective digital images with the encoded truth masks,
(iii) updating one or more parameters of the second convolutional neural network, such updating comprising
(A) calculating a gradient of a matrix of the calculated class probability values,
(B) starting from this calculated gradient of the matrix of the calculated class probability values, backpropagating through layers of the second convolutional neural network and calculating gradients for parameters associated with these layers, including weight parameters and bias parameters associated with these layers,
(C) performing, for each respective parameter of a set of one or more parameters of the second convolutional neural network, a parameter update based on a corresponding calculated gradient for that respective parameter and a step size value;
(c) training, using a third plurality of digital images of diapers with stool, a third convolutional neural network to classify stool depicted in an image, such training comprising
(i) generating, for each respective digital image of the third plurality of digital images, a respective first mask by providing the respective digital image to the first convolutional neural network as input and receiving as output the generated respective first mask representing an identification of one or more areas of the respective digital image that correspond to stool,
(ii) generating, for each respective digital image of the third plurality of digital images, a respective second mask by providing the respective digital image to the second convolutional neural network as input and receiving as output the generated respective second mask representing an identification of an area of the respective digital image that corresponds to a diaper,
(iii) generating, for each respective digital image of the third plurality of digital images, a respective third mask representing an intersection of the respective first mask for the respective digital image and the respective second mask for the respective digital image,
(iv) generating a fourth plurality of digital images from the third plurality of digital images by, for each respective digital image of the third plurality of digital images, applying the generated respective third mask to the respective digital image,
(v) for each respective batch of digital images of the fourth plurality of digital images, for each of a plurality of iterations,
(A) calculating, by the third convolutional neural network for each respective digital image of the respective batch of digital images, a respective set of class probability values for the respective digital image, each class probability value being calculated based on one or more parameters associated with one or more layers of the third convolutional neural network, the one or more parameters including one or more weight parameters and one or more bias parameters,
(B) calculating, by the third convolutional neural network based on a loss function, a respective loss value for the respective batch, such calculating comprising
(I) calculating, for each respective digital image of the respective batch, a respective loss value based on the calculated respective class probability values and a respective label associated with the respective digital image representing an indication of a classification of stool in the digital image on a rating scale by a human rater, and
(II) determining the respective loss value for the respective batch based on the calculated loss values for the digital images of the respective batch,
(C) updating one or more parameters of the third convolutional neural network based on use of gradient descent, such updating comprising
(I) calculating a gradient of a matrix of the calculated class probability values,
(II) starting from this calculated gradient of the matrix of the calculated class probability values, backpropagating through layers of the third convolutional neural network and calculating gradients for parameters associated with these layers, including weight parameters and bias parameters associated with these layers, this backpropagation involving at least some use of skip connections,
(III) performing, for each respective parameter of a set of one or more parameters of the third convolutional neural network, a parameter update based on a corresponding calculated gradient for that respective parameter and a step size value;
(d) providing, to a user via a web browser of the user, an interface configured to allow for upload of one or more digital images;
(e) receiving, at a server based on user input corresponding to interaction with the interface to upload digital images, a first set of digital images;
(f) utilizing the trained third convolutional neural network to evaluate the first set of digital images and generate a rating classification for each digital image of the first set of digital images, comprising
(i) generating, for each respective digital image of the first set of digital images, a respective first mask by providing the respective digital image to the first convolutional neural network as input and receiving as output the generated respective first mask representing an identification of one or more areas of the respective digital image that corresponds to stool,
(ii) generating, for each respective digital image of the first set of digital images, a respective second mask by providing the respective digital image to the second convolutional neural network as input and receiving as output the generated respective second mask representing an identification of an area of the respective digital image that corresponds to a diaper,
(iii) generating, for each respective digital image of the first set of digital images, a respective third mask representing an intersection of the respective first mask for the respective digital image and the respective second mask for the respective digital image,
(iv) generating a second set of digital images from the first set of digital images by, for each respective digital image of the first set of digital images, applying the generated respective third mask to the respective digital image,
(v) calculating, by the third convolutional neural network for each respective digital image of the second set of digital images, a respective set of class probability values for the respective digital image, each class probability value being calculated based on one or more parameters associated with one or more layers of the third convolutional neural network, the one or more parameters including one or more weight parameters and one or more bias parameters,
(vi) determining, for each respective digital image of the second set of digital images, based on the calculated respective set of class probability values for that respective digital image, a class to assign the respective digital image to, and associating the corresponding digital image of the first set of digital images with a rating corresponding to that class.