	US 12,266,158 B2
	Systems and methods for evaluating the error rate of human-generated data
Leon Bergen, San Diego, CA (US); Kenneth Ko, Santa Clara, CA (US); and Pelu Tran, San Francisco, CA (US)
Assigned to Ferrum Health Inc., Sunnyvale, CA (US)
Filed by Ferrum Health Inc., Sunnyvale, CA (US)
Filed on Mar. 21, 2022, as Appl. No. 17/700,053.
Claims priority of provisional application 63/168,566, filed on Mar. 31, 2021.
Prior Publication US 2022/0327808 A1, Oct. 13, 2022
Int. Cl. G06V 10/776 (2022.01); G06F 40/279 (2020.01); G06V 10/764 (2022.01); G06V 10/98 (2022.01); G06V 20/70 (2022.01)

CPC G06V 10/776 (2022.01) [G06F 40/279 (2020.01); G06V 10/764 (2022.01); G06V 10/98 (2022.01); G06V 20/70 (2022.01); G06V 2201/032 (2022.01)]

18 Claims

1. A method for evaluating an error rate of human-generated data used with a machine learning classifier, comprising:

obtaining a set of pairs of data, wherein each pair comprises an image and human-generated text describing an aspect of the image;

for each pair of data, evaluating the image in the pair using a trained image classifier and evaluating the human-generated text in the pair using a trained language classifier;

identifying one or more of the pairs of data for which there is a discrepancy between the evaluation by the trained image classifier and the evaluation by the trained language classifier;

for each of the identified pairs of data, obtaining a human-generated label of the image in the pair of data, where the human-generated label represents a second text describing the aspect of the image;

forming a joint distribution of the trained image classifier and the trained language classifier evaluations;

obtaining a performance characteristic of the trained language classifier; and

based on the human-generated labels, the joint distribution, and the performance characteristic, generating one or more of an estimate of the expected occurrence of each classification by the image classifier, an error rate in the human-generated text describing the aspect of the image, or a performance of the trained image classifier, to determine accuracy of the machine learning classifier's performance at correctly classifying input data and/or accuracy of human's ability to classify data which is used in training of the machine learning classifier, wherein the generating the one or more of the estimate of the expected occurrence of the each image classification by the trained image classifier, the error rate in the human-generated text describing the aspect of the image, or the performance of the trained image classifier further comprises:

defining a joint distribution between latent and observed variables of the image using an assumption of conditional independence;

generating a set of initializations for the joint distribution;

for each generated initialization in the set of initializations, performing an expectation operation;

for each result of performing an expectation operation, performing a maximization operation;

repeating the expectation operation and maximization operation for each generated initialization until a state of convergence is reached;

identifying an output of the expectation operation and the maximization operation corresponding to each generated initialization; and

selecting the output of the expectation operation and the maximization operation corresponding to each generated that maximizes the likelihood of observed data.