US 12,079,581 B2
Diagnosing sources of noise in an evaluation
Thomas Kehler, San Francisco, CA (US); Markus Guehrs, San Francisco, CA (US); and Sonali Sinha, San Francisco, CA (US)
Assigned to CrowdSmart, Inc., San Francisco, CA (US)
Filed by CrowdSmart, Inc., San Francisco, CA (US)
Filed on Aug. 3, 2022, as Appl. No. 17/880,534.
Application 17/880,534 is a continuation of application No. 17/492,504, filed on Oct. 1, 2021, granted, now 11,455,474.
Claims priority of provisional application 63/086,542, filed on Oct. 1, 2020.
Prior Publication US 2023/0067915 A1, Mar. 2, 2023
This patent is subject to a terminal disclaimer.
Int. Cl. G06F 40/30 (2020.01); G06F 16/245 (2019.01); G06F 18/20 (2023.01); G06F 18/214 (2023.01); G06F 18/22 (2023.01); G06F 18/2415 (2023.01); G06N 3/047 (2023.01); G06N 3/08 (2023.01); G06N 5/01 (2023.01); G06N 5/02 (2023.01); G06N 5/022 (2023.01); G06N 5/04 (2023.01); G06N 7/01 (2023.01); G06N 20/00 (2019.01); G06T 15/08 (2011.01)
CPC G06F 40/30 (2020.01) [G06F 16/245 (2019.01); G06F 18/214 (2023.01); G06F 18/22 (2023.01); G06F 18/24155 (2023.01); G06F 18/295 (2023.01); G06N 3/047 (2023.01); G06N 3/08 (2013.01); G06N 5/01 (2023.01); G06N 5/02 (2013.01); G06N 5/022 (2013.01); G06N 5/04 (2013.01); G06N 7/01 (2023.01); G06N 20/00 (2019.01); G06T 15/08 (2013.01); G06T 2200/24 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A computer-implemented method comprising:
obtaining a probabilistic graphical network model based on a data set including time series data for respective entities of a plurality of entities for one or more features corresponding to an evaluation;
obtaining observed data corresponding to a feature represented in the probabilistic graphical network model;
training a machine learning model based on a training data set of performance metrics corresponding to the feature;
determining, by the machine learning model, at least one score for observed performance or predicted performance based on the observed data corresponding to the feature;
determining a distribution of posterior predictions based on the probabilistic graphical network model, the distribution based on a posterior prediction determined for each entity based on respective time series data including one or more scores encoded by the probabilistic graphical network model; and
determining, on a normalized scale, a distance between a peak of the distribution and at least one respective score of observed or predicted performance, the distance being indicative of an amount of observed or predicted error in evaluation of the feature by the entities.