US 11,816,541 B2
Systems and methods for decomposition of differentiable and non-differentiable models
John Wickens Lamb Merrill, Los Angeles, CA (US); Geoffrey Michael Ward, Los Angeles, CA (US); Sean Javad Kamkar, Los Angeles, CA (US); John Joseph Beahan, Jr., Los Angeles, CA (US); Mark Frederick Eberstein, Los Angeles, CA (US); Jose Efrain Valentin, Los Angeles, CA (US); and Jerome Louis Budzik, Los Angeles, CA (US)
Assigned to ZestFinance, Inc., Burbank, CA (US)
Filed by ZestFinance, Inc., Los Angeles, CA (US)
Filed on Nov. 19, 2019, as Appl. No. 16/688,789.
Claims priority of provisional application 62/806,603, filed on Feb. 15, 2019.
Prior Publication US 2020/0265336 A1, Aug. 20, 2020
Int. Cl. G06F 16/00 (2019.01); G06N 20/00 (2019.01); G06F 16/901 (2019.01); G06N 5/045 (2023.01)
CPC G06N 20/00 (2019.01) [G06F 16/9027 (2019.01); G06N 5/045 (2013.01)] 19 Claims
OG exemplary drawing
 
1. A method implemented by a model evaluation system and comprising:
accessing model access information for a machine learning model, the model access information identifying at least one tree structure used by the machine learning model;
determining a feature contribution value for each of a plurality of features included in a reference input data set or an evaluation input data set, wherein the evaluation input data set represents a protected class population, each feature contribution value identifies a contribution by one of the features to a difference in output generated by the machine learning model for the evaluation input data set, relative to the reference input data set, and the determination of the feature contribution values comprises:
identifying a path between the reference and evaluation input data sets, boundary points along the path, and segments by segmenting the path at each of the boundary points;
determining, for each of the segments, a segment contribution value for each of the features by determining an integral of a gradient for the machine learning model along the segment for each of the features in parallel using a respective one of a plurality of processors in a multi-processor unit of the model evaluation system;
assigning a determined boundary contribution value and an end point contribution value to at least one of the features for each of the boundary points and each of a plurality of endpoints of the path, respectively; and
combining, for each of the features, the corresponding segment, boundary point, and end point contribution values to generate the feature contribution value for the feature;
generating model explanation information using the determined feature contribution values and monitoring execution of the machine learning model, wherein the model explanation information explains the difference in output generated by the model for the evaluation input data set, relative to the reference input data set, and includes information relating to a model-based decision; and
inserting the generated model explanation information into a stored knowledge graph for the machine learning model and generating from the knowledge graph, and outputting, a report for the machine learning model via a graphical user interface to an operator device, wherein the report comprises at least a portion of the model explanation information.