US 11,941,650 B2
Explainable machine learning financial credit approval model for protected classes of borrowers
Douglas C. Merrill, Los Angeles, CA (US); Michael Edward Ruberry, Los Angeles, CA (US); Ozan Sayin, West Hollywood, CA (US); Bojan Tunguz, Greencastle, IN (US); Lin Song, Sugar Land, TX (US); Esfandiar Alizadeh, Venice, CA (US); Melanie Eunique DeBruin, Northridge, CA (US); Yachen Yan, Los Angeles, CA (US); Derek Wilcox, Los Angeles, CA (US); John Candido, Burbank, CA (US); Benjamin Anthony Solecki, Los Angeles, CA (US); Jiahuan He, Los Angeles, CA (US); Jerome Louis Budzik, Altadena, CA (US); and Sean Javad Kamkar, Los Angeles, CA (US)
Assigned to ZestFinance, Inc., Burbank, CA (US)
Filed by ZestFinance, Inc., Los Angeles, CA (US)
Filed on Aug. 1, 2018, as Appl. No. 16/052,293.
Claims priority of provisional application 62/682,714, filed on Jun. 8, 2018.
Claims priority of provisional application 62/666,991, filed on May 4, 2018.
Claims priority of provisional application 62/641,176, filed on Mar. 9, 2018.
Claims priority of provisional application 62/540,419, filed on Aug. 2, 2017.
Prior Publication US 2019/0043070 A1, Feb. 7, 2019
Int. Cl. G06Q 30/0204 (2023.01); G06F 9/54 (2006.01); G06N 5/04 (2023.01); G06N 20/00 (2019.01); G06Q 30/0201 (2023.01)
CPC G06Q 30/0205 (2013.01) [G06F 9/54 (2013.01); G06N 5/04 (2013.01); G06N 20/00 (2019.01); G06Q 30/0201 (2013.01)] 16 Claims
OG exemplary drawing
 
1. A method of machine learning model dimensionality reduction evaluation, the method comprising:
receiving, by a model evaluation system, decisioning system information from a decisioning system communicably coupled to the model evaluation system via one or more communication networks and configured to apply a machine learning financial credit model, wherein the decisioning system information identifies predictors used by the machine learning financial credit model;
accessing, by the model evaluation system, input rows used by the machine learning financial credit model, including attributes, and protected class membership information corresponding to each of the input rows;
repeatedly testing and evaluating, by the model evaluation system, protected class machine learning models trained using the decisioning system information, the input rows, and the protected class membership information to automatically determine a set of the predictors that result in a protected class machine learning model that satisfies an accuracy threshold, wherein the protected class machine learning models are trained to predict membership in a protected class by using at least one of the predictors, each of the predictors is a set of one or more variables of the input rows, and the protected class includes at least one of a person's race, color, religion, national origin, sex, age, or disability status;
for each predictor of a plurality of subsets of the predictors, determining, by the model evaluation system, a protected class ranking value and a decisioning system ranking value, wherein the decisioning system ranking value is determined by providing one or more score requests to the decisioning system to cause the decisioning system to generate and return a decisioning output for at least one of the input rows by executing the machine learning financial credit model;
automatically identifying, by the model evaluation system, one or more of the subsets of predictors having decisioning system ranking values below a score impact threshold and protected class ranking values above a prediction impact threshold;
providing, by the model evaluation system, a modification request via the one or more computer networks to the decisioning system to cause the decisioning system to modify the machine learning financial credit model to remove the one or more of the subsets of predictors to thereby reduce a disproportionately adverse impact on protected class members of the one of the predictors on one or more subsequent decisioning outputs generated by one or more subsequent executions of the modified machine learning financial credit model; and
instructing, by the model evaluation system, the decisioning system to start operation in a production environment, wherein the modified machine learning financial credit model comprises a credit underwriting model for facilitating credit decisions and the operation of the decisioning system generates decisioning outputs responsive to evaluation requests and indicative of credit worthiness of persons associated with the evaluation requests.