US 11,748,667 B2
Methods and systems for the measurement of relative trustworthiness for technology enhanced with AI learning algorithms
Niraj Bhargava, Ottawa (CA); Fred Speckeen, Kitchener (CA); Evan W. Steeg, Kingston (CA); Jorge Deligiannis, Ottawa (CA); and Gaston Gonnet, Waterloo (CA)
Assigned to NuEnergy.ai, Ottawa (CA)
Filed by NuEnergy.ai, Ottawa (CA)
Filed on Jun. 27, 2019, as Appl. No. 16/454,613.
Claims priority of provisional application 62/690,519, filed on Jun. 27, 2018.
Prior Publication US 2020/0005168 A1, Jan. 2, 2020
Int. Cl. G06Q 30/00 (2023.01); G06N 20/20 (2019.01); G06N 5/04 (2023.01); G06F 17/18 (2006.01); G06N 20/00 (2019.01); G06F 18/22 (2023.01)
CPC G06N 20/20 (2019.01) [G06F 17/18 (2013.01); G06F 18/22 (2023.01); G06N 5/04 (2013.01); G06N 20/00 (2019.01)] 17 Claims
OG exemplary drawing
 
1. A computer implemented method for used in determining a machine trust index of artificial intelligence processes, the method comprising:
determining a plurality of evaluation criteria from a pool of evaluation criteria for use in generating a machine trust index (MTI) score for an artificial intelligence (AI) process, each of the evaluation criteria associated with an evaluation process to be used in evaluating the respective evaluation criteria;
evaluating each of the plurality of determined evaluation criteria according to the evaluation process associated with the respective evaluation criteria;
executing an MTI determination process to generate the MTI score based on the evaluated evaluation criteria;
storing in an MTI database the MTI score in association with a time the MTI score was generated;
receiving feedback associated with trustworthiness of one or more AI process each associated with a respective MTI score and storing in the MTI database
automatically applying a supervised or reinforcement learning algorithm to a plurality of generated MTI scores stored in the MTI database to determine which evaluation criteria in the pool of evaluation criteria are predictive of a particular feedback received about the plurality of AI processes;
automatically adjusting the MTI determination process based on which evaluation criteria are predictive of the particular user feedback;
receiving additional texts;
determining similarity matches between the additional texts and text of one or more evaluation criteria in the pool of evaluation criteria;
presenting matches to a user via a user interface;
receiving an indication of whether to add a new evaluation criteria to the pool of evaluation criteria based on the presented matches and adding the new evaluation criteria to the pool of questions
using a subset of generated MTIs stored in the MTI database, determining which evaluation criteria in the pool of evaluation criteria are significant in generating the MTI; and
suggesting possible enhancements for improving the results of the significant evaluation criteria.