US 11,922,126 B1
Use of semantic confidence metrics for uncertainty estimation in large language models
Jiaxin Zhang, Mountain View, CA (US); Kamalika Das, Saratoga, CA (US); and Sricharan Kallur Palli Kumar, Ann Arbor, MI (US)
Assigned to Intuit Inc., Mountain View, CA (US)
Filed by Intuit Inc., Mountain View, CA (US)
Filed on Jul. 28, 2023, as Appl. No. 18/360,956.
Int. Cl. G06F 40/30 (2020.01)
CPC G06F 40/30 (2020.01) 20 Claims
OG exemplary drawing
 
1. A method comprising:
receiving a user input for input to a machine learning model (MLM), wherein the MLM comprises a language processing MLM;
generating a plurality of modified inputs that are based on the user input, wherein the plurality of modified inputs each are semantically related to the user input;
executing the MLM to generate a plurality of model outputs of the MLM, wherein the MLM takes as input a plurality of instances of each of the plurality of modified inputs;
sampling the plurality of model outputs using a statistical sampling strategy to generate a plurality of sampled model outputs;
clustering the plurality of sampled model outputs into a plurality of clusters, wherein each cluster of the plurality of clusters represents a distinct semantic meaning of the plurality of sampled model outputs;
generating a confidence metric for the user input, wherein the confidence metric comprises a predictive entropy of the plurality of clusters; and
routing, automatically in a computing system, the user input based on whether the confidence metric satisfies or fails to satisfy a threshold value.