| CPC G06F 40/40 (2020.01) | 18 Claims |
|
1. A method, comprising:
obtaining a language model to be audited to determine an ability of the language model to incorporate common sense into processed responses performed by the language model;
providing a plurality of common sense tests to the language model in which two or more of the plurality of common sense tests correspond to different types of common sense tests, the plurality of common sense tests individually including one or more complex problems having multiple parameters or multiple answers, the plurality of common sense tests individually providing an indication of the ability of the language model to reflect laymen understanding of the world in the processed responses and including two or more of a crystallized ability test, a prototype analysis test, a rediscovery test, or a tacit knowledge test;
obtaining model results based on the processed responses of the language model with respect to the plurality of common sense tests;
determining, based on the model results, a first common sense reasoning ability of the language model with respect to a first type of common sense test;
determining, based on the model results, a second common sense reasoning ability of the language model with respect to a second type of common sense test;
obtaining one or more proposed changes to the language model based on one or more of the first common sense reasoning ability or the second common sense reasoning ability; and
implementing the one or more proposed changes to the language model.
|