US 12,223,270 B2
Systems and methods near negative distinction for evaluating NLP models
Philippe Laban, New York, NY (US); Chien-Sheng Wu, Mountain View, CA (US); Wenhao Liu, Redwood City, CA (US); and Caiming Xiong, Menlo Park, CA (US)
Assigned to Salesforce, Inc., San Francisco, CA (US)
Filed by Salesforce, Inc., San Francisco, CA (US)
Filed on Jun. 10, 2022, as Appl. No. 17/837,546.
Claims priority of provisional application 63/299,791, filed on Jan. 14, 2022.
Prior Publication US 2023/0229861 A1, Jul. 20, 2023
Int. Cl. G06F 40/284 (2020.01)
CPC G06F 40/284 (2020.01) 20 Claims
OG exemplary drawing
 
1. A method of evaluating a model via Near-Negative Distinction (NND), the method comprising:
receiving, via a communication interface, an evaluation dataset comprising a plurality of unit tests, the unit tests having:
an input context, and
a first candidate and a second candidate that are generated in response to the input context, wherein the first candidate is associated with a first quality notation, and the second candidate is associated with a second quality notation;
determining, via a model, a first likelihood of generating the first candidate in response to the input context;
determining, via the model, a second likelihood of generating the second candidate in response to the input context;
determining a count of unit tests in the evaluation dataset where the model passed;
determining an aggregate pass rate based on the determined count of unit tests, the aggregate pass rate indicating whether the model produces candidates that correspond to the first quality notation;
determining whether the first likelihood is greater than the second likelihood based on the determined aggregate pass rate; and
based on a determination that the first likelihood is greater than the second likelihood, determining whether the model passed a unit test in the evaluation dataset, wherein the first quality notation is associated with a higher quality candidate and the second quality notation is associated with a lower quality candidate.