US 12,147,513 B1
Dynamic evaluation of language model prompts for model selection and output validation and methods and systems of the same
Payal Jain, London (GB); Tariq Husayn Maonah, London (GB); Mariusz Saternus, Cracow (PL); Daniel Lewandowski, Cracow (PL); Biraj Krushna Rath, London (GB); Stuart Murray, London (GB); and Philip Davies, London (GB)
Assigned to Citibank, N.A., New York, NY (US)
Filed by Citibank, N.A., New York, NY (US)
Filed on Apr. 11, 2024, as Appl. No. 18/633,293.
Int. Cl. G06F 21/31 (2013.01); G06F 21/62 (2013.01); G06F 40/20 (2020.01)
CPC G06F 21/31 (2013.01) [G06F 21/6218 (2013.01); G06F 40/20 (2020.01)] 18 Claims
OG exemplary drawing
 
1. A non-transitory computer-readable storage medium comprising instructions thereon, wherein the instructions when executed by at least one data processor of a system, cause the system to:
receive an output generation request from a user device,
wherein the user device is associated with an authentication token, and
wherein the output generation request includes a prompt for generation of a text-based output using a first large-language model (LLM);
authenticate the user device based on the authentication token;
determine a performance metric value associated with the output generation request,
wherein the performance metric value indicates an estimated resource requirement for the output generation request;
identify, based on an attribute of the output generation request, a first prompt validation model of a plurality of prompt validation models;
provide the output generation request to the first prompt validation model to modify the prompt,
wherein modifying the prompt comprises:
determining that the prompt includes a forbidden token; and
generating the modified prompt by omitting the forbidden token;
compare the performance metric value of the output generation request with a first performance criterion associated with the first LLM of a plurality of LLMs;
in response to determining that the performance metric value satisfies the first performance criterion, provide the prompt to the first LLM to generate an output;
provide the output to an output validation model to generate a validation indicator associated with the output; and
in response to generating the validation indicator, transmit the output to a server system enabling access to the output by the user device.