| CPC G06F 40/40 (2020.01) [G06F 40/106 (2020.01)] | 20 Claims |
|
1. A method for evaluating and assessing performance of pre-trained large language models using layered assessments via a graphical user interface (GUI), the method comprising:
obtaining a set of application domains of a pre-trained large language model (LLM) in which the pre-trained LLM will be used,
wherein the pre-trained LLM is configured to generate, in response to a received input, a response;
using the set of application domains, obtaining a set of guidelines defining one or more operation boundaries of the pre-trained LLM,
determining a set of layers for the pre-trained LLM associated with one or more guidelines of the set of guidelines,
wherein each layer within the set of layers is mapped to: (1) a set of variables associated with the one or more guidelines of each corresponding layer and (2) a set of benchmarks associated with the one or more guidelines of each corresponding layer,
wherein each variable in the set of variables represents an attribute identified within the one or more guidelines of each corresponding layer, and
wherein each benchmark in the set of benchmarks is configured to indicate a degree of satisfaction of the pre-trained LLM with the one or more guidelines associated with the corresponding layer;
using the determined set of layers, dynamically evaluating the pre-trained LLM against the corresponding sets of benchmarks for the set of layers using a set of assessments, wherein each assessment of the set of assessments comprises (1) a layered prompt associated with a set of one or more layers of the set of layers and (2) a layer-specific expected response of the layered prompt, by, for each assessment:
supplying the layered prompt of the assessment into the pre-trained LLM,
wherein the layered prompt is configured to test the corresponding degree of satisfaction of the pre-trained LLM with the one or more guidelines associated with the corresponding layer,
responsive to inputting the layered prompt, receiving, from the pre-trained LLM, a layer-specific model response, and
using the received layer-specific model response, comparing the layer-specific expected response of the assessment to the layer-specific model response received from the pre-trained LLM,
wherein subsequent assessments of the set of assessments occurring subsequent to previous assessments of the set of assessments are dynamically constructed using the comparison of the layer-specific expected response of the previous assessments to the layer-specific model response received from the pre-trained LLM for the previous assessments;
using the dynamic evaluation of the pre-trained LLM, assigning a score, for each layer, to the pre-trained LLM, by:
for each assessment, comparing the layer-specific expected response of the assessment to the layer-specific model response received from the pre-trained LLM, and
using the comparisons, determining a particular degree of satisfaction of the pre-trained LLM with the one or more guidelines associated with the corresponding assessment, in accordance with the set of benchmarks for the corresponding layer; and
generating for display at the GUI, a graphical layout indicating the assigned scores, wherein the graphical layout includes a first graphical representation of each layer of the set of layers and a second graphical representation of the corresponding assigned score for each layer of the set of layers.
|