US 12,475,331 B2
Model capability extraction
Benjamin Goth Zorn, Woodinville, WA (US); Carina Suzana Negreanu, Cambridge (GB); Neil Blunt Toronto, Cambridge (GB); Brian Paul Slininger, Seattle, WA (US); Andrew Donald Gordon, Cambridge (GB); Advait Sarkar, Cambridge (GB); Elnaz Nouri, Seattle, WA (US); Vu Minh Le, Redmond, WA (US); Christian Leopold Bejamin Poelitz, London (GB); Shraddha Govind Barke, La Jolla, CA (US); and Sruti Srinivasa Ragavan, Oxford (GB)
Assigned to Microsoft Technology Licensing, LLC, Redmond, WA (US)
Filed by Microsoft Technology Licensing, LLC, Redmond, WA (US)
Filed on Oct. 20, 2022, as Appl. No. 17/969,922.
Prior Publication US 2024/0135113 A1, Apr. 25, 2024
Prior Publication US 2024/0232545 A9, Jul. 11, 2024
Int. Cl. G06F 40/51 (2020.01); G06F 40/35 (2020.01)
CPC G06F 40/51 (2020.01) [G06F 40/35 (2020.01)] 12 Claims
OG exemplary drawing
 
1. A computing system that determines a capability of a language model without directly querying the language model to ask what capabilities the language model has, said computing system comprising:
a capability extraction system that interacts with the language model to determine the capability of the language model, wherein the capability extraction system interacts with the language model by (i) providing language model input to the language model, (ii) receiving resulting output from the language model, and (iii) evaluating the output to identify the capability of the language model;
a model utilization system that receives a capability indication from the capability extraction system, where the capability indication indicates the capability of the language model, and that utilizes the capability of the language model by tasking the language model to assist in performing an operation;
one or more processors; and
one or more hardware storage devices that store instructions that are executable by the one or more processors to cause the computing system to:
causing the capability extraction system to formulate the language model input, wherein the capability extraction system structures the language model input to include natural language input;
causing the capability extraction system to feed the language model input to the language model;
causing the capability extraction system to access the output from the language model, where the output results from the language model input being provided to the language model, the output being in a form of a non-natural language output or a natural language output that is semantically non-responsive to the natural language input;
causing the capability extraction system to output a capability indication indicating an estimation or determination of the capability of the language model, wherein the capability indication is generated based on the output from the language model;
causing the capability extraction system to provide the capability indication and the output from the language model to the model utilization system;
as a part of a first operation of the model utilization system, causing the model utilization system to use the output, which is an earlier generated output, to perform the first operation, such that the model utilization system does not engage with the language model during the first operation; and
as a part of a second operation of the model utilization system, causing the model utilization system to utilize the capability of the language model by tasking the language model to assist in performing the second operation, such that the model utilization system does engage with the language model during the second operation,
wherein the language model is a large language model.