US 12,242,522 B2
Confidence enhancement for responses by document-based large language models
Matthew Jonathan Gardner, Irvine, CA (US)
Assigned to Microsoft Technology Licensing, LLC, Redmond, WA (US)
Filed by Microsoft Technology Licensing, LLC, Redmond, WA (US)
Filed on May 4, 2023, as Appl. No. 18/311,973.
Claims priority of provisional application 63/450,069, filed on Mar. 5, 2023.
Prior Publication US 2024/0296279 A1, Sep. 5, 2024
Int. Cl. G06F 16/332 (2019.01); G06F 16/3329 (2025.01); G06F 16/338 (2019.01); G06F 21/54 (2013.01); G06F 21/62 (2013.01); G06F 40/134 (2020.01); G06F 40/169 (2020.01); G06F 40/279 (2020.01); G06N 20/00 (2019.01); G10L 15/22 (2006.01)
CPC G06F 16/3329 (2019.01) [G06F 16/338 (2019.01); G06F 21/54 (2013.01); G06F 21/6218 (2013.01); G06F 40/169 (2020.01); G06F 40/279 (2020.01); G06N 20/00 (2019.01); G10L 15/22 (2013.01); G06F 40/134 (2020.01); G06F 2221/032 (2013.01); G06F 2221/033 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A system for implementing confidence enhancement for responses by document-based artificial intelligence (“AI”) and/or machine learning (“ML”) models, the system comprising:
a computing system, comprising:
at least one processor; and
a computer storage medium communicatively coupled to the at least one processor, the computer storage medium storing instructions that, when executed by the at least one processor, causes the computing system to perform operations comprising:
generating a first prompt requesting information about one or more data items, based on a natural language (“NL”) request that is received via a user interface;
providing the first prompt to an AI/ML-based system, the first prompt causing the AI/ML-based system to generate a structured object output including:
the requested information from the one or more data items; and
one or more citations, the one or more citations comprising one or more of a citation to each data item or a citation to each portion of each data item, from which the requested information or portions of the requested information were extracted;
generating a response to the NL request, the response comprising one or more representations of the requested information and the corresponding one or more citations;
causing, via the user interface, presentation of the generated response within a communication session between the user interface and the computing system;
generating a second prompt, the second prompt comprising the extracted information, the corresponding citations, and the one or more data items;
providing the second prompt to a second AI/ML-based system, the second prompt causing the second AI/ML-based system to:
for each citation, generate a first accuracy value corresponding to the accuracy of the citation based on the citation and the corresponding cited portion in the cited data item; and
for each of the one or more representations of the requested information, determine accuracy of said representation based on a comparison of said representation with original language in the corresponding cited portion in the cited data item based on the corresponding citation, and return a second accuracy value corresponding to the determined accuracy of said representation;
generating a reliability indicator for one or more of the citations, based on the returned first accuracy value;
causing each generated first reliability indicator to be displayed as visually coupled to the corresponding citations;
generating a second reliability indicator for each of the one or more representations of the requested information, based on the returned second accuracy value; and
causing each generated second reliability indicator to be displayed, within the structured object output, visually coupled to corresponding each of the one or more representations of the requested information.