US 12,450,422 B2
Intelligent capturing of user-viewed content for note keeping
David J. Conger, Issaquah, WA (US); Iman Abdullahi Yusuf, Kent, WA (US); Fnu Primadona, Mill Creek, WA (US); and Eric Anderson, Seattle, WA (US)
Assigned to Microsoft Technology Licensing, LLC, Redmond, WA (US)
Filed by Microsoft Technology Licensing, LLC
Filed on Aug. 9, 2023, as Appl. No. 18/446,812.
Prior Publication US 2025/0053728 A1, Feb. 13, 2025
Int. Cl. G06F 3/048 (2013.01); G06F 40/169 (2020.01); G06V 30/10 (2022.01)
CPC G06F 40/169 (2020.01) [G06V 30/10 (2022.01)] 13 Claims
OG exemplary drawing
 
1. A device comprising:
a display for presenting information to a user;
a user interface for receiving user input from the user;
a processor; and
a memory storing executable instructions which, when executed by the processor, cause the processor, alone or in combination with other processors, to perform:
displaying information on the display;
receiving the user input formatted in natural language, the user input including a request to capture the information on the display;
capturing the information responsive to the request, wherein the information includes at least one of a screenshot, text, metadata, or a spatial relationship between content elements;
operating a prompt generator to structure a prompt for a generative Artificial Intelligence (AI) model, wherein the prompt is based on the information captured from the display and the request and the prompt causes the generative AI model to generate a note based on the information and the request;
storing the note and the information in a user data structure;
receiving a second request from the user input to refine the note using the information;
operating the prompt generator to structure an updated prompt for the generative AI model, wherein the updated prompt is based on the note and the information retrieved from the user data structure and the second request, and the updated prompt causes the generative AI model to generate a refined note based on the information and the second request;
sending the updated prompt to the generative AI model to generate the refined note; and
presenting the refined note to the user.