US 12,254,785 B2
Dynamically adjusting instructions in an augmented-reality experience
Jessica Lee, Brooklyn, NY (US); David Trotter Oleson, Ruschlikon (CH); Fabian Roth, Zurich (CH); and Nils Grimsmo, Wollerau (CH)
Assigned to GOOGLE LLC, Mountain View, CA (US)
Filed by Google LLC, Mountain View, CA (US)
Filed on Oct. 19, 2022, as Appl. No. 17/969,303.
Prior Publication US 2024/0135835 A1, Apr. 25, 2024
Prior Publication US 2024/0233569 A9, Jul. 11, 2024
Int. Cl. G09B 7/04 (2006.01); G06F 3/04845 (2022.01); G06F 40/205 (2020.01); G06T 11/60 (2006.01); G06V 10/94 (2022.01); G06V 20/70 (2022.01); G06V 30/12 (2022.01); G06V 30/19 (2022.01)
CPC G09B 7/04 (2013.01) [G06F 3/04845 (2013.01); G06F 40/205 (2020.01); G06T 11/60 (2013.01); G06V 10/945 (2022.01); G06V 20/70 (2022.01); G06V 30/127 (2022.01); G06V 30/19133 (2022.01); G06V 30/19147 (2022.01)] 20 Claims
OG exemplary drawing
 
1. A computing system, the system comprising:
one or more processors; and
one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
obtaining image data, wherein the image data is descriptive of one or more images, wherein the one or more images are descriptive of an environment;
processing the image data with a machine-learned model to generate semantic data, wherein the semantic data is descriptive of a semantic understanding of at least a portion of the one or more images, wherein the machine-learned model comprises a language model trained for multi-part quantitative reasoning, wherein the language model was trained on a plurality of mathematical proofs;
processing the semantic data with the machine-learned model to generate a multi-part response for a detected problem in the one or more images, wherein the multi-part response is descriptive of a proof for the detected problem;
determining an error in the one or more images based at least in part on the multi-part response;
determining a corrective action based on the multi-part response and the error, wherein the corrective action is descriptive of at least one of a replacement for the error or an action to fix the error;
generating one or more augmented images based on the correction action and the one or more images, wherein the one or more augmented images comprise one or more user interface elements rendered into the one or more images, wherein the one or more user interface elements comprise text superimposed over at least a portion of the one or more images, wherein the text of the one or more user interface elements comprise informational data descriptive of the corrective action;
providing the one or more augmented images for display based on the corrective action;
obtaining additional image data after providing the one or more augmented images for display, wherein the additional image data is descriptive of one or more additional images, wherein the one or more additional images are descriptive of the one or more pages with user-generated text;
processing the additional image data and the multi-part response to determine a particular portion of the user-generated text deviates from the multi-part response; and
generating one or more second augmented images that indicate the particular portion of the user-generated text that has a determined error.