US 12,406,664 B2
Multimodal assistant understanding using on-screen and device context
Alkeshkumar M. Patel, San Jose, CA (US); Saurabh Adya, San Jose, CA (US); Karan M. Daryanani, San Francisco, CA (US); Myra C. Lukens, San Francisco, CA (US); and Aswath Manoharan, Tucson, AZ (US)
Assigned to Apple Inc., Cupertino, CA (US)
Filed by Apple Inc., Cupertino, CA (US)
Filed on Jan. 31, 2022, as Appl. No. 17/588,887.
Claims priority of provisional application 63/230,564, filed on Aug. 6, 2021.
Prior Publication US 2023/0042224 A1, Feb. 9, 2023
Int. Cl. G10L 15/22 (2006.01); G10L 15/18 (2013.01)
CPC G10L 15/22 (2013.01) [G10L 15/1815 (2013.01); G10L 2015/223 (2013.01); G10L 2015/227 (2013.01); G10L 2015/228 (2013.01)] 42 Claims
OG exemplary drawing
 
1. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device, the one or more programs including instructions for:
displaying, via a screen of the electronic device, a plurality of objects;
receiving an utterance including a user request;
determining whether at least a portion of the user request is ambiguous;
in response to determining that at least the portion of the user request is ambiguous:
determining a first set of context data based on the ambiguous portion of the user request, wherein the first set of context data includes data determined by a digital assistant from the plurality of objects displayed via the screen of the electronic device;
extracting a metadata from the first set of context data; and
determining a response to the user request based on the metadata extracted from the first set of context data, wherein determining the response comprises:
determining whether a meaning for the ambiguous portion of the user request can be determined based on the metadata extracted from the first set of context data; and
in accordance with a determination that the meaning for the ambiguous portion of the user request cannot be determined based on the metadata extracted from the first set of context data:
determining a second set of context data, wherein the second set of context data includes data determined by the digital assistant from sources other than the screen of the electronic device;
generating the response to the user request based on a comparison of the metadata extracted from the first set of context data to another metadata extracted from the second set of context data; and
updating the display, via the screen of the electronic device, to display an object of the plurality of objects as the response to the user request, wherein the object is identified from the plurality of objects based on the comparison of the metadata extracted from the first set of context data to another metadata extracted from the second set of context data.