US 11,783,815 B2
Multimodality in digital assistant systems
Pierre P. Greborio, Sunnyvale, CA (US); Didier Rene Guzzoni, Montsur-Rolle (CH); and Philippe P. Piernot, Palo Alto, CA (US)
Assigned to Apple Inc., Cupertino, CA (US)
Filed by Apple Inc., Cupertino, CA (US)
Filed on Apr. 28, 2022, as Appl. No. 17/732,243.
Application 17/732,243 is a continuation of application No. 16/441,461, filed on Jun. 14, 2019, granted, now 11,348,573.
Claims priority of provisional application 62/820,164, filed on Mar. 18, 2019.
Prior Publication US 2022/0262354 A1, Aug. 18, 2022
This patent is subject to a terminal disclaimer.
Int. Cl. G10L 15/22 (2006.01); G10L 15/18 (2013.01); G10L 15/30 (2013.01)
CPC G10L 15/1815 (2013.01) [G10L 15/1822 (2013.01); G10L 15/22 (2013.01); G10L 15/30 (2013.01); G10L 2015/223 (2013.01)] 42 Claims
OG exemplary drawing
 
1. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by one or more processors of an electronic device, cause the electronic device to:
receive a natural language input;
detect an object with a camera of the electronic device;
determine, based on the natural language input, a first value for a first node of a plurality of nodes of a parsing structure, the first node corresponding to a natural language data stream;
determine, based on the object and without using data from the natural language data stream, a second value for a second node of the plurality of nodes of the parsing structure; and
determine, using the parsing structure, the first value, and the second value, a user intent associated with the natural language input.