US 12,112,032 B2
Controlling a device based on processing of image data that captures the device and/or an installation environment of the device
Yuzhao Ni, Sunnyvale, CA (US)
Assigned to GOOGLE LLC, Mountain View, CA (US)
Filed by GOOGLE LLC, Mountain View, CA (US)
Filed on May 3, 2023, as Appl. No. 18/142,902.
Application 18/142,902 is a continuation of application No. 17/552,938, filed on Dec. 16, 2021, granted, now 11,644,964.
Application 17/552,938 is a continuation of application No. 16/339,269, granted, now 11,256,405, issued on Feb. 22, 2022, previously published as PCT/US2018/052460, filed on Sep. 24, 2018.
Prior Publication US 2023/0273716 A1, Aug. 31, 2023
This patent is subject to a terminal disclaimer.
Int. Cl. G06F 3/04847 (2022.01); G06F 3/04883 (2022.01); G06V 20/10 (2022.01); H04N 23/63 (2023.01)
CPC G06F 3/04847 (2013.01) [G06F 3/04883 (2013.01); G06V 20/10 (2022.01); H04N 23/631 (2023.01)] 17 Claims
OG exemplary drawing
 
1. A method implemented by one or more processors, the method comprising:
receiving image data corresponding to an output of a camera that is integral to a portable client device and is directed at an environment, wherein the portable client device includes an automated assistant interface that is configured to provide an interface between a user and an automated assistant;
receiving a spoken utterance from the user, wherein the spoken utterance is directed to the automated assistant;
processing the spoken utterance to identify a request of the user, wherein the request includes a command to adjust a state of a controllable electronic device;
identifying, based on the image data and contextual data that is in addition to the image data, a particular controllable electronic device that is included in the environment and that is captured in the image data;
determining, based on both (i) the command included in the request identified based on the spoken utterance and (ii) the particular controllable electronic device being identified based on the image data and the contextual data, that the request is directed to the particular controllable electronic device, wherein determining that the request is directed to the particular controllable electronic device further comprises:
identifying, based on the image data, an additional controllable electronic device that is included in the environment and that is captured in the image data;
determining a first metric associated the particular controllable device, wherein the first metric indicates a first probability that the request is directed to the particular controllable device;
determining a second metric associated with the additional controllable device, wherein the second metric indicates a second probability that the request is directed to the additional controllable device; and
determining, based on the first metric and the second metric, that the particular controllable electronic device, in lieu of the additional controllable electronic device, is the controllable electronic device to which the request is directed; and
causing, in response to receiving the request and in response to determining that the request is directed to the controllable electronic device, the state of the particular controllable electronic device to be modified in accordance with the command of the request.