US 12,111,834 B1
Ambient multi-device framework for agent companions
Victor Carbune, Zürich (CH); Arash Sadr, Belmont, CA (US); and Matthew Sharifi, Kilchberg (CH)
Assigned to GOOGLE LLC, Mountain View, CA (US)
Filed by Google LLC, Mountain View, CA (US)
Filed on Dec. 20, 2023, as Appl. No. 18/390,768.
Int. Cl. G06F 16/00 (2019.01); G06F 16/2455 (2019.01); G06F 16/248 (2019.01)
CPC G06F 16/24553 (2019.01) [G06F 16/248 (2019.01)] 20 Claims
OG exemplary drawing
 
1. A computing system for determining an output device for providing a query response, the system comprising:
one or more processors; and
one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
obtaining input data, wherein the input data comprises a query associated with a particular user;
obtaining environment data, wherein the environment data is descriptive of a plurality of computing devices in an environment of the user, wherein the plurality of computing devices are associated with a plurality of different output components;
generating a prompt based on the input data and the environment data, wherein the prompt comprises data descriptive of the query and device information associated with at least a subset of the plurality of computing devices, wherein generating the prompt comprises:
determining output capabilities for plurality of computing devices based on the environment data;
generating a representation of performance capabilities of the plurality of computing devices; and
generating the prompt based on the representation of performance capabilities of the plurality of computing devices and the input data;
processing the prompt with a generative model to generate a model-generated output based on the data descriptive of the query and the device information, wherein processing the prompt with the generative model to generate the model-generated output comprises:
generating a response to the query based on processing the data descriptive of the query with the generative model; and
generating, with the generative model and based on the representation and the response, a model-generated output to be provided with a particular computing device of the plurality of computing devices based on the device information; and
transmitting the model-generated output to the particular computing device.