US 12,242,824 B2
Creating user interface using machine learning
Zifeng Huang, Emeryville, CA (US); Yang Li, Palo Alto, CA (US); Gang Li, Mountain View, CA (US); Xin Zhou, Mountain View, CA (US); and John Francis Canny, Berkeley, CA (US)
Assigned to Google LLC, Mountain View, CA (US)
Filed by Google LLC, Mountain View, CA (US)
Filed on Oct. 13, 2022, as Appl. No. 18/046,446.
Claims priority of provisional application 63/255,366, filed on Oct. 13, 2021.
Prior Publication US 2023/0115185 A1, Apr. 13, 2023
Int. Cl. G06F 17/00 (2019.01); G06F 3/0484 (2022.01); G06F 8/33 (2018.01); G06F 8/38 (2018.01); G06F 40/40 (2020.01); G06N 3/0455 (2023.01)
CPC G06F 8/38 (2013.01) [G06F 3/0484 (2013.01); G06F 8/33 (2013.01); G06F 40/40 (2020.01); G06N 3/0455 (2023.01)] 15 Claims
OG exemplary drawing
 
1. A computer-implemented method, comprising:
receiving a training dataset comprising a plurality of training samples, each training sample comprising:
a graphical user interface that includes a plurality of graphical elements; and
a natural language textual description comprising a single phrase that describes the graphical user interface that includes the plurality of graphical elements;
generating, for each graphical user interface, graphical attribute data that describes, for each graphical element of the graphical user interface, an attribute type of the graphical element, and a position of the graphical element;
generating, for each natural language textual description, using a pre-trained word embedding model, an embedding vector of the natural language textual description; and
training a machine learning model, based on the graphical attribute data and the embedding vector for each training sample, to generate, as output, prediction data that is indicative of graphical elements in a graphical user interface, wherein the machine learning model comprises a transformer based model that includes:
an encoder that receives the embedding vector of the natural language textual description and processes the embedding vector to generate an output vector; and
a decoder that receives the output vector and the graphical attribute data generated from the training sample and is trained to generate the prediction data.