CPC G06F 9/451 (2018.02) [G06F 18/214 (2023.01); G06F 18/21355 (2023.01); G06N 3/045 (2023.01); G06N 20/00 (2019.01)] | 20 Claims |
1. A computer-implemented method for training and utilization of machine-learned models for user interface interaction understanding, comprising:
obtaining, by a computing system comprising one or more computing devices, interface data that comprises a sequence of two or more user interfaces obtained through performance of one or more user interactions which result in generation of the sequence of two or more user interfaces, wherein, for each user interface in the sequence of two or more user interfaces, the interface data comprises one or more interface images depicting the user interface;
determining, by the computing system, a plurality of intermediate embeddings based at least in part on the interface data;
processing, by the computing system, the plurality of intermediate embeddings with a machine-learned interface prediction model to obtain one or more user interface embeddings; and
performing, by the computing system, a pre-training task based at least in part on the one or more user interface embeddings to obtain a pre-training output.
|