US 12,332,770 B2
Automated locating of GUI elements during testing using multidimensional indices
Jin Shi, Ningbo (CN); Lu Chen, Ningbo (CN); Tang Xue Bo, Ningbo (CN); Ping Yang, Ningbo (CN); Meng Qi Chen, Ningbo (CN); and Rui Na Liu, Ningbo (CN)
Assigned to International Business Machines Corporation, Armonk, NY (US)
Filed by International Business Machines Corporation, Armonk, NY (US)
Filed on Jun. 28, 2021, as Appl. No. 17/360,390.
Prior Publication US 2022/0413997 A1, Dec. 29, 2022
Int. Cl. G06F 11/3668 (2025.01); G06F 3/0485 (2022.01); G06F 11/3698 (2025.01); G06N 20/00 (2019.01); G06T 7/70 (2017.01)
CPC G06F 11/3688 (2013.01) [G06F 3/0485 (2013.01); G06F 11/3698 (2025.01); G06N 20/00 (2019.01); G06T 7/70 (2017.01); G06T 2207/20081 (2013.01)] 15 Claims
OG exemplary drawing
 
1. A computer-implemented method (CIM) comprising:
receiving a first graphical user interface, including a first plurality of graphical user interface elements;
receiving a machine learning model trained to determine positions of groups of user interface elements and their corresponding constituent user interface elements in an image of a graphical user interface, wherein a serial number assigned to each of the corresponding constituent user interface elements are highlighted on the fly;
indexing, into a first position mapping table, each graphical user interface element of the first plurality of graphical user interface elements with a multidimensional index value corresponding to at least three dimensions based, at least in part, on the machine learning model, where at least one dimension of the at least three dimensions corresponds to grouping status within a hierarchical group of elements, and wherein the multidimensional index value comprises: (i) a first dimension indicative of horizontal position on the first graphical user interface, (ii) a second dimension comprising vertical position on the first graphical user interface, and (iii) a third dimension corresponding to relative placement within the hierarchical group of elements;
testing the first graphical user interface with a first automated graphical user interface test, where the first automated graphical user interface test locates graphical user interface elements for testing based, at least in part, on the serial number as a locator to locate the corresponding constituent user interface elements on the first position mapping table; and
applying a scrolling technique to generate an expanded virtual graphical user interface of one or more groups of user interface elements, wherein the machine learning model refreshes the multidimensional index values using the expanded virtual graphical user interface to reveal other graphical user interface elements not previously visible by: analyzing the expanded virtual graphical user interface (GUI) to identify newly revealed GUI elements; determining positions and hierarchical relationships of the newly revealed GUI elements; and updating the first position mapping table with new multidimensional index values for the newly revealed GUI elements, wherein the new multidimensional index values maintain consistency with an indexing scheme of previously indexed elements while incorporating the newly revealed elements into a hierarchical structure.