US 12,443,334 B2
Information processing method for displaying of graphical user interface screen using trained model
Yoshinori Sakabe, Hyogo (JP)
Assigned to MICWARE CO., LTD., Hyogo (JP)
Filed by MICWARE CO., LTD., Hyogo (JP)
Filed on Dec. 28, 2023, as Appl. No. 18/398,451.
Claims priority of application No. 2023-009209 (JP), filed on Jan. 25, 2023.
Prior Publication US 2024/0248585 A1, Jul. 25, 2024
Int. Cl. G06F 3/0484 (2022.01); G06F 3/14 (2006.01)
CPC G06F 3/0484 (2013.01) [G06F 3/1415 (2013.01)] 11 Claims
OG exemplary drawing
 
1. A non-transitory computer readable storage medium storing a program for causing a computer to perform operations comprising:
executing a target program to cause displaying of a graphical user interface screen;
detecting an object being operable and included in the graphical user interface screen displayed by execution of the target program, using a trained model being trained to output, in response to an input of an image of a graphical user interface screen, information about an object included in the graphical user interface screen,
wherein the trained model is generated using training data including an image of a graphical user interface screen and information about an object included in the graphical user interface screen, in which different designs associated with a same processing instruction are trained to be recognized as a same type of an object;
obtaining data about transition of a screen, wherein the transition of the screen is to be displayed in response to an operation on the detected object; and
generating screen transition data for the target program by associating the graphical user interface screen before transition, a type of the object in the graphical user interface screen, and the data about transition of the screen to be displayed in response to the operation on the object with one another,
wherein:
the object is assigned with a priority level;
the storage medium storing the program further causes the computer to perform operations comprising generating, upon detecting a plurality of objects in the graphical user interface screen, screen transition data by obtaining, in order of priority from an object with a highest priority level, data about transition of the screen to be displayed in response to an operation on each of the plurality of objects;
the graphical user interface screen has a plurality of hierarchical levels; and
the priority level is assigned to each of the plurality of objects based on whether screen transition data for a screen is generated by causing transition of the screen in a depth direction of the plurality of hierarchical levels of the graphical user interface screen or by causing transition to a screen at a same depth in the plurality of hierarchical levels for each depth of the plurality of hierarchical levels.