US 11,954,507 B2
GUI component recognition apparatus, method and program
Di Li, Musashino (JP); Haruhisa Nozue, Musashino (JP); Kazuaki Akashi, Musashino (JP); and Kenichi Tayama, Musashino (JP)
Assigned to NIPPON TELEGRAPH AND TELEPHONE CORPORATION, Tokyo (JP)
Appl. No. 17/908,233
Filed by NIPPON TELEGRAPH AND TELEPHONE CORPORATION, Tokyo (JP)
PCT Filed Mar. 2, 2020, PCT No. PCT/JP2020/008708
§ 371(c)(1), (2) Date Aug. 30, 2022,
PCT Pub. No. WO2021/176523, PCT Pub. Date Sep. 10, 2021.
Prior Publication US 2023/0153126 A1, May 18, 2023
Int. Cl. G06F 9/451 (2018.01); G06F 3/04812 (2022.01)
CPC G06F 9/451 (2018.02) [G06F 3/04812 (2013.01)] 18 Claims
OG exemplary drawing
 
1. A screen recognition device comprising:
a processor; and
a storage medium having computer program instructions stored thereon, when executed by the processor, perform to:
perform a marking process on a graphical user interface (GUI) component in an active display screen provided by an application software, wherein the marking process distinguishes a background of the display screen from the GUI component in the display screen of the application software;
extract the GUI component included in the display screen of application software from the display screen;
superimpose a mouse cursor on the GUI component based on at least one of a form and a position of the GUI component extracted;
identify a change in the displayed symbol image of the mouse cursor;
determine that a kind of the extracted GUI component cannot be identified based only on the change in the displayed symbol image of the mouse cursor;
in response to the determining, determine the kind of the GUI component extracted and an operation that can be performed to the GUI component, based on the change in the mouse cursor and a change in the GUI component at the same time the mouse cursor is superimposed on the GUI component; and
moving the mouse cursor sequentially to each of the plurality of graphical user interfaces in the display screen based on the form and the position of each GUI component, wherein the mouse cursor is moved to and superimposed on each GUI component to determine the kind and the operation of the graphical user interface.