US 11,726,802 B2
Robust user interface related robotic process automation
Marie-Claude Cote, Montreal (CA); Alexei Nordell-Markovits, Montreal (CA); and Andrej Todosic, Montreal (CA)
Assigned to ServiceNow Canada Inc., Montreal (CA)
Appl. No. 17/279,989
Filed by SERVICENOW CANADA INC., Montreal (CA)
PCT Filed Sep. 26, 2019, PCT No. PCT/CA2019/051376
§ 371(c)(1), (2) Date Mar. 25, 2021,
PCT Pub. No. WO2020/061700, PCT Pub. Date Apr. 2, 2020.
Claims priority of provisional application 62/738,319, filed on Sep. 28, 2018.
Prior Publication US 2021/0397157 A1, Dec. 23, 2021
Int. Cl. G06F 9/451 (2018.01); G06N 20/00 (2019.01); G06F 40/10 (2020.01); G06V 30/412 (2022.01); G06V 30/10 (2022.01); G06F 40/205 (2020.01); G06F 3/04886 (2022.01)
CPC G06F 9/451 (2018.02) [G06F 3/04886 (2013.01); G06F 40/10 (2020.01); G06N 20/00 (2019.01); G05B 2219/32128 (2013.01); G06F 40/205 (2020.01); G06V 30/10 (2022.01); G06V 30/412 (2022.01)] 10 Claims
OG exemplary drawing
 
1. A method for performing at least one task involving at least one interaction with a user interface, the method being executed by a processor, the method comprising:
a) receiving said user interface;
b) segmenting said user interface using a segmentation neural network to determine different areas of said user interface;
c) analyzing said user interface using a recognition neural network to determine data associated with each of said different areas;
d) determining which of the data associated with each of said different areas is relevant to said at least one task independently from a respective location of each of said different areas;
e) determining, using a text processing neural network, which areas in said user interface contain the data relevant to said at least one task by associating the relevant data with corresponding relevant areas independently from the respective location of each of said relevant areas, said determining comprising ignoring interface areas irrelevant to said at least one task;
f) executing said at least one task by executing at least one interaction with either:
at least one of said relevant areas determined in step e); or
relevant data contained in said at least one of said relevant areas.