US 12,111,646 B2
Robotic process automation with resilient playback of recordings
Virinchipuram Anand, San Ramon, CA (US); Sudhir Kumar Singh, Dublin, CA (US); and Harshil Lodhiya, Vadodara (IN)
Assigned to Automation Anywhere, Inc., San Jose, CA (US)
Filed by Automation Anywhere, Inc., San Jose, CA (US)
Filed on Mar. 18, 2021, as Appl. No. 17/206,029.
Claims priority of provisional application 63/139,757, filed on Jan. 20, 2021.
Claims priority of provisional application 63/060,541, filed on Aug. 3, 2020.
Prior Publication US 2022/0032457 A1, Feb. 3, 2022
Int. Cl. G05B 19/423 (2006.01); G06F 3/00 (2006.01); G06F 9/451 (2018.01); G06F 11/34 (2006.01)
CPC G05B 19/423 (2013.01) [G06F 3/00 (2013.01); G06F 9/451 (2018.02); G06F 11/34 (2013.01); G06F 11/3414 (2013.01); G06F 11/3438 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A computer-implemented method for playing back a previously established recording having a plurality of recorded actions undertaken with respect to one or more software programs, the method comprising:
initiating playback of the previously established recording, the previously established recording including at least properties and previously captured screen images for one or more of the recorded actions within the recording, the previously captured screen images are screen images captured at the time of the recording of the previously established recording;
determining if an error occurred during playback of a particular action of the previously established recording; and
resolving the error that occurred during playback of the particular action of the previously established recording, the resolving including at least:
identifying a pre-action captured screen image of the previously captured screen images that corresponds to a screen image occurring just prior to the particular action;
identifying a post-action captured screen image of the previously captured screen images that corresponds to a screen image occurring just after the particular action;
determining an amount of change between the pre-action captured screen image and the post-action captured screen image;
comparing the amount of change with a threshold amount to determining type of processing to be performed to resolve the error that occurred during execution of the previously established recording;
resolving the particular action associated with the error using a first type of processing technique if the comparing determines that the amount of change is greater than the threshold amount; and
resolving the particular action associated with the error using a second type of processing technique if the comparing determines that the amount of change is less than the threshold amount,
wherein the second type of processing technique includes at least:
identifying a particular captured screen image of the previously captured screen images that corresponds to the particular action;
detecting a position in the particular captured screen image that is associated with the particular action;
detecting one or more control objects in the particular captured screen image that are proximate to the position detected in the particular captured screen image;
identifying a corresponding control object within the one or more control objects detected in the particular captured screen image that corresponds to the particular action; and
facilitating playback of the particular action using the corresponding control object that has been identified.