US 12,437,055 B2
Stack pivot exploit detection and mitigation
Andrew Sandoval, San Antonio, TX (US)
Assigned to OPEN TEXT INC., Menlo Park, CA (US)
Filed by Open Text Inc., Menlo Park, CA (US)
Filed on May 17, 2024, as Appl. No. 18/666,973.
Application 18/666,973 is a continuation of application No. 18/158,621, filed on Jan. 24, 2023, granted, now 12,013,929.
Application 18/158,621 is a continuation of application No. 17/228,478, filed on Apr. 12, 2021, granted, now 11,593,473, issued on Feb. 28, 2023.
Application 17/228,478 is a continuation of application No. 17/088,285, filed on Nov. 3, 2020, granted, now 11,443,032, issued on Sep. 13, 2022.
Application 17/088,285 is a continuation of application No. 15/952,678, filed on Apr. 13, 2018, granted, now 10,853,480, issued on Dec. 1, 2020.
Prior Publication US 2024/0303319 A1, Sep. 12, 2024
This patent is subject to a terminal disclaimer.
Int. Cl. G06F 21/52 (2013.01); G06F 11/3668 (2025.01)
CPC G06F 21/52 (2013.01) [G06F 11/3688 (2013.01); G06F 2221/033 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A system for control flow exploit detection and mitigation, comprising:
a processor; and
memory storing instructions that, when executed by the processor, causes the system to perform a set of operations for control flow exploit detection and mitigation, the set of operations comprising:
defining, for a process executing in a plurality of different execution modes, thread information for the process, the thread information comprising:
a current memory pointer,
a memory base, and
a memory limit,
the memory base and the memory limit defining a memory range for the process;
based on the current memory pointer for the process executing in the execution mode, generating a corresponding memory base pointer for the memory base, and a corresponding memory limit pointer for the memory limit;
executing, during runtime of the process, a plurality of checkpoints, wherein the plurality of checkpoints comprises:
a first set of checkpoints specific to a first execution mode of the plurality of different execution modes; and
a second set of checkpoints specific to a second execution mode;
wherein the first execution mode is a kernel mode, wherein the first set of checkpoints comprises one or more checkpoints triggered by kernel-level events,
wherein the second execution mode is a user mode, wherein the second set of checkpoints comprises one or more other checkpoints triggered by user-level events, wherein executing each checkpoint comprises:
comparing the current memory pointer for the thread information to the memory base pointer and the memory limit pointer to determine whether the current memory pointer is within the memory range;
when the current memory pointer is determined to be within the memory range, permitting the process to execute; and
when the current memory pointer is determined to not be within the memory range, determining an occurrence of a control flow exploit and, in response, performing a remedial action; wherein the execution of each set of checkpoints allows for determining of the occurrence of the control flow exploit during the corresponding execution mode.