US 11,836,060 B2
System testing infrastructure with hidden variable, hidden attribute, and hidden value detection
Andrew C. M. Hicks, Highland, NY (US); Kevin Minerley, Red Hook, NY (US); Dale E. Blue, Poughkeepsie, NY (US); Ryan Thomas Rawlins, New Paltz, NY (US); and Daniel Nicolas Gisolfi, Hopewell Junction, NY (US)
Assigned to INTERNATIONAL BUSINESS MACHINES CORPORATION, Armonk, NY (US)
Filed by International Business Machines Corporation, Armonk, NY (US)
Filed on Oct. 25, 2021, as Appl. No. 17/509,432.
Application 17/509,432 is a continuation of application No. 17/121,850, filed on Dec. 15, 2020, granted, now 11,204,848.
Prior Publication US 2022/0188205 A1, Jun. 16, 2022
Int. Cl. G06F 11/36 (2006.01); G06F 11/263 (2006.01)
CPC G06F 11/263 (2013.01) [G06F 11/3664 (2013.01); G06F 11/3684 (2013.01); G06F 11/3692 (2013.01); G06F 11/3676 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A method for detecting and localizing a fault when testing a system under test (SUT), the method comprising:
modeling inputs to the SUT as a collection of attribute-value pairs;
generating an initial set of test vectors that provides complete n-wise coverage of a test space represented by the attribute-value pairs;
generating a set of testcases using the initial set of test vectors;
executing the set of testcases to obtain a set of execution results, the execution results being in binary form indicative that a testcase succeeded or failed, the set of testcases executed a plurality of times;
updating, for each execution of the set of testcases, for each attribute-value pair, a non-binary success rate (SAV) based on the set of execution results, wherein the non-binary success rate of an attribute-value pair is based on execution results of each testcase that uses said attribute-value pair;
in response to a success rate of an attribute-value pair being below a predetermined threshold:
identifying a subset of testcases that use the attribute-value pair;
identifying, based on execution of the subset of testcases, a first set of code paths that result in successful execution, and a second set of code paths that result in failed execution;
determining differences in the first set of code paths and the second set of code paths; and
highlighting, in a graphical user interface, the differences in the first set of code paths and the second set of code paths.