US 12,326,803 B2
Method and system for digital webpage testing
Matthew Isaac van Adelsberg, Falls Church, VA (US); Zhun Wang, Vienna, VA (US); Priscilla Alexander, Washington, DC (US); and Mackenzie Sweeney, Bristow, VA (US)
Assigned to Capital One Services, LLC, McLean, VA (US)
Filed by Capital One Services, LLC, McLean, VA (US)
Filed on Aug. 10, 2021, as Appl. No. 17/444,787.
Claims priority of provisional application 63/063,604, filed on Aug. 10, 2020.
Prior Publication US 2022/0043742 A1, Feb. 10, 2022
Int. Cl. G06F 11/3668 (2025.01); G06F 11/34 (2006.01); G06F 11/3698 (2025.01); G06N 20/00 (2019.01); G06Q 30/0201 (2023.01)
CPC G06F 11/3696 (2013.01) [G06F 11/3466 (2013.01); G06F 11/3698 (2025.01); G06N 20/00 (2019.01); G06Q 30/0201 (2013.01)] 18 Claims
OG exemplary drawing
 
1. A system for digitally testing one or more variants of a webpage, comprising:
a processor; and
a memory having programming instructions stored thereon, which, when executed by the processor, performs one or more operations, comprising:
initiating a testing service for one or more variants of a webpage of a website associated with a client computing system;
receiving, via one or more application programming interface (API) endpoints, a first set of user data comprising a first set of one or more indications of one or more users interacting with the one or more variants of the webpage;
receiving a client intervention to update the testing service;
upon receiving the client intervention, pausing the testing service;
modifying the testing service by injecting one or more extra explorations associated with the client intervention into the testing service;
upon injecting the one or more extra explorations, resuming the testing service;
receiving, via the one or more API endpoints, a second set of user data comprising a second set of one or more indications of one or more users interacting with the one or more variants of the webpage;
inputting the first set of one or more indications and the second set of one or more indications into a machine learning model comprising a Bayesian multi-arm bandit algorithm;
generating, using the machine learning model, one or more results comprising causal performance estimates based on the first set of one or more indications and the second set of one or more indications, the one or more results comprising indications of portions of the one or more variants of the webpage were underexplored based on the first set of user data and the second set of user data;
generating, using the machine learning model, a set of decision rules to adaptively design further testing experiments for other webpages based on the causal performance estimates learned by the machine learning model on the webpage, the set of decision rules used to inform allocation of users to additional variants of the other webpages for subsequent testing of the other webpages, the set of decision rules comprising extra explorations injected into the additional variants of the other webpages to account for the portions of the one or more variants of the webpage that were underexplored;
generating a portal comprising the one or more results, the portal accessible to one or more end users;
receiving, from the portal, one or more modifications to the one or more variants of the webpage;
applying the one or more modifications to the one or more variants of the webpage; and
re-initiating the testing service for the one or more variants of the webpage of the website for the one or more modifications.