US 12,332,765 B2
Systems and methods for variant testing at scale
Jae Hyeon Bae, San Jose, CA (US); Kurt M. Fredericks, San Francisco, CA (US); Nicholas Kistner, Cupertino, CA (US); Andrew T. Maher, London (GB); Nihar Ranjan Hati, Santa Clara, CA (US); Mahesh Molakalapalli, Santa Clara, CA (US); Srivas Chennu, Rochester (GB); and Jamie J. Martin, London (GB)
Assigned to Apple Inc., Cupertino, CA (US)
Filed by Apple Inc., Cupertino, CA (US)
Filed on Feb. 10, 2023, as Appl. No. 18/167,406.
Claims priority of provisional application 63/376,996, filed on Sep. 23, 2022.
Prior Publication US 2024/0103998 A1, Mar. 28, 2024
Int. Cl. G06F 9/44 (2018.01); G06F 11/3604 (2025.01)
CPC G06F 11/3608 (2013.01) 20 Claims
OG exemplary drawing
 
1. A method, comprising:
generating at least one variant object based on at least one variant feature indicated by variant testing input associated with a variant test, wherein the variant test is associated with a control object and a software application;
collecting first and second conversion data for the control object and the at least one variant object, respectively, wherein the first and second conversion data is based on first time downloads of the software application by users who access the control object and the at least one variant object;
storing, in a facts database, the first and second conversion data according to a collection date;
retrieving, from the facts database, a first and second subsets of the first and second conversion data, respectively;
computing at least one first conversion metric using the first subset and at least one second conversion metric using the second subset;
generating, for a collection period associated with the first and second subsets, a performance measurement for the variant test by applying at least one statistical hypothesis testing function to the at least one first conversion metric and the at least one second conversion metric; and
providing, at a display, the performance measurement.