US 12,216,569 B2
Systems and methods for configurable application feature experiment alerting using sequential testing
Charles Wu, Pittsburgh, PA (US); Gianpiero Condorelli, London (GB); Mansi Mohan, San Francisco, CA (US); Yoneo Arai, San Francisco, CA (US); and Gautham Sudarshan, San Mateo, CA (US)
Filed by Split Software, Inc., Redwood City, CA (US)
Filed on Mar. 17, 2023, as Appl. No. 18/123,192.
Prior Publication US 2024/0311281 A1, Sep. 19, 2024
Int. Cl. G06F 9/44 (2018.01); G06F 8/65 (2018.01)
CPC G06F 11/3688 (2013.01) [G06F 8/65 (2013.01); G06F 11/3692 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A method for configurable application feature experiments, comprising:
receiving, by a computer server from an application developer system, data indicative of: a metric to be collected after a feature treatment is deployed to a plurality of configurable applications executed by a plurality of end user systems, wherein the metric is associated with a performance characteristic of the plurality of configurable applications;
receiving, by the computer server from the configurable applications executed by the plurality of end user systems, feature treatment event messages generated by the configurable applications after deployment of the feature treatment, the event messages comprising metric values associated with the metric;
performing, by the computer server, a sequential testing process using the metric values from the event messages to determine when statistical significance has been reached for the metric values satisfying or not satisfying a significance threshold associated with the metric;
in response to detecting when the statistical significance has been reached based on results of generated by the sequential testing process, generating an alert message indicative of at least the metric, the significance threshold associated with the metric, and the determination of the statistical significance of the metric satisfying or not satisfying the significance threshold; and
transmitting, by the computer server to the application developer system, the alert message.