CPC G06F 11/3688 (2013.01) [G06F 8/65 (2013.01); G06F 11/3692 (2013.01)] | 20 Claims |
1. A method for configurable application feature experiments, comprising:
receiving, by a computer server from an application developer system, data indicative of: a metric to be collected after a feature treatment is deployed to a plurality of configurable applications executed by a plurality of end user systems, wherein the metric is associated with a performance characteristic of the plurality of configurable applications;
receiving, by the computer server from the configurable applications executed by the plurality of end user systems, feature treatment event messages generated by the configurable applications after deployment of the feature treatment, the event messages comprising metric values associated with the metric;
performing, by the computer server, a sequential testing process using the metric values from the event messages to determine when statistical significance has been reached for the metric values satisfying or not satisfying a significance threshold associated with the metric;
in response to detecting when the statistical significance has been reached based on results of generated by the sequential testing process, generating an alert message indicative of at least the metric, the significance threshold associated with the metric, and the determination of the statistical significance of the metric satisfying or not satisfying the significance threshold; and
transmitting, by the computer server to the application developer system, the alert message.
|