US 12,411,700 B2
Assigning control and variants to dynamically defined time periods in application state experiments
Yixiang Zeng, Fremont, CA (US); Aneesh Mannava, San Francisco, CA (US); Bing Hong Leonard How, South San Francisco, CA (US); Zhongqiang Liang, Fremont, CA (US); Wenhui Zhang, Redwood City, CA (US); and Lan Yu, Newark, CA (US)
Assigned to Maplebear Inc., San Francisco, CA (US)
Filed by Maplebear Inc., San Francisco, CA (US)
Filed on Jul. 17, 2023, as Appl. No. 18/353,744.
Prior Publication US 2025/0028540 A1, Jan. 23, 2025
Int. Cl. G06F 9/451 (2018.01); G06F 11/34 (2006.01)
CPC G06F 9/451 (2018.02) [G06F 11/3409 (2013.01); G06F 11/3438 (2013.01); G06F 11/3466 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A method, performed at a computer system comprising a processor and a computer-readable medium, comprising:
receiving data for an application state experiment for a client application of an online system, wherein the data for the application state experiment comprises a control application state for the client application, a test application state of the client application, a testing time period, and an interaction data criteria indicating a minimum number of interaction data points to be collected for each of the control application state and the test application state;
conducting the application state experiment by, for each of a plurality of time periods within the testing time period:
assigning one of: the control application state or the test application state to the time period;
transmitting instructions to a plurality of client devices to cause the client application on the plurality of client devices to present a user interface in accordance with the assigned application state;
collecting a set of interaction data points describing user interactions with the user interface of the assigned application state;
ending the time period when the collected set of interaction data points meets the interaction data criteria; and
collecting another set of interaction data points during a transition time period following the time period;
computing a success metric for each of the control application state and the test application state based on the sets of interaction data points for the application states collected during the time periods, wherein the success metric is not computed based on the set of interaction data points collected during transition time periods
transmitting instructions to a client device of the plurality of client devices to cause the client application on the client device to present a user interface in accordance with the control application state or the test application state based on the success metric.