	US 12,450,650 B2
	Method, device, and non-transitory machine readable medium for content item selection and measurement determination
Joel Barajas Zamora, Fremont, CA (US); Lakshmi Narayan Bhamidipati, Sunnyvale, CA (US); Ilya Nathaniel Slain, Santa Clara, CA (US); Lioz Medina, Jerusalem (IL); and Milena Petrova Krasteva, Mountain View, CA (US)
Assigned to Yahoo Ad Tech LLC, New York, NY (US)
Filed by Oath Inc., New York, NY (US)
Filed on Dec. 9, 2019, as Appl. No. 16/707,431.
Prior Publication US 2021/0174433 A1, Jun. 10, 2021
Int. Cl. G06Q 30/00 (2023.01); G06Q 30/0241 (2023.01); G06Q 30/0601 (2023.01); G06Q 30/08 (2012.01); G06Q 30/0273 (2023.01)

CPC G06Q 30/08 (2013.01) [G06Q 30/0241 (2013.01); G06Q 30/0631 (2013.01); G06Q 30/0275 (2013.01)]

19 Claims

1. A method for performing an experiment using a plurality of client devices, comprising:

generating a test group and a control group for the experiment by splitting the plurality of client devices into a first set of client devices associated with the test group and a second set of client devices associated with the control group;

receiving, by one or more servers, a first request for content associated with a first client device of the first set of client devices, wherein the first request for content is indicative of a first quantity of content items;

extracting at least one of a format or a structure of content items from the first request for content;

searching a content item database for content items having at least one of the format or the structure of content items extracted from the first request for content;

retrieving a first plurality of content items from the content item database based upon a determination that content items of the first plurality of content items match at least one of the format or the structure of content items extracted from the first request for content;

determining a first plurality of bid values associated with the first plurality of content items;

selecting, by the one or more servers, a second plurality of content items from the first plurality of content items based upon the first plurality of bid values, wherein:

the second plurality of content items is associated with a first plurality of rankings; and

a second quantity of content items of the second plurality of content items is greater than the first quantity of content items associated with the first request for content;

extracting one or more experiment statuses associated with the second plurality of content items;

identifying, by the one or more servers, one or more first ghost content items of the second plurality of content items based upon a determination that the one or more first ghost content items are each associated with an experiment status indicative of an ongoing experiment, wherein the one or more first ghost content items are associated with one or more first rankings of the first plurality of rankings;

selecting, by the one or more servers, a third plurality of content items, of the second plurality of content items, that are not ghost content items for presentation via the first client device of the test group of the experiment, wherein:

a third quantity of content items of the third plurality of content items is equal to the first quantity of content items associated with the first request for content; and

the third plurality of content items is associated with a second plurality of rankings;

providing, by the one or more servers, a first content item, of the third plurality of content items that are not ghost content items, to the first client device of the test group of the experiment, wherein:

the first content item is associated with a second ranking of the second plurality of rankings;

a second content item, of the third plurality of content items, is associated with a third ranking of the second plurality of rankings; and

each content item is displayed in a serve area on the first client device corresponding to a rank of the content item;

providing, by the one or more servers, a first ghost content item, of the one or more first ghost content items, to the second set of client devices of the control group of the experiment;

determining whether a first ranking associated with the first ghost content item, matches one or more rankings associated with one or more presented content items;

responsive to determining that at least a threshold proportion of the first ghost content item was displayed on the first client device, generating and storing, by the one or more servers, a first ghost impression indicator associated with the first ghost content item based upon a determination that the first ranking associated with the first ghost content item matches the second ranking associated with the first content item presented via the first client device;

after storing the first ghost impression indicator, determining, by the one or more servers, one or more measurements associated with the first ghost content item;

tracking, by the one or more servers, at least some of the plurality of client devices that presented one or more content items;

based upon the tracking the plurality of client devices, retrieving at least one of one or more instant messages or one or more emails associated with one or more devices of at least one of the first set of client devices comprising the first client device or the second set of client devices that at least one of received or presented the first ghost content item;

analyzing at least one of the one or more instant messages or the one or more emails to determine (i) first activity associated with the first set of client devices associated with the test group and (ii) second activity associated with the second set of client devices, associated with the control group, that at least one of received or presented the first ghost content item;

determining a first relationship between (i) the first activity associated with the first set of client devices comprising the first client device and (i) the second activity associated with the second set of client devices that at least one of received or presented the first ghost content item;

modifying, based upon the first relationship, a transmission control data structure associated with the first ghost content item; and

controlling, by the one or more servers and based upon the modified transmission control data structure, transmission of one or more content items to one or more client devices, wherein the controlling comprises increasing a rate of transmission to client devices associated with first characteristics and decreasing a rate of transmission to client devices associated with second characteristics based upon the one or more measurements, wherein the controlling creates a closed-loop process that uses results of the experiment as feedback to tailor one or more parameters of a content system.