US 11,756,101 B2
Methods and apparatuses for evaluating substitutions made during order fulfillment processes
Sushant Pralhad Joshi, Fremont, CA (US); Kamiya Motwani, Madhya Pradesh (IN); Prashant Chandrakant Saundade, San Jose, CA (US); Sushant Kumar, Sunnyvale, CA (US); Vidya Sagar Kalidindi, Milpitas, CA (US); and Kannan Achan, Saratoga, CA (US)
Assigned to Walmart Apollo, LLC, Bentonville, AR (US)
Filed by Walmart Apollo, LLC, Bentonville, AR (US)
Filed on May 16, 2022, as Appl. No. 17/745,113.
Application 17/745,113 is a continuation of application No. 16/752,521, filed on Jan. 24, 2020, granted, now 11,367,119.
Prior Publication US 2022/0277377 A1, Sep. 1, 2022
This patent is subject to a terminal disclaimer.
Int. Cl. G06Q 10/0639 (2023.01); G06Q 30/0601 (2023.01); G06N 20/00 (2019.01)
CPC G06Q 30/0631 (2013.01) [G06N 20/00 (2019.01); G06Q 10/06393 (2013.01); G06Q 30/0633 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A system comprising at least one computing device coupled to one or more data sources comprising at least one server, the at least one computing device configured to:
obtain test configuration information via a user interface of a test configuration manager;
retrieve a first version of a trained machine learning customer understanding model and a second version of the trained machine learning customer understanding model from the one or more data sources;
receive order data characterizing orders placed by one or more customers on an e-commerce platform;
filter orders defined by the test configuration information to obtain filtered orders by comparing the received order data to the test configuration information;
assign orders from the filtered orders to a first group or a second group;
determine recommended first test substitute items based on overall test substitution scores of possible substitute items using the first version of the trained machine learning customer understanding model;
determine recommended second test substitute items based on overall control substitution scores of possible substitute items using the second version of the trained machine learning customer understanding model;
send the recommended first test substitute items for the first group of orders and the recommended second test substitute items for the second group of orders to a store order fulfillment system;
determine one or more performance metrics of each of the first version of the trained machine learning customer understanding model and the second version of the trained machine learning customer understanding model; and
compare one or more performance metrics of the first version of the trained machine learning customer understanding model to the second version of the trained machine learning customer understanding model.