US 11,983,649 B2
Targeted training of inductive multi-organization recommendation models for enterprise applications
Kiran Tomlinson, Ithaca, NY (US); Longqi Yang, Issaquah, WA (US); Mengting Wan, Bellevue, WA (US); Cao Lu, Bellevue, WA (US); Brent Jaron Hecht, Redmond, WA (US); and Jaime Teevan, Bellevue, WA (US)
Assigned to Microsoft Technology Licensing, LLC, Redmond, WA (US)
Filed by Microsoft Technology Licensing, LLC, Redmond, WA (US)
Filed on Oct. 26, 2021, as Appl. No. 17/510,523.
Prior Publication US 2023/0128832 A1, Apr. 27, 2023
Int. Cl. G06Q 10/00 (2023.01); G06N 5/00 (2023.01); G06N 5/022 (2023.01); G06N 5/04 (2023.01); G06Q 10/063 (2023.01)
CPC G06Q 10/063 (2013.01) [G06N 5/022 (2013.01); G06N 5/04 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A method implemented in a computing system comprising a processor, and wherein the method comprises:
receiving, via a network, enterprise application data from remote organization computing systems executing an enterprise application, wherein each remote organization computing system corresponds to an organization that subscribes to the enterprise application;
training, via the processor, a per-organization recommendation machine learning model for each of a subset of the organizations that subscribe to the enterprise application using at least a portion of the enterprise application data received from the corresponding remote organization computing systems;
validating each per-organization recommendation machine learning model on at least a portion of the enterprise application data corresponding to at least one other organization;
calculating a transferability metric for each per-organization recommendation machine learning model based on results obtained during the validation of the per-organization recommendation machine learning model, wherein the calculated transferability metric includes a mean average precision (MAP) for the results obtained during the validation of each per-organization recommendation machine learning model;
determining a specified number of organizations comprising best-transferring per-organization recommendation machine learning models based on the calculated transferability metrics;
training an inductive multi-organization recommendation machine learning model using at least a portion of the enterprise application data from the specified number of organizations comprising the best-transferring per-organization recommendation machine learning models;
transmitting, via the network, user recommendations to the remote organization computing systems during execution of the enterprise application, the user recommendations being derived based on the trained inductive multi-organization recommendation machine learning model; and
receiving new enterprise application data from at least one of the remote organization computing systems, wherein the new enterprise application data is used to update the inductive multi-organization recommendation machine learning model.