US 12,238,188 B2
Offensive chat filtering using machine learning models
Monica Tongya, Redmond, WA (US)
Assigned to Microsoft Technology Licensing, LLC, Redmond, WA (US)
Filed by Microsoft Technology Licensing, LLC, Redmond, WA (US)
Filed on Oct. 13, 2023, as Appl. No. 18/486,868.
Application 18/486,868 is a continuation of application No. 17/190,530, filed on Mar. 3, 2021, granted, now 11,805,185.
Prior Publication US 2024/0048637 A1, Feb. 8, 2024
This patent is subject to a terminal disclaimer.
Int. Cl. H04L 67/50 (2022.01); A63F 13/75 (2014.01); A63F 13/87 (2014.01); G06N 3/04 (2023.01); G06N 3/08 (2023.01); H04L 51/212 (2022.01)
CPC H04L 67/535 (2022.05) [A63F 13/75 (2014.09); A63F 13/87 (2014.09); G06N 3/04 (2013.01); G06N 3/08 (2013.01); H04L 51/212 (2022.05)] 18 Claims
OG exemplary drawing
 
1. A server system comprising:
one or more processors configured to:
execute a platform for an online multi-user chat service that communicates with a plurality of client devices of users of the online multi-user chat service, the platform being configured to exchange user chat data between the plurality of client devices;
execute a user chat filtering program that performs filter actions for incoming user chat data exchanged on the platform for the online multi-user chat service, the user chat filtering program including a plurality of trained machine learning models and a filter decision service that determines a filter action to be performed for target portions of incoming user chat data based on respective outputs of the plurality of trained machine learning models for those target portions of incoming user chat data;
wherein the user chat filtering program is configured to:
receive incoming user chat data that includes a first user chat data and a second user chat data;
process the first user chat data and the second user chat data using the plurality of trained machine learning models;
determine that the first user chat data that includes a target word or phrase is offensive language based on the respective outputs of the plurality of trained machine learning models for the first user chat data;
determine that the second user chat data that includes the same target word or phrase is not offensive language based on the respective outputs of the plurality of trained machine learning models for the second user chat data;
determine a filter action to be performed for the first user chat data;
perform the filter action on the first user chat data without performing the filter action on the second user chat data;
send a feedback package to a moderator user computer device, the feedback package including the first user chat data and each of the respective outputs of the plurality of trained machine learning models for the first user chat data;
receive a feedback response from the moderator user computer device that indicates a moderator selected label of offensive language for the first user chat data; and
perform batched feedback training on at least one of the plurality of trained machine learning models based on the received feedback response from the moderator user computer device.