US 11,888,792 B2
Bot supervision
Michael Higgins, New York, NY (US); Xi Chen, New York, NY (US); Chuqinh He, New York, NY (US); and Ender Ricart, Bellingham, WA (US)
Assigned to LIVEPERSON, INC., New York, NY (US)
Filed by LIVEPERSON, INC., New York, NY (US)
Filed on Oct. 18, 2022, as Appl. No. 17/968,252.
Application 17/968,252 is a continuation of application No. 17/475,725, filed on Sep. 15, 2021, granted, now 11,522,818.
Claims priority of provisional application 63/078,718, filed on Sep. 15, 2020.
Prior Publication US 2023/0143863 A1, May 11, 2023
This patent is subject to a terminal disclaimer.
Int. Cl. G06N 3/044 (2023.01); H04L 51/02 (2022.01); G06N 20/00 (2019.01); H04L 51/216 (2022.01); G06F 18/24 (2023.01); G06F 18/214 (2023.01); G06N 5/04 (2023.01); G06Q 10/10 (2023.01); G06N 3/08 (2023.01)
CPC H04L 51/02 (2013.01) [G06F 18/214 (2023.01); G06F 18/24 (2023.01); G06N 3/044 (2023.01); G06N 20/00 (2019.01); H04L 51/216 (2022.05); G06N 3/08 (2013.01); G06N 5/04 (2013.01); G06Q 10/10 (2013.01)] 30 Claims
OG exemplary drawing
 
1. A computer-implemented method comprising:
storing a set of supervision criteria in memory, the set of supervision criteria including a threshold level of confidence;
monitoring one or more messages in a conversation between a bot and a user of a client device in accordance with the set of supervision criteria;
detecting that a bot prediction associated with the conversation fails to meet the threshold level of confidence associated with the set of supervision criteria;
requesting input from an agent device based on the bot prediction failing to meet the threshold level of confidence, wherein the agent device provides annotation input identifying one or more of the messages in the conversation and identifying contextual or supplemental information related to classification of the one or more identified messages in the conversation; and
updating a learning model based on the one or more messages identified by the annotation input and the classification according to the identified contextual or supplemental information, wherein the bot uses the updated learning model to make a next bot prediction regarding a next conversation having one or more similar messages.