US 12,261,827 B1
Proxy servers for managing queries to large language models
Tao Xu, Saratoga, CA (US); and Barun Kar, Saratoga, CA (US)
Assigned to Auradine, Inc., Santa Clara, CA (US)
Filed by Auradine, Inc., Santa Clara, CA (US)
Filed on Jan. 19, 2024, as Appl. No. 18/417,186.
Int. Cl. G06F 16/00 (2019.01); G06F 40/35 (2020.01); G06F 40/58 (2020.01); H04L 9/40 (2022.01); H04L 41/16 (2022.01)
CPC H04L 63/0281 (2013.01) [G06F 40/35 (2020.01); G06F 40/58 (2020.01); H04L 41/16 (2013.01)] 15 Claims
OG exemplary drawing
 
1. A method performed by one or more computers for managing network traffic to and from a server configured to: (i) receive, from a client device, a query in a natural language, and (ii) generate a response to the query in the natural language, the method comprising:
receiving, from the client device via a network connection, a network message comprising a new query for the server,
wherein the one or more computers are communicatively coupled to the server;
processing the new query, using a text encoder, to generate an embedding vector of the new query;
identifying, from amongst a plurality of entries of a vector database, a particular entry based on a similarity metric between: (i) the embedding vector of the new query, and (ii) an embedding vector of a particular query stored in the particular entry,
wherein each of the plurality of entries comprises: (i) an embedding vector of a respective query, and (ii) a corresponding response to the respective query;
determining whether the similarity metric is greater than a threshold similarity value;
based on determining that the similarity metric is greater than the threshold similarity value, sampling, from a distribution of random numbers, a random number;
determining whether the random number satisfies a threshold condition; and
based on determining that the random number satisfies the threshold condition,
transmitting, to the server, the new query,
receiving, from the server, a response to the new query,
processing the response to the new query and the response corresponding to the particular query, using the text encoder, to generate embedding vectors of the response to the new query and the response corresponding to the particular query,
calculating a second similarity metric between: (i) the embedding vector of the response to the new query and (ii) the embedding vector of the response corresponding to the particular query,
determining whether the second similarity metric is greater than a second threshold similarity value, and
based on determining that the second similarity metric is greater than the second threshold similarity value, sending the response to the new query or the response corresponding to the particular query to the client device.