US 12,271,399 B2
Cross-modal search method and related device
Ke Mei, Shenzhen (CN); Huan Zheng, Shenzhen (CN); and Ming Li, Shenzhen (CN)
Assigned to TENCENT TECHNOLOGY (SHENZHEN) COMPANY LIMITED, Shenzhen (CN)
Filed by TENCENT TECHNOLOGY (SHENZHEN) COMPANY LIMITED, Shenzhen (CN)
Filed on Jul. 18, 2023, as Appl. No. 18/353,882.
Application 18/353,882 is a continuation of application No. PCT/CN2022/134918, filed on Nov. 29, 2022.
Claims priority of application No. 202210222089.0 (CN), filed on Mar. 7, 2022.
Prior Publication US 2023/0359651 A1, Nov. 9, 2023
Int. Cl. G06F 16/28 (2019.01); G06F 16/24 (2019.01); G06F 16/2455 (2019.01)
CPC G06F 16/285 (2019.01) [G06F 16/2455 (2019.01)] 15 Claims
OG exemplary drawing
 
1. A cross-modal search method, performed by a computer device, and comprising:
displaying, on a display screen of the computer device, a session interface of a social session;
displaying, on the display screen, a session record detail interface in response to a selection of viewing a historical session record of the social session, the historical session record of the social session comprising second modality data;
displaying, on the session record detail interface, a search box and a plurality of options about modality types;
detecting a selected option of modality type and first modality data inputted in the session record detail interface;
searching in a second modality database based on content information of the first modality data to obtain a first set, the first set comprising at least one piece of the second modality data matched with the content information of the first modality data, the second modality data having the selected modality type;
searching in the second modality database based on semantic information of the first modality data to obtain a second set, the second set comprising at least one piece of second modality data matched with the semantic information of the first modality data; and
presenting, in the session record detail interface on the display screen, a cross-modal search result corresponding to the first modality data based on the first set and the second set,
wherein:
the second modality database stores N pieces of second modality data and respective attribute information of the N pieces of second modality data, and N is a positive integer;
the attribute information comprises first modality description information; one of the N pieces of second modality data is denoted as an ith second modality data, i is a positive integer, and i is less than or equal to N; and
the searching in the second modality database to obtain the first set comprises:
determining a match degree between the content information of the first modality data and the attribute information of a piece of the second modality data in the N pieces of second modality data as a match degree corresponding to the piece of second modality data, comprising:
determining semantic similarity between the content information of the first modality data and the first modality description information of the ith second modality data in the N pieces of second modality data as the match degree corresponding to the ith second modality data; and
adding the piece of the second modality data with the corresponding match degree satisfying a match condition to the first set, comprising:
adding the ith second modality data to the first set in response to that the semantic similarity between the content information of the first modality data and the first modality description information of the ith second modality data is greater than a first similarity threshold.