The models for semantic similarity calculation mentioned above are basically double tower structures. The main advantage of the double tower structure is the fast similarity calculation. The fast here refers not to the inference speed of the model's single data, but to the calculation in a large number of question scenarios, such as the recall scenarios of similar questions. Because the double tower model actually produces a single question expression, the similarity calculation is just a simple calculation at the end, and the most time-consuming question expression operation can be completed offline. Cross-encoder performs splicing input when inputting the model, so that the two questions interact deeper, and similar sentence tasks are completed directly in the model, rather than just the semantic representation model of the questions. Therefore, the similar calculation effect of cross-encoder is also significantly better than that of bi-encoder with a double tower structure, but cross-encoder cannot get the vector representation of the question. Similar calculations between a large number of questions require real-time model inference calculations and consume more time. Cross-encoder is suitable for a small number of candidate question scenarios, so we can use it in the sorting stage of question questions to obtain better similar recognition effects.

The model structure is as follows. The bi-encoder with a double tower structure on the left is the cross-encoder structure on the right.

Source of the picture:21 classic deep learning inter-sentence relationship models｜Code & Skills - Zhihu

Experiment       

Tests were carried out on different pre-trained models. As in the table, hfl/chinese-roberta-wwm-ext(390M), hfl/chinese-roberta-wwm-ext-large(1.2G), and hfl/chinese-electra-180g-base-discriminator(1.2G) were used respectively. Perhaps because the training set data volume is already relatively large (50w+), the benefits of the pre-trained model after it becomes larger are not particularly obvious. During training, it was found that the effect of cross-encoder will also improve after exchanging two questions and adding them to the training set again, because the difference in sentence position was eliminated. Another point is that cross-encoder does not require too much training time. It has achieved better results with 1 epoch, which is better than 5 epoch.