Abstract:

The rise in duplicate questions has an impact on a variety of applications, including tests and assignments in higher education programmes, in addition to degrading the quality of the content. Therefore, it is crucial to address the problem of question duplication. In this research, we present a novel deep learning similarity checking method for efficiently detecting question duplication. With the help of deep neural networks, our method is able to fully comprehend both the semantic content and syntactic structure of questions. This makes it possible to compare and categorise question pairs precisely.

The representations obtained from the suggested method are used by the similarity measurement module to calculate the similarity score between pairs of questions. To account for multiple aspects of similarity, it integrates numerous similarity measures including cosine similarity and word embedding distances. The method also makes use of an attention mechanism to highlight key elements of the questions that significantly contribute to their shared sentence structure.

A sizeable dataset made up of question pairs with tagged duplications is used to evaluate and validate the effectiveness of the suggested strategy. This dataset includes both identical and paraphrased question pairings and spans a wide range of subjects. The proposed approach is thoroughly tested through considerable experimentation, proving its effectiveness and robustness. The outcomes demonstrate cutting-edge performance on benchmark datasets.

In order to successfully eliminate question duplication, this research introduces a precise and effective deep learning similarity checking method. The method consists of two main parts: a similarity measuring module and a multi-layered filter system that extract high-level features from input questions. This approach has the potential to improve the calibre and usability of question-based platforms by successfully capturing both syntactic structure and semantic content. Future research projects can examine the method's applicability in multilingual and multimodal environments as well as its integration into practical systems.

Keywords:

Syntactic structure, Question-based platforms, Multimodal contexts, Duplicated questions, Deep learning similarity checking, Cosine similarity.