Volume 17 , Issue 1 , PP: 01-09, 2026 | Cite this article as | XML | Html | PDF | Full Length Article
Sura Mahroos 1 * , Rihab Hazim 2 , Yaqeen Saad 3 , Nadia Mohammed 4
Doi: https://doi.org/10.54216/JCIM.170101
There are several real-world uses for the duplication system or record linkage. In order to help the system make the best judgments, it appears in a broad area of recognizing similar data, joining online papers in the wide web, detecting plagiarism, and allowing several applications to enter it. To improve the financial interest and applicability of logistics project, routing is crucial. The following is the issue with this study: Because duplicate receipts contain the same significant change in data restrictions and limitations, and the data change itself is minor, the duplicate record data is ambiguous to other redacted records that are reassembled with the same customer. The purpose of this study is to use statistical techniques and the Q-gram to discover the best method for the detection and removal of duplicate records. We propose the following goals to help achieve that goal: Reduce the size of the data warehouse (DW) by providing a data warehouse free of duplicates. Decrease the amount of time spent looking for the (DW) and improve the DSS. The approach is divided into two stages: first, identify similarity records based on Q-gram similarity; second, determine whether classification records may be improved by statistical methods. The percentage threshold of 0.68 has been determined. It goes through a statistical process that decides whether this record is duplicated if the key ratio similarity is surpassed. The accuracy of the suggested work is 79%.
Duplicate Elimination , Data Cleaning , Similarity Score , Q-Gram Similarity , Statistical Tools
[1] P. Bhatia, Data Mining and Data Warehousing: Principles and Practical Techniques. Cambridge University Press, 2019.
[2] Darch Abed Dawar, “Enhancing Wireless Security and Privacy: A 2-Way Identity Authentication Method for 5G Networks,” International Journal of Mathematics, Statistics, and Computer Science, vol. 2, pp. 183–198, 2024. doi: 10.59543/ijmscs.v2i.9073.
[3] Kennedy et al., “Epidemiology of homicide in community-dwelling older adults: a systematic review and meta-analysis,” Trauma, Violence, & Abuse, vol. 24, no. 2, pp. 390-406, 2023.
[4] M. Anitha et al., “Duplicate Detection of Records in Queries Using Clustering,” International Journal of Research in Computer Science, eISSN: 2249-8265.
[5] M. Padmanaban and T. Bhuvaneswari, “A technique for data deduplication using Q-gram concept with support vector machine,” International Journal of Computer Applications, vol. 61, no. 12, 2013.
[6] B. Khan et al., “Identification and removal of duplicated records,” World Applied Sciences Journal, vol. 13, no. 5, pp. 1178-1184, 2011.
[7] M. Padmanaban and R. Radha, “PSO Algorithm to Select Subsets of Q-Gram Features for Record Duplicate Detection,” International Journal of Computer Applications, vol. 82, no. 12, 2013.
[8] O. Azeroual, A. Nikiforova, and K. Sha, “Overlooked Aspects of Data Governance: Workflow Framework For Enterprise Data Deduplication,” in 2023 International Conference on Intelligent Computing, Communication, Networking and Services (ICCNS). IEEE, 2023.
[9] O. Alotaibi, S. Tomy, and E. Pardede, “A Framework for Cleaning Streaming Data in Healthcare: A Context and User-Supported Approach,” Computers, vol. 13, no. 7, p. 175, 2024.
[10] L. Yang et al., “Authenticating q-gram-based similarity search results for outsourced string databases,” Mathematics, vol. 11, no. 9, p. 2128, 2023.
[11] M. Sagheer, S. S. Salih, and S. M. Searan, “Design and Implementation of Secure Stream Cipher Algorithm,” International Journal of Computing and Digital Systems, vol. 7, no. 03, pp. 127-134, 2018.
[12] S. Yakhni et al., “Using fuzzy reasoning to improve redundancy elimination for data deduplication in connected environments,” Soft Computing, vol. 27, no. 17, pp. 12387-12418, 2023.
[13] C. H. Lindquist et al., “When pre-release optimism meets post-release reality: Understanding reentry success through a longitudinal framework assessing pre-and post-release perceptions,” Crime & Delinquency, vol. 71, no. 1, pp. 144-174, 2025.
