Figure 15 – uploaded by Imene Bensalem

See full PDF downloadDownload figure

Figure 15 Chapter V. Character N-grams as the Only Intrinsic Evidence of Plagiarism Figure V-3. Steps for computing the n-gram classes of a document. The parameter n is the length of n-grams and m is the number of classes. In this example m = 3 (class labels are from 0 to 2) following subsections provide further details on these three stages. 3.2. N-gram Classification

Related Figures (62)

AEPULV ALE CLI MME Se. ME MED VD oOwLtatrit, tli me MADER LEE ES WEI A Gai Usow tL VUeU Ul VV. Chapter II provides a survey on the current methods of detecting plagiarism in Arabic documents. The survey shows that almost all the methods are based on uncovering plagiarism by comparing the suspicious document to the potential sources of plagiarism (the external approach). This motivates us to conduct the first experiments on Arabic documents that attempt to detect plagiarism by spotting the writing style changes (the intrinsic approach). In the light of these experiments, that utilise a small ad-hoc corpus, we felt the necessity to build a larger evaluation corpus that allows for a better assessment of the task performance on Arabic documents. a larger evaluation corpus that allows for a better assessment of the task performance on Arabic Besides the technical aspect of Arabic plagiarism detection, this chapter discusses another

Figure II-1. External plagiarism detection methods building blocks coining standard terminology. Therefore, we refer the reader to PAN overview papers

Chapter II. Arabic Plagiarism Detection: Critical Review Figure II-2. Intrinsic plagiarism detection methods building block

Figure II-3. Arabic plagiarism detection papers published from 2008 to June 2019 As shown in Figure II-4, almost all the collected publications are papers that describe

Chapter II. Arabic Plagiarism Detection: Critical Review

Table Il-1. Bad smells that we detected in Arabic plagiarism detection papers 10 See Chapter HI (Section 5.3.1) for details on the standardised evaluation measures of plagiarism detection. 'l Most of the “bad smells” that we did not consider concern the statistical significance, which is usually not utilised in plagiarism detection studies. Note that this does not mean that this technique is not applicable for plagiarism detection evaluation but rather its use is uncommon even in the best studies. This fact might be attributed to the lack of practical guidelines on hypothesis testing that may accompany the current plagiarism detection evaluation measures. Instead of a simplistic approach of including/excluding papers from our literature review. we applied the approach proposed in (Menzies and Shepperd 2019), which consists in assessing the quality of papers in terms of twelve criteria. The authors called these criteria “bad smells” and defined them as the surface issues that might be detected in research publications and that can be indications of serious problems. The scope of Menzies anc Shepperd’s investigation is software analytics. Still, the authors noted that while some of the proposed “bad smells” are specific to software analytics, others are general and then applicable to other scientific domains. Hence, we selected four “bad smells” (from twelve) that we judged appropriate for plagiarism detection research". Table II-1 lists them in the first column. In the second column, we determine exactly how these “bad smells” emerged in the examined Arabic plagiarism detection papers.

Table II-2. Overview of the number of papers considered in our study 3.2.2 Results

Figure II-5. Proportion of Arabic plagiarism papers with and without “bad smells” Chapter II. Arabic Plagiarism Detection: Critical Review

Table II-3. Scope of the examined Arabic plagiarism detection papers 3.3. Methods and Evaluation Corpora Chapter II. Arabic Plagiarism Detection: Critical Review In this section, we review the 24 selected works (from the previous step) in terms of their

Table II-4. Papers on Arabic plagiarism detection using the external approach 'S This method does not use exactly the principle of creating queries to a search engine to retrieve the candidate document but it compares the suspicious and the source documents at three levels starting from the document level then th paragraph level and finally the sentence level. If no similarity is detected at the document level, the followings level will not be considered. Thus, we consider the document-level comparison as the candidate retrieval module.

Table II-5. Description of the corpora used to evaluate plagiarism detection methods on Arabic documents. The character '-' is used when no information is provided.

Table II-7. Performance evaluation Chapter II. Arabic Plagiarism Detection: Critical Review 4.2.2 Results and Discussion We used three measures to evaluate the performance of discriminators: Precision (equation 1)

Table II-8. Combination’s results: baseline vs. the most precise voting schemes 1.3, Experiment 2: Combining Discriminators

Chapter Ill. Evaluation of Plagiarism Detection on Arabic Documents Figure III-1. The insertion based approach of building plagiarism detection evaluation corpor important books of building corpora (McEnery et al. 2006), the copyright-free documents are

Table III-1. Comparison between approaches to creating suspicious documents. The symbol V indicates an advantage, and * indicates a disadvantage. Chapter III. Evaluation of Plagiarism Detection on Arabic Documents

Table Ill-2. AraPlagDet shared task schedule *July 16 is the release date of a sample of the training corpus. The complete training corpus has been released on August 10. Chapter Ill. Evaluation of Plagiarism Detection on Arabic Documents

Figure III-2. Different representations of the same word with and without letters’ diacritics. Chapter Ill. Evaluation of Plagiarism Detection on Arabic Documents

Table III-3. Statistics of the ExAra corpus Chapter III. Evaluation of Plagiarism Detection on Arabic Documents

Table IIl-4. Source retrieval approaches with their building blocks used in the participants’ methods. Each column describes an approach in terms of its building blocks. The first line provides a concise description of the approach, and the second line indicates the methods that employed each approach. For example, Ma- gooda_2 method used two approaches: sentence-based and keyword-based indexing. With respect to Alzahrani’s method, it is suitable to an offline scenario, i.e., when th source of plagiarism is local and not too large, as in the case of detecting plagiarism betwee: students’ assignments. This is for two reasons: (i) its retrieval model is not structured to b used with search engines (for example, there is no query formulation, see Table III-4); and (ii it is based on fingerprinting all the source documents and entails an exhaustive compariso: between the n-grams of the suspicious document and those of each source document, which i not workable if the source of plagiarism is extremely large, like the web. Still, even with th intention to be used offline, it would be better to use retrieval techniques that allow for th processing of a large number of documents in a reasonable time such as inverted indexes Malcolm and Lane (2009) discuss the importance of scalability even for offline plagiarisn detectors.

Table III-5. Text alignment approaches with their building blocks used in the participants’ methods Language Dependence Chapter Ill. Evaluation of Plagiarism Detection on Arabic Documents

respectively. The symbols |saci] and |Sde| are, respectively, the lengths of Sac and Sdet in For a single actual plagiarism case, Sact, a plagiarism detection method may output multiple detections (separate or overlapping). Thus, granularity is used to average the number of the detected cases for each actual case as depicted in formula 3. Actge, & Act is the set of the actual cases that have been detected, and Det,,., © Det is the set of the detected cases that intersect with a given actual case Sact. The optimal value of the granularity is 1, and it means that for each actual case sac, no more than a single case has been detected (i.e. not many overlapping or adjacent cases). Detected cases). The symbols |Ac¢| and |Det| are the number of actual and detected cases,

Method Precision Recall Granularity Plagdet Table III-6. Performance of the external plagiarism detection methods on the test corpus comparison with what has been achieved by the state-of-the-art methods (see for example the

Table III-7. Detailed performance of the participants’ methods. In each measure, the underlined values are the highest per parameter.

Table III-8. Statistics of InAra corpus criterion 1 Each host document must be written by one author only. If the document is multi- Chapter Ill. Evaluation of Plagiarism Detection on Arabic Documents

Table Ill-9. Sources of texts used to build InAra corpus Chapter III. Evaluation of Plagiarism Detection on Arabic Documents

Table III-11. Performance of the intrinsic plagiarism detection methods Chapter Ill. Evaluation of Plagiarism Detection on Arabic Documents 6.3.2. Detailed Results

Table III-12. Detailed performance of the intrinsic plagiarism detection methods a module that detects and filters out the Quranic citations. Such a module can rely on the Chapter III. Evaluation of Plagiarism Detection on Arabic Documents external approach, whereby the whole text of the document is compared to the Quran corpus

Chapter IV. Intrinsic Plagiarism Detection: a Survey Figure IV-1. Timeline of some milestones related to intrinsic plagiarism detection

Table IV-1 Intrinsic plagiarism detection and its related research areas The major drawback of this perception emerges when the plagiarism constitutes the Intrinsic plagiarism detection in its essence could be seen as an anomaly-of-authorship detection at fragment level (Guthrie et al. 2007), where plagiarism is the anomaly, and the text written in the plagiarist’s own style is the normal part. In fact, most of the current IPD methods deal with IPD as an anomaly detection problem. That is, they are based on the assumption that the normal data (original part) is the majority, and hence can be characterised, and the abnormal data (plagiarised part) is sparse and thus difficult to characterise. Therefore, methods based on this assumption build a writing style model of the whole suspicious document, and consider as plagiarism any fragment deviating from this general style (Mahgoub et al. 2015; Muhr et al. 2010; Oberreuter and Velasquez 2013; Stamatatos 2009a; Suarez et al. 2010; Zechner et al. 2009).

Figure IV-2. Taxonomy of the building blocks of intrinsic plagiarism detection methods The pre-processing heuristics are called so because they operate before the fragment-level analysis. These heuristics aim to filter out the irrelevant information that may disrupt the style analysis (through cleaning, normalisation and genre analysis) or reduce the computation by taking an early decision on the document (through checking whether the document is taking an early decision on the document (through checking whether the document i:

Table IV-2. Pre-processing heuristics in intrinsic plagiarism detection methods document as plagiarism-free if the variance of the style change function is not significant. Practically speaking, this implementation checks the significance of the style variance by comparing the standard deviation 6 of the style change function to a predefined threshold Ts. If 5 < ts then the heuristic marks the document as plagiarism-free.

it in a structured manner. Feature extraction in natural language processing (NLP) structures

Figure IV-3. Feature extraction at fragment and document levels. The symbols s,, ..., Sn denote the fragments and f,,..., fm denote the features. kinds of units: (i) one character, (ii) a sequence of characters, or (iii) a class of characters. See,

Table IV-4. The units from which the character features are extracted with examples extracted from a sentence features are computed, we classify character features depending on whether the unit is defined Character n-grams are sequences of contiguous characters of a predefined length extracted from the text without considering any linguistic relationship between them. Despite their simplicity, these features have proven their effectiveness in many NLP applications, such as authorship attribution (Cavnar and Trenkle 1994; Stamatatos 2016), native language identification (Kulmizev et al. 2017) and opinion spam detection (Hernandez Fusilier et al. 2015). Based on their reputability as stylistic markers notably for authorship attribution, they have been employed in intrinsic plagiarism detection. As a matter of fact, Stamatatos (2009a) was the first to develop a character-n-grams-based IPD method. Although it utilises only these anguage-independent features, Stamatatos’ method was ranked first in the PAN 2009 intrinsic plagiarism detection shared task. This seminal method, by its simplicity, inspired other researchers, who reproduced it partially or fully in their works (Kasprzak and Brandejs 2010; Kestemont et al. 2011; Kuta and Kitowski 2014; Rao et al. 2011). Character n-grams are sequences of contiguous characters of a predefined length extracted

Table IV-6. Some linguistic aspects manipulated to produce different sentence structures detection wherein the writing style is analysed at the fragment level. Nonetheless, these measures are included in numerous intrinsic plagiarism detection methods, which are: (Meyet zu Eien et al. 2007), (Stein et al. 2011), (Kern et al. 2012), and (Carnahan et al. 2014). Or the other hand, Meyer zu EiBen and his colleagues (2007; 2006) proposed a new vocabulary richness measure called Average Word Frequency Class, which is argued to be ideal for IPD due to its stability with different text lengths. Later, this feature has been used in other methods. such as (Stein and Meyer zu Eifen 2007), (Zechner et al. 2009), and (Carnahan et al. 2014) In addition, variants of this measure are used in (Polydouri et al. 2017). measures are included in numerous intrinsic plagiarism detection methods, which are: (Meyer

writing style of a fragment and that of the whole document. Then, all fragments with a a wor Chapter IV. Intrinsic Plagiarism Detection: a Survey

The IPD methods that use supervised learning are listed in Table IV-9. It remains to say that the pitfall of IPD methods based on supervised learning is that they may suffer from the lack of training data. And even if it is available, there will be an imbalance in the number of plagiarised and the non-plagiarised examples since naturally the original texts are more abundant. This issue renders the IPD a classification problem with skewed classes, which is a known problem in machine learning that may lead to training biased classifiers. In (Polydouri et al. 2017, 2018), the authors attempted to mitigate this problem by using sampling techniques on the training corpus aiming to construct a balanced dataset. This problem can be also tackled by using classification algorithms designed to function with datasets of skewed classes, such as Complement Naive Bayes (Rennie et al. 2003). In that context, we used this algorithm in one of our IPD experiments and it proved its effectiveness in comparison with the original Naive Bayes (Bensalem et al. 2014b)*.

Table IV-9. The supervised learning-based methods used for intrinsic plagiarism detection 4.4.2 Clustering Clustering is an unsupervised machine learning approach that creates, from a given set 0: elements, subsets that group together the similar elements. The similarity between the element: is assessed based on their feature vectors. The number of clusters to create should be determined a priori for most of the algorithms. This paradigm is well suited for multi-autho: documents segmentation wherein each cluster involves the fragments of similar writing style (see, e.g., (Akiva 2012; Kern et al. 2012)), and hence, the number of the clusters represent: the number of the authors involved in writing the document. In the existing intrinsic plagiarisn methods, the number of the clusters created from the suspicious document fragments i: typically two; one of them groups the plagiarism-free fragments and the other one contains the plagiarised fragments. Clustering is an unsupervised machine learning approach that creates, from a given set of elements, subsets that group together the similar elements. The similarity between the elements

Chapter IV. Intrinsic Plagiarism Detection: a Survey Figure IV-6. Illustration of the density-based outlier detection for intrinsic plagiarism detection. Plagiarised and non-plagiarised sections can be separated if their values of a feature fi are differently distributed (adapted after (Stein et al. 2011)). As for the priors, i.e., P(Class = plago) and P(Class = plagi) —which is the portion of each class among all the fragments”°— the authors of this approach stated that they are estimated either by an impurity assessment (meta information on the document) or by the maximum likelihood estimator which assumes that the classes are uniformly distributed, i.e., half of the fragments is plagiarised and the other half is not. However, it has not been stated in the paper (Stein et al. 2011) which of these two options is adopted in the conducted experiments. As for the priors, i.e., P(Class = plago) and P(Class = plag:) —which is the portion of each

Figure IV-7. Steps of the distance-based outlier detection for intrinsic plagiarism detection. In the figure (A), the distance is computed between the fragments and the document; and in the figure (B), the distance is computed between each pair of fragments. are averaged. Hence, for both cases, the document is represented with a vector of distances

Table IV-11. Post-processing heuristics in intrinsic plagiarism detection methods 8 The performance measure used by Polydouri et al. (2017, 2018) are computed based on the number of sentences an not the number of characters. For example, given a plagiarised fragment composed of 4 sentences, if the software detect 2 of them, the recall measured on this fragment, according to Polydouri et al. would be 0.5. However, the standardise: recall score (Potthast et al. 2010c) could be more or less different since it is the ratio of the length, in characters, of th 2 detected sentences to the length of the full plagiarised fragment. To complete the picture on intrinsic plagiarism detection, it is necessary to talk about its effectiveness. In fact, despite the variety of heuristics and stylistic features used in the methods (as shown in Section 4), their performance scores are still poor. To the best of our knowledge. few methods, such as (Stamatatos 2009a) and (Oberreuter et al. 2011b; Oberreuter and Velasquez 2013), reached an F-measure greater than 0.3 using a standardised evaluation framework. Other methods, for instance Stein et al. (2011), Tschuggnall and Specht (2013c) and Polydouri et al. (2017, 2018) obtained relatively higher scores. Nonetheless, the twc former methods have been evaluated on only subsets of the evaluation corpus, and the evaluation of the latter method is based on a modified version of the performance measures”*. evaluation of the latter method is based on a modified version of the performance measures”

Table V-1. The frequency and length of character n-grams in intrinsic plagiarism detection methods 5 The table lists only the methods that provide information on the used character n-grams. I ON For example, in (Kestemont et al. 2011), representing the text using only the most frequent n-grams extracted from a corpus was based on an efficiency reason which is to reduce the computation. However, no experiment has been done to check the impact of this reduction of the number of the used n-grams on performance or to prove that high-frequency n-grams are more effective than the rest of n-grams with lesser frequency. In (Kuznetsov et al. 2016), the frequencies of both rare and frequent n-grams in a sentence were among the features used to quantify the writing style incoherence between this sentence and the rest of the document. However, the rationale behind these choices has not been explained.

Chapter V. Character N-grams as the Only Intrinsic Evidence of Plagiarism

Computing the NFCP features by considering the repetition of n-grams in the fragment fragment, and its maximum value is the number of fragments in d if ng; occurs in each

Computing the NFCP features without considering the repetition of n-grams in the fragment Figure V-5. Illustration of two ways of computing the proportion of n-gram classes in a fragment

Figure V-6. Average of InfoGrain of the features generated by different variants of the extraction method Chapter V. Character N-grams as the Only Intrinsic Evidence of Plagiarism the subsequent experiments, we will adopt the best variant (S1RO) without mentioning that every time.

Table V-3. Statistics on the evaluation corpora Chapter V. Character N-grams as the Only Intrinsic Evidence of Plagiarism 1 Datasets and Performance Measures We used for our experiments three evaluation corpora in English and one corpus in Arabic with its two parts training and test. The English corpora (Potthast et al. 2010c) have been developed for the international competition on plagiarism detection (PAN) of the years 2009, 2010 and 2011 to evaluate the IPD methods (Potthast et al. 2009, 2010a, 2011). We used specifically the test part of each corpus'*. The Arabic corpus (InAra) (Bensalem et al. 2013a, 2013b) has been built by ourselves, following PAN annotation standards, and has been used in AraPlagDet 2015", the first plagiarism detection competition on Arabic documents (Bensalem et al. 2015). We used for our experiments three evaluation corpora in English and one corpus in Arabic

Figure V-7. F-measure of our method in comparison with the best methods in the PAN intrinsic plagiarism detection competitions Chapter V. Character N-grams as the Only Intrinsic Evidence of Plagiarism

Figure V-8. The 54 classes obtained from the n-grams of a document by classifying them into different number of classes, m. For example, when m = 2 (the top of the figure), this means that the n-grams of the document are classified into 2 classes labelled 0 and 1. The former represents n-grams of low frequency, and the latter represents n-grams of high frequency Practically, for each language, a total number of 540 classifiers (in each iteration), corresponding documents including 34765 and 5547 plagiarism cases, respectively. Once the 540 features

Table V-4. Evaluation setting of NFCP features to the 540 NFCP features, have been trained and tested using the five datasets described in Section 5. Explicitly, cross-validation has been performed between each couple of corpora, 1.e., each corpus is used separately, on the one hand, for training a classification model and on the other hand, for testing the models trained on the other corpora of the same language. Consequently, we obtained for each NFCP feature six classification results on English corpora and two classification results on the Arabic corpus as illustrated in Table V-4. Then, the F- measure scores are averaged for each language to be used in our analysis. to the 540 NFCP features, have been trained and tested using the five datasets described in Section 5. Explicitly, cross-validation has been performed between each couple of corpora, Chapter V. Character N-grams as the Only Intrinsic Evidence of Plagiarism

Figure V-9. The distribution of performance of the NFCP features computed on English text (a) and Arabic text (b) Chapter V. Character N-grams as the Only Intrinsic Evidence of Plagiarism

Figure V-11. Sensitivity of NFCP features performance to the n-gram length on English (left) and Arabic (right) Chapter V. Character N-grams as the Only Intrinsic Evidence of Plagiarism

Table V-5 The configurations that produce the best NFCP features Chapter V. Character N-grams as the Only Intrinsic Evidence of Plagiarism

Chapter V. Character N-grams as the Only Intrinsic Evidence of Plagiarism Figure V-12. Performance of combined NFCP features selected using different techniques 7 Sensitivity Analysis of Stamatatos’ Method Performance to N-grams Frequency and Length

Table V-6. Cumulative percentages computed on the 3-grams of the suspicious-documento1020 of PAN-PC-09 profiles is based on the cumulative percentages that we computed on the frequency distribution

Figure V-13. Sensitivity of Stamatatos’ method performance to the size of the selected subset of the n-grams (in percentage) and n-gram length. N-grams are selected from profiles sorted according to frequencies starting from the least frequent n-grams (En-1 and Ar-1) or the most frequent n-grams (En-2 and Ar-2). The performance is computed on English (En-1 and En-2) and Arabic (Ar-1 and Ar-2) documents. In the charts En- 1 and Ar-1, the values of the x-axis labelled with an asterisk (*) represent the sizes of sub-profiles that contain only n-grams whose frequency = 1 whatever their proportion in the document’s full profile. Chapter V. Character N-grams as the Only Intrinsic Evidence of Plagiarism

Figure 15 – uploaded by Imene Bensalem

Related Figures (62)

Connect with 287M+ leading minds in your field