Figure 1. Internet Patrol Process Table 1. Summary of Previous Research in Cyberbullying Detection. Figure 2. A Graphical Summary of the Whole Method. Table 2. Comparison of Capabilities of Different Language Models 00 Capture Certain Patterns from 2 Hypothetical Corpus Consisting of Two Sentences, (1) and (2)? An example of such a sophisticated pattern can be explained as follows. The following sentence (in Japanese) Kyd wa nante kimochi ti hi nanda ! (What a pleasant day it is today!) contains a common and widely studied language pattern nante * nanda !1° Similar cases can be easily found in other languages; for instance, in English, the exclamative sentence “Oh, she is so pretty, isn’t she?” contains a pattern “Oh * is so * isn’t *?,” which is a typical example of a wh- exclamative sentence pattern (Beijer, 2002, Potts & Schwarz, 2008). The existence of such patterns in language is common and well recognized. However, it is not possible to discover them using an n-gram approach. Moreover, the number of skipped elements is recorded for each gap. For example, a 2-skip-3-gram can only allow 2 skips (omitting two words) between the elements, which means that the model considers as different patterns two cases in which the first gap has 2 skips and the second has 5 skips. Since the model assumes full control of the skip-length, the 2-skip-3- gram and 5-skip-3-gram consisting of the same elements (words) are represented as different entities and can never refer to the same pattern in a corpus. This assumption is unrealistic, since one can easily imagine that the same pattern, appearing in two sentences of different lengths, will be separated by gaps of different sizes. To illustrate this problem, in Table 2 we compare which of the above-mentioned language models are capable of discovering particular patterns present in the two sentences below. The last column on the right represents the capability of the language model based on the idea of language combinatorics (LC), applied in this research. Figure 3. Examples of Various Length (i.e., Number of Elements) Patterns Extracted from One Sentence. From the fact that the method first extracts all possible patterns from a sentence with a brute-force-inspired algorithm, we call the method pattern extortion, to distinguish it from typical pattern extraction methods based on n-grams or single tokens. Next, all nonsubsequent elements are separated with an asterisk (“*’’). All patterns generated this way are used to extract frequent patterns appearing in a given corpus. Exact examples of patterns created from one sentence are represented in Figure 3. In particular, this method, first, orders nonrepeated combinations generated from all elements of all input sentences in a training set. In every n-element sentence there is k-number of combination clusters, such as that 1 S k Sn, where k represents all k- element combinations being a subset of n. The number of combinations generated for one k-element cluster of combinations is equal to a binomial coefficient, represented in Equation 1. In this procedure, the system creates all combinations for all values of k from the range of {1, ... , n}. Therefore, the number of all combinations is equal to the sum of combinations from all k-element clusters of combinations, like in Equation 2. Table 5. Comparison of Best Precision, F-Score and Accuracy Within the Threshold Spar for Each Version of The Classifier for All Data Sets. Note: Best scores for each classifier version underlined; best scores for each preprocessing method in bold font. No scor means no BEP within the threshold range. Table 6. Break-Even Points for all Feature Sets and All Classifier Versions. Notes: Ordered according to highest F-score from left to right with corresponding precision, recall, and accuracy. The classifi version that achieved the score is in parentheses. Table 7. Analysis of Influence of Data Set Generalization on Results. words, the method provides better results when it can extract large number of features, even if they occurred infrequently. The data set preprocessing method that usually scored highest was the lemmatized data set combined with part-of-speech information. The highest score of all, mentioned above, was also achieved by this feature set. Table 8. Results of T-test for Best F-scores for Each Data Set. Figure 5. Comparison Between the Proposed Method (Best and Worst Performance) and Previous Methods. Table 9. All Results of Experiments on Traditional Classifiers on All Data Sets. Table 10. Comparison of Different Features for Two Methods. lable Al. Comparison of Best Precision withir the Threshold Span for Each Version of the Classifier for Tokenized Dataset. Table A2. Comparison of Best F-Score Within the Threshold Span for Each Version of the Classifier for Tokenized Dataset. Table A2. Comparison of Best F-Score Within the Threshold Span for Each Version of the Classifier for Tokenized Dataset. Note: In case of identical results for f-score, the best score was optimized for F-score. ‘able A3. Comparison of best Accuracy Within the Threshold Span for Each Version of the Classifier for Tokenized Dataset. Table A6. Comparison of Best F-Score Within the Threshold Span for Each Version of the Classifier for Lemmatized Dataset. Table A5. Comparison of Best Precision Within the Threshold Span for Each Version of the Classifier for Lemmatized Dataset. Table A7. Comparison of Best Accuracy Within the Threshold Span for Each Version of the Classifier for Lemmatized Dataset. lable A10. Comparison of Best F-Score Withir the Threshold Span for Each Version of the Classifier for POS-Tagged Dataset. Table A10. Comparison of Best F-Score Within the Threshold Span for Each Version of the Classifier for POS-Tagged Dataset. Table A9. Comparison of Best Precision Within the Threshold Span for Each Version of the Classifier for POS-Tagged Dataset. fable A9. Comparison of Best Precision Withit the Threshold Span for Each Version of the Classifier for POS-Tagged Dataset. ‘able A11. Comparison of Best Accuracy Withi the Threshold Span for Each Version of the Classifier for POS-Tagged Dataset. Table A11. Comparison of Best Accuracy Within the Threshold Span for Each Version of the Classifier for POS-Tagged Dataset. Table A13. Comparison of Best Precision Within the Threshold Span for Each Version of the Classifier fo Dataset Containing Tokens with POS Information. Table A14. Comparison of Best F-Score Within the Threshold Span for Each Version of the Classifier for Dataset Containing Tokens with POS Information. Table A15. Comparison of Best Accuracy Within the Threshold Span for Each Version of the Classifier fo1 Dataset Containing Tokens with POS Information. Table A18. Comparison of Best F-Score Within the Threshold Span for Each Version of the Classifier for Dataset Containing Lemmas with POS Information. Table A17. Comparison of Best Precision Within the Threshold Span for Each Version of the Classifier fo Dataset Containing Lemmas with POS Information. Table A19. Comparison of Best Accuracy Within the Threshold Span for each Version of the Classifier for Dataset Containing Lemmas with POS Information. Table A21 Comparison of Best Precision Within the Threshold Span for Each Version of the Classifier fo. Chunk-Separated Dataset. Table A22. Comparison of Best F-Score Within the Threshold Span for Each Version of the Classifier for Chunk-Separated Dataset. fable A23. Comparison of Best Accuracy Within the Threshold Span for Each Version of the Classifier for Chunk-Separated Dataset. Table A26. Comparison of Best F-Score Within the Threshold Span for Each Version of the Classifier with Dependency Parsed Dataset. Table A25. Comparison of Best Precision Within the Threshold Span for Each Version of the Classifier with Dependency Parsed Dataset. fable A27. Comparison of Best Accuracy Within the Threshold Span for Each Version of the Classifier witl Dependency Parsed Dataset. Table A29. Comparison of Best Precision Within the Threshold Span for Each Version of the Classifier fo: Chunks with NER. Table A30. Comparison of Best F-Score Within the Threshold Span for Each Version of the Classifier fo Chunks with NER. Table 41. Comparison of Best Accuracy Within the Threshold Span for Each Version of the Classifier for Chunks with NER. Table A34. Comparison of Best F-Score Within the Threshold Span for Each Version of the Classifier for Dependency Parsed Set with NER. Table A33. Comparison of Best Precision Within the Threshold Span for Each Version of the Classifier for Dependency Parsed Set with NER. Table A35. Comparison of Best Accuracy Within the Threshold Span for Each Version of the Classifier for Dependency Parsed Set with NER.
