Table 20 Univariate feature ranking based on F-Score
Related Figures (28)
Automated Essay Scoring is defined as the act of assigning scores to essays using an AES system. Throughout this document I use the acronym AES by itself when discussing general aspects of automated essay scoring. I use “AES system” when discussing specific aspects related to functions and their implementations. Figure | presents an overview of the functions of an AES in operation. It receives an essay in digital form as an input and it outputs a score. The AES system consists of software functions. There are functions to pre-process the essays into an internal format that is accepted by the other functions, there are functions to extract the required features from the essays anc there are functions to perform the classification task to decide which score should be assigned to an essay. The classifier is based on supervised machine learning where the classes are the scores and eacl essay is represented by a set of features. (Dikli, 2006) The life cycle of an AES system can be divided into two main phases: 1) a development phase and 2) an operational phase. During the development phase the AES system software is created and it may be a time consuming activity. An important input for this phase is the essays with the corresponding scores that are used as the training data to build a supervised machine learning classifier. When the development phase is completed a fully operational AES system is available. One characteristic of the classifier is its training error that shows how well it does when classifying the training data. The fact that the classifier is able to classify all classes in the training data correctly (training error = 0%) does not assure that it will perform well when classifying new data. The key question is how well the classifier is able to generalize and to rate new essays in a consistent way. Generalization error is the error that the classifier makes when classifying new data. There are different ways to measure the generalization error. One way is to use a set of test data that was not involved in the training process. The test data contains essays and the corresponding scores assigned by human raters. It is important that that all the data used during training and test are random samples Figure 4 Support vectors that violate the SVM margins (Manning , Raghavan, & Schutze, 2008) Table 1 Frequency distribution of blind raters and teachers in training data The bar chart in figure 6 shows the proportional distribution of number of essays per score related to the total number of essays in the training data. The distribution resembles a normal distribution. There are relatively few “tail scores” and relatively many “central scores”. In the blind raters’ scores, 77% of the essays are scored with “G” or VG (central scores). But there are only 7.53% of the essays that are scored with “MVG” and 15.8% that are scored with IG (tail scores). Table 2 Confusion matrix showing absolute numbers of blind scores teachers’ scores in training data fe el he confusion matrix in table 2 describes the absolute values for agreements and disagreement between the teachers and the blind scores. The value in the cell (i, j) indicates the number of essays with blind score “1” that were classified by a teacher with the score “j”. For example, the cell defined by the row “IG” and the column “VG” contains the value 42. It indicates that there are 42 essays that are scored with “IG” by the blind raters and with “VG” by the teacher. The values in diagonal cells (i 1) indicate the exact agreements for the ith grade. The ith row total describes the total number essays with blind score “i”. The ith column total describes the total number essays with teacher’s score cos rhea Table 3 Normalized confusion matrix showing the proportion of agreements and disagreements Table 3 shows the normalized confusion matrix where each row is normalized by the total number of essays that have the corresponding blind score. The value in the cell (i, j) indicates the proportion of essays with blind scores “i” that were assigned score “j” by the teachers. For example, the cell defined by the row “G” and the column “VG” has the value 34.6%. It indicates that 34.6% of all the essays that have the blind score “G” were assigned score VG by the teachers. A value in the diagonal (dark shaded) indicates the proportion of essays with the corresponding score where the blind raters and the teachers are in exact agreement. Table 4 Summary of agreements between blind raters and teachers Figure 7 Overview of the grid search procedure for SVM with kernels The SVM package provided by Scikit-learn (Pedregosa, et al., 2011) is used. The packages are based on LIBSVM (Chang & Lin, 2011). The basic recommendations for the training process for the SVM classifier are defined in (Hsu, Chang, & Lin, 2010). Figure 7 presents an overview of the main activities required to train SVM classifiers. The different activities are described in more detail further down. Table 5 Multivariate Feature ranking based on extremely randomized trees As a comparison, the appendix A shows the results for a univariate feature ranking based on F-score (ANOVA) where the importance of each feature is measured without consideration of the other features. Table 6 Summary of agreements between blind raters and LDAC scores using 20 features Table 7 Confusion matrix with absolute values for number of blind scores versus LDAC scores The confusion matrix in table 7 describes the absolute values for agreements and disagreement between the LDAC and the blind scores. The value in the cell (i, j) indicates the number of essays with blind score “1” that were classified by LDAC with the score “j”. For example, the cell defined by the row “IG” and the column “VG” contains the value 15. It indicates that there are 15 essays that are scored with “IG” by the blind raters and with “VG” by the LDAC classifier. The values in diagonal cells (i, 1) indicate the exact agreements for the ith grade. The ith row total describes the total number 66599 of essays with blind score “i”. The ith column total describes the total number of essays with LDAC 66599 score 1. Table 8 Normalized confusion matrix showing percent of agreements and disagreements Table 8 shows the normalized confusion matrix where each row is normalized by the total number of essays that have the corresponding blind score. The value in the cell (i, j) indicates the proportion of essays with blind scores “1” that were assigned score “j” by the LDAC classifier. The cell defined by the row “G” and the column “VG” has the value 10. 5%, It indicates that 10.5% of all the essays that have the blind score “G” were classified as “VG” by the LDAC classifier. A value in the diagonal (dark shaded) indicates the proportion of essays with the corresponding score where the blind raters and the LDAC classifier are in exact agreement. Table 9 Summary of agreements between blind raters and SVM-RBF Table 10 Confusion matrix with absolute values for number of blind scores versus SVM-RBF scores Table 11 Normalized confusion matrix showing percent of agreements and disagreements The t-test for related data (SVM-RBEF scores and blind scores) shows that there is a significant difference between the mean essay scores generated by the AES based on SVM-RBF (mean=1.185, standard deviation=0.564) and the mean scores generated by the blind raters (mean=1.259, standard deviation=0.811). The results suggest that the blind raters tend to assign higher scores than the AES using SVM-RBEF. 7 [his is based on the t-statistics v alue of -3.873 and a corresponding p-value that is much less than 0.01(two tailed). The Spearman Rank Correlation test shows that there is a positive correlation between the SVM-RBEF scores and the 0.603 and the p va ue is much less than 0.01. blind raters that is significant. The rho-coefficient is Table 13 Confusion matrix with absolute values for number of blind scores versus SVM-POL scores Table 14 Normalized confusion matrix showing percent of agreements and disagreements The t-test for related data (SVM-POL and blind scores) shows that there is a significant difference between the mean essay scores generated by the AES based on SVM-POL (mean=1.195, standard deviation=0.597) and the mean scores generated by the blind raters (mean=1.259, standard deviation=0.811). The results suggest that the blind raters tend to assign higher scores than the AES using SVM-POL. This is based on the t-statistics value of -3.188 and a corresponding p-value that is much less than 0.01(two tailed). The Spearman Rank Correlation test shows that there is a significant positive correlation between the SVM-POL scores and the blind raters. The rho-coefficient is 0.570 and the p value is much less than 0.01. Table 17 Normalized confusion matrix showing percent of agreements and disagreements The t-test for related data (ERT and blind scores) shows that there is a significant difference between the essay scores generated by the AES based extremely randomized trees (mean=1.195, standard deviation=0.554) and the scores generated by the blind raters (mean=1.259, standard deviation=0.811). The results suggest that the blind raters tend to assign higher scores than the AES using ERT. This is based on the t-statistics value of -3.407 and a corresponding p-value that is much less than 0.01(two tailed). The Spearman Rank Correlation test shows that there is a positive correlation between the ERT’s scores and the blind raters. The rho-coefficient is 0.613 and the p value is much less than 0.01. Table 18 Summary of measurements for agreements between the two best classifiers and blind raters Table 19 describes the relative agreements per score. When we break down the agreement figures for each score, we see a clear relationship between the number of training examples per score and the performance of the classifiers. The classifiers have an excellent performance when classifying essays with score G that correspond to the majority of the training data (classifiers have more than 80% agreement with blind raters). But the classifiers have an agreement with blind raters below 50% for all other scores. When comparing the agreement results with human raters, the classifiers do better for the essays with scores G and IG. The human raters have a much better agreement for the essays with score MVG. However, it should be noted that the human raters have a more uniform distribution of the scores but the level of agreement barely reaches 50% in any score. The results show that the classifiers are not able to handle the scores with few essays in the training data (“tail scores”). A requirement for the classifiers should be to perform better than 50% (chance) in any scores. This area must be further studied in future projects. Table 19 Summary of average accuracy for exact agreements per score The t-test for paired dependent data indicates that all AES systems are assigning scores with mean scores that are lower than the blind raters (on average). The differences in the mean scores are statistically significant (p-values much less than 0.01 two tailed). Table 21 describes the different functions used to create the different features. Stagger software is the basis for the creation of all the features because it produces the files in “.conll format” that contains the information about “word”, “lemma” and “POS” for each word. All essays and news corpus are available in “.conll”.
