Automated Evaluation

Nowadays, manually assessing students’ programming exercises has been identified as among of the toughest tasks to lecturers of programming courses on top of their high routine workloads.Thus, Automatic Programming Assessment (or APA) has turn out to be an alternative method to automatically assess these students’ programming exercises effectively.In APA, test data generation process has mainly being a part of performing dynamic testing on students’ programming solutions. Diverse of automated test data generation (or ATDG) methods have been researched in past few years.However, these methods have not been sufficiently and systematically adapted by researches in APA. Nonetheless, it appears very limited studies have put in some efforts to realize an integration of APA and ATDG as a complete APA system (or APAS) so as to provide a better quality and precise program testing on students’ programs.So as to realize this limitation, this paper presents the design and implementation of an A...

Introduction. Writing foreign-language creative writing assignments is one of the goals of foreign language
teaching in higher education. Modern AI tools (ChatGPT 4.0) are able to provide learners with evaluative feedback
and recommendations for essay revision. However, the feedback quality from ChatGPT 4 and other AI tools is a
subject of discussion in the teaching community. The aim of this paper is to compare the feedback quality provided
by ChatGPT 4.0 and teachers in evaluating students' essays.
Materials and methods. A bank of English essays (N=350) written by linguistics students (A2-B1 level) was used
as the material. The participants of the research were 12 teachers of English at Derzhavin Tambov State University
(Russian Federation). For every essay, one teacher and ChatGPT gave evaluative feedback on the following criteria:
1) content of the essay; 2) organisation and structure of the essay; 3) supporting ideas and arguments; 4) language
of the essay (lexical aspect of speech, grammatical aspect of speech, syntax); and 5) originality of the idea. The
recommendations received from the teacher and ChatGPT 4.0 were evaluated on the basis of norm-referenced
testing. The data analysis was carried out using the Student’s t-test.
Research results. It was found that ChatGPT matched the teacher in terms of the quality of evaluative feedback
for the following criteria: ‘content of the paper’ (t=0.24; p>0.05), ‘organisation and structure’ (t=1; p>0.05), and
‘supporting ideas and arguments’ (t=1.43; p>0.05). Moreover, ChatGPT outperformed the teacher (not a native
speaker) for the criteria ‘language of the essay’ (t=1.67; p≤0.05) and ‘originality of the essay’ (t=1.78; p≤0.05), which
is explained by the fact that the GPT language model was developed based on large English textual data. This
allowed the AI tool to be more accurate in assessing specifically the linguistic correctness of a written expression.
Conclusion. The novelty of the study is in the confirmation of the ability of the AI tool ChatGPT 4.0 to provide
qualitative feedback in assessing creative writing at the teacher's level or even better. The results of the study
support more intensive implementation of ChatGPT 4.0 in the process of teaching foreign language and assessing
the development level of students' writing skills.

Introduction. The development of students' foreign language creative writing skills is a component
of the goal of foreign language teaching in higher education. The effectiveness of the development of
students' writing skills is largely determined by the method and tools of teaching. Artificial intelligence
(AI) tools are able to provide learners with evaluative feedback that can be used to finalize written creative
works in a foreign language. The aim of the paper is to develop a method of teaching students to write
foreign language creative works on the basis of evaluative feedback from artificial intelligence, to test its
effectiveness in the course of experimental training.
Materials and methods. The study involved 1st year students (N=50) of Derzhavin Tambov State University
(Russian Federation), majoring in English as a Foreign Language. In order to test the effectiveness of the
authors’ method, stages of training were developed during which students received evaluative feedback
from the AI tool in order to further refine their papers. Students of the control group developed writing
skills using the traditional method. The objects of control were: a) content of the paper; b) organization
and structure; c) support of ideas and arguments; d) vocabulary; e) grammar. The data analysis was carried
out using the Student’s t-test.
Research results. It was found that the method of teaching students foreign language creative writing
on the basis of evaluative feedback from artificial intelligence is more effective in comparison with the
traditional method in terms of the following criteria: a) content of creative paper (t=2,75; p≤0,05), b)
organization and structure (t=3,05; p ≤ 0,05), and c) argumentation (t=2,44; p ≤ 0,05). At the same time,
statistics did not reveal any growth in the development of students' lexical (t=2,13; p>0,05) and grammar
(t=2,13; p>0,05) skills, which is explained by the individual nature of AI recommendations and the
objectively lack of sustainability in the lexical component of the teaching content when studying different
topics within the framework of one academic course.
Conclusion. The novelty of the study includes the development of stages of teaching students foreign
language creative writing based on evaluative feedback from artificial intelligence. The results obtained
can be used in the development of methods of teaching foreign languages with the use of AI-technologies.

Technology is defined as the use of scientific knowledge to solve practical problems. However, educators’ initiatives to integrate technology have been mostly prohibitively expensive. In this context, researchers proposed the automation of one of the most important processes but highly repetitive tasks among educators, the processing of student test results. The aim was to determine the alignment with evaluation standards and the acceptability of a cost-effective alternative. This study utilized a mixed-method approach, specifically concurrent triangulation. Quantitative and qualitative data were gathered concurrently, and then compared and combined the results to get a comprehensive understanding of the topic. Quantitatively, it involved the use of mean, standard deviation, t-test, and Cohen’s d to evaluate Alternative Optical Mark Recognition (AOMR) according to the required educators’ evaluation standards and its impact on reducing educators’ clerical workload. Qualitatively, semi-structured interviews and thematic analysis were employed to elucidate educators’ perspectives regarding the use of AOMR and the broader integration of technology as a whole. Results showed a one-hundred-thirty times efficiency compared to the manual process without losing the accuracy and reliability of data. Participants underscored the positive effect of AOMR in diminishing the labor-intensive nature of a crucial yet arduous clerical task for educators. Additionally, participants also emphasized unexpected benefits, including email results distribution, backup e-copies of sheets, ease of data management, class record integration, and automated student ranking. These findings offer valuable insights into the challenges surrounding the integration of technology in educational contexts in general, shedding light on the advantages of AOMR in the evaluation of student test results, in particular.

Technology is defined as the use of scientific knowledge to solve practical problems. However, educators’ initiatives to integrate technology have been mostly prohibitively expensive. In this context, researchers proposed the automation of one of the most important processes but highly repetitive tasks among educators, the processing of student test results. The aim was to determine the alignment with evaluation standards and the acceptability of a cost-effective alternative. This study utilized a mixed-method approach, specifically concurrent triangulation. Quantitative and qualitative data were gathered concurrently, and then compared and combined the results to get a comprehensive understanding of the topic. Quantitatively, it involved the use of mean, standard deviation, t-test, and Cohen’s d to evaluate Alternative Optical Mark Recognition (AOMR) according to the required educators’ evaluation standards and its impact on reducing educators’ clerical workload. Qualitatively, semi-structured interviews and thematic analysis were employed to elucidate educators’ perspectives regarding the use of AOMR and the broader integration of technology as a whole. Results showed a one-hundred-thirty times efficiency compared to the manual process without losing the accuracy and reliability of data. Participants underscored the positive effect of AOMR in diminishing the labor-intensive nature of a crucial yet arduous clerical task for educators. Additionally, participants also emphasized unexpected benefits, including email results distribution, backup e-copies of sheets, ease of data management, class record integration, and automated student ranking. These findings offer valuable insights into the challenges surrounding the integration of technology in educational contexts in general, shedding light on the advantages of AOMR in the evaluation of student test results, in particular.

Technology is defined as the use of scientific knowledge to solve practical problems. However, educators’ initiatives to integrate technology  have  been  mostly  prohibitively  expensive.  In  this context,  researchers  proposed  the  automation  of  one  of  the  most important processes but highly repetitive tasks among educators, the processing  of  student  test  results.  The  aim  was  to  determine  the alignment with evaluation standards and the acceptability of a cost-effective alternative. This study utilized a mixed-method approach, specifically  concurrent  triangulation.  Quantitative  and  qualitative data were gathered concurrently, and then compared and combined the  results  to  get  a  comprehensive  understanding  of  the  topic. Quantitatively, it involved the use of mean, standard deviation, t-test, and  Cohen’s  d  to  evaluate Alternative  Optical  MarkRecognition (AOMR) according to the required educators’ evaluation standards and    its    impact    on    reducing    educators’clerical    workload. Qualitatively, semi-structured interviews and thematic analysis were employed to elucidate educators’perspectives regarding the use of AOMR and the broader integration of technology as a whole. Results showed  a  one-hundred-thirty  times  efficiency  compared  to  the manual process without losing the accuracy and reliability of data. Participants underscored the positive effect of AOMR in diminishing the labor-intensive nature of a crucial  yet arduous clerical task for educators.  Additionally,  participants  also  emphasized  unexpected benefits,including  email  results  distribution,  backup  e-copies  of sheets,  ease  of  data  management,  class  record  integration,  and automated  student  ranking.  These  findings  offer  valuable  insights into  the  challenges  surrounding  the  integration  of  technology  in educational contexts in general, shedding light on the advantages of AOMR in the evaluation of student test results, in particular.
Keywords: alternative optical mark recognition; Evalbee; integrating automation; student test evaluation; technology in education

Computer programming captivates the attention of both professionals and young learners due to its multidisciplinary applications and high paid jobs. However, mastering computer programming requires critical thinking and consistent practice under the guidance of expert teachers. Consistently assessing the programming laboratories and giving detailed feedback are curial for novice learners. However, manual grading is daunting and timeconsuming. Moreover, novice programmers often do not identify all the test cases and sufficient test data to test their programs thoroughly. This research introduces a sophisticated web-based system to aid instructors in grading programming assignments. Instructors can create questions and corresponding solutions, shared with students, who then write their programs. The system generates test cases for both student and correct solutions, assessing performance through comparison. A survey with 30 teachers affirmed the system's effectiveness, rating it 4.00 for simplifying assessment and 4.13 for user-friendliness on a 5point Likert scale. Statistical analysis, yielding a p-value of 0.034, supported its positive impact. This web-based grading system not only streamlines assessment but also enhances the educational experience, fostering a user-friendly environment for both instructors and students.

This paper proposes a solution to evaluate summary of a scientific article through diagram analysis. The model diagram used for evaluation is constructed solely base on the reading text, and does not require extra input from human experts. By fully automate this evaluation process, it helps bring a tight feedback loop that will help students to have a positive reinforcement of the knowledge of the article.

Within this study, the authors want to address the problem of overworking of teachers in Philippine schools due to their excessive clerical responsibility, which could lead to teacher attrition. The authors propose to automate the process, particularly the evaluation of student test results since it could improve human well-being by reducing the burden of manual labor. Automation using OMR has not been widely applied in Philippine schools due to cost issues. The authors want to observe whether an alternative OMR - EvalBee - can meet the evaluation standard even though it is free of charge. The study employed a quantitative-evaluative research method. Validated questionnaires on paper and a Google Form survey were used to collect relevant data in a secondary public school in the City Schools Division of Cabuyao – Southville1 Integrated National High School. Statistical tools such as the mean, standard deviation, t-test, and Cohen's d were used in this research to determine the ef...

Being the most used method for dissemination of information, especially for public services, it is of paramount importance that the Web is made accessible as to allow all its users to access the content of its pages. In this paper, we evaluated 2250 Governmental Web pages from each one of three different African countries (i.e., Angola, Mozambique and South Africa). This report compares the accessibility quality and the level of structural complexity of these African countries government's Web pages. We found that hand coded pages tend to have larger number of HTML elements and also to present higher number of accessibility problems. Finally, it suggests some recommendations to repair the most common problems in these pages.

Within this study, the authors want to address the problem of overworking of teachers in Philippine schools due to their excessive clerical responsibility, which could lead to teacher attrition. The authors propose to automate the process, particularly the evaluation of student test results since it could improve human well-being by reducing the burden of manual labor. Automation using OMR has not been widely applied in Philippine schools due to cost issues. The authors want to observe whether an alternative OMR - EvalBee - can meet the evaluation standard even though it is free of charge. The study employed a quantitative-evaluative research method. Validated questionnaires on paper and a Google Form survey were used to collect relevant data in a secondary public school in the City Schools Division of Cabuyao – Southville1 Integrated National High School. Statistical tools such as the mean, standard deviation, t-test, and Cohen's d were used in this research to determine the ef...

This paper proposes a new idea for grading multiple-choice test which is to develop a method to use a personal computer plus a scanner and a program based application, that will grade a specially designed MCQ exam test and feedback analysis paper. The program will test to detect the correct answers by comparing each paper with a correct answer which will be pre stored in database,. Also students can give the feedback about the lecturer who conducts lectures during the semester. It will thereby help in analysis of the staff record on grade entered by student in the feedback form in schools, colleges ,and other institutions .This project use C# an object-oriented programming language from Microsoft that aims to combine the computing power of C++ and contains features similar to those of Java, a software application that will have simple interface that would be understandable to all with minimal cost and can be used in various education institutions for conducting multiple choice quest...

Within this study, the authors want to address the problem of overworking of teachers in Philippine schools due to their excessive clerical responsibility, which could lead to teacher attrition. The authors propose to automate the process, particularly the evaluation of student test results since it could improve human well-being by reducing the burden of manual labor. Automation using OMR has not been widely applied in Philippine schools due to cost issues. The authors want to observe whether an alternative OMR - EvalBee - can meet the evaluation standard even though it is free of charge. The study employed a quantitative-evaluative research method. Validated questionnaires on paper and a Google Form survey were used to collect relevant data in a secondary public school in the City Schools Division of Cabuyao – Southville1 Integrated National High School. Statistical tools such as the mean, standard deviation, t-test, and Cohen's d were used in this research to determine the ef...

Within this study, the authors want to address the problem of overworking of teachers in Philippine schools due to their excessive clerical responsibility, which could lead to teacher attrition. The authors propose to automate the process, particularly the evaluation of student test results since it could improve human well-being by reducing the burden of manual labor. Automation using OMR has not been widely applied in Philippine schools due to cost issues. The authors want to observe whether an alternative OMR - EvalBee - can meet the evaluation standard even though it is free of charge. The study employed a quantitative-evaluative research method. Validated questionnaires on paper and a Google Form survey were used to collect relevant data in a secondary public school in the City Schools Division of Cabuyao – Southville1 Integrated National High School. Statistical tools such as the mean, standard deviation, t-test, and Cohen's d were used in this research to determine the ef...

A preliminary version of a platform for automated, remote, insitu user experience measurement called TUMCAT was evaluated. The use of peer-to-peer software was monitored with it during five weeks and subjective data were gathered with the through experience sampling. Findings from two alternative ways of evaluating the software were compared to findings yielded by TUMCAT. The results of the comparison clearly show how findings from the different evaluations complement each other. Suggestions are formulated on how these findings can be used to improve TUMCAT.

Within this study, the authors want to address the problem of overworking of teachers in Philippine schools due to their excessive clerical responsibility, which could lead to teacher attrition. The authors propose to automate the process, particularly the evaluation of student test results since it could improve human wellbeing by reducing the burden of manual labor. Automation using OMR has not been widely applied in Philippine schools due to cost issues. The authors want to observe whether an alternative OMR-EvalBee-can meet the evaluation standard even though it is free of charge. The study employed a quantitative-evaluative research method. Validated questionnaires on paper and a Google Form survey were used to collect relevant data in a secondary public school in the City Schools Division of Cabuyao-Southville1 Integrated National High School. Statistical tools such as the mean, standard deviation, t-test, and Cohen's d were used in this research to determine the effectiveness of using alternative OMR in evaluating examinations. This study shows that alternative OMR is efficient, accurate, and reliable, despite the fact that it is free, and recommends it be used as part of the teachers' standard method of test evaluation.

A preliminary version of a platform for automated, remote, insitu user experience measurement called TUMCAT was evaluated. The use of peer-to-peer software was monitored with it during five weeks and subjective data were gathered with the through experience sampling. Findings from two alternative ways of evaluating the software were compared to findings yielded by TUMCAT. The results of the comparison clearly show how findings from the different evaluations complement each other. Suggestions are formulated on how these findings can be used to improve TUMCAT.

El programa EssA fue creado en 2004 como un instrumento para ayudar a valorar narraciones en el área de lengua extranjera. El análisis de regresión múltiple, según variables léxico-gramaticales, determinó una ecuación de regresión que predice la calificación de los textos con un coeficiente de regresión del 0.79. EssA ha demostrado realizar la valoración gramatical de las narraciones con mayor objetividad y fiabilidad que los docentes. Por otra parte, proporciona feedback inmediato al estudiante sobre el uso que hace de los distintos recursos lingüísticos en el contexto de la narración, al mismo tiempo que le muestra la valoración del mismo.
EssA software was created in 2004 as an instrument to help EFL teachers to assess narrative stories. Multiple regression analysis, using lexical-grammar variables, determined a regression equation to predict the score of each text with a regression coefficient of 0.79. The final result we found was that the EssA's analysis of the stories has proven to be more reliable and more objective compare to the scores given by the teachers. It also provides immediate feedback on those linguistics resources of the academic curriculum the student uses in their story, as well as the score.