The Use of Person Fit to Model Panelist Fit

1995

This study examines the effects on the National Assessment of Educational Progress (NAEP) achievement levels of using item response theory (IRT) models that have nominal missing-response parameters. It compares cutpoints based on item parameters that were fitted using two different models. The first set of cutpoints were based on parameters for the two-and three-parameter logistic model, and the second set of cutpoints were based on R. Bock's (1972) nominal model. Data are from the 1992 NAEP in mathematics and reading for grade 12. For reading, data included 1,966 responses to a block of items, and for mathematics, data included 2,192 responses. Other data were the item-by-item ratings by each panelist who participated in the Achievement Levels Setting process for NAEP 1992. For each subject, the cutpoints set using the probability curves obtained by the different models were compared. The percent of students scoring at each level were also compared. For reading, the logistic model, when fitted to the data, converged in 25 iterations and yielded a marginal reliability of 0.67 with maximum information of 4.3 at theta equals-0.5. The nominal model converged in 88 iterations and yielded a marginal reliability of 0.85 with a maximum information of 14.2 at theta equals 0. For mathematics, the logistic model converged in 184 iterations and yielded a marginal reliability of 0.61 with maximum information of 23.9 at theta equals 1.5. The nominal model, with mathematics data, converged in 46 iterations, and yielded a marginal reliability of 0.62, with maximum information of 7.3 at theta equals-0.2. When the percentages of students scoring at or above each cutpoint were compared, none scored at the Advanced level using the nominal model. These preliminary results suggest the direction of future studies, but cannot be generalized to the NAEP assessment program. An appendix contains achievement levels descriptions. (Contains three tables, six figures, and five references.) (SLD)

Analyzing Data from a National Assessment of Educational Achievement deals with how to generate information on test items and test scores and how to relate the test scores to educational and social factors. Like other volumes in this series, the volume is intended primarily for teams in developing and emerging economies with responsibility for conducting national assessments.

The analysis of test-generated data or survey responses often utilizes means, total scores, or latent scores from factor analysis as outcome variables in a regression-based procedure

ETS Research Report Series, 2011

Research Association, American Psychological Association, & National Council for Measurement in Education, 1999) demands evidence of model fit when an item response theory (IRT) model is used to make inferences from a data set. We applied two recently suggested methods for assessing goodness of fit of IRT models-generalized residual analysis (Haberman, 2009) and residual analysis for assessing item fit )-to several operational data sets.

Educational Measurement: Issues and Practice, 2005

Many educational and psychological tests are inherently multidimensional, meaning these tests measure two or more dimensions or constructs. The purpose of this module is to illustrate how test practitioners and researchers can apply multidimensional item response theory (MIRT) to understand better what their tests are measuring, how accurately the different composites of ability are being assessed, and how this information can be cycled back into the test development process. Procedures for conducting MIRT analyses-from obtaining evidence that the test is multidimensional, to modeling the test as multidimensional, to illustrating the properties of multidimensional items graphicallyare described from both a theoretical and a substantive basis. This module also illustrates these procedures using data from a ninth-grade mathematics achievement test. I t concludes with a discussion of future directions in MIRT research.

Large-scale Assessments in Education, 2017

Background Recent publications have re-ignited interest in the approach to modeling used to generate achievement measures for large scale assessments such as NAEP and PISA. Even though the foundations and statistical methodology behind these models have been extensively covered for over 3 decades (Mislevy 1984, 1985; Mislevy and Sheehan 1987), there continue to be concerns about their ability to provide appropriate estimates of

Journal of Integrated Elementary Education

This study determined the mean ability estimates of students in public and private secondary schools when their scores are equated through linear equating, and it examined which of the equating methods is more efficient. This study adopted a descriptive research design. The population for the study comprised 24,874 candidates that registered and sat for the 2016 June/July National Examinations Council (NECO) mathematics examination in Ogun State. A sample of 1139 candidates were selected from both public and private schools using the multi-stage sampling procedure. The research instruments used for the study were secondary data sources. The data was analyzed using Multidimensional Item Response Theory (MIRT), Separate and Linear Equation. The results showed that private school candidates’ ability score (x̅ = -0.001, SD = 0.961), (x̅= -0.001, SD = 0.961) was higher than public school candidates’ ability score (x̅= -0.865, SD = 1.058), (x̅= -0.626, SD = 0.970) when equated using separ...

1970

The relationships between teacher ratings and standardized achievement tests as measures of student progress are investigated. Initial teacher ratings of pupils in grade 3 and subsequently in grade 6, were contrasted to scores obtained on the Metropolitan Achievement Test (MAT) . It was found that a teacher's rating does not provide the same information about the pupil's level of achievement as do the MAT scores. Although a relationship between the two measures was found, the degree of relationship dependefl upon what areas the teachers had ttted. An examination of the data revealed a consistent pattern in the relationship between various subsections of the Teacher Rating Questionnaire and the MAT, the discrepancy scores, and the I.Q. scores. Teacher ratings of performance were much more closely related to these measures than were their ratings of pupil adjustment and creativity. Despite differences among teachers and among classes, as well as the development in the pupils' characteristics flout grade 3 to grade 6, there was a reasonable level of agreement among teachers' ratings of student performance. The data supports the conclusion that teachers' ratings of academic achievement are an important and dependable adjunct to the use of standardized achievement tests. The development of the instrument is described briefly and copies of two versions are appended.

2008

2013

The question of how to handle missing responses in psychological and educational measurement has been repeatedly and controversially debated for decades. Even in highly respected international studies and large scale assessments, such as the PISA (Program for International Student Assessment), TIMSS (Third International Mathematics and Science Study), and PIRLS (Progress in International Reading Literacy Study) a generally accepted methodology for missing data is still lacking. Since the late 1990s multidimensional item response theory (MIRT) models for item nonresponses have been developed. These models become quickly complex in application and rest upon assumptions that are usually not critically addressed. Although statistical theory of missing data suggests adequate handling of missing responses to avoid inefficient and biased parameter estimation, there is empirical evidence that IRT-based parameter estimation is fairly robust against missing responses. That may question the ne...