Inspired by four recent decisions to change achievement tests used in the Austin Independent Scho... more Inspired by four recent decisions to change achievement tests used in the Austin Independent School District, the separate forms used and in_ followed have been combined into a systematic approach intern ed for use in future achievement test selections. A rating scale (Attachment 1) was developed to expedite a systematic comparison among possible achievement tests, and to allow a weighting of the factors to be rated according to the school system's needs. Five groups of experts (parents, teachers-and principals, testing staff, central administration, and the board of trustees) have varying responsibility for rating the fives factors critical to making the best choice: technical soundness; logical feasibility; instructional validity; financial affordability; and interpretational ease. The Faeal Flaw Principle (occurring-when an essential factor is rated unacceptable) can eliminate a testoutright, and the Shoo-In Principle (occurring when a clearly superior rating is given on a critical factor) will'select a single test outright..An outline of eight procedural steps for the selection process and the contexts in which they are appropriate is attached. (BS) 4
°Six anomalies in achievement test scores encountered by the'Aditin Independent'School District a... more °Six anomalies in achievement test scores encountered by the'Aditin Independent'School District are described. These include crossing gaps with uninterpolated medians; total group median '4., declines while all subgroups:. medians rise; outlying total percentiles; percentile andgrade equivalent growth antithesis; same wjade equivalent earning a.different percentile, in each content area; and the median,, does not represent any group. Evaluators and . 4reseirchers must.know how t distinghish real achievement/gains from tartifactual: gains result4hg from anomalies such as those discussed in this paper. It is necessary to determine when an inconsistency is an error and when it 'Oar) explainable anomaly. When -interpreting achievement test sco4es, ihteragtion of types of scores such as percentiles and grade equPoglentsp shifts in student demdgraphics, and non-noimal distributions within groups being tested need to be carefully considered: The factors causing the anomalies and possible I solutions ire discussed. (DWH) 4.==,
A comprehensive guide for observers using the Pupil Activities Record--Revised (PAR-R), this manu... more A comprehensive guide for observers using the Pupil Activities Record--Revised (PAR-R), this manual covers procedures followed in systematic observation of a student's classroom . activities for the period of cve instructional day. The manual provides general information atout the purpose and design of the PAR and explains the revisions, which were made in order to collect additional data including the amount of lancivage arts instruction received, mode of instruction, and characteristics cf tlw, teacher's questioning strategy. Guidelines are includedfor recording each of the twelve variables observed: place cf instruction, activities during no instruction, basic skills instruction, instruction other than basic skills, adult contact and group size during basic skills instruction, time spend on-and off-task,,teachers' use of verbal and nonverbal signal systems, language of instructicn, Rode and content of instruction, and types of instructional questions used by the teacher. Appendices contain the PAR-R ob6ervaticn fcrs, interrater reliability correlations, and a sample computer printout giving time spene on classroom activities by students observed in Title I, Non-Title I, Title I Migrant, and LESA (Limited English Speaking Abiiity) programs. (JH)
1 So that comparisons of achievement test scores across campuses would be unaffected by variation... more 1 So that comparisons of achievement test scores across campuses would be unaffected by variations in test preparation activities, the Austin Independent School District has set out to standardize all testwiseness instructions and practice testing across the school system. Appropriate preparation activity was defined as that which contributes to students' performance on the test near their true achievement levels, and which contributes more to their scores than would an equal amount of regular classroom instruction. The program concentrated on basic testwiseness skills, as advanced tactics often required more skill than the skill being measured by the test item. Research showed that test practice was beneficial if done once, near the time of the actual test. Skills taught to all students included: following directions, understanding item formats, understanding a test's terminology, symbols and procedures, using time wisely, checking answers, marking answers properly, knowing how to ask questions about the test, being physically ready to take the test, and thinking logically. The attached materials for preparing students (and teachers) for standardized tests include: Guidelines a for Test Administrators, Packets for the Preparation of Students for Standardized Testing, All-Purpose Answer Sheet, and Explanation and Examples of Testwiseness Cues. (PN)
Careful management of a testing program can greatly a teacher's opportunity to cheat while admini... more Careful management of a testing program can greatly a teacher's opportunity to cheat while administering s..andardized tests to students and, can increase the likelihood that such cheating will be detected. The Austin Independent School District's systemwide testing staff's plan for controlling cheating has three basic premises: (1) plan and organize for security by reducing the opportunities available for cheating; (2) look everywhere for clue:. and discrepancies; and (3) follow up swiftly on violations to let the school personnel know how seriously cheating is viewed. The following are included as attachments: the "Profile of a Teacher Who Cheats on Standardized Tests"--a composite that in some ways contradicts usual pe:ceptions; six brief case studies of cheating in the Austin Independent School District; and "Guidelines for Test Administrator." (PN)
Linking Outcomes to Organizational Planning (LOOP) was initiated during the 1984-85 school year i... more Linking Outcomes to Organizational Planning (LOOP) was initiated during the 1984-85 school year in the Austin (Texas) Independent School District. LOOP was designed to ensure that evaluation, research, and informal findings became part of the instructional planning loop; to provide information to the Superintendent on progress toward priorities and to focus staff on the goals; and to provide a reporting format. Conditions for success included a clear mandate from the Superintendent, and designation of a key person to follow through and provide reports. LOOP procedures were integrated into established procedures. Findings for action were generated by diverse sources, including evaluation reports, construction progress reports, and committees. LOOP has ten steps: (1) identify relevant evaluation finding; (2) establish goal; (3) assign responsibility for meeting the goal; (4) identify means of achieving the goal; (5) update activity status and action required, using the data processing system; (6) have continual monitoring of progress by Office of Research and Evaluation (ORE); (7) ORE to make monthly and annual reports; (8) Superintendent and Cabinet to provide feedback regarding progress toward each goal; (9) Superintendent and Cabinet to declare that the goal has been met; and (10) Superintendent to report to Board and Public. Although a great deal of effort was involved, LOOP has been effective. About two-thirds of the document consists of an appendix containing monthly and other reports on LOOP activities for selected priorities especially those involving achievment gains and computers. (GDC)
The impact on some student behavior of the no pass/no play rule enacted in Texas was studied for ... more The impact on some student behavior of the no pass/no play rule enacted in Texas was studied for the variables of: (1) grades; (2) enrollment in honors courses' and (3) dropping out. Under the rule, enacted in 1985, a student must pass every course (with an average of at least 70) cr sit out extracurricular activities in the next 6-week grading period. Focus is on the rule's impact on high school students in the Austin (Texas) Independent School District. Selected student data for the school years between 1982 and 1988 are provided. Students did fail fewer courses under the new rule, particularly in the fall semester. The decline in failure was greater for those participating in entracurriculars. Overall, the dropout rate did not increase, although there was a possible increase for those participating in varsity sports. Enrollments in honors courses did not decline under the new rule. Students agreed that the rule encouraged them to make better grades. On balance, the new rule appears to have had a positive effect. Since many other changes were implemented at the same time, it is difficult to be sure that these changes are attributable to no pass/no play, but there are no evident negative effects in the areas of course enrollments and overall dropout rates. Attachments include a list of courses involving extracurricular activities and a table showing percentages of failures. Six statistical figures are included. (SLD)
This paper describes the development of the Basic Educational Skills Test (BEST) in the Austin In... more This paper describes the development of the Basic Educational Skills Test (BEST) in the Austin Independent School District. In 1982 future options for the AISD minimum competency program, which requires all students to meet minimum proficiency levels in reading and mathematics before graduation, were evaluated against legal and local criteria. It was decided to develop an item bank that would allow the generation of multiple, parallel forms of a competency test to be administered: to students in tutorial courses; to senior transfers; and in special testing sessions to students meeting the course requirements, but not yet meeting the minimum competency requirement, for graduation. Important issues during the BEST development process fell into six general categories: (1) identification of objectives; (2) item acquisition; (3) item calibration; (4) test formatting; (5) test production; and (6) item and test booklet security. (BS)
Issues related to achieving adequate national norms are reviewed, and a new methodology is propos... more Issues related to achieving adequate national norms are reviewed, and a new methodology is proposed that would work to provide a true measure of nPtional achievement levels on an annual basis and would enable reporting results in current-year norms. Statistical methodology and technology could combine to create a national norming process that would publish an annual national norm soon after test users complete their regular annual testing programs. This approach would be supplemented by a small national normative sample and adjusted by a factor to reduce the influence of users. This approach would also include almost immediate turnaround of the current-year norms to allow schools to report their annual scores using those norms. Schools already giving a test would transmit their data electronically to a central location. Schools in the normative sample would test and transmit their data. A current year norm table could then be produced. This type of norming procedure would probably require a national center for test norming, as well as cooperation among test publishers. Ccoperation from school districts would be secured by the .LIct that they could have national norms within weeks of transmitting their raw data. It should ultimately be possible to establish a national educational achievement indicator to rival Scholastic Aptitude Test scores--an indicator with the simplicity of the Dow zones average or the Consumer Price Index. Nine figures illustrate the discussion. (SLD)
Upon learning that a form of the Sequential Tests of Educational Progress was incorrectly distrib... more Upon learning that a form of the Sequential Tests of Educational Progress was incorrectly distributed to an unidentified number of high school students along with an answer sheet pregridded with an alternate test form, the Austin Independent School District performed the following research analyses: (1) scored the tests using the key for each form, calculating a daily total raw score; (2) Rasch-calibrated the tests using the key for Form B, the form the students were supposed to have taken; (3) obtained an average person-fit stntistic for each day; (4) created a file which included, for both days of testing, the average person-fit score and the total raw score for both Forms A and B, and also the person-fit scores for each subset; (5) sorted the file by Day 1 average person-fit score and print; (6) sorted the file by Day 2 average person-fit score and print; and (7) examined the output. Six samples illustrate the case types found. Apparently 63 of 65 booklets were used with 78 students taking Form A at least one day. However, there is no empirical evidence to suggest what kind of error rate there is with this procedure. (PN)-
For Texas's Career Ladder System of rewarding good teachers, teachers' performance evaluations fr... more For Texas's Career Ladder System of rewarding good teachers, teachers' performance evaluations from 1981 to 1984 were used to rank teachers in the Austin Independent School District. Significant biases were noted between raters, between years, and between elementary and secondary teacher ratings. To adjust for these biases, each teacher's raw score average was converted to a z-score, and adjusted for the factors associated with bias. The final z-score was used to determine selection for the Career Ladder; quota systems were not used. A small correlation was noted between teachers' z-scores and elementary school students' reading and mathematics achievement on the Iowa Tests of Basic Skills. Teachers' raw scores were less able to predict student achievement. However, z-scores were not well received. This technique was eventually abandoned due to: (1) dissatisfaction and distrust; (2) changes in legislation and confused communication; (3) the non-selection of some key teachers; (4) differing goals of the teachers' associations; and (5) the provision of additional funding for a greater number of career ladder teachers, which made z-scores unnecessary. (GDC)
Based on the experiences of four equating studies conducted by the Austin (Texas) Independent-Sch... more Based on the experiences of four equating studies conducted by the Austin (Texas) Independent-School District, a practical "cookbook" approach to test equating is presented. Th types of equating procedures are discussed: choosing a cutoff on a new instrument, predicting Y from X, and symmetric equati and F. (MW) 4 . .
In certain situations, grade equivalent scores are the most appropriate statistic available for r... more In certain situations, grade equivalent scores are the most appropriate statistic available for reporting achievement test data. It is noted that testing practitioners have found that raw scores, normal curve equivalents, stanines, and standard scores are very useful. However, it is best to convert to either grade equivalents or percentiles before communicating them to lay audiences. In the Austin, Texas Independent School District, both grade equivalents and percentiles are routinely reported to high school students' parents. Elementary school parents receive percentile scores routinely, but may request grade equivalents. Both percentiles and grade equivalents are often misinterpreted; the shortcomings of percentiles are merely less well known. Seven critical questions that can be used to decide whether achievement test results should be reported in grade equivalent or percentile scores are presented and briefly discussed. These questions can be located along two dimensions: comparison standard and time. The five standards of comparison are: (1) grade level; (2) peers; (3) all students, all grades; (4) self; and (5) students at the same achievement level. The two criteria involving time periods are: achievement status to date; and pretest posttest gain. Appendices include a 1977 publication describing myths in interpreting grade equivalent scores and an information kit containing training materials on grade equivalents, percentiles, and norming a standardized test. (GDC)
Uploads
Papers by Glynn Ligon