Koenig, J.A. Examination of the Comparability of MCAT Writing Sample Test Forms. Paper presented at the Annual Meeting of the American Educational Research Association, 1995. San Francisco, CA

PURPOSE: This study describes procedures used in connection with constructing the MCAT Writing Sample test forms, which included pretesting, form assembly, range finding, and scoring. The procedures utilized were implemented to control form difficulty level and to achieve form comparability. Performance and reliability data from six MCAT administrations are presented to evaluate the effectiveness of these procedures.

METHODS: April 1985 marked the start of the Writing Sample pilot testing project. Originally designed to continue for two years, the pilot project was extended through September 1990 due to problems with obtaining acceptable reliability levels and developing comparable test forms. This paper provides a detailed account of the changes made during the pilot project to increase reliability levels of the test and to maintain form comparability. To investigate form comparability the author presents data showing: (1) means, standard deviations, and score distributions; (2) correlations between the Writing Sample and other test sections; and (3) reliability levels.

RESULTS: The author presents results demonstrating that the efforts during the pilot project served to improve the psychometric characteristics of the Writing Sample. Examination of descriptive statistics revealed similar means, standard deviations, and score distributions across administrations. Intercorrelations were similar ranging from .26 to .34 for the Writing Sample and Physical Science sections; .28 to .33 for Writing Sample and Biological Science sections; and .35 to .43 for the Writing Sample and Verbal Reasoning sections. Reliability estimates based on generalizability analyses ranged from .76 to .86.

CONCLUSIONS: The author concluded that the procedures used during the pilot project proved to increase comparability between Writing Sample forms. However, the author noted that to maintain acceptable levels of comparability, Writing Sample test data will continue to need to be monitored.

Contact Us    © 1995-2008 AAMC    Terms and Conditions    Privacy Statement