McLarty, J.R.; Mitchell, K.J. Effects of Prompt Context and Order on Creating Equivalent Essay Test Forms. Paper presented at the annual meeting of the American Educational Research Association, 1991. Chicago, IL

PURPOSE: The purpose of this study was to examine the extent to which pretest data on MCAT Writing Sample prompts were affected by the order in which prompts were administered and/or by the subject matter of the prompt with which each was paired.

METHODS: Nine prompts were selected from the pool of prompts being pretested for the 1991 MCAT. Seventy-two possible combinations of prompts were created by pairing each prompt with every other prompt and administering each pair in both orders. The different forms were administered to randomly equivalent groups of examinees taking the MCAT on a regularly scheduled test date. For each form, 25 examinee responses to each prompt in each context and order were collected for a total of 3,600 examinee responses. Analysis of variance (ANOVA) techniques were used to assess the effects of context, order, and their interactions.

RESULTS: Only one prompt was found to have a significant order effect, in that examinees consistently scored higher when this prompt was in the first position than when it was in the second position. Context effects were also found, but the authors concluded that they were generally very small.

CONCLUSIONS: The authors discussed the practical implicationsof these findings. They maintained that due to the small number of significant order effects, Writing Sample test forms could be assembled according to pretest statistics without regard to the order in which a prompt was pretested. Further, given the relatively small size of the context effects, Writing Sample forms need not be pretested intact.

Contact Us    © 1995-2008 AAMC    Terms and Conditions    Privacy Statement