AAMC Reporter: September 2005

AAMC Examines Low Response Rate for 2005 GQ

A lower-than-anticipated response rate to the 2005 Graduation Questionnaire (GQ) has prompted the AAMC to examine why participation has dropped and how to improve involvement in next year's survey. Strategies to increase responses will be presented to the Council of Deans later this month.

Based on data from this year's GQ, which was available online from February to July, 20 percent of the 125 member schools had a student-response rate greater than 90 percent. An additional 35 percent of schools reported that between 60 percent and 90 percent of their students responded to the survey. Less than 60 percent of students participated at the remaining schools.

Starting in July, staff from the AAMC's Division of Medical Education (DME) contacted a representative sample of the schools and conducted a telephone evaluation of the institutions' experience with the GQ. The dean of education, the dean of students, or their designee provided, among other data, information about how the AAMC's GQ promotional materials were used in addition to successful dissemination tactics employed by the school, and whether problems resulted from the school conducting its own institutional review board (IRB) review along with the IRB review done for AAMC. School officials were also asked for their recommendations on how to improve the survey process.

"The deans see the Graduation Questionnaire as a very important tool for program evaluation," said M. Brownell Anderson, DME senior associate vice president. "They recognize its importance and have been helpful in working with us on the recent challenges in implementation and to increase participation."

Initial changes to the GQ began in late 2003, when a complaint from Public Citizen, a national non-profit public interest organization, was filed on behalf of a graduating medical student who felt that answering survey questions about financial indebtedness and career choice could put a student's potential livelihood at risk. Because the AAMC analyzes the data and occasionally publishes research papers based on the information collected, the American Institutes for Research Institutional Review Board (IRB) advised that participating medical students could be considered human subjects and must be protected.

Last year, schools could not require students to complete the GQ before they graduated, so many institutions implemented methods to boost involvement and data collection. Some schools advertised the GQ with posters and postcards from the AAMC, and others sent repeated reminders to students through e-mail, letters, or phone calls.

Some students, however, still resisted answering questions about their medical school experience. To help address this issue, the AAMC made changes to the survey's structure for 2005. The GQ was split into two surveys—a Program Evaluation Survey and a Supplemental Survey on Financial Aid, Career Intentions, and Diversity.

The IRB agreed that the survey should be presented in two parts. The Program Evaluation section contains questions about medical school experiences, student support programs, and includes a subset of questions on potential mistreatment. Responses are anonymous, and schools can require students to complete this section.

The Supplemental Survey asks for information about levels of educational and non-educational debt, specialty choice, and the cultural environment of medical school. This section, which includes some personally identifiable information, is optional. If students choose to complete the survey, they must provide explicit consent before survey data can be released to their school.

However, making the Supplemental Survey optional presents a problem for the AAMC's data collection efforts. The low response rate limits the amount of information the AAMC can provide to constituents and federal agencies about the level of student indebtedness and the continuation and enhancement of loan or loan deferment programs, Anderson said.

She added that the AAMC must be able to match some information with individual students in order to track the way in which students' finances and career choices change as they progress through medical school and longitudinally across years.

Separating the GQ into two parts has also been problematic, according to Henry Pohl, M.D., vice dean of academic administration at Albany Medical College, who believes students are more concerned about the time needed to complete two questionnaires than they are about providing personal information.

"I've had to bribe my students to complete the first section with lotteries for iPods or movie tickets," Pohl said. "It's a time issue more than anything. Two surveys means two logins. If you require students to take the survey, they'll do it, but if it's voluntary, it's just a pain for them."

Despite the difficulties of getting students to take the GQ, Pohl said Albany did have a high response rate and he will make every effort to continue strong student participation. Not only is the GQ the most effective method of compiling and comparing national data about student services and quality improvement, he said, but the information it provides is also used by the Liaison Committee on Medical Education (LCME) during accreditation site visits.

"GQ information is key to LCME site visits," said Pohl, who serves on the LCME. "Without high student participation, the information is tainted."

Ken Simons, M.D., senior associate dean for academic affairs at the Medical College of Wisconsin, said that even though the student response rate was high at his institution, he is unsure about the data's accuracy. This year, students who did not complete the survey with their first log-in were able to log in a second time and allowed to complete the questionnaire.

"I think the GQ went over really well, but I'm concerned that I don't know for sure," Simons said. "If I look at the numbers, I have more responses than actual students that are in the fourth year. That problem should be fixed."

"Changing the GQ to a two-part survey has presented some unanticipated challenges, and Dr. Simons is one of the many medical school administrators who have been incredibly helpful as we have worked to correct the problems that have accompanied a new approach," Anderson said. "The results of the telephone survey and the input from people like Dr. Simons and Dr. Pohl make me hopeful that the response rate and the value of the responses will be higher in 2006."

—Whitney L.J. Howell, whowell@aamc.org

Contact Us    © 1995-2008 AAMC    Terms and Conditions    Privacy Statement