AAMC Reporter: October 2006

Some Doctors Fall Short in Assessing Own Educational Needs

Many physicians have difficulty assessing their own strengths and weaknesses — a finding that could have wide implications for continuing medical education (CME) as well as other phases of medical education, according to a new study.

"Accuracy of Physician Self-assessment Compared With Observed Measures of Competence: A Systematic Review" appeared recently in the Journal of the American Medical Association and found a significant number of physicians are not as adept at grading their own performances as they should be and recommended a series of concrete steps toward improvement.

The review's primary author, David A. "Dave" Davis, M.D., professor of health policy, management and evaluation, and professor of family and community medicine at the University of Toronto Faculty of Medicine, conducted much of the research between late 2005 and mid-2006 while studying CME as a Robert G. Petersdorf Scholar-in-Residence at the AAMC. Davis also leads the knowledge translation program of the Li Ka Shing Knowledge Institute at St. Michael's Hospital and the University of Toronto.

According to Davis, this is the only study of its kind.

To conduct the study, Davis and team members examined 17 academic articles on the subject of physician external and self-assessment. Of the 20 comparisons between self- and external assessment, 13 demonstrated little, no, or an inverse relationship between physicians' opinions of themselves and others' opinions of their performance, and seven demonstrated positive associations. According to the review, several studies found the worst self-assessment accuracy among physicians who were the least skilled and those who were the most confident. These results are consistent with those found in other professions, according to the study.

"About a third of the doctors in these studies could judge their own performances pretty well," Davis said. "But the remainder failed to assess themselves wholly."

Davis and Carol A. Aschenbrener, M.D., senior vice president of the AAMC's Division of Medical Education, agreed that self-assessment is vital to all areas of medical education.

"We want to be moving to a system where physicians are encouraged to assess their own needs and given the tools to address them," Aschenbrener said. "And if we're really going to work to advance medical education, we have to do it along the entire continuum, and emphasize this in undergraduate medical education, graduate medical education, and continuing medical education."

However, Davis said the findings could have ramifications predominantly for CME — specifically, the manner in which physicians select which CME courses they wish to take.

"Virtually all of the mandatory hours of CME are based on doctors' own assessments of themselves. And there seem to be two lists for a physician's CME requirements — what they think they need, and what they really need," Davis said. "We need to provide doctors with more feedback and more mechanisms by which they can assess themselves."

The study recommended several major changes for the CME system, including increasing the objectivity and number of sources involved in physician feedback. Davis added that, overall, the process of selecting CME topics should include more external feedback.

"I'm a family physician, and currently I can determine all my credits on my own recognizance," Davis said. "And that should change."

Overall, the study suggested that a general increase in physician training at every stage of education and career "may reduce the variation between self- and external assessments by encouraging the internalization of objective measurements or benchmarks of performance." As an example, it noted the results of a hospital-wide initiative to improve the quality of handwashing that narrowed the differences between observed and self-reported handwashing techniques.

"Similarly," the study continued, "physician trainees may be able to self-rate more accurately when they compare their ratings with those of others. Attention to this effect of training and the comparative feedback phenomenon in undergraduate medical education as well as graduate and CME appears both appropriate and timely."

At the GME level, the study recommended adhering closely to the general competencies formulated by the Accreditation Council for Graduate Medical Education, including a more holistic continuing professional development process involving learning portfolios, documenting practice-based learning and improvement activities, and creating less general and more detailed learning and practice objectives.

Perhaps most importantly, however, Davis called for more investigation in the area.

"We need to understand this phenomenon more," he said. "Is it personality-driven? Is it cognitive? There's a big research agenda here."

The other study authors were Paul E. Mazmanian, Ph.D., associate dean of CME at Virginia Commonwealth University School of Medicine; C. M. Fordis, M.D., senior associate dean and director of CME at Baylor College of Medicine; R. Van Harrison, Ph.D., director of continuing medical education at the University of Michigan Medical School; Kevin E. Thorpe, assistant professor, Department of Public Health Sciences, University of Toronto Faculty of Medicine; Laure Perrier, M.Ed., University of Toronto Faculty of Medicine.

— By Scott Harris

Contact Us    © 1995-2008 AAMC    Terms and Conditions    Privacy Statement