Viewpoint

ePortfolios and Assessing Competence: The Western Reserve2 Curriculum

By Terry Wolpaw, M.D., MHPE
Associate Dean for Curricular Affairs
Case Western Reserve University School of Medicine

Case Western Reserve University School of Medicine launched the Western Reserve2 (WR2) curriculum in July 2006. This student-centered curriculum focuses on the development of skills physicians need for the practice of healthcare in the 21st century—skills in reflection, self assessment and self-directed life-long learning. Taking its lead from the six ACGME competencies, the WR2 curriculum was built with a competency-based educational design.

Nine core competencies describe the knowledge, skills and behaviors a student must demonstrate to qualify for an M.D. degree in the Western Reserve2 curriculum (Table 1). Early in the planning process, curriculum leaders recognized two important design challenges. First, many key components of the nine competencies could not be assessed through usual methods such as multiple choice tests. Second, if assessment drives learning, the goals of the WR2 curriculum could only be realized if student assessment, teaching methodology, and goals were aligned. Student assessment strategies would need to evaluate and acknowledge the importance of skills in reflection, self-assessment, and self-directed life-long learning.

Table 1. Western Reserve2 Curriculum Competencies

The nine competencies in Table 1 define the performance and professional development that students in the WR2 curriculum at Case must achieve to receive an MD degree. However, students (and faculty) need some benchmarks along the way to know if the learners are on track in their development. These benchmarks—called achievement levels or standards—attempt to define student progress in each competency along a developmental continuum. Achievement levels for the nine competencies in the WR2 curriculum have been described at three time points—the end of the first year of study, on entry into clinical rotations, and in the fourth or final year.

Table 2. Example of Achievement Levels for the Life-long Learning and Personal Development Competency

Portfolios have emerged as a way to assess competence across a broad range of performance and professional development¹. As described by Holmboe, Davis, and Carraccio, portfolios are "collections of work, evaluations, products, and similar material of the learner collected over time that reflect professional development, annotated by the trainee's reflection on what has been learned in terms of learning outcomes."

There are three key aspects to a portfolio that distinguish it from a logbook. First, a critical part of the portfolio is the learner's reflection on his/her development in a competency, including an analysis of strengths and areas for improvement. Second, the learner rather than the teacher has the major responsibility for developing the portfolio. Third, the learner receives feedback from portfolio reviewers to help inform a personal quality improvement process.

The portfolio is a natural fit for a self-directed, competency-based medical school curriculum such as WR2 because of its alignment with curricular goals and its ability to focus on each competency. Students have the opportunity to utilize longitudinal perspectives and insights based on collections of their work, rather than self-limited examinations, to demonstrate their development in a competency. Perhaps most importantly, the portfolio process utilizes the same skills the WR2 curriculum considers as core to its mission - reflection, self assessment, and self directed life-long learning. Students have to engage in guided reflection and self assessment in building their portfolios. Because the ACGME is already piloting a portfolio for resident evaluation, the portfolio process is one that students will need to use they move forward in their careers.

A five-year physician investigator track within Case Western Reserve University School of Medicine (Cleveland Clinic Lerner College of Medicine-CCLCM) implemented portfolio-based assessment with its first class of 32 students in 2004². The challenge for the Western Reserve2 curriculum was to build on best practices from CCLCM and adapt portfolio based assessment to a medical school class of 150 students per year.

How Does a Student Prepare a Portfolio: A Brief Overview

Step 1
Students need a place to store their educational work and evaluations on an ongoing basis. In WR2 this is accomplished through an ePortfolio. The ePortfolio is an organized and confidential repository for storing work. Student examination results, faculty and peer feedback, reflection papers, physical diagnosis and communication skills check lists, etc. are stored in a student's ePortfolio.

Step 2
At three points in time students prepare a summative ePortfolio to demonstrate that 1) they have met the achievement levels for a given competency and 2) they can reflect on their areas of strength and their areas for improvement. A student's summative ePortfolio is an electronic collection of reflective essays, one per competency, each no more than two pages in length and accompanied by supporting evidence. To prepare for writing a reflective essay, students look over the evidence in their ePortfolio repositories and choose which pieces best support their progress. Their essays must convince the reviewers that they have met the achievement levels for a given competency and have provided a balanced assessment of their strengths and areas for improvement. Throughout the essays, they reference evidence to support their narrative³.

Step 3
Faculty reviewers read each student's portfolio. After each reflective essay, the reviewer answers the following questions:

• Is the essay consistent with the evidence?
• Is the essay balanced, i.e., includes areas of strength and areas for improvement?
• Is this a thoughtful, insightful essay?
• Is the essay well-organized and information communicated clearly and convincingly?
• Have all achievement levels for the competency been addressed?

Step 4
Reviewers determine if a portfolio meets or does not meet expectations:

Meets Expectations: Meets achievement levels for all required competencies and demonstrates satisfactory self-reflection on strengths and areas for improvement.

Does Not Meet Expectations: The portfolio does not meet expectations for one or more competencies. Indicate all that apply:

• Does not meet achievement levels for one or more competency
• Does not demonstrate satisfactory self-reflection on strengths and areas for improvement for one or more competency
• Evidence cited is insufficient or inconsistent with reflection essay for one or more competency
• The reflection essay does not show adequate thought and insight for one or more competency
• Information in not sufficiently organized or communicated clearly in one or more competency

Step 5
A student whose portfolio does not meet expectations meets with an advising dean and develops remediation plans and portfolio revisions.

Challenges

Like any curriculum change, implementing a portfolio assessment of student performance is an iterative process. We have encountered many challenges as we implement portfolios in the WR2 curriculum at Case. One is the need to identify curriculum sessions and activities that relate to the competencies and develop tangible products students can use as evidence of their progress. Another is encouraging students to regularly consider what contributions an educational session or activity might make to the portfolio and to save any work or feedback they obtain. And finally, helping faculty to appreciate how much students seek high quality narrative feedback that informs their development and provides rich evidence for their portfolio essays.

References

1. Holmboe ES, Davis MH, and Carraccio C. Portfolios. In Practical guide to the evaluation of competence. Holmboe ES and Hawkins
   RE, editors. Mosby 2008.

2. Dannefer E and Henson L. The portfolio approach to competency-based assessment at the Cleveland Clinic Lerner College of Medicine. Acad Med 2007;82:493-502.

3. Driessen E, Van Tartwijk J, Overeem K, et. al. Conditions for successful reflective use of portfolios in undergraduate medical education. Medical Education 2005;39:1230-1235.