Measuring the Effectiveness of CEhp Activities and Overall CEhp Program Impact - CE Competency Series #3

By Derek T. Dietze, MA, CHCP, FACEHP, President, Improve CME LLC

Alliance National Learning Competency Area #3: Measuring the Effectiveness of CEhp Activities and the Impact of Overall CEhp Program

Some CEhp professionals cringe when they hear the word “outcomes.” Clearly, many of us recognize our own needs for continuing professional development (CPD) on this topic, and one section of the Alliance National Learning Competencies provides a framework to help us build greater confidence and expertise with respect to outcomes: Area 3.

Area 3 of the Alliance National Learning Competencies focuses on using data to evaluate the effectiveness of CEhp activities/ interventions and the impact of our organization’s overall CEhp program (i.e., everything it does with respect to CEhp).

Building competencies in this area and implementing them in practice provides ongoing opportunities to build a pattern of continuous quality improvement (CQA) that can improve the efficiency, quality and impact of our efforts in CEhp. In addition, exercising competencies in this area will help us comply with and exhibit exemplary performance with respect to ACCME, AAFP, AOA, ACPE, AAPA, AANP, ANCC and other organizations’ requirements for planning, implement­ing, assessing and improving our CEhp.

In addition to providing specific competency statements, behaviors that demonstrate competence in this area — at novice (just starting to learn about outcomes), intermediate (proficient) and expert levels — are also provided to help us assess current performance and make plans for CPD. While not comprehensive, the following sections provide an orientation to this competency area and suggestions for CPD. Click here for more detailed information on this and all other competencies.

Evaluating the Effectiveness of CEhp Activities/Interventions
Competency 3.1 centers around the use of outcomes data to assess and determine the outcomes or results of CEhp activities on learner attitudes, knowledge, skills, performance and/or patient outcomes. It also includes the use of that data to determine unmet learning needs and the quality and success of our CEhp interventions.

To accomplish this, CEhp professionals need to be able to 1) identify the level or type of outcome(s) associated with the learning objectives and expected results of the activity; 2) select and use appropriate assessment methods and tools in line with the type of outcomes to be measured, resources and budget; 3) analyze the data collected to draw conclusions about the effectiveness of the activity compared to its expected results; and 4) analyze the same data to identify practice gaps and/or learning needs for future interventions.

If you are a novice with respect to this competency (i.e., have little to no knowledge or experience in this area), below are some suggestions for where to begin your CPD:

  • Attend a pre-conference workshop at the Alliance An­nual Conference. The first pre-conference workshop listed for the 2017 Annual Conference is “Introduc­tion to CME/CE Outcomes.”
  • Study the 2015 Almanac series Beginner’s Guide to Measuring Education Outcomes in CEhp.
  • Ask a colleague to review an outcomes report with you and discuss how the outcomes tool(s) were selected, how the data were collected and how the results were used.

Those who are “competent” in this area currently engage in the types of actions listed below:

  • Select assessment methods/tools that are aligned with the objectives, outcomes levels based on your assess­ment resources and stakeholder expectations.
  • Implement assessment tools to collect data on the quality, independence and balance of interventions.
  • Analyze qualitative/quantitative data collected using basic methods (e.g., summarizing comments, review­ing descriptive statistics like percentages, frequencies and mean scores).
  • Draw conclusions regarding the extent to which the activity met its goals, objectives and desired outcomes.
  • Translate analysis results into opportunities for future activities (i.e., new or ongoing educational needs).

To further improve your competence in this specific area, or to develop an area of expertise, you could do the following to continue your CPD:

  • Study learning and assessment theories, frameworks, methods and tools that you are not familiar with, and experiment with them in practice, for instance: Prochaska and DiClemente’s Transtheoretical Model (or Stages of Change)
  • The Expanded Learning Model for Systems (TELMS)
  • Bandura’s Social Cognitive Theory and Self-Efficacy
  • Brian McGowan’s Learning Actions Model
  • The Alliance’s SQUIRE Tool

Look outside of traditional forms of CEhp assessment to identify practical methods for assessing effectiveness and identify opportunities for implementation in your practice. You might investigate:

  • Health Economics and Outcomes Research (HEOR)
  • The Association for Talent Development (formerly American Society for Training & Development)
  • Net Promoter Score®
  • Confidence-Based Learning
  • The Patient Education Materials Assessment Tool (PEMAT by AHRQ)
  • Pursue additional training on statistical methods.

Evaluating the Impact of Your Overall CEhp Program
Competency 3.2 focuses on the use of evaluation and outcomes data to evaluate the impact of the overall CEhp program (“program” being defined as everything your organization does in CEhp) and its effectiveness in meeting your organization’s mission and goals. While the rationale for inclusion of this competency may be directly tied to compliance with the ACCME’s Accreditation Criteria 12 and 13 (overall program-based analysis and improvement), following a CQI process is clearly beneficial for any type of organization and CEhp professional. To attain this competency, a CEhp professional needs to be able to: 1) outline and plan steps to evaluate all aspects of the CEhp program and its effectiveness in meeting its mission and goals; 2) use (combine, analyze and report) activity evaluation and outcomes data to determine overall program effectiveness; and 3) use analysis results to identify areas for improvement of individual activities and the overall program.

A novice level practitioner with respect to this competency could:

  • Review your organization’s last overall program evalua­tion and discuss the process and results with a mentor or colleague.
  • Identify data and information resources within your organization that can be used for an assessment of overall program effectiveness.
  • Attend an Alliance conference session on overall pro­gram assessment.

If you are competent in this area, then you currently engage in the actions listed below:

  • Use data/information collected from individual CEhp activities or initiatives and summarize or “draw up” this information into reports.
  • Use the summary reports to assess the level of achieve­ment of specific elements of your mission and organi­zational goals and to identify areas for improvement.
  • Have identified stakeholders and collaborators who can contribute to your overall program evaluation.

To further improve your competence in this area, or to develop an area of expertise, you could do the following to continue your CPD:

  • Actively seek the opportunity to participate in your orga­nization’s next overall program evaluation. Often, this is a multiyear process, and your help may be appreciated.
  • Identify new sources of information, either internally or externally, and use them to add value to your over­all program evaluation. These might include:
    • Interviews with patients or patient surveys
    • Feedback from educational partners and vendors
    • Comments from the quality or patient safety departments in your institution
    • Information on the financial impact or cost-savings resulting from your activities/interventions
  • Develop executive summaries of your overall program evaluation results, then actively seek opportunities to present and discuss these results with key stakehold­ers within or outside your organization. Record the feedback you receive.

Developing a personal and organizational CPD plan using Area 3 of the Alliance National Learning Competencies can help us all build greater confidence and expertise with respect to evaluating the effectiveness of individual activities/initiatives and our overall CEhp programs. If you have had success in building your competencies in this area, we would like to hear your story. Please send your story and any CPD resources you used to almanac@acehp.org.

Recent Stories
Seamless MOC Part 2 Integration: Add Value Not Work

Tiny Moments Matter: Promoting Professionalism in Everyday Practice

AMA Congratulates IU Medical Students