Incorporating Meta-Analysis into Educational Outcomes to Demonstrate and Drive Success

Track: Innovation

Session Number: 2013
Date: Tue, Sep 26th, 2017
Time: 1:30 PM - 2:15 PM

Description:

Introduction

Most medical education providers focus on outcomes for single activities. While this is helpful for gauging the success of a single activity, it does not provide information regarding whether or not the provider’s instructional models and overall strategies are having a positive impact on healthcare provider (HCP) performance and patient health. Meta-analyses can provide that insight, and should become part of standard procedures for medical education in order to better understand the overall impact of multiple formats and activities. Meta-analyses in the context of medical education can be thought of as akin to clinical guidelines, which pull information from multiple sources to develop a comprehensive evaluation and strategy for tackling a particular problem, in this case, how to best educate HCPs.

In addition to the overall benefit of gaining insight regarding an organization’s success as a medical education provider, there are 3 distinct advantages to conducting meta-analyses. First, they increase statistical power due to having multiple sources of input. Second, they resolve uncertainty regarding an overall educational program if results from individual activities disagree. Third, they can inform strategic planning for future activities by prompting an evaluation of patterns or outliers. This can be achieved for each of Moore’s Levels, mainly knowledge, competence, and behavior. For example, if the results of a meta-analysis were not favorable (i.e., the overall effect size was low), drilling down to data from individual activities can help determine if the poor results were due to content, design, format, or other barriers. Conversely, if the results are favorable, elements of success can be determined and continued. In order for the meta-analysis to be representative of the provider’s programs, it is important to establish criteria for which activities to include, while avoiding selection bias or “cherry picking.”

The two main goals for this workshop are to provide learners with an appreciation for the value of meta-analysis as well as the skills to conduct a meta-analysis. It is hoped that following this workshop, participants will be motivated to take what they have learned back to their workplaces, begin conducting meta-analyses, and use the information obtained from the meta-analysis to guide strategy in developing future activities to maximize success.

Workshop Methodology

This workshop will provide an overview of meta-analysis and provide an example of how to conduct a meta-analysis using Excel or vetted online calculators. Participants will be able to actively conduct a sample meta-analysis on their laptops, using data provided by the presenter. Participants who do not have a laptop or do not wish to use their laptop will still find the session valuable and be able to contribute. All participants will be encouraged to share their ideas on benefits of conducting a meta-analysis, challenges, criteria for inclusion or exclusion of data, interpretation of results, and discussion of advantages and limitations (see below).

The general format of the workshop will include:
• Overview of challenges related to reporting results of individual activities, with solicitation from participants
• Discussion of how meta-analyses can benefit providers, the medical education industry, and patients, with solicitation from participants
• Overview of steps in conducting a meta-analysis, including establishing criteria for data inclusion
• Presentation of results from a meta-analysis conducted by CME Outfitters
• Example walk-through, which will include:
o Describing hypothetical activities that would be included in the meta-analysis
o Providing data from a hypothetical activity to participants so they can follow along on their laptops. (Data will be provided via email or cloud. Alternatively, participants will be able to hand-enter the data.)
o Step-by-step instruction on how to conduct meta-analysis using Excel or vetted online calculators, with participants following along
o Reviewing and interpreting results, with solicitation from participants
• Discussion of methods and results, including limitations of the approaches, with solicitation from participants

It is not the intention of this presentation to provide a full understanding of the mathematics and statistical theory behind meta-analysis rather a basic overview.
Session Type: Skill Building Workshop

Learning Objective 1: Upon completion, participants will be able to be able to identify criteria for selecting activities to include in a meta-analysis
Learning Objective 2: Upon completion, participants will be able to conduct a meta-analysis
Learning Objective 3: Upon completion, participants will be able to interpret results of their meta-analysis, including patterns and outliers, to aid in strategic planning of future activities
Session Type: Skill Building Workshop

Learning Objective 1: Upon completion, participants will be able to be able to identify criteria for selecting activities to include in a meta-analysis
Learning Objective 2: Upon completion, participants will be able to conduct a meta-analysis
Learning Objective 3: Upon completion, participants will be able to interpret results of their meta-analysis, including patterns and outliers, to aid in strategic planning of future activities