By the Alliance Research Committee: Greg Salinas (Chair), Andrew Bowser, Elizabeth Franklin, Alexandra Howson, Jan Perez, Pesha Rubinstein and Wendy Turell
In late 2016, the Alliance Research Committee conducted an environmental scan, utilizing unique surveys for educational providers and supporters, respectively. The study aim was to ascertain practices and member-organization direction in five domains of interest: 1) quality improvement; 2) adult learning principles/educational methods; 3) outcomes measurement; 4) Maintenance of Certification (MOC); and 5) patient inclusion in CE activities. Study findings were reported using descriptive statistics and were published in two parts in the Almanac in September 2017.
Of all the domains measured, findings from the questions pertaining to Quality Improvement Education (QIE) stood out as the most surprising to the committee. More provider groups (58 percent) and supporter organizations (77 percent) reported involvement in QI initiatives than previously thought, and barriers stated by non-QIE players did not tell a full enough story to explain why certain groups enter this area of education, while others refrain. The Research Committee explored the topic further by gathering more information with stakeholders about barriers and facilitators related to successful QIE efforts.
Qualitative research methodologies offer a means of exploring topics about which little is known. We designed a qualitative study to elicit a broad range of perspectives on barriers to and facilitators of successful QIE initiatives and used focus groups to collect discussion data. Focus groups are considered an effective method of data collection when the goal is to explore perspectives from different participant groups or stakeholders. Discussion itself can prompt participants to disclose information about their experience of the topic under investigation.
In order to simplify participation, ensure ease of discussion, and limit cost, we conducted focus groups via a closed online portal (www.zoom.com). The Alliance sent out two recruitment e-blasts to membership that detailed plans for the online focus group sessions. Respondents who met the following inclusion criteria were invited to participate: 1) being an educational provider from any type of organization (e.g., medical schools, medical education companies (MEC), specialty associations); and, 2) had proposed or implemented at least one QIE program, regardless of outcome. Of the 26 members who expressed interest in the focus group discussion, 18 were able to commit to one of two focus groups conducted in November 2017. Demographics reflected a diverse group of hospital, medical associations and MECs (Figure 1).
Figure 1: Participant Demographics
We developed an open-ended, semi-structured question guide to direct focus group discussion (Figure 2). The guide included optional probing questions to use when needed to gain more detail or context.
Questions focused on the following broad areas:
- Defining QIE
- Designing QIE
- Barriers to QIE
- Facilitators to QIE
We used web conferencing software (www.zoom.com) and utilized its text-based chat function to support a fully text-based group chat, such as one might see in an online instant messaging platform with multiple participants. The groups were fully closed to outside public, and the software generated instant transcripts following group completion. One Research Committee member served as moderator for the two groups, which involved managing the introduction of questions from the pre-written guide and following up with the probes when additional information was desired. To ensure reliability, qualitative researchers independently coded the data, generated themes, and compared codes and themes to reach agreement.
There was broad consensus across both groups that QIE is a form of education that supports wider efforts to improve practitioner processes and/or healthcare outcomes. Participants distinguished between education in general and QIE in particular by emphasizing that whereas education addressed individual knowledge and competence, the focus of QIE was generally on improving process, performance and practice at a systems level. There were three main goals of QIE within this broad definition. First, participants identified measurable improvement as a goal in relation to physician delivery of care and patient health across a variety of settings. Addressing specific care metrics was a second QIE goal that participants identified, for instance, in relation to adherence to new guidelines, incorporating new diagnostics, managing diabetic patients with high blood glucose and metrics linked to patient outcomes. A third goal for the QIE programs in which participants were involved concerned specific issues such as improving foot care for patients with diabetes, reducing admission/readmission and increasing preventive screening rates in primary care. Most participants also agreed that QIE goals “evolve over time” and may result in “unanticipated outcomes.”
Planning and Implementation: Internal and External Drivers
Both internal and external factors prompted participants to develop QIE programs. Hospital system participants described internal drivers of QIE such as an existing health system culture that valued QIE, support for QIE by senior leadership and physician faculty, and the presence of an established QIE process within participants’ organizations. For instance, one hospital system participant noted the presence of:
…a monthly PDSA for improving cardiovascular surgery in which there are a list of improvements to be implemented and tested every month. It's a[n] RSS CME.
Another hospital stakeholder noted that her organization had been doing QIE for at least nine years, following an initial response to a statewide initiative. As a result, QIE is “part of what we do now.”
External drivers for QIE cited by hospital system participants included alignment between the availability of funding and health system goals or state directives to achieve particular public health goals.
MECs and specialty association participants mostly described external factors — principally funding availability or funder goals — as drivers of QIE. As one MEC participant noted:
Outside catalysts often drive programming, especially funding opportunities. We are trying to reverse that a bit so that we seek funding aligned with our vision, rather than just being reactive to RFPs.
QIE as an Resource-Intensive Endeavor
Participants unanimously agreed that QIE is a resource-intensive endeavor and referred to rigorous planning processes designed to anticipate likely resources; however, there was a significant distinction between resources anticipated prior to commencement of a QIE initiative and the need for resources that arose during the implementation of QIE.
Both focus groups identified similar resources as anticipated prior to commencement of a QIE initiative. Such resources included access to data, software and personnel for data analysis, leadership to drive QIE forward, and human capital support from faculty and project management. The establishment of strategic alignment and funding were also frequently identified as baseline resources to support effective QIE. Strategic alignment was key, because as one participant put it, “once you get the right people involved, the projects take off.”
Unanticipated resources included personnel, time and technology. For instance, for MEC and specialty association participants, healthcare staff turnover posed a temporal barrier to QIE implementation. As staff left the health system, new staff needed to be trained in data collection for the initiative. Challenges in participant recruitment also slowed down the process of QIE implementation. Notably, the lack of technology and standardized methodology for data collection, extraction and analysis posed a common challenge for most participants, including those in hospital systems. Indeed, data seldom seemed to exist in a sufficiently robust form that was amenable to collection and analysis. This lack of standardization inevitably extended the time frame for QIE implementation and added unanticipated costs to QIE budgets. The need for course content modification was also a challenge for those providing the education component of QIE.
Partner Identification and Desired Characteristics
Most participants emphasized the importance of having partners in the QIE process, especially partners who brought skills that participants’ own organizations lacked. Data collection and analysis was a core skill that most participants sought in a partner, while organizations with a robust QI process looked for education innovators as partners. Aligned mission was an important characteristic in a partner. For some participants, QIE partnerships were forged through established relationships with medical specialty societies, patient advocacy groups and identified experts. Other participants strategically sought out partners with known QI expertise or were tasked to do so by their funders or community partners (e.g., foundations, the state). Networking at Alliance meetings provided another route to partner identification.
Despite the value conferred on partnerships, participants reported challenges in the partner relationship. Lack of partner understanding about the CME/QIE grant environment and process was a potential obstacle to getting a QIE off the ground at all. Some QI organizations had a low level of risk tolerance should their QIE proposal be rejected, and participants expressed concern that when QIE proposals are not funded, this can trigger a loss of trust between education and QI partners. Further, although many organizations ostensibly convey a desire to participate in QIE, they do not wish to have their “bad” data on display. As a MEC participant put it:
The biggest challenge is getting the people with power to drive change to sit around a table, look at their performance data, and figure out how they can get better. If you can get the right people doing that, the rest takes care of itself. They don't need coaching, they only need to be convinced that there is improvement to be made and they will figure it out pretty quickly.
This concern about negative data has the potential to undermine transparency in the QIE partner relationship. Finally, changes in leadership could derail a QIE, as could competing priorities or a breakdown in understanding about QIE project roles and goals. As one participant put it:
An initial challenge was doing some basic ‘translation’ to make sure we were all speaking the same language. QI and CME use some very similar terms, but they don't always mean the same thing.
Barriers and Roadblocks
The challenges of QIE planning and implementation described above are both compounded by and exacerbate other barriers, including scalability challenges, leadership changes, funding, and navigating timelines that can be short and intense. For instance, one participant noted that while scalability is of interest to systems and supporters, there is currently little guidance on how to implement QI, which is, by definition, typically a locally focused initiative. Another participant noted that timelines can really make or break the success of QIE, noting:
I think timelines are a roadblock, especially in a commercial support atmosphere. A comprehensive QI project may need 18 months to run its course, especially in disease states where the patient volume isn't huge. Supporters usually need to see results quicker than that.
Ineffective communication was another commonly reported barrier to effective QIE implementation, especially in relation to communicating project goals and roles to learners and partners. Such communication becomes especially critical when partners are unfamiliar with key components of their professional cultures. For instance, participants in the MEC culture understood how CME grants work, though not necessarily the funding cycles of non-commercial grants, while non-MEC partners were unfamiliar with CME funding cycles and processes, calling attention to the frustration of the disconnect between upfront investment in identifying potential data sources for QIE without being awarded a grant.
Participants unanimously identified several barriers to obtaining good data. For instance, it was hard to achieve an appropriate QI sample size and the extraction process for baseline and outcomes data from electronic health records was rarely seamless. Invariably, for many participants, there was too much data without the proper expertise and resources to analyze it. One participant described how:
One of the clinic networks could not run an EHR report on the data I specifically wanted, so they gave me their entire record of 60,000+ visits to sift through.
These and other roadblocks negatively impacted QIE efforts by, for instance, creating uneven results across multiple sites, extending timelines and over-recruiting in order to achieve an adequate QI sample size.
Yet, participants were resourceful in the face of these challenges. They were adamant that QIE is a constantly evolving process. As one participant noted, “QI is not about ‘perfect,’ it's about ‘good enough’ and making an impact on patient outcomes now and sustaining that improvement.” In order to tackle roadblocks and counteract the effects of these challenges to QIE success, participants adopted “workaround” strategies, such as identifying surrogate metrics redefining metrics, as described by a MEC participant:
Instead of being hyper-focused on getting that exact NQF measure, we ask questions to find out how a physician would know, for example, if a patient's asthma is managed. Turns out lots of docs look at rescue inhaler refill rates. We need those kinds of surrogate measures to answer the most important questions.
Other strategies included reapportioning budget to hire a data analysist where EHR data were inaccessible, and communicating directly with physicians via the CME provider, the QIE champion, or the partner. Indeed, participants emphasized the importance of champions for ensuring the success of QIE.
Participants in these two focus groups represented a relatively seasoned group of QIE practitioners. Although much of the discussion focused on challenges and barriers, participants also shared hard won expertise on QIE planning and implementation. Key insights include the following:
- Identify general and local QIE champions.
- Train stakeholders on the necessary skills; embrace and train new stakeholders in the event of staff turnovers.
- Create a team charter and set clear definitions, including goals, metrics, roles and responsibilities.
- Identify capable partners, especially with the tasks of data collection and analysis.
- Redefine scope of project as needed.
- Remain focused but flexible throughout the life of the project.
- Create a realistic timeline.
- Choose measures that are important and relevant to learners.
- Choose measures so that outcomes data can be readily retrieved.
- Create education that is simple and focused for learners so that they can easily apply the concepts in the real world.
The 2017 environmental scan of Alliance members reported that more than half (58 percent) of provider organizations had ever begun or completed one or more QI project; 28 percent had never engaged in a QI initiative; and 14 percent stated QI was not relevant to them. A majority of supporters (77 percent) had supported one or more QI projects at any point in time. This survey identified several barriers to QIE engagement including access to healthcare system data, human or financial resources, organizational culture, expertise and other reasons such as the time frame for conducting QIE, which often exceeds a 12-month grant cycle, and physician engagement.
Our subsequent analysis of focus group data provides additional insight to these QIE barriers and facilitators. First, QIE represents an ambitious endeavor in any health system, even when there is broad support for it. Focus group participants emphasized that identifying a QI champion to communicate QIE goals to faculty, learners and support staff is a crucial ingredient to effective QIE. Second, participants felt that QIE was most effective when embedded in a wider process as part of organizational culture, versus as a one-off, standalone education intervention. However, achieving such embeddedness requires substantial effort to identify partners, build relationships, and achieve goal alignment across partner (and funding) organizations. Third, deficits in human capital and technical expertise pose threats to any QIE initiative.
As CME/CPD evolves as a mechanism for driving QI, our community is in need of education and resources. The combined findings from the environmental scan and focus groups suggest that such education and resource-building could be profitably directed toward the following areas:
- Establishing a database of providers with data extraction, analysis and reporting expertise
- Creating more opportunities for dialogue and exchange to enable Alliance members identify potential QIE partners
- Educating members on the elements of QI and how it differs from education (CME/CPD)
Hall LL, Brown M, Rosof BM, et al. American Medical Association. Steps Forward. Quality Improvement Using Plan-Do-Study-Act. https://www.stepsforward.org/modules/pdsa-quality-improvement
Health Resources and Services Administration. Quality Improvement. 2011. https://www.hrsa.gov/sites/default/files/quality/toolbox/508pdfs/qualityimprovement.pdf
Health Resources and Services Administration. Developing and Implementing a QI Plan. 2011. https://www.hrsa.gov/sites/default/files/quality/toolbox/508pdfs/developingqiplan.pdf
Hughes RG. Tools and Strategies for Quality Improvement and Patient Safety. In RG. Hughes (ed). Patient Safety and Quality: An Evidence-Based Handbook for Nurses. Rockville, MD: Agency for Healthcare Research and Quality. 2008. https://www.ncbi.nlm.nih.gov/books/NBK2682/#ch44.r28
Institute for Healthcare Improvement (IHI). Quality Improvement Essentials Toolkit http://www.ihi.org/resources/Pages/Tools/Quality-Improvement-Essentials-Toolkit.aspx
SQUIRE 2.0 (Revised Standards for Quality Improvement Reporting Excellence). http://www.squire-statement.org/index.cfm?fuseaction=page.viewpage&pageid=471
Figure 2: Focus Group Discussion Questions
- How would you describe or define QIE to a colleague?
- What did you hope to achieve (educational and/or quality goals)?
- Did the QIE program result in the educational outcomes you anticipated?
- Did the QIE program result in the improvement outcomes you anticipated?
- What prompted your organization to consider developing a QIE program?
- Which resources did you anticipate ahead of time to support the QIE program?
- Which resources did you not anticipate that you realized you needed after starting the QIE program?
- At what point in the process did you learn this?
- How did you identify partner organizations to collaborate with for this QIE project?
- What characteristics were you looking for in a QIE partner organization?
- Please describe any challenges you experienced with the partner organization.
- What were some of the roadblocks you anticipated to developing a QIE program?
- At what point in the process did you encounter those roadblocks?
- How did these roadblocks affect the QIE program?
- Planning? Implementation? Evaluation? Feedback?
- How did you identify solutions to these roadblocks?
- What were the key differences between what you expected and what the outcomes were?
- How did you explore the reasons for those differences?
- Knowing what you know now, what would you say are the most important factors to consider in launching a QIE program?
- Would you plan other QIE program?
- What would you do differently next time?
- Why would you do these things differently?
- What top 3 tips would you offer CME colleagues who are thinking of developing a QIE program?
- Anything else we didn’t cover that you feel is important for CME practitioners to think about in starting a QIE program?