Continuing medical education has long relied on a familiar set of evaluation practices. We launch a program, track participation, analyze pre- and post-test scores, review self-reported practice behavior and confidence scores and collect information about perceived care barriers in standard post-activity surveys.

But everyone who has spent time designing educational interventions knows the lingering frustration of this model.  A clinician can easily pick the correct guideline recommendation on a multiple-choice question yet still hesitate to implement that exact therapy when they return to a busy clinic the next morning.

Traditional assessment tools often fail to capture how clinicians experience and operationalize new knowledge. They are frequently blind to the invisible friction points that stall real-world behavior change. Competing workflow demands, fear of toxicities, institutional inertia and systemic anxieties do not always show up clearly on a Likert scale.

To bridge this gap, we developed a framework that leverages AI to scale sentiment analysis, providing deeper insight into how learners experience and apply new knowledge. Traditional evaluation methods typically rely on structured survey items or on manual coding of a limited sample of comments. An AI-supported approach could make it feasible to review the full body of unstructured learner dialogue, such as discussion transcripts and free-text responses, and to identify recurring emotional and attitudinal patterns across many hours of content in a fraction of the time.

Unlocking the "Why" Behind the Data

It’s important to understand the role that sentiment analysis plays in qualitative research. Sentiment analysis is the process of determining the emotional tone, attitude or opinion expressed in communication.  It is not fundamentally new. Evaluators have historically used manual text coding and rule-based word counts to track learner sentiment. In the past, however, analyzing hours of spoken clinician dialogue or thousands of free-text survey responses required prohibitive amounts of human labor.

Advances in AI-enabled natural language processing (NLP) have created new opportunities to scale this methodology and analyze unstructured qualitative data, including clinician discussions and program transcripts. Rather than manually parsing text, we can use large language models (LLMs) to efficiently process massive volumes of unstructured data, including free-text survey responses, chat boxes and discussion transcripts, synthesizing raw text into meaningful behavioral patterns. Instead of evaluating whether a learner's response is factually correct, the technology allows us to scale the detection of deeper signals such as underlying systemic perceptions, emotional cues, attitudes, anxiety, hesitation, uncertainty, confidence and perceived barriers that influence clinical decision-making. Understanding these signals allows educators to tailor content, reinforce confidence and better support real-world decision-making.

It is important to note that sentiment analysis is not a replacement for traditional educational outcomes. Knowledge and competency improvements remain foundational assessments. Instead, sentiment analysis acts as a complementary layer and answers the "why" behind the numbers, showing us the contextual and emotional barriers that dictate whether learning ultimately translates to the bedside.

Case Study

To illustrate how this approach could work in practice, we applied it to two educational programs in oncology.

Despite transformative advances in relapsed/refractory Diffuse Large B-Cell Lymphoma (DLBCL), including chimeric antigen receptor T-cell (CAR-T) therapy and bispecific antibodies, real-world adoption of these therapies remains uneven across practice settings. Logistics, toxicity concerns, institutional infrastructure and care coordination pathways all influence whether evidence-based therapies are ultimately implemented.

 To better understand these dynamics, we examined how clinicians discuss CAR-T and bispecific antibody therapies in structured educational settings, focusing specifically on how language reflects real-world barriers to implementation. 

Methodology: Capturing Real-World Clinical Dialogue

This analysis drew on live, CME-accredited peer-to-peer educational discussions, representing more than 19 hours of clinician dialogue focused on advanced DLBCL management. The dataset included both community-based and academic oncologists engaged in case-based discussions of CAR-T cell therapy and bispecific antibody treatment strategies.

Rather than relying on traditional survey instruments, which might fail to capture real-world nuance, we analyzed verbatim transcripts of these discussions to understand how clinicians naturally express decision-making in the context of the available resources in their practices.

An AI-enabled LLM approach was applied to identify recurring patterns in clinician language. Importantly, the focus was not on evaluating clinical correctness, but on understanding how clinicians describe the feasibility of implementing care in their practice environments.

The analysis included the following steps:

1. Identifying top themes that emerge from transcript-based analysis.
2. Engineering sentiment-based prompts for each of the top themes.
3. Engineering prompts for a sentiment scoring system for each theme, along with justification and specific examples.
4. Highlighting the highest positive and negative perceptions.
5. Reviewing AI findings by subject matter experts to generate actionable educational insights for clinical practice improvement.

Summary of Findings

Across the 19 discussions, oncologists demonstrated a strong understanding of advanced DLBCL guidelines, and knowledge was not the limiting factor. Instead, the analysis isolated a clear tension between clinical agreement and practical feasibility as seen in the summary of findings below (Figure 1):

Figure 1: Color scheme: Red indicates scores below 5 out of 10, Orange indicates a score of 5 out of 10, and Green indicates scores above 5 out of 10. The square boxes on the upper left side of the figure represent the actual score. For example, four red boxes indicate a score of 4 out of 10.

Interpretation of Findings

These findings suggest that the primary barriers to adoption of advanced DLBCL therapies are not rooted in knowledge deficits, but in implimentation readiness. Clinicians demonstrate strong familiarity with CAR-T and bispecific antibodies, but their language reflects uncertainty about feasibility within their local practice environments.

Importantly, hesitation does not equate to resistance. Instead, it reflects a layered decision-making process in which clinical evidence is weighed alongside logistical constraints, institutional support and perceived risk.  Clinicians are not questioning efficacy but rather evaluating whether their system is equipped to safely and effectively deliver these therapies.

This distinction is critical in understanding why guideline-aligned therapies may still experience delayed uptake in real-world oncology practice.

Implications and Limitations

Our results demonstrate that, when paired with structured prompts and expert oversight, sentiment analysis:

• Adds depth to traditional outcomes measures.
• Captures learner experience, not just knowledge acquisition.
• Identifies educational gaps rooted in emotion, confidence and feasibility.
• Can be conducted efficiently using AI with structured prompts and expert oversight.

Despite growing interest, AI-driven sentiment analysis in healthcare education remains early-stage.  They are often limited in scope with inconsistent methodology and underdeveloped for real-world implementation (e.g., dashboards, workflows, decision support). More rigorous, interdisciplinary approaches are needed to validate and operationalize these tools at scale.

Several additional considerations warrant attention. LLMs can reflect biases present in their training data.  Sentiment is context dependent and the same emotion may signal different barriers or opportunities across specialties, practice settings, clinical situation and cultural communication styles. This is one reason expert review of AI-generated findings remains essential. Learner data privacy is also an important consideration. Transcripts and free-text responses may contain identifiable information about clinicians, institutions, or patients, so programs should take appropriate steps to de-identify data before using AI tools for analysis.

Conclusions

AI-enabled sentiment analysis makes it possible to capture learners’ emotional and cognitive experiences at scale, transforming what used to be time-intensive qualitative work into an efficient, actionable process.

By revealing not just what learners know but how they feel, this approach uncovers drivers of confidence, uncertainty and engagement, highlighting areas where learners may struggle despite strong knowledge scores.

This deeper understanding allows health professional education providers to design interventions that are more responsive, effective and learner-centered, targeting both knowledge and the emotional experience of learning.

Here are some practical steps for CPD professionals interested in exploring this approach:

Start with qualitative data you have already collected such as open-ended survey responses, learner reflections, or discussion transcripts.

1. Choose a set of learner comments focused on a particular topic area.
2. Use an AI tool to identify recurring sentiments such as enthusiasm, uncertainty, anxiety, frustration, or resistance.
3. Ask what is driving those sentiments and identify recurring barriers or facilitators.
4. Validate the interpretation by asking a subject matter expert or other knowledgeable reviewer to assess the findings within the clinical and educational context.
5. Compare the findings with your traditional outcomes. Do they help explain why the education did, or did not, have the intended impact?

---

Interested in this article? Join the discussion on the Alliance Community.