Takeaways
- States convening Medicaid beneficiary advisory councils (BAC) can use evaluation approaches to increase effectiveness of BAC processes, demonstrate impact, and ultimately, better serve Medicaid members.
- States can engage BAC members and other stakeholders in the evaluation process, to understand what questions they have, how they prefer to share feedback, how data align with their experiences, and what suggestions they have for making improvements.
- Findings from evaluation can be used to refine BAC activities, improve services and supports, and strengthen trust with members.
A 2024 Centers for Medicare & Medicaid Services (CMS) final rule required state Medicaid agencies to establish Beneficiary Advisory Councils (BACs) by July 2025. These councils — composed of current and former Medicaid members, their families, and caregivers — are designed to elevate member roles and perspectives in shaping Medicaid policy and program implementation. BACs offer a meaningful way to strengthen policy making and improve Medicaid member experience by engaging member feedback to identify unmet needs, inform program design, improve services, and highlight what is working and what is not. While some states are new to this approach, many have long engaged community members prior to the BAC requirement. Regardless of where states are starting, strengthening BAC processes and understanding their impact is important.
As states implement and expand BACs, they can use evaluation approaches to monitor progress, demonstrate impact, and improve BAC functionality and member experience. Information gathered through evaluation can be used to refine future BAC activities.
This tool presents strategies and examples to help states effectively integrate evaluation into their BAC efforts. It outlines three key steps:
- Clarify Evaluation Questions
- Gather Information to Answer Evaluation Questions
- Share Lessons to Drive Improvement and Demonstrate Impact
While this tool focuses on BACs, the guidance may also be relevant for other member-focused advisory groups at the state, plan, system, or community levels.
Clarify Evaluation Questions
The first step in effective evaluation is clarifying what you want to learn and how you will use that information. States can connect evaluation questions to improvement goals or agency priorities. Be sure to engage state colleagues and BAC members in identifying key questions.
Two types of evaluation may be especially helpful for BAC work: process evaluation and impact evaluation. Process evaluation assesses how BACs function and how members experience participation, offering opportunities for continuous quality improvement. Impact evaluation seeks to understand and demonstrate the impact of the BAC and its recommendations on Medicaid policies, programs, and ultimately, outcomes. Both types of evaluation are important for helping BACs operate effectively and inform Medicaid policies and programs. These types of evaluation can work together, as the results of a process evaluation can guide improvements that lead to greater impact.
Process Evaluation
- Purpose: Understand which BAC processes are working well, which aren’t, and how to improve
- Potential Uses: Guide process improvements, strengthen effectiveness, plan changes or expansions, promote BAC membership
- Audiences: BAC members, state Medicaid agency staff
- Example process evaluation questions:
- Do BAC members have the support they need to prepare for meeting discussions?
- What do BAC members say about their experience in the first year?
- How and when are key points from BAC discussions shared within the agency?
Impact Evaluation
- Purpose: Demonstrate impact and outcomes of BACs
- Potential Uses: Create transparency, sustain buy-in, inform decisions
- Audiences: Medicaid leadership, BAC members, public
- Example impact evaluation questions:
- What action has the agency taken in response to BAC feedback? What wasn’t acted on? Why?
- How were changes based on BAC feedback shared with the broader Medicaid agency and community?
- What program improvements or outcomes has the BAC supported?
Try these prompts to clarify evaluation questions and planned use:
- We’d really like to know ___________________________.
- We will use that information to _____________________.
- We will share what we learn with _______________
and share that information via ________________.
Tips
- Talk with BAC members, Medicaid staff, and other stakeholders about what questions they have and how they would use answers to those questions.
- Think about timing: when do you need to act on or share evaluation learnings? Build these timeframes into your evaluation questions and approach.
Gather Information to Answer Evaluation Questions
Next, consider what information can help answer evaluation questions. Start with information you already have and use a mix of information sources, including quantitative and qualitative data. Combining multiple types of information provides richer insights. For example, quantitative data like meeting attendance reports can show how BAC participation changed after a shift in meeting schedule or format, while qualitative interviews with BAC members can help explain why attendance changed. There are many ways to gather information that don’t require a formal or extensive data collection process.
What data are you already collecting?
Whenever possible, use information that’s already available. Many states already track information about BAC composition, participation, and feedback. To demonstrate impact, consider how BAC guidance informs agency actions or may lead to changes in data the agency already tracks, such as member satisfaction, grievances, call center volume, or utilization.
EXAMPLE: The California Department of Health Care Services funded an evaluation of their BAC predecessor organization, the Medi-Cal Member Advisory Committee (MMAC), to inform the implementation of their BAC. They gathered information on domains related to BAC member experience, impact, differences in MMAC perceived value and outcomes across different groups, and lessons learned. The evaluation included an analysis of member surveys collected regularly after meetings.
What additional information gathering method is feasible and preferred?
When looking for additional information, start with easy-to-gather options. This can include informal conversations with BAC members or colleagues, additional questions on existing feedback forms or meetings, or a simple tracking tool to make data collection routine. Ask BAC members how they would like to share information and feedback. Consider offering a variety of methods to accommodate different preferences. Periodically reassess information gathering techniques to make sure they are still helpful and feasible to answer evaluation questions.
What information is most compelling to key audiences?
Think about who will receive or act on evaluation learning. What kinds of information do they prefer? What information sources do they trust? To evaluate impact, consider how engagement with BAC members relates to agency priorities, metrics, or accountability mechanisms and link those priorities together. Ask BAC members what examples of impact are most meaningful to them.
EXAMPLE: Oregon created an Action Tracker that provides its advisory committee members and the public with a record of completed actions from meetings, as well as pending items.
Case Scenario: Using Evaluation to Connect BAC Insights to Agency Priorities
Naomi is leading the BAC at her state Medicaid agency. BAC members have asked her how the Medicaid agency is using their feedback and want to know how the BAC is making a difference. Medicaid colleagues have also asked about the BAC’s impact. Naomi decides to pursue an impact evaluation of the state’s BAC work.
Clarify impact evaluation questions: As a first step, Naomi includes time on the next BAC meeting agenda to talk with members about the impact they want to have. She asks questions like, “What would make you feel like your time on this committee is well spent?” and “What are some ways you’d like to make a difference?”
She also spoke to Medicaid colleagues about how they are getting feedback from the BAC, and if and how they are acting on it. She hears that her colleagues are committed to improving policies and programs with BAC feedback but have challenges in how to operationalize that feedback. She also notes that her colleagues are feeling a lot of pressure to make progress on agency priorities, including improving member satisfaction.
Gather examples of impact: To understand how BAC feedback is impacting agency programs and policies, Naomi creates a tracking tool that connects feedback from BAC members to agency priorities and follow-up actions. She updates the tracker regularly, including noting who within the agency is responsible for following up and what actions they take, if any.
| BAC Comment | Related Agency Priorities | Responsible for Follow-Up | Follow-Up Actions | Status |
|---|---|---|---|---|
| We have a lot of questions about our care and costs but trying to get a question answered on the phone takes much too long. | • Improve member satisfaction • Decrease call center hold times | Customer service team | • Update FAQs • Revise informational materials | Planning revisions |
| Navigating the referral process is a huge pain. You get the run-around every time. | • Improve access to care • Improve member satisfaction | Care navigation team | • Create new step-by-step referral guidance | Drafting content |
Sharing progress: After nine months, Naomi shares a progress update with BAC members, highlighting three examples of agency improvement efforts guided by BAC comments. These include: (1) adding frequently asked questions to the website; (2) revising informational materials to make it easier for members to get questions answered without calling customer service; and (3) creating step-by-step guidance on referral processes. BAC members appreciate seeing the progress and suggest additional ways to improve materials and member satisfaction.
Naomi also shares these examples of impact with her team and supervising director. The director includes these in a progress report to leadership alongside the latest data on member satisfaction scores and improvement efforts.
Share Lessons to Drive Improvement and Demonstrate Impact
Finally, share learning widely with BAC members, Medicaid agency leadership, and other stakeholders, including the public, to drive improvements and demonstrate impact. Frame learning for key audiences and translate impact into terms that are relevant and useful to different stakeholders.
Review the evaluation results you collect with BAC members. Share initial ideas about what you’re learning and ask questions to understand how it fits with their experiences.
Try these prompts for engaging BAC members and other stakeholders in making sense of evaluation lessons:
- We think this means ____________________. What do you think?
- What other explanations can you think of for what we’re seeing?
- Is there anything important we’re missing? What else do we need to know?
- Who else should we share this learning with?
Frame Findings and Lessons for Key Audiences
Translate impact into terms that are relevant and useful to different stakeholders. For Medicaid colleagues and leadership, show how examples of BAC impact connect to agency priorities or public concerns. For example, if feedback from BAC members informed improvements to communication materials and strategies about a new or underutilized benefit, examine whether this changed how information is communicated to members, utilization of the benefit, and related outcomes. For BAC members and the public, highlight ways that the agency is responding to Medicaid member feedback.
Conclusion
Integrating evaluation into BAC efforts is a valuable strategy to understand and demonstrate the impact of engaging Medicaid members and identify opportunities to make BAC processes more efficient, effective, and person-centered. Evaluation doesn’t need to be formal or resource intensive — it can begin with clarifying questions, identifying the information needed to answer them, and using insights gleaned to guide next steps. Process evaluation can help states refine BAC operations, while impact evaluation can demonstrate the effectiveness of the BAC in informing Medicaid programs and policies and strengthening trust with BAC members.
Engaging BAC members in evaluation will help states clarify key questions to be asked, identify relevant information sources, and understand how preliminary findings align with members’ experiences. Discussing evaluation insights during BAC meetings can generate new ideas for improving programs and services, connect BAC engagement to agency priorities, inform members how their feedback informed Medicaid decision-making, and ultimately increase BAC impact.
