Program Evaluation (2023 Update!) - Social Work Shorts - ASWB Study Prep (LMSW, LSW, LCSW Exams)

Summary

In the enlightening video by Megan Mitchell from Agents of Change, viewers are guided through a thorough understanding of program evaluation, a fundamental aspect of macro social work that serves to enhance program effectiveness and justify resource allocation. A six-step method, based on the CDC's guidelines, is outlined, helping clinicians deploy evaluations efficiently. Using an engaging example of a social skills group for sixth-grade girls, viewers learn to engage stakeholders, describe the program, focus the evaluation design, gather evidence, draw conclusions, and share lessons learned. The video emphasizes the importance of data-driven decision-making to improve services and secure funding, ultimately aiming for a broader impact on clients and systems.

Highlights

• Program evaluation steps are adapted from CDC guidelines. 🧑‍⚕️
• Sixth-grade girls' social skills group used as a case study. 👧
• Engage stakeholders such as teachers, parents, and funders. 🎓
• Outcome evaluation design was utilized to measure effectiveness. 📈
• 85% increase in self-reported self-esteem was observed. 🏆

Key Takeaways

• Program evaluation is vital for ensuring resource efficiency and program impact in social work. 🎯
• Engaging stakeholders is the first step in the program evaluation process. 👥
• A detailed program description aids in setting measurable goals and outcomes. 📋
• Collecting credible data is essential for a successful evaluation. 📊
• Lessons learned from evaluations help in refining future programs. 🚀

Overview

Megan Mitchell, from Agents of Change ASWB Test Prep, takes viewers on an insightful journey into the realm of program evaluation. This educational video serves as a guide for social workers, especially those more familiar with clinical work, to delve into macro aspects of program evaluation. By breaking down the meticulous process into six understandable steps, Mitchell simplifies the seemingly daunting task of evaluations, showcasing the significance of deriving data to foster effectiveness and efficiency in social programs.

The video ingeniously utilizes a real-world example by highlighting a social skills group designed for sixth-grade girls, aimed at boosting their self-esteem and social skills. This case study brings clarity and relatability to the evaluation process, demonstrating how engaging stakeholders, gathering credible data, and grounded evaluation designs lead to meaningful results. With 85% of participants reporting increased self-esteem, the example underscores the tangible benefits of well-structured program evaluations.

As the video unfolds, viewers are not only equipped with knowledge of program evaluation but also inspired to implement these insights into their own practices. Whether for improving program offerings or securing competitive funding, the emphasis is on the importance of assessing efficacy and impact. By concluding with ways to share learned lessons, the video ensures that future iterations of programs are more effective, utilizing data to inform decisions and optimize resource use.

Chapters

• 00:00 - 03:00: Introduction and Importance of Program Evaluation The chapter titled 'Introduction and Importance of Program Evaluation' hosted by Megan Mitchell discusses the basics of program evaluation within the field of social work. The video is popular as it addresses an area often less familiar to clinicians who predominantly focus on direct practice. Program evaluation is presented as a macro-level aspect crucial for those unfamiliar with its steps or facing challenges in understanding its importance.
• 03:00 - 05:00: Purpose and Example Used for Program Evaluation The chapter introduces the concept of program evaluation, describing it as the systematic collection of information about a program or a specific aspect of it. This process is essential to inform decision-making and ensure the effectiveness of the program. The chapter also mentions additional resources for further understanding, directing readers to a website dedicated to agents of change.
• 05:00 - 08:30: Reasons for Conducting Program Evaluation The chapter titled 'Reasons for Conducting Program Evaluation' explores the importance and purpose of evaluating programs within the context of social work. It discusses how evaluations help determine the efficacy and impact of a program. By carrying out program evaluations, social workers can gather essential data points to understand whether a program is working effectively and what changes might be necessary. The chapter suggests that by purchasing the associated course, readers can gain a deeper understanding of different types of evaluations and their significance.
• 08:30 - 18:00: Six Steps in Program Evaluation Framework The chapter introduces a framework for program evaluation consisting of six steps. The speaker emphasizes the importance of following these steps to improve services for clients and client systems. The use of examples aids in understanding and retention of the information presented.
• 18:00 - 28:00: Detailed Example of Program Evaluation The chapter titled 'Detailed Example of Program Evaluation' discusses the importance and process of evaluating a program to determine its effectiveness. The example used is an agency's parenting group, illustrating how program evaluation helps in gathering information about the program's impact. Emphasis is placed on the necessity of evaluation due to the consumption of resources such as personnel and time.
• 28:00 - 30:30: Additional Resources and Conclusion The chapter discusses the significance of resource efficiency and program evaluation in agency work, particularly within the macro realm of Social Work. Emphasizing on creating a substantial impact on clients and client systems, the text highlights the necessity of evaluating programs to ensure resources are not wasted and the desired results are achieved.

Program Evaluation (2023 Update!) - Social Work Shorts - ASWB Study Prep (LMSW, LSW, LCSW Exams) Transcription

• 00:00 - 00:30 welcome this is Megan Mitchell with agents of change Social Work test prep and today I'm here to bring you an updated Social Work shorts on program evaluation so this is one of our most popular videos because generally most clinicians work with clinical work and program evaluation is definitely more in the macro realm of social work so if this is something that is new to you or you're struggling to identify the steps
• 00:30 - 01:00 in program evaluation this will be a great video for you to watch and if you have any questions or are looking for more information about agents of change you can check out our website at agentsofchangeprep.com so let's go ahead and jump in so what is program evaluation program evaluation is carefully collecting information about a program or a particular aspect or part of a program in order to make necessary decisions
• 01:00 - 01:30 about the program including efficacy and impact so if you purchase the course we go deeper into what types of evaluations there are but program evaluation is looking at a program to see is it working is it making an impact what are some changes that we need to make but why we do as social workers program evaluation or why we complete program evaluations is so that we can have a data point so we can use information to
• 01:30 - 02:00 help better our services for our clients and for our larger client systems for example and I'm going to use an example throughout this video because as you have seen from my previous videos when there is an example it helps the information stick so we're going to walk through an example today but I also encourage you to take the steps that we are going to outline and come up with either your own
• 02:00 - 02:30 program or something from your agency or your internship and walk through those steps and come up with your own example so our example here if you're if you work at an agency and run a parenting group a program evaluation would help you collect information about your program and whether it's effective or not why do we have to do this because we know that when we're running programs they take up resources right that could be Personnel that could be time it could
• 02:30 - 03:00 be money we want to make sure that we are not wasting any resources we're being efficient and that like I said we are making the greatest amount of impact on our clients and our client systems so program evaluation is a very very very important component of any sort of program that you're running it's very important to agency work and it definitely Falls more into that macro realm of Social Work so why do we use program evaluation
• 03:00 - 03:30 there's a variety of different reasons but some of the key reasons why we use it to prove results right is the program working is it not working to secure funding we know that resources can be very scarce and they can be very competitive so you might be applying for a grant or you might be applying for some sort of money and if you have solid results from your program evaluations that might be more compelling in securing that funding for your services
• 03:30 - 04:00 to improve Services we should always be striving to improve our services to make them the most efficient that they can be to make them the most effective so we use program evaluation to guide that process why else might we use it to cut costs or justifying justify needing more money right so we might do a program evaluation and see that we are wasting a lot of resources that could be monetary resources or we could say this program
• 04:00 - 04:30 is very effective so we need to put more money into the program so we can service more people there could be a variety of different monetary ways that you are using evaluation of a program and then to compare and contrast efficacy with other programs right so you might have to make hard decisions right which program is going to stay so you would need to use data we know what we are always striving to use data to inform our decisions we need evidence to show why something's working to guide our decision making so program
• 04:30 - 05:00 evaluation gives us those data points so there are six steps and people always ask me where did you get these steps from I use the guidance from the CDC so this is the guidance that is put out by the obviously that very large organization so these are the six steps that we use for agents of change uses as the guideline for program evaluation six steps the first is to engage stakeholders the second is to describe
• 05:00 - 05:30 the program design three Focus evaluation design four gather credible evidence five justify conclusions and six use and share Lessons Learned and we're going to dive into each of these six steps a little bit further so I said that we would be using a example throughout these steps in this video so here is the framework and the example of a program that we will be
• 05:30 - 06:00 working with today you are the facilitator of an eight-week social skills group for sixth grade girls to promote positive self-esteem and pro-social skills you will conduct a program evaluation upon completion to determine results so what's going on here we are facilitating a social skills group for sixth grade girls so most likely a middle schools group the purpose of the group is that the hope would be after eight weeks the girls are developing
• 06:00 - 06:30 positive self-esteem and pro-social skills but we want to see did this have positive effect did this show positive results so let's go ahead and jump into the six steps the first step is to engage our stakeholders so who is a stakeholder a stakeholder is someone who's interested in knowing the results of this evaluation and this will be different depending on what program is running there might be a variety of different
• 06:30 - 07:00 interested people interested parties based on the program you're evaluating so for the sixth grade group this girls social skills program that's being run who might be some stakeholders teachers right they probably want to know if there's going to be a positive effect for this group because these are going to be students that go into classrooms School leaders that could include principals Deans superintendents parents right because they're going to
• 07:00 - 07:30 need to consent for the child to be in the group so parents are going to want to know what is this group and why is this going to help my child the school board may be interested right if this is something that is on a broader scale depending on the size of the district outside funder so sometimes grant money is used or there are different people that have different donations that are made and there might be certain initiatives that they put money towards so an outside funder might be a
• 07:30 - 08:00 stakeholder that is important to no note here the students themselves right they are middle schoolers but it's super important that they are engaged in the process and they also know the intent of the group and the results of the group the facilitator right lots that could be done to maybe improve the group what worked what didn't work and then Community Partners so maybe you have in your school community-based organizations that this would be helpful for or Community Partners that could
• 08:00 - 08:30 also benefit from the results of this evaluation this is not an exhaustive list this is just an example it's going to depend on your setting in a variety of other different factors but always you want to say who are my stakeholders first because that's going to be important to know who cares about the results of this program the second step in program evaluation is to describe the program we need to know what the program's doing right
• 08:30 - 09:00 so in this stage you're describing what the program does you are not describing your evaluation design or tool just yet you will get to that but you're going to describe the program including what outcomes or goals are set for the program or we're set for the program and this should be comprehensive with as many details as possible so you don't want to just say I ran a social skills group for middle school girls that doesn't give us any tangible outcomes tangible goals or anything to really measure what was going on so here is an example of how we
• 09:00 - 09:30 could describe the program this program was a school-based social skills curriculum for sixth grade girls identified as having low self-esteem from teacher and parent reports that one sentence sentence there gives us a great deal of information we know that it was a curriculum that was used we know the population sixth grade girls and how were these group participants selected they were identified from parent and teacher reports the program ran for eight weeks so
• 09:30 - 10:00 duration is important and it was held for 60 minutes two times per week so eight weeks 16 sessions the goal of the program was to increase self-esteem as evidence through teacher and student reports this here is a very solid example of what the program did who was involved how frequently it met and how was Data collected foreign step is to focus your evaluation design
• 10:00 - 10:30 right this is where you have to pick what type of evaluation design you're going to choose there's several types of evaluations like was mentioned before so there's a variety of different types there is formative evaluation process or implementation evaluation outcome or Effectiveness evaluation impact evaluation or cost benefit evaluation so for this specific program we are going to choose an outcome or
• 10:30 - 11:00 Effectiveness evaluation so we're going to determine if the social skills group achieved the established goals and if the program produced the intended results so was it effective what were the outcomes that's what we're going to be looking for with this program evaluation design so we've picked our design in step three we're doing an outcome evaluation step four is to gather credible evidence
• 11:00 - 11:30 right we know we always need to have data points we need to have evidence so you need to collect the data right it doesn't just appear for us so things that you need to consider when collecting data to evaluate your program what data are you going to collect there are a variety of different data points that could be collected but it all depends on what type of evaluation you're doing what your stakeholders want to see so this is not going to be a uniform Choice it's very tailored to the
• 11:30 - 12:00 program how are you going to collect it right is it going to be pre and post tests is it going to be done via an app is it going to be done via observations how are you going to get this data and then does your data measure program goals you want to make sure that the data you're collecting is aligned with the goals of the program right it would not make sense otherwise so our example here to gather data we will use pre and post intervention self-esteem rating scales from students and teachers or parents right so we are
• 12:00 - 12:30 using a rating skill pre-test so we're going to have a data point before that day participants and teachers and parents took and then after the program to see was this program effective did it meet the outcomes that we wanted to Staffing cost of the program and Fidelity of the program will also be reviewed because we need to know who is running the group how much did it cost were they able to run the group with Fidelity what does this mean the group
• 12:30 - 13:00 was intended to run for eight weeks twice a week for 60 minutes did that happen right we know particularly in a school setting things come up so maybe there was a holiday and school did not happen so they didn't actually get to do the the 16 sessions in eight weeks maybe it had to be extended to 10 weeks what was attendance like in the group this is all really important data points that would be needed to help shape your program evaluation
• 13:00 - 13:30 step five you gotta take that data and draw conclusions right so we have all this data how do we make sense of it what does the data mean what does the data show us so we're using the results of the data to draw conclusions so what is what do we do here this step includes analyzing the evidence so all that data we've just collected making claims about the program so connecting the dots I like to say did this program work did it not work was it effective
• 13:30 - 14:00 did it meet the program goals did it meet the outcomes and then comparing the results with expectations so whatever our goal was that was set did we meet those goals did the participants meet the outcomes so this is just an example but let's see what our results show for this program the evaluation data shows that 85 percent of participants saw an increase in self-reported self-esteem from pre-test to post-test there you go that
• 14:00 - 14:30 is a data point 85 percent of people that participate in the group had an increase in self-esteem seventy percent of parents and teachers reported an increase of self-esteem from pre to post-test right so parents and teachers are saying they saw the effects of this program as well the program ran with Fidelity and within budget so it was supposed to run eight weeks two sessions a week it did just that and it
• 14:30 - 15:00 was within budget so you're going to have costs generally Associated whether that be time or resources and it was within budget we can conclude that the program met targeted goals and improved participant self-esteem right you could get much deeper into the data here but this is just for example purposes of some Justified conclusions that we can draw from the data that we got so step six share Lessons Learned
• 15:00 - 15:30 what can we do better in the future what did we learn from running this program so what did you learn from evaluating this program the data and observations from program evaluation can help improve programming in the future that's why you don't want to skip this last step right you want this to inform your future work so the data from the evaluation can help you determine where to cut costs so was there costs that could be cut was there
• 15:30 - 16:00 more staff needed was there more resources needed this can also be used to inform programming how you can improve the programming right maybe you were running it during lunch time and the time that you ran it was not necessarily the most beneficial time what are the training needs you might need to train the trainer right we can't assume that everyone that's running the group is going to come in with the skills able to be the facilitator so you might need to identify where training needs are or maybe they had the basic training but
• 16:00 - 16:30 they need a brush up or they need to go a little bit deeper and then you might be able to identify outside Partnerships right so maybe this was really great and you want to partner with the community mental health agency because you think that bringing them into the partnership could be beneficial as well so what's an example here from this evaluation we can conclude that two step staff numbers will be needed to run the group it is important to note that training is needed for all program facilitators so that's a data point here
• 16:30 - 17:00 in order for it to be effective training is needed the time and location of the group needs to stay during lunch as it is difficult to pull students from academic time students that completed homework outside of groups showed more Improvement on surveys that's another great data point right the the people who the participants that were doing work outside of the group actually saw more improvements so that could be a really good point to make sure that you are focusing on if you run
• 17:00 - 17:30 the group again right we always want to improve our services step six allows us to do just that so if you're still um wanting some more program evaluation resources like I said we follow the CDC guidance for program evaluation but University of Kansas has a very very very good toolkit on all things macro so if you have not checked that out I definitely encourage you to there's an intro to evaluation that gives much more
• 17:30 - 18:00 detailed information there that you can check out as well if you're looking to kind of supplement your learning about program evaluation and if you're looking for more study content you can check out our paid study materials we have a audio visual program that we offer if you are looking for information on that check out our website and here is our contact information um obviously I am Megan Mitchell I'm the founder of Agents of change and you can always reach out if you have any
• 18:00 - 18:30 questions comments or concerns at agents of change prep gmail.com and thank you for tuning in I want you to know this is a tough task to sit for your aswba exam but it is doable remember you got this believe in yourself and best of luck on your studying Journey thank you