Chapter 11: Measuring Program Impact and Designing Evaluation Instruments

Summary

In Chapter 11, the video focuses on measuring program impact and designing evaluation instruments, produced by the Children's Hospital of Philadelphia Center for Injury Research and Prevention. It provides insight into selecting key performance indicators (KPIs) that align with program goals and introduces both quantitative and qualitative methods to assess these KPIs. The chapter emphasizes performance measures like attitudes, knowledge, skills, behavior, and self-efficacy, specifically in teen driver programs. The video also discusses data collection methods and offers examples of questions to assess KPIs such as distracted driving.

Highlights

• Understanding the importance of performance measures in setting goals and allocating resources. 🎯
• How to choose the right performance indicators for your program evaluation. 💡
• Exploring both quantitative and qualitative methods for measuring program impact. 🔬
• Specific KPIs for teen driver programs, including attitudes, knowledge, and self-efficacy. 🚘
• Sample questions for evaluating KPIs such as distracted driving behaviors and attitudes. 📋

Key Takeaways

• Performance measures are essential for setting goals, monitoring progress, and communicating program value to society. 🏆
• Key Performance Indicators (KPIs) help track program progress over time and compare across different sites and populations. 📊
• For teen driver programs, going beyond participation and satisfaction to include attitudes, knowledge, skills, and behaviors is crucial. 🚗
• Two main methods for data collection are quantitative (numbers) and qualitative (words and concepts), with detailed exploration in upcoming chapters. 🔍
• Example questions for performance indicators help in evaluating aspects like self-efficacy, attitudes, behaviors, and behavioral intentions. 🌟

Overview

In this engaging chapter, we delve into the world of measuring program impact and designing evaluation instruments, especially crafted for teen driver programs. The creators highlight the significance of selecting and applying the right key performance indicators (KPIs), which serve as the backbone for evaluating program success. You'll learn to connect goals to action, set objectives, and allocate resources effectively, while ensuring your program's value is recognized in society.

The video unfolds the nuances of different performance measures, focusing on five key aspects: attitudes, knowledge, skills, behaviors, and self-efficacy. For those invested in teen driver safety, these indicators are not only vital but also specific, measurable, achievable, and realistic. By watching this chapter, you gain a clearer perspective on how to track progress, compare between varied programs and sites, and ultimately inch closer to achieving your program’s goals.

Data collection is a crucial component, and this chapter prepares you for it by introducing both quantitative and qualitative methods. Whether you're focusing on numbers or narratives, the video has you covered. The chapter wraps up with insightful examples of questions aimed at evaluating specific KPIs, like distracted driving, setting the stage for the next chapters that delve deeper into data collection techniques. Get ready to turn insights into impactful action!

Chapters

• 00:00 - 08:00: Measuring Program Impact and Designing Evaluation Instruments In this chapter, the focus is on evaluating the impact of a program and designing appropriate evaluation instruments. It covers the importance of identifying key performance indicators (KPIs) to measure the success of the program. The discussion includes guidance on selecting the most suitable performance measures or indicators for your specific program needs.

Chapter 11: Measuring Program Impact and Designing Evaluation Instruments Transcription

• 00:00 - 00:30 hello pendock Community Traffic Safety Partners thank you for joining us for another video which is being produced for you by the Children's Hospital of Philadelphia Center for injury research and prevention in this video Chapter 11 we'll talk about measuring your program impact and how to design evaluation instruments by considering the key performance indicators that you want to have in your evaluation for your program we'll talk about how to choose which performance measures or which performance indicators are right for you
• 00:30 - 01:00 and we'll present you with some of the quantitative and qualitative methods that you can use to do that and then provide you an example of what this looks like when you pull it all together for a program evaluation performance measures are used for several different purposes for example to set specific goals for your program to connect your goals to actions and allocate resources to Monitor and evaluate your progress and to communicate your priorities results and the value of what you do to society but perhaps the way
• 01:00 - 01:30 that performance measures or key performance indicators are most important for your evaluation is that they can also track the progress of your program over time and help you to compare how your program operates across different sites and across different programs and populations you may be familiar with some of these key performance measures which are recommended by the national highway traffic safety administration these include outcome measures
• 01:30 - 02:00 behavioral measures and activity measures and you might have seen these before but there are some additional performance measures for teen driver programs specifically that go beyond measures of participation and satisfaction which we think will be particularly helpful for you because they're specific measurable achievable reachable and time bound these include measuring the attitudes Knowledge and Skills of your participants judging their overall Behavior change their self-efficacy the frequency with which
• 02:00 - 02:30 they engage in certain behaviors or disengage in certain behaviors and their behavioral intentions let's talk a little bit more about what each of these mean if you're looking to measure attitudes with your program then you're looking to understand a relatively stable belief or feeling that your program participants have about a particular concept or a person or an object and we infer somebody's Attitudes by observing their behaviors if we're looking to influence
• 02:30 - 03:00 individuals knowledge then we want to increase their awareness or their familiarity with a certain concept if we want to increase their level of ability or expertise with regard to driving then we'd be looking to change their skills if we want to affect behaviors then we're interested in the way in which an individual Acts or conducts themselves self-efficacy refers to the Judgment of one's capability to accomplish a certain level of performance so if we want to increase
• 03:00 - 03:30 the likelihood that somebody is going to engage in a particular Behavior we often want to start by increasing their selfefficacy or their self-confidence to do that particular Behavior frequency is a performance measure which refers to the rate at which a particular Behavior occurs perhaps how often somebody does something or doesn't do something and behavioral intentions refers to the person's perceived likelihood or their subjective probability that they will
• 03:30 - 04:00 engage in a given Behavior which we're asking them about in our program what teen driving issues could you evaluate with these measures for example seat belt use of the driver and the passenger speeding practices distracted driving through phones or by other passengers in the car driving skills and experiences impaired driving and drowsy driving now that you've chosen your performance indicators how are you going to collect information about them so
• 04:00 - 04:30 that you know the degree of change that your program is causing for these performance indicators you have two basic options remember that quantitative methods are those that are going to express your results in numbers they'll help you to answer the questions how many how much or how often while qualitative methods are those that will Express their results in words ideas and Concepts they'll answer the questions how and why this table provides you with a brief snapshot of some of the options that are
• 04:30 - 05:00 available to you for quantitative and qualitative data collection we'll talk in more detail about these in chapters 12 and 13 quantitative data that may be helpful for your measuring your performance indicators for your evaluation includes surveys record R viws indicator data GIS or geographical app data and performing an environmental assessment qualitative methods that might be helpful for you include open-ended survey interview interviews
• 05:00 - 05:30 in-depth interviews Diaries of your participants focus groups observations and newspapers or other media again we'll talk more about these in chapters 12 and 13 this table shows you one of many options that we have designed for you in your resource book to get you started with the types of questions that you could include for your performance indicators for example if you chose your performance indicator was distractor driving you might ask questions like for
• 05:30 - 06:00 self-efficacy How likely are you to do or say something to your driver if they're talking on a handheld cell phone while driving for Attitudes about distracted driving do you support a state law Banning talking on a handheld cell phone while driving behaviors toward distractive driving might include questions like when you receive a text message while driving how often do you answer the text and for Behavioral intentions toward distractive driving we might ask somebody to rate the agreement or
• 06:00 - 06:30 disagreement that they have with statements like I would be more likely to give up using my cell phone while driving if my mom or dad made me do it my friends gave up their cell phones while driving I got a discount on my insurance Etc again please see the resource book for a number of examples of different types of questions that match different key performance indicators that we feel would be most helpful that you would be likely to use for your program evaluations what does this look like when we pull it all together recall from our earlier chapters that our programs
• 06:30 - 07:00 have an overall goal for example to reduce teen driver crash rates and as a followup to that goal we have a basic performance measure which is that we're looking to measure distracted driving behaviors we hope that you will have developed some smart objectives objectives which are specific measurable achievable realistic and time bound you can learn more about smart objectives in our earlier chapters as we've been discussing in
• 07:00 - 07:30 this chapter you may also have different measurement categories perhaps you're interested in looking at the frequency of distracted driving from participants in your program their attitudes toward distractive driving or their behavioral intentions toward distractive driving this table shows you some of the questions that you might want to ask under each of these categories this concludes chapter 11 measuring program impact and designing evaluation instruments in our next chapter we'll get started with with quantitative data collection methods
• 07:30 - 08:00 that you can use for your program evaluation thank you so much for watching