SEER - Strengths Enhancing Evaluation Research

Summary

The video by molokaimatt showcases the journey of creating SEER, an evaluation tool for Ina-based programs in Oahu. This tool integrates video for capturing cultural processes often missed in written formats, aiming to support community program developers, evaluators, and funders. Over a year, a culturally resonant, adaptable, and inclusive framework was developed to improve the evaluation of Ina-based programs, addressing concerns about indigenous voices being overlooked in traditional evaluations. The video emphasizes the importance of cultural context, storytelling, and mutual respect in understanding and evaluating program effectiveness and community impact.

Highlights

• SEER is an innovative evaluation tool focusing on cultural context. 🎥
• The video documentation enhances experiential understanding of programs. 👀
• Indigenous voices and cultural context are central to SEER's approach. 🗣️
• Storytelling and community input are vital to program evaluations. 📚
• The process fosters sustainable relationships and understanding between cultures. 🌱

Key Takeaways

• SEER is a culturally driven evaluation approach for Ina-based programs in Oahu. 🌺
• The tool uses video to capture cultural nuances often lost in text. 📹
• SEER promotes the inclusion of indigenous voices in program evaluation. 👥
• Interactive storytelling aids in understanding program impact and effectiveness. 📖
• A strong emphasis is placed on mutual respect and understanding in the evaluation process. 🤝

Overview

SEER, or Strengths Enhancing Evaluation Research, is a groundbreaking tool designed to evaluate Ina-based programs on the island of Oahu. Created by molokaimatt, it leverages video documentation to provide insights into cultural processes that are often overlooked in traditional written evaluations. This approach ensures a richer, more comprehensive understanding of the programs' impact.

The core of SEER is about amplifying the voices and perspectives of indigenous communities within the evaluation process. It moves beyond conventional data and evidence-focused evaluations to include storytelling and cultural context, providing a holistic view of the programs' effectiveness. This culturally resonant method is adaptable and inclusive, ensuring the programs' processes and impacts are accurately represented and understood.

The video highlights the collaboration between evaluators, program developers, and community members to create tailored evaluation tools. By fostering mutual respect and understanding, SEER not only assesses program effectiveness but also aims to build lasting, meaningful relationships that support community development and cultural preservation.

Chapters

• 00:00 - 01:00: Introduction and Purpose of the Video This chapter introduces the purpose of the video, which is to document the experience of developing an evaluation tool involving two INA (Indigenous Native Applications) based programs on Oahu Island. It emphasizes the unique capability of video to capture aspects that written words might miss, aiming to offer an experiential understanding of cultural processes and perspectives of these programs. The video is intended to complement a handbook, serving as an educational tool.
• 01:00 - 02:30: Culturally Resonant Evaluation Practices The chapter focuses on culturally resonant evaluation practices, highlighting the importance of supporting community-based program developers, evaluators, and funders. Over the course of a year, efforts were dedicated to developing a process that aligns with cultural practices and values. Though the initial segment indicates the presence of music, it suggests a commitment to blending traditional methods with innovative, culturally-sensitive evaluation strategies.
• 02:30 - 04:00: Challenges in Indigenous Evaluation The chapter discusses the development of a tool for evaluating Ina-based programs. This tool emphasizes culturally resonant practices that are rigorous, adaptable, and inclusive of diverse perspectives and data collection methods. The goal is to enhance the strength and representation accuracy of evaluation research related to Ina-based program processes.
• 04:00 - 06:00: SEER Framework and Community Engagement This chapter discusses the SEER Framework and its role in community engagement, particularly focusing on research and evaluation involving indigenous peoples. A key concern highlighted is the lack of opportunities for indigenous voices to be properly heard and integrated within evaluation processes.
• 06:00 - 09:00: Community Program Descriptions The chapter discusses the importance of focusing on data and evidence when working with communities, while also emphasizing the need to consider cultural and contextual factors. In particular, indigenous evaluation is highlighted as an approach that places importance on understanding culture and place to provide context to data and numbers, thus giving voice to the people whose stories might otherwise be lost.
• 09:00 - 13:00: Process of Engagement and Data Collection The chapter titled 'Process of Engagement and Data Collection' discusses the importance of evaluating and possibly changing the way stories are told about language and background history. It highlights the collaboration with the Consuelo Foundation and the Haoli Mooa Foundation in reassessing the evaluation process to ensure it is fitting and valuable.
• 13:00 - 16:00: Collaborative Storytelling and Analysis The chapter titled 'Collaborative Storytelling and Analysis' addresses the limitations of traditional logical models and theories of change in storytelling and analysis processes. It argues that answers to complex questions often lie beyond linear frameworks and templates that many individuals are used to. The solution is not simply about modifying questions within existing frameworks but requires a transformation of the entire process. The chapter emphasizes a more comprehensive and holistic approach, likely referring to how collaborative storytelling can offer deeper insights and more meaningful analysis by stepping outside of conventional methodology.
• 16:00 - 19:00: Development of Customized Evaluation Tools The chapter titled "Development of Customized Evaluation Tools" discusses a framework for engaging with INAB (Indigenous Native American-Based) programs. It emphasizes the importance of approaching these programs with a mindset of collaboration and respect. Researchers are encouraged to engage as guests and acknowledge the expertise of community program developers and providers, rather than assuming they have all the answers.
• 19:00 - 22:00: Connection Between Land, People, and Culture The chapter explores the unique development of tools customized by individuals involved in running music programs. It highlights the role of an educational nonprofit that provides services to families, particularly focusing on those with young children in certain locales. The narrative evidently intertwines with themes of land, people, and culture, though these aspects need further elaboration based on the provided transcript.
• 22:00 - 25:00: Long-term Impact and Generational Goals The chapter titled 'Long-term Impact and Generational Goals' discusses a program centered around family and child interactions. It seems to emphasize a specific interaction or methodology referred to as 'Hur.' However, there are interruptions by music and applause in the transcript, making the details of the discussion incomplete.
• 25:00 - 28:00: Building Sustainable Relationships The chapter titled 'Building Sustainable Relationships' focuses on the use of the Punalu Discovery Garden as a tool for engaging families in sustainable practices. The garden spans an acre and a half and was once a landfill area that has been revitalized through the planting of native species and strategic planning. Every Friday, families participating in the program visit the garden to explore, discover, and learn within a natural environment, guided by adults. This interaction aims to foster a deeper understanding of nature and sustainable living.
• 28:00 - 30:00: Conclusion and Mutual Responsibility The chapter titled "Conclusion and Mutual Responsibility" discusses the Hua project, which is part of the kokua khhi Valley's comprehensive family services. This initiative, known as Hu AA to grow land, focuses on promoting mutual aid and responsibility within the community.

SEER - Strengths Enhancing Evaluation Research Transcription

• 00:00 - 00:30 the following video documents the experience of developing an evaluation tool with two Ina based programs on the island of Oahu the use of video with its ability to capture what is often lost in written word is intended to provide a more experiential articulation of the cultural processes and perspectives of inab based programs designed to complement the handbook the video may be used to
• 00:30 - 01:00 support the work of community-based program developers evaluators and [Music] funders over the course of a year efforts were made to develop a process
• 01:00 - 01:30 based tool which would lend to more effective and accurate evaluation of Ina based programs what emerged are ongoing culturally resonant practices that are rigorous adaptable and inclusive of many perspectives and ways of collecting data the resultant product strengths enhancing evaluation research aims to provide a trer representation of inab based program processes and thereby
• 01:30 - 02:00 expand their potential and [Music] Effectiveness one of the critical concerns that indigenous peoples have about research and about evaluation in particular is that there doesn't seem to be an opportunity for their voices to be heard for the essence of the program most evaluations
• 02:00 - 02:30 uh focusing on on data and evidence but there's a lot more to it than that when it comes to working with people in communities and and obviously it's the same with other cultures I guess not just with indigenous but there's a need to reflect the voices more of people and I think one thing about indigenous evaluation especially is that we put it in a context of culture and place and that will tell the story behind the numbers when you cut off the context of culture
• 02:30 - 03:00 and place even language background history you're changing the story that you're telling people now we're working with the Consuelo foundation and um for myself also the haoli mooa foundation and I really give these foundations credit for saying okay well let's look at the evaluation process and does it really fit and um that's that's really valuable even as the foundation is asking that
• 03:00 - 03:30 question they don't I don't think they realize what the answer is you know I don't think they realized that the the answer is way far away from the logic model theory of change um uh linear uh template that they're accustomed to it's not just changing the questions it's changing the entire template it's changing the entire process the process of seer is a general
• 03:30 - 04:00 framework for approaching inab based programs it's a general framework for engaging becoming a guest and then being respectful all the time throughout the process making sure that we don't that researchers don't come in acting like they have all the answers because in fact the experts are the community program developers and providers
• 04:00 - 04:30 therefore the tools that actually are developed are unique to or customized by those individuals who run the [Music] programs theay is an educational nonprofit that Services U families in Co La law families with young children and
• 04:30 - 05:00 the focus of our program is a family and child interaction called Hur [Music] [Applause]
• 05:00 - 05:30 one of the tools that we use in h is the punalu Discovery Garden and it's about an acre and a half of uh forly dump lands that we have tried to bring back to life through native plants and some careful planning and we bring all of the families in the program out here every Friday and we've tried to create a a natural environment for kids to explore and discover and learn um with an adult
• 05:30 - 06:00 member of their [Music] family this is Hua the name of our project is Hu AA to grow land we're part of kokua khhi Valley which is a comprehensive Family Services here at U know we have four
• 06:00 - 06:30 program areas we have Loa listening to the land which is our wahana and moo revitalizing ancient sits and ancient stories we have kaaa which is our native reforestation project revitalizing the Kaa mic Forest here in khi we have our Mahi AA program which is our community food production program where elders and children create food for a sustainable food system in khhi and our last area is haa AA sharing the land and friendship that's our community engagement
• 06:30 - 07:00 program the most exciting thing for me to see as an educator when I see the kids come they're afraid to get dirty you know they're like you know and so we get them a big pile of compost and we get them the sifters and they go to town in 10 minutes they got a we wheelbarrow full of the stuff they didn't want to touch at first and then they they're like don't spill it don't spill it so in
• 07:00 - 07:30 10 minutes they go from I'm afraid of dirt to I value dirt and that's it's a it's metaphoric but I think that that's a seed that can last for a long time and that idea of growing land um that you're growing actually growing healthy land but you're growing because of that relationship too if you're going to be aware on the land you have to be aware with people too and you should be able to go back and forth between the natural world and the human world
• 07:30 - 08:00 I mean if you're a successful human being you should be able to migrate that so that the things that you learn in the garden the small little things if you're aware and you understand it you're going to apply it to the human world and when you get out in into the land and you're working with the land I mean the land is what it is you can't force it to change just like you can't force people to change we are all indigenous to some place that at some point in our genealogies every person came from a land-based culture and it
• 08:00 - 08:30 might be that your particular group of people your particular family has come away from their land-based culture a long time ago that doesn't mean that you're not indigenous that doesn't mean that you don't have the capacity to understand the value of relating to humans on a cultural level on a land level and that's that's what we want people to experience here and that's what inab based culture-based programming is is helping to get get children especially but humans in
• 08:30 - 09:00 general to remember that part of themselves that doesn't see the land as separate from [Music] themselves the core elements of sear involve a process of Engagement and that really means that we are asking permission to become guests and hoping not only to be invited but be invited to to develop a
• 09:00 - 09:30 relationship with those individuals well our first uh meetings our first uh invitations to meet with uh nakal Anda we really about building a sense of who we are and who they are so we weren't really focused on our um questions about the program we were really introducing ourselves this then takes at least two or three sessions it involves us figuring out what they want
• 09:30 - 10:00 to do what their goals are what their expectations are and how we can work together everything has a meaning there is not there's no such thing as it doesn't mean anything and when Paula I remember Paula I was trying to get to know her and we were driving around the property looking at the various areas where we conduct program and I remember Paula telling me
• 10:00 - 10:30 at the very end of the conversation about her family from mikai and to be honest with you that was the exact minute I was sited and I think it was because I realized that Paulo was going to know what it was like to live in a rural area that was um very family based and landbased and the community supported itself and I I I think it was that I could trust Paula to understand that
• 10:30 - 11:00 because I knew her background if you show up with yourself if you show up as who you are then the person that you're interviewing the person that you're sharing stories with is free to be who they are too and you can respect each other and at that point that's when you get to the best [Music] stories we then collaborate towards data
• 11:00 - 11:30 collection and they the individuals let us know who's going to be in this uh data collection process it at nakamal we did individual interviews and focus groups at ho ulua we did a learning circle learning from each other talking about Concepts Beginnings how they go about doing their program what kind of concepts are
• 11:30 - 12:00 important to them that they communicate to the individuals who come to the property and it wasn't always an interrogation process or judgment process it was like they really wanted to find out what was happening um in our organization what our challenges were we sit together as as human beings and we sit and we have conversations about who are you and who are we and how do we connect to to these places you know that's the essence of indigenous uh work
• 12:00 - 12:30 ways of doing things is building the relationships uh we listen and participate and listen and uh contribute and listen and so a lot of it is listening I was uh very pleasantly surprised at how open the conversation has been and how much um
• 12:30 - 13:00 uh how much creativity and freedom and and expression and time has gone into into the discussion about evaluation process but for me personally the discussion of what we on this land in this space for this program in this time frame what are the stories that we need to be looking at and what do what do we need to um how do we need to elicit those stories and how are we
• 13:00 - 13:30 able to present those stories We Begin the session by catching up talking story what's been happening in your life what's happening down here and then we move towards well what is the topic for today what are we going to be discussing thanks so much for spending time with us allowing us to talk to you we really appreciate this you're welcome and first we usually like to get to know each other so so we'll share something
• 13:30 - 14:00 about ourselves and you know as we go along this talking story and we're using the camera primarily as a note-taking device all of our conversations with individuals at each of these sites has been storytelling our hearing their stories they're hearing our stories and in in the process almost mutually constructing or understanding where they're coming from and where what
• 14:00 - 14:30 we you know potentially have to share with [Music] them we are not only interviewing people taping them looking at the tapes afterwards discussing them developing charts in which we put the main issues up we're doing some situational mapping which involves looking at the human and
• 14:30 - 15:00 non-human elements and then we're drawing out of that themes important processes which we can then list and then take a look at take back to those that we've been working with to verify to complete to enhance the explanation of their processes and taking a look at what evolved from the data up to this point and how they would like to use it how would they
• 15:00 - 15:30 like to have it presented what were some of the major issues that came up that were strengths and others that may need Improvement or more support from funders we're we're collecting your stories your manal and we're taking a look at what are some of the main Essences that we see and so we're bringing that back to you
• 15:30 - 16:00 to see what you think that this represents what you're doing it's based of what you have written here for me to to see that this is what I firmly believe it takes more than just a language that was put into a written form it takes people just being people but what is more important is that we treat the different is with the value
• 16:00 - 16:30 which was respect [Music] love one of the observations is that you know um in many many respects uh we are all in in in some in some sense structured and and goal orientated people so you know often they're sitting with us and they and they're looking for answers too and they're looking at us
• 16:30 - 17:00 for those answers and what we try to say is yeah you you've actually explained and given your solution we just we just want to help you try to frame that in a way that uh your your funding agencies um and um have a sense of what what it is that you're doing what do we do now from here what that's a good question because we've been sort of running along you know we've met with you several times you've
• 17:00 - 17:30 gotten themes going and I think at this point we we want to bring those ideas together and come out with a product that also has some indicators like we said for this particular project the tools that actually are developed are unique to or customized by those individuals who run the programs H ulua for example is going to develop
• 17:30 - 18:00 a multimedia type uh evaluation tool that will enable anyone who comes to their website to see the connection between the land and the people the way in which both have benefited the stories behind each of those participants as well as the areas on the land and so when we first sat down to tell our stor together we understood that that we were kind of birthing something we were but
• 18:00 - 18:30 we didn't know what it was we were like it could be something very simple it could be a paper tool a digital tool a video tool it could be anything what is a tool we we thought we talked at length about all kinds of um options all kinds of um formats we looked at mapping formats and um and the storytelling was the most important part what I'm imagining in it its physical manifestation is that you
• 18:30 - 19:00 might click on a particular part of the AA and it will have the story of a particular group that planted a particular tree and maybe within levels that tree will grow the group will grow the group will change the tree will change around the tree changes I just I love that feeling like seeing something that we planted like knowing that once it starts growing like a piece of my history is a part of this place's history it feels like you're the leader and you LED that path you made that path
• 19:00 - 19:30 and now others are following our path within those images that can be produced are the stories from the children what it meant for them to plant that tree what it means for them to come visit that tree 2 years later what it means what what the metaphor is for that I I'm a leader because I created a path like that that is that's invaluable but I cannot explain that I can write that down in a survey form but it doesn't make sense of unless you can see the
• 19:30 - 20:00 path unless you can walk the path I can create that walking in a digital format so that people can understand that the transformation of the community and the transformation of the land is not two separate things they're interdependent they're the same [Music] thing one of the things that uh Peter and I have talked about is that connection between the in the people and the culture and how in
• 20:00 - 20:30 this process we're attempting to look at what is that connection for this program how well are they connecting not only the participants but themselves and what they are doing to enhance sustainability in all areas so the essence of what we're doing is looking at the heart and soul of of the the relationships and the work
• 20:30 - 21:00 they do I mean the program is what gives them the kind of structure but I think what a lot of them are looking at is beyond the life of the program you know the generations to come how will what we're doing impact the land in in Seven Generations time and how will what they're doing with the Ki impact the Next Generation so I think that that's something that we're working on and that's why it's kind of like a our our product products are really
• 21:00 - 21:30 what they are committed to in in the long term thank you for allowing us to hear your stories for trusting us with those stories uh and and the thing the important thing with what we're doing is is the we are building sustainable relationships because this isn't just about research this is about connecting um at at a community level in in a very deep
• 21:30 - 22:00 significant uh and respectful sensitive way that we're both committed ourselves to supporting the programs after the the life of this evaluation I mean it's okay to have an evaluator who comes and looks at your program and you have contact with them and then they go away but it's another thing when someone comes in and you know they're going to be working with you professionally but suddenly they remind you about things they remind you about stuff that you may have always knew but
• 22:00 - 22:30 you don't talk about they remind you of things that you always thought but you were too scared to say and it's also a learning issue about learning what else is going on out there in the world beyond the community that you're working in and I think that having having that relationship doesn't change the evaluation I think it actually heightens it because I think how having someone who
• 22:30 - 23:00 understands where you are at can help you improve the services that you're providing through their evaluation and I think the story is Fuller richer more detailed and really helps um the larger Community to understand what it is you're really [Music] doing Koka Uka Koka
• 23:00 - 23:30 K those of the Upland those of the shore the deeper meaning of this Hawaiian proverb is the notion of mutual responsibility for each other and nature strengths enhancing evaluation research embodies this Mutual responsibility it is both a culturally resonant process of empowering inab based programs as well as a tool for delivering reliable evidence for funer decision making decisions that will
• 23:30 - 24:00 actively support healthful relationships between people culture and the AA [Music]