Measuring the Impact of Community Engagement for Empowerment
A summary of the Webinar “Measuring the Impact of Community Engagement for Empowerment” in the framework of the series of webinars “Community Engagement Days”.
Moderator: Mr Carlos Barahona, Managing Director, Stats4SD Presenters:
Video recording: https://fao.zoom.us/rec/play/cP92vMfXQ2nAMoVSqOGHFyLgDX6M2guQXUWbV3sGg7dfDoORON0QgCphndKYYJEPRFynAlvoY-6E-l6g.baHtWQuZL6SV6ykM
Article: https://www.fao.org/flexible-multipartner-mechanism/news/news-detail/en/c/1401147/ |
How do we know that community-led empowerment is happening and what the impacts are? Are community-led measurement and analysis of impact possible? This approach turns on its head the notion of outsiders extracting information from communities that they have decided is needed to track and understand local change.
Carlos Barahona posited that we need to ask ourselves difficult questions when tackling this challenge. Who gets to decide what impact is valuable or life changing? Is it fair to impose our criteria of what is important? Is it realistic to expect that those who normally decide – the power holders – will open up spaces to allow communities to engage and influence what is to be measured and what are the impacts of interest? Why should they do that?
To illustrate the size of this challenge, Sonal Zaveri, Co-chair of EvalGender+, reflected on a case of failure by outsiders to understand the transformative change that they envisaged for marginalized communities. She recounted her involvement in a project that supported female tea garden workers in India. The women were members of tribal communities brought to Assam by the British in the 1800s. Over hundreds of years, several generations of their communities worked on these tea gardens with very little contact with the outside world. Very few cases of rights violations were reported.
The project assumed that giving skills and resources to those with less power would enable them to make an impact or be impacted. A grassroots NGO provided tea garden workers with cell phones and codes for maternal health rights violations that would theoretically be geomapped and be used as the basis for litigation. Sonal supported the project staff to have conversations with the tea workers and at the same time to question their own assumptions about how change happens. It emerged that self-stigma was preventing many of these women from reporting violations. With this understanding, the project managers overhauled training for their staff so that they first facilitated group discussions about oppression and empowerment and only then went on to discuss digital literacy and geomapping:
“So often the criteria that we use to measure success don’t reflect the realities on the ground. Unless we put on our community lens, our gender lens and our social justice lens then we won’t be supporting communities to understand what really needs to be measured.” (Sonal Zaveri)
Participants reflected on the question: How do we change our lens, and is changing our lens enough to lead to empowerment? Disempowering ourselves is really important so that we can step into other people’s spaces without expecting them to be in our time, our space, our minds. Participation researcher Dee Jupp recalled her work on immersion, most recently in Indonesia, to illustrate this point. It starts with us, the outsiders, immersing ourselves and being totally open minded and “'non- experts'”, staying with people, listening and observing, while giving them the inspiration and support to design and measure the change they want. Dee reflected that “it’s a very different approach when you move into other people’s space.”
Yet disempowering ourselves in this way is not to deny that we as outsiders have a position – broadly liberal and progressive – whilst also having agency in the process. Typically adopting a feminist position, for instance, means using a lens to see false consciousness or a “consciousness gap” that collective reflection and action can bridge. Collaborative Impact’s Jeremy Holland used the example of UN Women Nepal’s ambition to support the end of harmful cultural practices such as chhauppadi ("untouchability during menstruation”) in Nepal as part of a broader country programme supporting change to gendered social norms.
Recognizing our positionality when we talk about unlearning does not mean shortcutting the community engagement approach that we are supporting or facilitating as outsiders. Ben Cislaghi talked about “sticking with the process” rather than getting too hung up on impact and validating our external goals for a community.
Participants discussed the argument that when we support communities in the measurement of their engagement processes, we remain positioned as external agents. Moreover, sometimes our commitment is to the community, while sometimes it is to other agents that come and impose, contribute, or partner with communities. In both scenarios – when we are with the community and when we are with those outside agents – we need to be rigorous but we need to be sensitive and empathetic with the community and sensitive to gender, ethnicity and marginalization. Sonal reflected that the donor-driven methodological trend (for externalized measurement) has “swept away the community dialogue approach that I have come out of.”
Jeremy Holland argued that as outsiders we must try our best to create space for local processes of community-driven social change even in the face of political imperatives to deliver narrow, linear logframe-driven programming. Our role combines, first, a facilitation role to encourage inclusive processes at local levels as well as an interpretive role for others who were not there with us on the ground. Participatory, community-led impact research is a “win-win” for development. First and foremost, participatory research can empower local people in a sphere of research that has traditionally been highly extractive and externally controlled. At the same time, participatory research can generate accurate and generalizable statistics in a timely, efficient (value for money) and effective way for outsiders working to contribute to changes in impact level that they see as progressive and developmental.
To illustrate this point, Jeremy described a participatory mass storytelling instrument that is being adopted by UN Women Nepal to measure changes in social norms and harmful cultural practices across the country. Storytelling provides the basis for locally-led individual and collective reflection dialogue and action, empowering women and men to challenge gendered social norms and behaviours. At the same time, it generates “data points” at scale that can be coded and analysed for patterns and trends. In particular, the initiative will utilize a mass storytelling tool such as SenseMaker in tracking and interpreting programmatic contributions linked to the SDG 5 indicators to changes in social norms and gender equality. The mass storytelling research tool combines the interpretive depth of storytelling with the statistical power of aggregated data for tracking patterns and trends in social norm behaviours. With this tool, the aim is to generate a “feedback loop” of evidence and learning into long-term programming for better impact in influencing social norms change and ending harmful practices.
Participants reflected that community engagement in, and community dialogue around, impact measurement can at its best be integrated into programmatic support for community transformation processes. This is in stark contrast to the dominant approach adopted by an industry driven by expensive, large-scale surveys that measure global metrics. Carlos Barahona, of Statistics for Sustainable Development described what he called “single factor targeting of development”, simplistic models that do not embrace complexity:
“Technocratic positions prevail. That prevailing way of thinking attempts to simplify too much and generates resistance. We can work with other partners that bring time, patience and solidarity to the way that we do interventions. Correcting the route that we follow so that we make small adjustments rather than step changes that shift from one indicator to another.” (Carlos Barahona)
To illustrate, Dee Jupp’s powerful case study from Bangladesh explains how members of a land rights coalition analysed the impacts of gaining access to land entitlements. Coalition members identified locally-meaningful metrics – changes that they valued – for measuring their empowerment. A dramatic change in gender roles and relations was powerfully brought to life through participatory theatre, with women explaining how they were now decision-makers in their households and had a voice in their communities.
“What does community engagement for measuring impact mean to me? Measuring what is valued (by people themselves), not valuing what is measured” Dee Jupp, Participation researcher |
Similarly, Dee’s work in Malawi involved community action groups working in education, health, markets and roads achieving positive change at different speeds and with different issues valued by different groups. Dee’s question was: “what were people valuing and what data did they need to engage with the system?”
‘This is ours. This is not for the partner organization. They may use some of the information from time to time but this is ours and ours alone’ Village Development Committee member, Malawi |
For Dee Jupp, community engagement is what happens within systems. At one level, this means starting from local people’s priorities and interests, but more fundamentally, it means recognizing both that they “are part of a system and that they have their own information to engage with the system.” Becoming (more) active participants in the system through a community engagement process, they can then make demands, share experiences and build bridges. This is what is “empowering”; this is what is community-driven transformation.
If we shift to a way of thinking in which local knowledge is generated and owned by local people, where does that leave “us” as external agents in someone else’s empowerment process? What is our role as external agents? Are there difficult trade-offs we need to make? Reflecting on our role as facilitators of internal processes, Dee flagged up principles from her experience of facilitating participatory research: using approaches that are informal and engaging; using approaches in community members’ space and time; and letting community members use approaches in their own ways. This then allows us the outsider to collect and cluster change statement information/pattern finding “aside from what the community needs from the data.”
Participants concluded that, on the face of it, community engagement in measuring such results seems like a lot of effort. Community involvement requires significant effort, from many people, and the process is not valued by everyone. But if we think about community-led measurement as an empowerment process in its own right, then this becomes the value that we would like to see.
What of the legacy to date of participatory or community-led research and the way forward for community engagement in impact measurement? We have seen surges of interest and troughs of business as usual. What should we learn from the last 30 years of work in this area as we look to the future?
Dee Jupp set out a vision of future impact measurement systems that communities own and use to measure their engagement and feed collective reflection and action. Locally-generated and owned data feeds collective action and enhances accountability. In this future scenario, the key for “us” as outsiders is to develop systems which borrow “their” data for “our” purposes. We can be so much smarter in finding ways to borrow their data and fit into results frameworks, manage data which is detailed and local to track trends and identify impact in more relevant and meaningful ways. This does not mean that we deny ourselves agency in local development. Instead, we need to reposition our role as outsiders as providers of inspiration, advice and links: “on tap, not on top”.
Ms. Andrea Sánchez Enciso