Evaluating Progress: Key Takeaways

“We cannot direct the wind, but we can use data and metrics to adjust our sails,” said Yvonne Belanger, the Barr Foundation’s Director of Learning & Evaluation. Her statement perfectly set the stage for the Social Innovation Forum’s virtual event, “Evaluating Progress for Continuous Learning: Lessons Learned from Our 2018 Innovators,” on July 14th.  This event allowed the 2018 Innovators the opportunity to reflect on their performance measurement goals, as well as the challenges they have encountered carrying out their missions since the novel coronavirus pandemic hit the US hard earlier this year. It also provided an opportunity for nonprofit leaders to discuss how funders and partners can better help these organizations continue to serve their communities, especially when the pandemic has caused a large, unexpected strain.

Above and Beyond

Even as nonprofits focus on a range of specialties, their main focus is always on strengthening, connecting, and empowering marginalized communities. For the 2018 Innovators, focuses ranged from empowerment through mental health services and substance abuse treatments, to financial literacy services, art therapies, and recreational boating. The populations served by these nonprofits have been disproportionately impacted by the effects of COVID-19 due to the intersection of systemic issues such as class, poverty, and institutional racism.

It was truly an inspiration to hear how the Innovators have gone above and beyond to make sure they can best serve their communities. ACT Lawrence, which aims to help families become financially stable through coaching and resources, spoke about the volatility of need in the face of COVID-19. They realized they would need to start helping families find free food and guide them through filing for unemployment, even though these tasks weren’t originally part of their services. And this is only one example. As Innovators exchanged commentary and tips surrounding their journeys of evaluating progress during the pandemic, we picked up a few key takeaways.

Takeaway #1: Both internal and external data are important to consider for determining focus areas and evaluating impact.

Fathers’ Uplift, which provides mental health and substance abuse programs to aid fathers and their relationships with their families, noted that externally sourced data were critical for shaping their goals during the pandemic. Their staff had reviewed studies indicating that increases in unemployment rates lead to increases in suicide rates, so when the pandemic and ensuing unemployment spike hit, they doubled down in their efforts to provide remote mental health resources. They also understood that, during this time, their internal data would become even more important for evaluating their impact. They made more use of digital tracking of client forms rather than handwritten forms, allowing for consolidated data collection and greater accessibility. Fathers’ Uplift additionally began to provide new services like grocery and clothing delivery and maintained a tracking process for this in spreadsheets.

Analysis Group has been working with Fathers’ Uplift since March 2019, when we began helping them analyze data from their existing surveys. After our initial analyses, we then worked together to revamp their surveys, optimizing data collection to focus on measures that would quantify the impact. Since then, we have assisted with impact reports for donors, as well as the creation of a master dataset to track all data collected for individual clients over time. Like other Innovators, Fathers’ Uplift spoke about the challenges of pivoting to provide groceries and clothing to fathers in need when that had not previously been a part of their services. As an organization partnering with Fathers’ Uplift, we saw how hard they worked to swiftly adapt, all while continuing toward their original goals, such as a grant application due in July that we helped them prepare. We are inspired by their continued, successful use of data to chart their path and track progress.

Takeaway #2: As nonprofits shift focus to adapt to community needs, it is helpful for funders to remain flexible and patient as the organization pivots to evaluate new initiatives.

Leslie Dominguez-Santos, Director of Development at GreenRoots, explained how the measurement and metrics processes for planned programmatic work had to become a lower priority as the organization immediately shifted towards crisis intervention. GreenRoots, which provides community-based environmental justice work in Chelsea and East Boston, immediately jumped into action by convening community stakeholders in what became a daily call to organize the community emergency response. When their Board President identified early statistics of high coronavirus infection rates in Chelsea, they convened public health officials and elected officials to jointly write a letter to the Governor which resulted in additional COVID19 testing capacity, community-wide food distribution, and National Guard assistance. Organizational funders were able to contribute by relaxing previously restricted funding allocations, and allowing for increased expenditures on WiFi capabilities and other necessary technologies for their staff to work remotely. The necessity of flexibility was echoed by other Innovators throughout the event.

Takeaway #3: Surveys can help measure or quantify non-numeric impacts or experiences to tell compelling stories.

As a firm dedicated to providing data-driven consulting work to clients, Analysis Group is no stranger to the uses of data and quantifying the seemingly unquantifiable. Lisa Thurau, Executive Director of Strategies for Youth, discussed how data have been critical for her organization to demonstrate the effectiveness of their police officer training and its resulting impact on officers’ responses to the trainings, as well as rates of juvenile arrests and, to a lesser degree, police use of force. Lisa also mentioned Analysis Group’s past work with Strategies for Youth, in which we helped design surveys that captured how community youth feel about their interactions with the police. The results allowed Strategies for Youth to inform and improve their police training by communicating and quantifying trends in attitudes and experiences of the youth.

Takeaway #4: Data is more powerful with context. Seek to combine numbers with personal stories for maximum impact.

While data and metrics are immensely powerful, the Innovators’ stories also emphasized the importance of context and the power of individual stories when used in combination with numbers. The Political Asylum/Immigration Representation (PAIR) Project, which provides legal services to asylum seekers, discussed the challenge of accurately capturing their powerful impact with just internal data. Analysis Group has seen this type of question across many Innovators, and in 2018 worked with PAIR to help craft surveys to measure different types of outcomes, such as language acquisition and community involvement, for their clients. This helped them frame their impact in different ways. However, it can still be hard to accurately collect self-reported data from communities who face many risk factors. PAIR noted at the event that, due to factors beyond their control, such as government agencies and political influence on the court system, their numbers may look lower than previous years in terms of clients who have successfully achieved asylum. As a result, these numbers may not accurately reflect their efforts, influence, or true impact. PAIR has overcome this challenge by emphasizing that each individual success is hugely important and can dramatically change lives, as demonstrated by personal success stories that help paint a more specific, colorful picture of the nonprofit’s work. This has helped funders understand how critical and impactful their contributions are.

Learning Together

The Barr Foundation spoke about how the work of nonprofits and their partners requires a core value of humility. If we are going to find true solutions, then we need to learn in close proximity to the communities in need. We also must acknowledge that we will have to take risks in order to find these solutions, and will inevitably make mistakes to learn. Overall, the event was a great space to reflect broadly on the path the 2018 Innovators are on towards effectively quantifying their impact, as well as to learn about the steps they are taking to adapt and respond to the pandemic crisis at hand. Analysis Group looks forward to continuing our relationship with the Social Innovation Forum to help its Innovators harness the power of data collection and metrics to quantify their impact on stakeholders in various ways. Analysis Group will be hosting a workshop on the use of surveys for impact evaluation in August. We hope to see you there!

About the Authors

Natalie Fox and Natalie Bina are Senior Analysts in the Boston office of Analysis Group, a global firm that provides data-driven economic and strategy consulting work to Fortune 500 companies, government agencies, and leading law firms. They also serve as pro bono coordinators to drive and maintain Analysis Group’s in-kind partnership with Social Innovation Forum and its Innovators.

Learn more about Analysis Group’s pro bono work here.