Type “social impact” into Google and you get four billion results. While it might seem like a fairly new term popularized by impact investors and philanthropists in recent years, it was first used in a 1969 Yale University seminar on the ethical responsibilities of institutional investors.
Since then, it has gone from mostly being used to assess potential negative impacts of large government projects to becoming an integral part of the work done by everyone from multilateral development banks and non-governmental organizations to social entrepreneurs and the private sector.
Despite its importance, measuring social impact is far from a straightforward exercise and more than 150 different methodologies have been developed for wide use, according to the Journal of Global Policy. More recently, many social innovators and the philanthropic donors that fund them have started questioning how to meaningfully measure the impact interventions – and funding – are having. To many in the social impact space, the current practice of impact measurement is coming up short, with too much focus being placed on the reporting of outputs such as the number of women reached by a skills training program or the number of jobs created by a startup.
The complex challenges facing the world today require interventions that go beyond band-aid solutions to tackling the root causes of poverty or inequality by actually changing the system that upholds them in the first place. And trying to measure it requires a totally different approach – and mindset.
“Complex issues like systems change are context-sensitive. We
all have intent, but there is no recipe for systems change. So when you’re in a game of systems change, you’re in a game of maybe. We improve the probability that we can get an outcome, but there’s nothing guaranteed,” said Mark Cabaj, an expert in social innovation, strategic learning, and evaluation and the president of the consulting company From Here to There.
H&M Foundation spoke to Cabaj about why many organizations still struggle to measure systems change and why organizations should go beyond just tracking outputs if they want to contribute to lasting change.
The conversation has been edited for length and clarity.
For organizations looking to achieve social impact, it’s easier to collect low-hanging fruit and strive for easily quantifiable impact – such as the number of mosquito bed nets handed out – than to tackle the root causes of poverty or inequality. But why is it important to also measure deeper systems change if more organizations are to take on this challenge?
It’s a classic challenge showing how it’s easier to default to tracking outputs and just assume that something magical will happen. It’s natural behavior ever since we started normalizing logic models and results frameworks. I’m really interested in measurement, but I’m very cautious about measurement because it’s only one small part of evaluation. I think we have obsessive measurement disorder – we follow the principle that it’s really important to focus on outcomes and outputs. The question is, when do you go beyond simply tracking outputs? How do you do it in a feasible way? And how do you actually get measurements that mean something?
“I’m really interested in measurement, but I’m very cautious about measurement because it’s only one small part of evaluation.”
Mark Cabaj
There are even cases when measurement isn’t necessary. Take vaccination campaigns – if you follow a logic model that lays out the outcomes and research has already shown that vaccines reduce COVID-19 cases, you might not have to measure all of the outcomes. Why would you track a first principle there’s already evidence for? Instead, you might want to evaluate the quality of the implementation of the campaign or see which communities you might be missing. We should in principle measure impact, but depending on the strategy and the existing evidence base, it may not be a good use of time and resources.
How can funders who want to see at least some feedback on the impact their money is having approach impact measurement?
The challenge of philanthropy is investing in social innovators and wanting to get some kind of feedback about whether your dollars are making a difference. But it actually takes a lot of expertise and resources to do impact measurement well. When we started doing evaluations in the 1970s and ’80s, we had big think tanks and research institutes trying to track outcomes with a lot of expertise and money.
And then there was this accountability shift and instead of financing these think tanks and having them do the evaluation, we downgraded the responsibility of tracking impact to non-governmental organizations and those doing the work. While NGOs are mostly really good at what they’re doing, they hardly ever get resources to do decent tracking, and almost never have the required expertise. We don’t have the infrastructure for NGOs to properly measure impact and we need to invest in their capability to do so.
I spent my whole life trying to track outcomes. And sometimes I’m so desperate to find out, but there’s actually no feasible way of knowing the impact. But that doesn’t mean I would not invest in an idea that logically seems good. For example, I would invest in almost any clever communication campaign that focuses on the beliefs of those reluctant to admit climate change is real. It’s probably not going to be possible to know the exact impact of that work, but it’s still worth spending money on it and it’s the right thing to do.
You mentioned that measurement is only a small part of evaluation. So what else goes into impact evaluation if done well?
Good evaluation of systems change actually starts by asking questions about strategic learning. Let’s say a farmer growing coffee beans only gets $9 dollars a day, so what are we learning about the systems that prevent him from earning more? Strategic learning is when you intervene, you get feedback on the outcomes, and then you ask ‘What are we learning about the nature of the problem we’re trying to address and the strengths and limitations of our strategy?’.
Most evaluations are not used, and that’s partly because we’re data and measurement-rich, but sense-making and use-making poor. Our sense-making processes aren’t very good and we don’t pay attention to how we facilitate the insights gathered from evaluation to make actual changes because we turn it into a measurement exercise.
“Most evaluations are not used, and that’s partly because we’re data and measurement-rich, but sense-making and use-making poor.”
Mark Cabaj
When I read a report, I have three questions: what is the change, how do they describe it, and is there at least enough sense-making to provoke me into agreeing or disagreeing? Then the next question is, ‘So what now? Tell me the implications of the things you measure.’ That’s when it starts bridging into the zone of usefulness. Without it, it’s just noise.
There’s been a recent shift towards international organizations focusing more on addressing unequal power structures to shift power to local organizations and communities. But how do you measure and quantify something as complex as power?
Tracking change – particularly power and systems change – almost always requires mixed methods, meaning qualitative and quantitative, because there are no narratives without numbers and no numbers without narratives. So one of the challenges of the evaluator when looking at something like shifting power is to ask: ‘What does that outcome look like? What do you mean by power? And how is it shifted?’
There’s this instinct and an impulse in a group to say we want to change power, and the evaluator’s job is to help people make that explicit and say ‘Well, what does it look like? What’s the theory of change?’. So we actually have to help people sharpen up what they mean by these outcome statements, and then we can do something about it. Just saying what you mean by power is a very useful exercise for innovators.
Measuring long-lasting and meaningful systems change might seem like an insurmountable task to many organizations, especially in the beginning. So where do you start?
First, you need to ask what system you’re trying to change. If it’s the health system, for example, how big is that system, is it the health system in Nepal, or across the planet? Or is it just in one city? Then you want to consider the characteristics and try to describe the system as much as possible. How is it behaving now? And how do you want the system to change and behave after your intervention?
“For systems change, you have to be able to handle ambiguity and have a tolerance for risk, and you have to be able to accept failure.”
Mark Cabaj
You also need to consider that systems change happens through little tipping points – slowly, but then suddenly. And everyone wants to be that tipping point, that last snowflake to break the branch. But it’s difficult to know how much of that “tipping” you’re responsible for. Consider your ability to create the conditions for systems change without ever pretending that you will know for certain that you’re gonna be the last snowflake to break the branch, but you can track to what degree you’re adding on snowflakes.
And so when people say I want systems change but I want predictable outcomes, my response is always ‘Get out of that game, you won’t handle it.’ Go take on simple things like vaccination rates, you can handle that. For systems change, you have to be able to handle ambiguity and have a tolerance for risk, and you have to be able to accept failure.