How do people experience impact?

Research across Myanmar, Bolivia, Belgium, the US and the UK – with all kinds of different people – has shown that various things really matter to us when we think about the impact we experience when engaging with an enterprise:

  • We generally want to participate in defining the impact that enterprises have on our lives.
  • We may describe impact in terms overcoming challenges or achieving certain outcomes. Whether framed as a challenge or an opportunity, we have a view about which products, services or jobs would be more effective than others. We also broadly know how vulnerable, disadvantaged or underserved we are in relation to diferent types of outcome – for example, being healthy but poorly educated – and feel improvements in underserved areas of our life are more impactful than in aspects where we are already well-served.
  • Many of us agree that a product, service or opportunity that has a positive impact is most impactful if it is the only available option. It does not immediately lose its value if something else meets the same need but becomes less impactful if it is aggressively competing with other products or services that offer the same (or greater) benefit.
  • It matters whether or not a product, service or job creates the change we were looking for. We are often willing to take higher risks that our experience will differ from our expectation when the potential benefit is greater

Understanding how people (and the planet) directly experience impact is the foundation of impact management.

What is our shared understanding of impact?

In finance, we have developed shared fundamentals, which include risk, return, liquidity, volatility – and asset classes, which group investments with similar characteristics – to describe, align and manage our respective financial goals. For impact, we now have a shared understanding that impact refers to material effects experienced by people and planet, both positive and negative – and that to understand and manage impact, we need to consider five dimensions of any effect:

  1. What

    What outcome, positive or negative, the effect drives and whether it is important to the people or planet experiencing it. We may decide that an outcome is important based on our own opinion, or guided by professional experts, or through shared consensus like the Sustainable Development Goals or Social Progress Index.

    We may relate our experience of secondary education to our ability to have financial security and we may think that financial security is a positive and important outcome.
  2. How Much

    How much of an effect occurs by considering its significance:

    • How big a driver the effect is of the outcome (Depth)
    • How many people the effect occurs for (Scale)
    • How long the effect lasts for (Duration)
    Data may show that secondary education is a big driver of financial security or that a statistically significant number of people perceive secondary education to contribute ‘a lot’. It may show us that 200, or 2 million, people have experienced the effect. It may (in the form of feedback data from alumni) tell us that students’ ability to secure a job with the skills they gained from a training programme has endured for at least 5 years.
  3. Who

    Who experiences the effect and whether they are underserved in relation to the outcome(s).

    Since different effects can lead to the same outcome – and it’s the outcome that matters to us – this is what we focus on when thinking about whether a population is under- or well-served.  Do the people (or planet) have the opportunity to experience the outcome that the effect relates to?

    We may be a farmer’s son in Nigeria and be more or less underserved when it comes to our ability to have good health; or a girl who speaks English as a second language in the United States, with more or less access to financial security. For environmental issues, we may be thinking about certain species and how near or far they are from endangerment.
  4. Contribution

    The contribution that the effect makes to what is likely to occur anyway, based on whether the effect:

    • Leads to more or less important positive or negative outcomes, and/or
    • Is more or less significant (in terms of depth or the number of people it occurs for or how long it lasts for or how long it takes to occur) and/or
    • Occurs for people (or planet) who are more or less underserved than those currently experiencing it
    For example, a new health service might contribute the same amount to our physical health as a service we are currently using but contribute less to negative greenhouse gas emissions for the planet. It might be likely to contribute more or less significantly to our physical health than if we continue to use our current service. It might reach other people who do not currently have access to the service at all.
  5. Risk

    The risk that the effect is different from our expectation.

    For example, an innovative health service might make our physical health worse when we expected it would make it a little better – but it could also make our health significantly better than expected. Or, it might make our health better but lead to negative effects for other people or the planet.

Do we have to manage everything that people and planet experience?

No – impact refers to material effects experienced by people and/or the planet.

Material effects are those that relate to important positive or negative outcomes (what), are significant (how much) and occur for groups of people and/or the planet who are not already well-served in relation to the outcome (who).

Impact management involves trying to prevent material negative effects and may involve trying to make positive effects more material, depending on our intention.

For example, a business may assess that their current mix of employees does not reflect the diversity of the community in which they operate. They might decide that this is a material negative effect because, as a large business in the local community, they have a significant effect (how much) on inequality (what) among the local population (what). The same business may assess that their healthcare product for women is not having a material effect as these women will experience only a marginal effect (how much) on their physical health (what) and are easily able to afford alternative treatment to achieve the same level of good health (who). The business may or may set goals to have a more material effect with their product.

What information do we need to understand impact?

The five dimensions of impact guide the data that we collect, analyse and assess, so that we can understand our impact and improve (or re-set our goals). The impact experienced by people and the planet cannot be understood or benchmarked if we only have data about one or two dimensions out of context of the others. We therefore try to collect data as a ‘set’ across the five dimensions of impact, alongside data about financial performance.

For example, imagine two investment opportunities, both aiming to improve educational outcomes for kids by increasing the student transition rate from one level of school to the next. Both provide counselling to 11-16 year olds and they even work in the same city. But Investment A is a proven intervention to re-motivate teenagers who are skipping school, while Investment B is testing a new approach to helping teenagers with learning disabilities. A’s transition rate shoots up from 50% to 85% and the investment generates attractive financial returns, while B’s moves from 50% to 65% and the financial return doesn’t justify the risk taken. If you benchmark what A achieves versus B (the student transition rate and the financial risk-adjusted return), what have you really learned? That A is more efficient than B – and resources should therefore be diverted away from B? Or just that B’s goal was to take a higher level of risk to impact a harder-to-reach demographic? By comparing these two opportunities without data across the other dimensions – for example data on who is being affected – we would diminish our impact. It could lead us not to make that investment B at all, pulling resources away from our toughest problems.

Over time we might be able to build up enough evidence to illustrate how a diference in one or two dimensions of impact affects the result across other dimensions but, for now, information is useful for managing impact if kept in the context of all dimensions.

When do we collect information to understand our impact?

We collect information about the experience of people and planet on an ongoing basis – because it informs our decisions throughout impact management. This information guides our intentions in the first place (e.g. what do people or the planet tell us they want and need and what should we therefore aim for?) and our choice of goals (e.g. what material effects is an enterprise having or likely to have and what effects should we therefore set goals to try to prevent or increase?) – both of which guide how we deliver and improve impact.

Since the initial impact goals we set are often based on incomplete information and are different to the impact that people and planet go on to experience, we often have to redesign how we deliver impact or re-set our goals based on information about what people actually want. We will only can only do this if we’re collecting information about impact on an ongoing basis.

Who bears the cost of collecting information about impact?

For example, information about the demographics we reach (who), an understanding of the significance of effects we have on someone’s life (how much) and a benchmark of our performance against what the market is likely to do anyway (contribution) all help us to tailor, improve and differentiate our products or services, or the quality of our employment.

Much of the data we collect to understand the five dimensions of impact is data that is also commercially useful.

This means that much of the collection of information about our effects on people and the planet can form part of a general management approach and the cost is a legitimate use of capital from investors or funders. However, when the field as a whole will benefit from a deeper understanding of an enterprise’s impact (for example, if looking to replicate what looks like a successful model), philanthropic or government sources can play an important role in helping fund studies. We might choose to do a deeper study of impact to understand what impact achieved can be directly attributable to the enterprise, or whether customer feedback is sufficiently unbiased, or how the enterprise interacts with a wider system.

What is ‘good’ information?

There are some rules of thumb that help us to identify good information:

  • Completeness: consider whether the data is good quality, and whether it has been collected from a range of relevant stakeholders.
  • Accuracy: consider whether the information has been substantiated by a counterfactual, and if it has been subject to external review or was the product of an independent assessment.
  • Relevance: consider whether the information directly corresponds to the same impact goals across all the dimensions of impact.

When we examine an existing evidence base, the more numerous the individual sets of information that demonstrate these features and share the same findings, the stronger your evidence base.

These rules of thumb can be converted into what are known as ‘Standards of Evidence’ to help us determine what level of confidence we can have that the information we collect is telling us about impact.

  1. At a minimum, we collect activity or output data that existing sources of information tell us are a good proxy for change in effect, i.e. there is existing evidence that the activities or outputs translate into the effect in sufficiently comparable circumstances.
    For example, counting a child that has received a vaccination against measles as a proxy for that child not developing measles.

    We also recognise that people and planet will likely experience other impact too, both positive and negative and therefore draw on existing information about the material effects that comparable approaches delivered in comparable contexts to help us judge other effects we might want either to mitigate (if negative) or increase deliberately (if positive).

    For example, by consulting existing research on the effects of offshore wind farms, we might learn that some marine animals experience negative impact due to the noise and regular disturbance. This information helps us to dedicate resource to trying to mitigate these negative effects.
  2. We collect direct feedback from people experiencing effects, which may serve as a more reliable proxy for change in effect because there is existing evidence that people’s feedback is strongly correlated with the desired effect (or lack of effect), without circumstances needing to be comparable.
    For example, maternal health clinics have tracked that 470 mothers received treatment in 2008, 89% of whom safely delivered a healthy child and 97% of whom reported satisfaction with the service.
  3. Using a control, or comparison, group helps to establish causality (where possible)
    For example, a survey of a similar group of mothers in a neighbouring region reported a 65% safe, healthy delivery rate in the same period, while the maternal mortality rate is 43% below the regional average.
  4. An independent evaluation helps to confirm (or contest) the assessment, including removal of bias, and often enables a greater understanding of the outcomes experienced.
    For example, an evaluation shows that local schools report better average attendance and attainment rates for the children born since provision of the service.
  5. Being clear about how context affects the delivery of these outcomes enables consistent replication.
    For example, there are multiple drivers of positive maternal health outcomes – such as access to good nutrition or a sanitary delivery environment – and people in different contexts will be underserved in different ways in relation to this outcome. This will influence the effectiveness and/or applicability of post-natal care.

How do we share information with others?

Enterprises collect information that enables them to make decisions about whether and how to improve performance or re-allocate resources. When this information across all dimensions is communicated as a “set”, other organisations relying on that information (such as investors, funders or policymakers) can use it to make decisions and improve their own allocation of resources, helping them compare ‘apples to apples’ when looking at the performance of different enterprises.

Enterprises who share information about their impact performance publicly help everyone to improve. As we understand what different people want and need – and which enterprises do and do not work – we contribute to existing bodies of information (including our own). By contributing to existing bodies of information, we either reinforce the validity of existing information (does the information we have support what others have found?), or challenge it. This helps us all to set and re-set our goals and design or identify approaches with the greatest likelihood of success.

How do we share information about a portfolio of many diverse enterprises?

Sometimes there are too many sets of information for an organisation to analyse and use for decision-making. For example, a passive asset owner may have a range of investment managers through whom they invest who, taken together, are investing in hundreds of underlying and diverse enterprises. The asset owner will typically not want or have the time to process data sets from every underlying enterprise. We therefore need to be creative about how we summarise impact performance, without losing its value for decision-making.

There are different approaches to tackling this issue:

  1. An investor may choose to assess an enterprise’s impact performance on each dimension and convert the assessment into a quantitative score or rating. Scores of individual enterprises can then be summed to provide a view of portfolio-level performance. Sometimes investors use conversion to socio-economic value (expressed in monetary terms) to express performance, which can also be summed to show the performance of the portfolio as a whole.
  2. An investor may choose to assess an enterprise’s impact performance on each dimension and convert the assessment into a quantitative score or rating. Scores of individual enterprises can then be summed to provide a view of portfolio-level performance.
  3. An investor may choose to conduct due diligence on the strength of impact management processes. For example, they might ask questions about how an investment manager has built impact management into their governance, incentives, processes and systems. Being transparent about what our impact management process involves is therefore very helpful for everyone and we anticipate accredited assessments of this kind in future.
  4. An investor may choose to sum data that is comparable across enterprises and relates to one or two dimensions of impact only – for example the “total number of low-income people accessing services that relate to good health and well-being”. This sort of information can give a ‘flavour’ of the effect achieved but it does not give the whole picture. Since individual metrics like this typically includes a mix of performance across the other dimensions (for example, the depth of improvement in people’s health), it cannot help us to learn whether one enterprise, or portfolio of enterprises, is performing better than another, and therefore improve. When investors or funders want this kind of data for communication (rather than improvement) purposes, they can ideally extract it from the data sets that enterprises collect anyway, so that enterprises can always focus their limited resources on collecting data sets that enable learning and improvement.

There is a growing number of resources that enable investors and funders to collect and analyse either data ‘sets’ (to support their investees and improve their allocation of resources), or to score and sum performance, or to assess the strength of an organisation’s impact management processes. See the relevant resources highlighted below.

Relevant Resources

Latest

Bringing impact management to life