Why most NGO impact measurement is technically true and practically meaningless, and what honest outcome reporting and real transparency actually require.
Read enough annual reports from the social sector and a pattern starts to show itself. The numbers are large, the numbers are round, and the numbers always go up. Nobody is lying. Every figure could probably survive a spot check by an auditor with a stack of attendance registers. And yet, having read the report, you know almost nothing about whether anyone’s life is materially different. This is the quiet problem at the centre of NGO impact measurement: most published figures are technically true and practically meaningless, and a great many people working in the sector know it.
This is not a story about fraud. It is a story about what is cheap to count. Counting people who attended a workshop costs almost nothing, because you already have the register. Finding out what those same people were earning a year later costs money, staff time, working phone numbers and a tolerance for bad news. Given a reporting deadline and a limited budget, organisations reach for the number they already have. Repeat that decision across a sector for a couple of decades and you get the impact reporting culture we currently have, where activity is presented as achievement and nobody quite has to say so.
The purpose of this essay is to set out the mechanics of how that happens, without pointing at anyone, and then to describe what honest outcome measurement would look like instead. The second half matters more than the first. Criticism of the sector’s reporting habits is easy and, at this point, fairly common. Doing the harder version, and publishing it where funders and partners can interrogate it, is rarer.
Almost every failure of social impact reporting can be traced back to a confusion between three different kinds of number. They are not interchangeable, but they are routinely printed side by side as though they were.
Inputs are the resources that went in. Budget disbursed, staff hired, centres opened, equipment purchased. Inputs tell you an organisation was funded and spent the money on something. They tell you nothing at all about whether the spending worked. A programme can spend its entire budget correctly, with clean books and a satisfied auditor, and achieve nothing.
Outputs are the activities the money produced. People trained. Workshops held. Toolkits distributed. Sessions delivered. Villages covered. Outputs are the workhorse of the sector’s reporting because they have three convenient properties: they are cheap to count, they are available immediately, and they are always flattering. There is no such thing as a disappointing output number, because an output measures effort rather than result. If you ran forty workshops, you ran forty workshops. The figure cannot turn against you.
Outcomes are the only category that describes a change in someone’s life. People earning more than they were before. Businesses still trading after eighteen months. Children still enrolled in school at the end of the year. Workers who stayed in a job past the sixth month rather than returning home after the first. Outcomes are expensive, slow, and capable of embarrassing you. That combination explains their scarcity in published reports far better than any theory about bad intentions.
The test is simple enough to apply to any report you are handed. Take each headline number and ask whether it describes something the organisation did, or something that changed for a person. If the whole page describes things the organisation did, you are reading an activity report wearing the vocabulary of impact.
Of all the numbers in circulation, reach is the one that does the most damage, because it sounds like an outcome and behaves like an output. An organisation reports that it reached two hundred thousand people. What does that mean? It could mean two hundred thousand people completed a year-long programme and were tracked afterwards. It could equally mean two hundred thousand people walked past an awareness stall, or received a text message, or sat in an assembly hall while somebody spoke.
The word does no work. It has no threshold, no minimum dose, no definition that anyone is obliged to honour. And because it has no definition, it is unfalsifiable, which is precisely why it survives. Two organisations can report the same reach figure while doing work that differs by an order of magnitude in intensity and cost.
Being reached by a programme is not the same as being changed by one.
That sentence is the whole argument. A number that cannot distinguish between the two is not measuring impact; it is measuring exposure, and exposure is a marketing concept. When you see reach quoted as a headline, the useful follow-up question is not how many, but what happened to them, and how would you know.
Suppose an organisation does the honourable thing and runs a follow-up survey a year after a training programme. It calls the people who went through the course and asks what they are doing now. The results come back encouraging. Most respondents are working, many report higher earnings, and the programme looks like a success.
Here is the difficulty. The survey did not reach everyone. It reached the people who could be reached. And the people who can still be reached a year later are not a random sample of the original group. They are disproportionately the people for whom things went well.
Think about who becomes hard to contact. Someone whose number stopped working because they could not keep paying for it. Someone who migrated for work in distress and changed circuits entirely. Someone who felt the course had been a waste of their time and has no interest in a call from the organisation that ran it. Someone whose small enterprise closed and who would rather not discuss it. Failure is quieter than success, and it is quieter in exactly the way that removes it from your dataset.
This is survivorship bias, and in livelihood work it is severe. If you survey the sixty percent of a cohort you can still reach and report their results as the programme’s results, you have not measured the programme. You have measured its most successful fraction and quietly relabelled it as the whole. The correction is not complicated, though it is uncomfortable: report the people you could not trace as a category of their own, and never as a success.
A young woman completes a three-month course and finds work two months later. The programme records a placement. But she is capable, motivated and actively looking, in a district where some hiring was happening anyway. Would she have found that job without the course? Possibly. Possibly not. Possibly she would have found a worse one, later, and the honest description of the programme’s contribution is that it made the search shorter and the job better, which is a real and valuable thing, and is not the same as claiming the entire outcome.
Attribution is the hardest problem in outcome measurement and there is no cheap solution to it. Rigorous methods for isolating a programme’s specific contribution exist, and they are genuinely expensive and genuinely difficult to run well. Most organisations, including most good ones, cannot deploy them at scale.
What every organisation can do is stop pretending the problem does not exist. Reporting that a certain number of people were employed twelve months after a programme is a defensible statement of fact. Reporting that the programme created those jobs is a claim about causation that the data usually does not support. The gap between those two sentences is where a great deal of the sector’s inflated arithmetic lives. Writing the first sentence instead of the second costs nothing and immediately makes a report more credible to anyone who knows the difference.
None of this persists because people are dishonest. It persists because the incentives point that way, consistently, year after year.
Funding relationships are built on growth. A funder who supported a programme last year would like to see the numbers rise this year, and the organisation knows it. Renewal conversations, internal targets and board reviews all reward an upward line. Now consider what that does to a programme manager who has a genuine, well-documented decline to report. Perhaps the labour market in one district softened. Perhaps a cohort was weaker. Perhaps something in the design stopped working and needs rebuilding. Reporting that honestly is the professionally dangerous option, even when it is the truth, and even when it is the single most useful piece of information the organisation has generated all year.
Nobody has to falsify anything for this to distort the output. There are enough legitimate choices available. Choose the more generous definition of reach. Measure at course completion rather than a year out, when enthusiasm is highest. Emphasise the cohort that did well. Present a cumulative total in a year when the current-period figure is flat. Every one of those decisions is defensible in isolation. Taken together they produce a report that is true in every particular and misleading as a whole.
Some tells are easy to spot once you know to look for them.
The alternative is not a research institute. It is a set of disciplines that a working organisation can adopt, most of which cost more in courage than in money.
Outcome metrics selected after the results are in will, without anyone intending it, be the metrics that make the results look good. Fixing them in advance, and publishing them in advance, removes that freedom. It also forces a useful conversation at the design stage about what the programme is actually for.
A defensible standard is to record employment status and income at a set point after a person leaves the programme, twelve months being a reasonable choice in livelihood work. Twelve months is long enough for the first job to have been kept or lost, for a small enterprise to have met its first real difficulty, and for early enthusiasm to have been tested by reality.
Every outcome figure should carry the population it came from. How many enrolled, how many completed, how many were surveyed, how many could not be found. Treat untraceable as unknown, never as success. An organisation that reports a lower headline number because it refuses to assume the best about the people it lost is telling you something valuable about the rest of its numbers.
Say how the survey was conducted, by whom, at what interval, with what response rate and what known limitations. Methodology is what allows a reader to disagree with you, and a number nobody can disagree with is not evidence of anything. This is the practical meaning of transparency in impact reporting: not more numbers, but numbers that can be checked.
A report that includes a programme that underperformed, a district where the model did not transfer, or a cohort where placement collapsed is more useful and more credible than a page of triumphs. It is more useful because it is the only part of the document anyone can learn from. It is more credible because an organisation willing to publish its failures has demonstrated something about its treatment of its successes.
Report this period’s performance in its own right, then the cumulative history separately and clearly labelled. Both are legitimate. Merging them is the trick.
The usual case for honest reporting is ethical, and it holds: donors, CSR partners and government programme partners are entitled to know what their money did. But the stronger argument is self-interested. An organisation that reports only flattering numbers has blinded itself. It cannot tell which of its districts is working, which curriculum is worth expanding, which employer relationship produces jobs that people keep. It has no error signal, and without an error signal there is no improvement, only repetition at increasing scale.
The funding environment is moving in this direction anyway. Indian CSR rules require certain companies to spend on social programmes and to report on what they funded, which has put a class of partner in the market who must be able to defend the claims they publish. That partner increasingly wants defensible, auditable outcome data rather than a glossy impact claim, because the glossy claim is the one that becomes a liability when someone asks how it was calculated. Organisations that can show their working are becoming easier to fund, not harder. Anyone assessing a potential CSR partnership on the strength of its reporting discipline is asking the right question.
There is a corollary worth stating plainly. A funder who cannot handle a bad number is not a good long-term partner. The relationship in which every report must improve on the last is a relationship that will eventually require someone to shade the truth, and the organisation, not the funder, will carry that. Better to establish early that the reporting will be accurate in both directions.
Lakshmi Foundation works in Jharkhand, a state in eastern India with twenty-four districts and its capital at Ranchi. We describe our work in a specific way: we connect Jharkhand’s talent to opportunity through skills, mentorship, and access to employers, markets, and capital. The framing matters. Jharkhand is not a place that needs rescuing. It is a place with a great deal of ability and not enough access, and access is a solvable problem.
We are building a livelihood ecosystem rather than a training institute, which changes what we are obliged to measure. A training institute can reasonably report on training delivered. An organisation that claims to connect people to work has to report on the work. Skill opens the door. Livelihood is what walks through it. Dignity is what stays. If our numbers only describe the door, we have not reported on the thing we said we do.
So we have made a set of commitments about measurement, and we would rather be held to them than praised for them. We publish outcomes rather than activity. We publish methodology alongside every number, so that anyone can interrogate how it was produced. We report what did not work, because that is the part of the record with any teaching value in it. Our approach to how we define and publish outcome data is set out in full, and it is written to be argued with.
One consequence of this is visible on our own site. While our measurement systems are being established and our first cohorts reach the twelve-month mark, parts of our impact reporting carry placeholders rather than figures. This is deliberate and we are not apologising for it. Publishing an unverified number to fill a space is how the habits described in this essay begin. A placeholder is honest about the state of our evidence in a way that an estimate would not be. When we publish figures, they will carry their denominators, their attrition, their untraceable cases and their method, and readers will be able to see how performance differs across the districts we work in rather than a single state-level average that hides the variation. The same principle governs how we describe our governance, board oversight and financial reporting.
The sector does not need more impact numbers. It needs fewer numbers that mean more. That means outputs described as outputs, outcomes measured long after the certificate, denominators printed beside numerators, untraced people counted as unknown, methodology published for anyone to attack, and failures reported alongside successes because they are the more instructive half of the record.
None of this is technically difficult. It is culturally difficult, because it requires an organisation to publish a number that might disappoint someone, and to trust that the partners worth having will respect it more for that. We think they will. And even if some do not, an organisation that measures itself honestly gets something no report can supply from the outside: it finds out what is actually working, while there is still time to do more of it.
Why digital skills training rural India often fails, and what actually turns connectivity into income: honest sequencing, phone-first teaching and real work.
What really blocks women livelihood Jharkhand outcomes is not motivation but access: distance, timing, safety, childcare, documents and invisible work.
Why rural entrepreneurship India needs real support: markets, working capital, mentors and follow-up, not a business plan template and a one-day workshop.
Back a training seat, mentor someone through their first interview, or open a hiring conversation at your organisation.
