Here is a situation that will be familiar to a lot of organisations that support young people in schools.
Teachers say the young people on your programme have grown in confidence. Parents describe children coming home “buzzing”. Young people tell you, unprompted, that they want to come into school now. And then you open the spreadsheet, run the pre/post analysis on your validated wellbeing scale, and the number barely moves.
That was roughly where Power2 found itself. Power2 Thrive is a 16-week programme in which young people aged 11–16 mentor nursery-age children, alongside classroom learning and an NCFE qualification. Power2 has used the Warwick–Edinburgh Mental Wellbeing Scale (WEMWBS) since 2021. When we analysed 490 young people with matched before-and-after scores, the average change was +0.5 points, which is small, and not statistically significant. Thirty-nine per cent improved meaningfully, 35% declined, 27% stayed roughly where they were.
The temptation in that moment is to reach for one of two bad conclusions: the programme doesn’t work, or the data doesn’t matter. The below describes our attempt to work through that with the team in an honest and rigorous way.
Firstly, nothing lies like an average
Cohort averages are very good at hiding things, and the first step is almost always to disaggregate that in some way.
When we split the Power2 Thrive cohort by initial wellbeing level, the pattern changed completely. Young people who began in the lowest wellbeing band gained an average of more than four points, and 42% of them moved up a wellbeing level. There was a programme wellbeing benefit, and it was concentrated among the people who arrived with the greatest need. This diluted to near-zero once averaged with everyone else.
Then ask whether you’re measuring the right thing
The deeper issue wasn’t the average. It was the fit between what the programme supports and what the scale is built to detect.
This is where our mixed method study paid off. We had two quantitative data sources for outcomes: what teachers observed, and what young people reported on WEMWBS. In this cohort they were essentially unrelated and correlated at less than 0.15. Teachers were emphatic about a specific cluster: 83% of young people improved in confidence, 82% in communication, 80% in self-esteem, 75% were less shy or withdrawn. The qualitative fieldwork said the same thing in different words: young people said they felt they were capable of things they previously couldn’t do.
This relates to self-efficacy: a person’s belief in their own capacity to handle situations and succeed at tasks. WEMWBS measures general mental wellbeing and measures it well. Self-efficacy is a related but distinct construct, and WEMWBS was never designed to isolate it.
The finding wasn’t “the programme isn’t working”. It was “the tool isn’t measuring the thing the programme is best at changing”.
Why we didn’t recommend dropping WEMWBS
The next step could have been that WEMWBS was the problem and we should replace it. We didn’t think that, for three reasons that will apply to most organisations in this position.
It is recognised. Funders know WEMWBS, several require it, and it functions as a form of currency in bids and eligibility criteria. It is general, but that’s its strength.
It enables alignment. Power2 runs Thrive alongside other programmes, and a common measure lets the organisation compare across its portfolio rather than holding a set of incompatible datasets.
It is a time series. There are years of WEMWBS data behind this. Switching instruments doesn’t just change your next report it breaks your ability to compare it to the last five.
A measure that undersells one programme can still be the right measure for an organisation. So we proposed adding an additional measurement tool. However, we wanted to be cautious about over-burdening young people, and so we needed something that would not be too difficult to run alongside WEMWBS.
What we landed on: NGSE alongside WEMWBS
We assessed five candidate measures and recommended the New General Self-Efficacy Scale (NGSE).
It is eight items, free and open access and it has been validated with young people from low-income and first-generation backgrounds — a genuine overlap with Power2’s cohort, rather than a scale borrowed from an adult workplace population. It has been found to have higher construct validity than the original General Self-Efficacy Scale, which came a close second on our ranking. And run together with WEMWBS it produces 22 items in total, which is manageable.
The practical outcome is that Power2 can now put a specific self-efficacy change score in front of a funder, next to a WEMWBS score, and have the numbers (hopefully) say what the teachers, the parents and the young people were saying during this evaluation.
Four questions worth asking about your own measurement
- Before you conclude “no significant change”, have you disaggregated by starting point? Who is your average hiding?
- Is your headline measure pointed at the construct your programme actually changes — or at a broader one that contains it?
- What does your existing measure provide in funder recognition, internal comparability and historical continuity? Would replacing it cost you those?
- What is the honest total item count that a young person faces, and can you manage or stagger that?
Stories of Change conducted an independent mixed-methods evaluation of Power2 Thrive, combining fieldwork across three school settings with analysis of programme data from 666 participants. [Contact / link to full report / author byline to be added.]
The Power2 Thrive programme is delivered by Power2, a children and young people’s charity helping schools and local authorities across the North West, London, Yorkshire and the West Midlands respond to an escalating social, emotional and mental health crisis and the growing number of school absences.
