In October 2023, UK Research and Innovation and the Economic and Social Research Council announced £100 million of funding for Understanding Society – with Minister of State at the Department for Science, Innovation and Technology, George Freeman, saying “Understanding how the UK lives and changes over time is key to ensuring services like healthcare, transport and education best serve the public and that hardworking taxpayers’ money is spent wisely”.
A National Audit Office report in 2021 on government spending said that only 8% of major government projects are robustly evaluated, with 64% not being evaluated at all. With the UK government spending something in the region of £1 trillion a year, though, it’s important to know how well different policies and programmes are working.
To do so requires accurate figures, but also a long-term perspective – which governments may not have – and an interest in people’s daily lives, and how they’re affected.
That has been the thinking behind our ongoing project to encourage evaluation of policies using our long-term panel data. To that end, we have produced six case studies showcasing how the data have been used to assess policies as diverse as the ‘bedroom tax’, automatic pension enrolment, and same-sex marriage legislation. We have also recruited a number of Fellows, some of whom have submitted proposals to evaluate, for example, the right to flexible working and shared parental leave.
Before and after
One of the advantages of longitudinal data is that researchers can assess impact by looking at people’s lives before and after a policy was implemented. The first of our policy evaluation case studies summarised how research used Understanding Society data from 2009 to 2015 to look at housing tenants who received housing benefit in order to look at the ‘under-occupancy penalty’ before and after its introduction in 2013.
What quickly became known as the ‘bedroom tax’ reduced housing benefits for social tenants deemed to have a ‘spare’ bedroom. It aimed to cut the costs of social housing, and to match households to properties better.
The research found that the housing benefit bill fell by around £350 million, but that this was only 70% of the projected savings – and the government was forced to spend an extra £60 million in discretionary payments to protect people from the impacts of the policy.
Those affected saw their income fall by around 3.5% – a figure the researchers described as “not negligible for such a low-income group”. The policy had some success, then, but it was distinctly qualified.
Schools good and bad
One big policy challenge for government is the law of unintended consequences. Ofsted inspections were introduced in 1992 with the aim of improving educational standards across the country. However, research published in 2023 shows that when an Ofsted report shows school quality is improving, parents reduce the time they spend supporting children with school work. Children are likely to increase the time they spend on their homework, partly to compensate.
The researchers linked Understanding Society data with school performance tables from the Department for Education, school census data, and Ofsted data – benefitting from the fact that both school inspections and Understanding Society interviews can take place in any month of the (academic) year. These results, then, were random and independent of one another, creating a very strong control group for comparison. Unfortunately, progress on linking survey and administrative data is slow, and research like this is far from routine as a result.
Vaccines and the feel-good factor
Of course, some types of unintended consequence can be a good thing. The COVID-19 vaccination programme simply aimed to reduce severe disease (in the form of hospitalisations and deaths), but also led to a significant improvement in mental health for people who were most at risk of these outcomes. This was probably down to the reduced fear of becoming seriously ill, combined with the benefits from loosening self-imposed social restrictions. There was no material effect on the mental health of the younger groups of people who were least at risk – and this was also the group least likely to seek vaccination. The researchers suggested a campaign to encourage them, perhaps in the form of financial incentives, in order to achieve herd immunity.
Lockdowns and exercise
Something few of us miss about the pandemic – which vaccines thankfully helped to do away with – was lockdowns, and our data show the effect that different periods of lockdown had on exercise for different groups. Activity levels held up from the beginning of the pandemic to September 2020, despite the first lockdown, but there was a marked decline in activity between September 2020 and January 2021, with a drop from 43 to 33% of people achieving the Chef Medical Officer’s recommended levels of physical activity.
People aged 74+ had more marked decreases in activity levels to January 2021 than those aged 65-73 – although they had lower levels of activity even before the pandemic. The researchers concluded that these reductions in activity put older groups “at risk of becoming deconditioned and developing adverse health outcomes”, and suggested government action to promote physical activity in older adults to reverse these effects.
Pensions and health equality
Sticking with the subject of older age, another example of unintended good consequences: the introduction of automatic pension enrolment after the 2008 Pensions Act has reduced a previously unmeasured mental health gap.
Before automatic enrolment, people with poor mental health – particularly men – were more likely to work for employers who were less likely to offer a workplace pension scheme. Men with poor mental health working in the private sector were 3.7% less likely to take part in a workplace pension scheme, while female private sector employees with poor mental health were 2.9% less likely to participate. Understanding Society allowed the researchers to eliminate other factors which might have explained the gap.
After the legislation kicked in in 2012, the mental health disparity in pension participation disappeared.
Same-sex marriage
The results of policies may also be affected by factors outside the scope of the legislation. The aim of the Marriage (Same Sex Couples) Act 2013 was to equalise the rights of gay and lesbian people – and we might expect that to have a natural effect on mental health for people whose lives are changed by it.
Research using Understanding Society showed that mental health improved for all sexual minorities when marriage equality was introduced, but the improvement was biggest for men who also reported good levels of family support.
This was another policy evaluation that benefitted from the longitudinal nature of the data, using Waves 3-7 of our main survey, covering 2011-17, before and after the legislation. The improvement in mental health was lowest among sexual minority men who had previously reported below-average levels of family support. The mental health of sexual minority women also improved, but family support was less of a factor.
Why Understanding Society?
The sheer range of questions we ask – on education, employment, health, happiness, money, family life, and politics, among other subjects – means researchers can use our data to explore how policies touch on some very different areas of life.
Our data has been used to examine:
- NHS staff ‘burnout’
- which policies might work best to tackle climate change
- the effect of Universal Credit on mental health
…and the Treasury used our data in its analysis of the government’s support for people who couldn’t work during the Covid lockdowns. Our current call for Research Fellows has invited further applications for policy evaluations. The deadline is 6 December 2023.
In the meantime, we’re planning our next research springboard on the topic of children’s lives. Contact us if you are interested in learning more.
Further reading: Raj Patel, Associate Director of Policy and Partnerships at Understanding Society on why we need a long-term evaluation culture
Authors
Chris Coates
Chris is Research Impact and Project Manager at Understanding Society
