Aveek Bhattacharya is Chief Economist at the Social Market Foundation, a non-partisan think-tank
Does English higher education have a problem with “low quality degrees”? The Government certainly thinks so. In July Michelle Donelan, the Universities Minister, suggested that young people had been “taken advantage of” by purveyors of ineffectual courses, and last year’s Conservative manifesto explicitly pledged to tackle the issue.
Such a project faces two problems, however. First, claims of university underperformance are based on narrow and limited measures of the benefits of a degree. Second, even if we accept that low-quality courses are out there, developing the tools to identify them would involve substantial costs and regulatory burdens.
The idea that many universities are failing their students appears to come primarily from data on graduate earnings. For example, a 2019 release from the Department of Education said on one in ten courses, 75 per cent of students were earning less than £25,000 five years after completion.
Yet it is just not enough to evaluate university courses solely on the financial gains they generate for students. Even without even getting into charges of philistinism and neglecting knowledge for knowledge’s sake, the amount of money a person makes is a simply an imperfect guide to their social and economic contribution. The logic of this assessment produces perverse conclusions: measured on an earnings basis, institutions and courses producing highly paid bankers are many multiples better than those producing nurses and teachers.
It is also questionable how fair it is to judge universities on the basis of graduate wages that are shaped a host of factors beyond their control: macroeconomic policy, labour market regulation, shifting consumption patterns, the business cycle, and international trade flows, to name a few. Graduates tend to stay close to the town where they studied: if there is a downturn in a local industry to pushes down earnings, why should that count against the nearby university?
In fact, at present the Teaching Excellence and Student Outcomes Framework (TEF), the main official measure of higher education quality, does not make use of earnings data at all. It focuses instead on graduate employment, drop-out rates, and self-reported student satisfaction.
However, the future of the TEF is unclear. The 2020 iteration was suspended. An independent review, which is believed to have recommended substantial changes, was due to have been published in summer 2019 but has still not emerged. The data sources underpinning much of the TEF are also being replaced or reviewed.
So the Government needs a new way of evaluating universities, and in particular of identifying supposedly low-quality degrees. If ministers recognise that focusing on graduates’ economic outcomes is too narrow, they could adopt a more holistic measure that accounts for the full social value of a degree, such as the proportion of graduates who work in public services or areas of skills shortage. Measurement might also include non-economic benefits, such as life satisfaction, health and community participation. Universities UK, an industry body, is in the process of developing just such a measure.
As with earnings, though, it is unclear how much responsibility universities should bear for such outcomes. In which case, the natural question is why we don’t judge universities on the thing everybody agrees they are there to do: help their students learn. We don’t, for the most part, judge how well a school is doing by asking what students do when they leave. We look at their exam results (adjusting for differences in students’ background and attainment) and send inspectors to visit the school to assess its teaching.
Unlike schools, we cannot compare university degrees in terms of student results because institutions set their own assessments, reflecting often very different courses. We cannot, to use a well-worn example, assume that top marks from Oxford University reflect the same level of achievement as top marks from Oxford Brookes.
The Quality Assurance Agency, which is nominally responsible for safeguarding standards, does little to ensure comparability because it does not assess teaching directly, but rather checks institutions’ own quality management processes. Though university departments invite external examiners to assess their programmes against subject benchmarks, this process has been criticised as too informal and lacking ‘teeth’.
To increase comparability, we could have some shared exam questions, or a centrally-pooled, institution-blind marking system for at least some assessments. That would be a major departure from the status quo, though. so alternatively we could introduce a separate standardised test, in the way that the OECD’s PISA programme compares school students from different countries with different exam systems. In fact, the OECD has floated the idea of a higher education equivalent to PISA, but the UK opted not to participate in 2015. At the time, the Government claimed to be working on its own measure of ‘learning gain’, but the idea appears to have been dropped.
How about a university equivalent of Ofsted (perhaps a more hands-on QAA), directly inspecting universities and examining how teaching operates first-hand? The idea has been mooted, and indeed FE colleges already are regularly inspected by Ofsted. From 1992 to 2001, universities faced Teaching Quality Assessments, closer to the inspection model with outside academic assessors carrying out site visits and interviews.
Proper quality assessment, though, can’t be done on the cheap. The Government spends £130 million a year inspecting schools. The last TEF cost around £3 million. If the Government really believes low quality degrees are such a pressing problem, it should invest more in the processes necessary to identify them. That would come at a cost to taxpayers and divert more of academics’ time and energy away from teaching and research. Alternatively, the Government may need to just trust universities a bit more.
A sensible national conversation about the quality of degrees needs a proper system of assessment. That won’t be simple or cheap, but it can be done if the will is there. Time for critics of “low-quality” degrees to put up or shut up.
This article draws on the Social Market Foundation’s briefing paper, ‘Elusive quality: how should we evaluate higher education?‘, published today.