Accurately ranking university innovation
Impact, innovation and the whole process of turning research into reality have an ever-growing role in university life yet remain overlooked by global rankings. Should the rankings adapt to reflect this?
Written by Adam Stoten, Chief Operating Officer, Oxford University Innovation
Oxford is the world’s number one university three years running, according to the Times Higher Education (THE) World University Rankings. Obviously, with Oxford University Innovation (OUI) being an integral part of the Oxford machine, this was excellent news for us. It also reflected a strong performance by UK universities in general, with Oxford, Cambridge and Imperial all in the top 10, and, of course, the results confirmed the THE as the only rankings that really matter (for this year at least).
Tempting as it was to sit back on the shoulders of Oxford’s many giants, I felt compelled to better understand the methodology that goes into generating the rankings. In particular, to what extent do OUI’s activities in supporting impact contribute to the overall result?
The weightings make for interesting reading. Teaching, Research and Citations each comprise 30% of the overall score. Of the remaining 10%, International Outlook constitutes 7.5%. Finally, sneaking in with a mere 2.5% is Knowledge Transfer.
The methodology defines this as “A university’s ability to help industry with innovations, inventions and consultancy” and confirms it “has become a core mission of the contemporary global academy”. All well and good, but how do you — or rather the THE analysts– measure Knowledge Transfer? The metric used turns out to be how much research income an institution earns from industry. This figure is then adjusted for purchasing power parity (as for other income metrics in the methodology) and scaled against the number of academics the university employs.
Compare this with the definition of impact used as part of the Research Excellence Framework, the exercise by which UK university research quality is assessed; “..an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia”. The contrast is striking; the THE ranking definition very narrow, the REF definition very wide. Now clearly we are comparing apples and pears, but if the Knowledge Transfer category in the THE ranking aims to capture how good a given institution is at transferring its knowledge, then surely part of that equation must be an assessment of the impact said dissemination has in society — the “so what?” question.
In the world of knowledge transfer, institutional income from industry is a valid and important metric and is certainly a good indicator of the ability of an institution to engage with the private sector. However, it is only part of the picture, and to put all our eggs in that basket risks ignoring other very important — and impactful — forms of knowledge transfer. For instance, while OUI spins out around 20 new companies every year based on research outputs, we are witnessing an increasing number of projects proceeding down a social enterprise route, with a social mission written into their corporate DNA (i.e. Articles of Association). These include diverse offerings such as a validated index to help corporations and governments understand if and how they are contributing to poverty amongst their people (sOPHIa), and a smart phone-based game to help train healthcare workers in Africa to respond to paediatric emergencies (LIFE). These are projects that may be hugely impactful but may well not return huge amounts of money to the University.
It is also worth noting that for the majority of universities the main source of commercial income will be sponsored research, rather than revenue derived from more conventional downstream knowledge transfer activities such as IP licensing and spinout creation. This is a very important activity for universities, especially where industry is able to pay the full cost of research, but again it potentially distorts the picture.
There are also issues in terms of timing. As with most offices supporting technology transfer, we are focused on long term value creation associated with early stage, high risk technologies and IP. This means that a) most of the licence deals we conclude or spinouts for which we secure investment will never yield a product due to high risk of technical failure, b) even when we get lucky and a product makes it to the market, very few of these ever return really significant quantities of money, and c) the very occasional big payday typically comes many years after the point at which the technology transfer activity took place, and so doesn’t really reflect the current ability of an institution.
If the above picture sounds a little depressing, it shouldn’t. If technology transfer offices can’t take a punt on potentially world-changing technologies when they are in their infancy (or even, to stretch the analogy, neonatal) then who can? We expect most of our technologies to fail for many different reasons (technical, changing market, left-field competition) but we shouldn’t forget that along the way our licensees and spinouts build further on the transferred knowledge, develop new products, employ people, use 3rd party services and at some point might even pay taxes. Likewise, our academic consultants contribute to commercial product development programmes, training of industry personnel and many other activities across a wide range of sectors. The point being that the economic impact is much wider than simply how much cash a university receives from industry, and hence why Oxford and many of its peers have commissioned Economic Impact Assessment reviews to better understand and communicate this. https://www.ox.ac.uk/about/facts-and-figures/economic-impact?wssl=1
So should THE consider a more inclusive approach to Knowledge Transfer metrics? Ideally, almost certainly yes. While the fundamental mission of most universities remains teaching and research, their ability to generate impact in society from both through knowledge transfer and entrepreneurial activities is of increasing importance not only within governments but also amongst prospective staff and students. In terms of the former, the UK government has set itself the goal of increasing the percentage of GDP spent on R&D to 2.4% by 2027 — bringing us in line with the 2014 OECD average — with a longer-term goal of hitting 3%. This will only be achieved through more investment in public sector research that translates into growth in private sector R&D. For staff and students, the ability and reputation of a university to support translational and entrepreneurial activities is a material factor in where they choose to work or study; in fact, a colleague at UC Berkeley’s technology transfer office told me earlier this year that the main reason they exist is for staff retention purposes, such was the extent of entrepreneurial culture amongst their academics.
Practically, however, there are of course challenges. Accessing a metric that is both fully inclusive of knowledge transfer and can be reliably derived for universities not just in the UK but globally is difficult, and perhaps impossible. It is also noteworthy that two of the other major global university rankings (QS World University Rankings and Academic Ranking of World Universities) do not allocate any portion of the overall score to knowledge transfer activities, so arguably THE is already ahead of the game. Nonetheless, in the absence of any reliable metric that truly captures downstream impact, I would advocate re-focusing on numbers of transactions that reflect a genuine transfer of knowledge, and which could include IP licences, spinout and social enterprise creation, academic consulting agreements and commercially sponsored research — all which include both direct and indirect benefits and which I would argue are better predictors of future societal benefit.
Unless of course this would jeopardise our number one spot, in which case just leave well alone.
