The Trouble with Citations

For the past two years the Times Higher (THE) has published a league table of the top 100 universities under 50. This is a spin off of its more established World University Rankings. Whilst I accept that we live in a league table world, and have joked about this in the past on this blog (‘Almost Everyone Can Score Well at Something’), I do have a niggling worry about this latest manifestation of the genre.

My worry comes from a concern about the methodology the table uses. THE understandably lays less store in the subjective ‘reputation’ of these young upstarts (given that they’ve had less time to develop laurels on which to rest), and concentrates instead on ‘hard, objective ­performance indicators’:

• Research: volume, income and reputation (30 per cent)
• Citations: research influence (30 per cent)
• Teaching: the learning environment (30 per cent)
• International outlook: people and research (7.5 per cent)
• Industry income: innovation (2.5 per cent).

The trouble is that ‘hard, objective performance indicators’ are never quite as hard and objective as you think.

Take, for example, citations. These have increasingly become the gold standard for research reputation over the last few years. With the rise in the h-index and the g-index, and the ease with which we can access these through Scopus, Google Scholar and elsewhere, citations have become shorthand for ‘excellent research.’ Indeed, the science of citations, bibliometrics, came close to being used as a proxy for research excellence when HEFCE was deciding on the shape of REF2014.

So it’s understandable that THE has chosen to give almost a third of its weighting for its league table to citations. Its understandable, but I don’t think its acceptable.

See, if you dig just a little below the surface, you find that citations are riven with difficulties, not least of which is the gender gap that underlies them. It’s only recently that some quantitative analysis has been undertaken to assess this. The first time I became aware of such an example was when Daniel Maliniak, Ryan Powers and Barbara F. Walter published their findings on The Gender Citation Gap in International Relations in International Organization in October last year.

Looking at over 3,000 articles published between 1980 and 2006, the authors showed that ‘women are systematically cited less than men’. On average, men are cited one and a half times as often as women, getting a mean citation rate of 18.47 compared to 13.65. Maliniak et al offered two essential reasons for this:

• that women tend to cite themselves less than men;
• that men tend to cite men more than women.

Going deeper, they suggest that this could be due to the following:

• that there are ‘informal networks’ underlying the citation culture. ‘In a field numerically dominated by men, men tend to cite men, and women tend to cite women, resulting in disproportionately fewer citations for women’;
• second, that ‘women may publish less in the early years of their career as a result of their need to take parental leave.’ As citations are, to a degree, dependent on profile and recognition, it may just be that women have not had the same opportunities to climb as high, and are reliant on a smaller body of publications;
• thirdly, in International Relations, there is a (broad) gender divide in topics: men tend to write on security, US foreign policy and methods; women on human rights, comparative foreign policy, health, international law and the environment. These topics may be less popular and well cited.

Sugimoto et al focussed on this issue in Nature in December:

‘We find that in the most productive countries, all articles with women in dominant author positions receive fewer citations than those with men in the same positions. And this citation disadvantage is accentuated by the fact that women’s publication portfolios are more domestic than their male colleagues — they profit less from the extra citations that international collaborations accrue.’

They concurred with one of the conclusions of the earlier article, that ‘it is likely that many of the trends we observed can be explained by the under-representation of women among the elders of science.’

But, as The Specials said, it doesn’t make it alright. However, the situation might not be as gloomy as it appears. The IR Blog reported on an interesting session at the American Political Science Association Conference in 2013 at which Beth Simmons suggested that there may be a glimmer of hope:

• the citation gender gap seems to be narrowing. This was also picked up in the Washington Post’s Monkey Cage blog.
• the effect may be more extreme in the States (Maliniak et al were focused on there), and that it may be less pronounced elsewhere;
• and – hell! – people now know about it, so we could and should act on it.

Which brings me back to The Times Higher. Concentrating on a bald indicator like citations is retrogressive. It positively encourage universities to focus on the citation rates of its staff. By so doing it exacerbates inequality within academia, entrenching an unrepresentative and unhelpful system that may give a very broad sense of research excellence, but in essence just demonstrates what self belief and lucky breaks an academic’s had.

Photo by Shane on Unsplash