Breaking ranks

©Faber

University league tables are fashionable, but should not be taken as accurate measures of the quality of education. The OECD is investigating other tools to measure performance, policymakers and educators heard at a recent conference.

Rumour has it that the two biggest fans of university league tables are vice-chancellors and mothers in China. Do league tables accurately measure what they claim to measure? Or are they, as their critics claim, simplistic and damaging?

Excellence is the quarry. The trouble is, excellence only seems to appear in the top ranks. Surely no university can be “world class” in every aspect. Franz Van Vught of the European Centre for Strategic Management of Universities put the problem succinctly at a recent conference called “Higher education: quality, relevance and impact” held at the OECD in Paris (see references). He said that if only 3% of universities out of 17,000 worldwide can be considered “world class”, it cannot mean that the rest are failures.

Exhortations to become world class have tucked universities into a Procrustean bed of indicators. Presidents anxiously cut back programmes, reorient their university’s mission, swell application numbers to tighten student selectivity, and seek mergers with higher-ranking institutions–conversely, those higher up jealously guard their hard-won reputations and shy away from collaborating with anyone but their peers. Deep excisions may be made into the social sciences and humanities to leave more room for the natural sciences and research. Speaking at the biennial conference of the OECD’s Programme on Institutional Management in Higher Education (IMHE) in September, Ellen Hazelkorn of the Dublin Institute of Technology, cited a respondent to an international survey which said that “the easiest way to boost rankings is to kill the Humanities”. Clearly not a realistic proposition, though another survey told her that “reputation, unfortunately, is always based on research, and research attracts the best talent”.

Research is considered the most salient example of a nation’s intellectual resources, economic strength and global competitiveness. It is no surprise then that real research rather than just teaching and studying for exams figures highly in two of the world’s most influential league tables. The Shanghai Jiao Tong University (SJT) gives research a thumping 40% weighting, and includes the number of faculty publications and citations in journals. The Times Higher Education rank gives research 20%. But the Shanghai rank also attributes another 40% to faculty who have won Field Medals and Nobel Prizes. This is a bit distorting, since not every university or college focuses on maths or science, and even if they do, there is no clear evidence that having a Nobel Prize winner on campus now, still less having had one fifty or more years ago, helps students learn. The Shanghai survey cautions against using its tables as an overall assessment of a university’s performance. But that, of course, is exactly what people and the headline-hungry media do.

Short of hiring a Nobel Prize winner, what can a university do to improve its standing in these rankings, assuming they should try? Alas, the best way to gain prestige is to already have it.

The old schools like Harvard and Oxford are consistently in the top ten. Their mythic status draws the highest-calibre students and faculty, and guarantees generous endowments. Does reputation give them an unfair advantage and make them hard to dislodge?

Not necessarily. Other established universities as old or older than Oxford, such as Freiburg in Germany, figure much lower in the Shanghai top 100, while Italy’s Bologna, one of the oldest of all, does not even figure.

Reputations may be based partly on assumption. In its peer review weighting, the Times Higher Education queries some 200,000 academics worldwide via email, asking them to name what they feel are the top thirty institutions. With no clear indicators as to what they should evaluate, compounded by the fact that the Times Higher Education is published in English-speaking countries and that the average response rate is 1%, it is hardly a surprise when British institutions come out on top. Also, it depends on what is being measured: while neither Shanghai nor Times rate French universities that highly, in the FT 2007 European Business Schools rankings six of the top 10 are French. Incidentally, the Financial Times offers quite a range of rankings of business schools, MBAs and the like, as much to earn revenue from the “infotainment” business as anything else.

There seems to be a whimsical element in the rankings. A university high in the ranks one year may suddenly plummet forty or more places in the next. Is this possible? If so, what is to prevent it from rising fifty places the following year? Nothing, one may say. Also, there may be very little difference between institutions placed many tens of places apart, while small changes in the underlying factors have dramatic impacts. So why pay attention to rankings at all? To underline the point, the 2008 Times Higher Education survey just published as we write, reduced Britain’s rankings quite markedly across the board, leading to more gloomy headlines about the state of the country.

To squeeze out of such indicators, some universities are turning the tables upside down, using their lower position as a marketing strategy. Student mobility is higher than ever before in history. In most OECD countries, the international student body is about 6.7% of the total; in Australian universities it is 19.3%, and in some as high as 50%.

One of the biggest problems with rankings is not the accuracy of the data, but how that data is used, and to whom it is applied. Another is that a good deal of data has never been collected, how students are progressing in the classroom for instance, or levels of students who leave compared to when they joined. This makes it hard to build a full picture of university performance. The most serious omission is that certain data leaves out what for many people is the very reason universities exist: actual coursework.

Indicators such as “teacher/student ratio” are too feeble to tell us much about learning outcomes, nor anything about a teacher’s ability to teach. The Centre for Higher Education in Germany (CHE) publishes a variety of data from which students can construct their own rankings, according to their needs. The CHE does not stamp them with a number but places them in categories of “good”, “medium” or “bad”, and lists them alphabetically. Universities in one category are of comparable quality, whereas those in different categories show a marked contrast in performance. Unlike rankings, this method prevents trivial differences from creating the false impression that one university is clearly better than another.

Rankings also overlook the “value added” component of degree programmes. As might be expected, top universities draw A+ students and turn out A+ graduates. But what of universities that accept B students and produce A-level graduates? The added value of the B-student’s degree programme would be considerably higher.

The OECD Assessment of Higher Education Learning Outcomes (AHELO) study aims at determining whether it is possible to make meaningful statements about the education provided in universities in different countries, taking into account different “strands” of competence: skill in a chosen discipline, and generic skills such as critical thinking or the ability to apply knowledge practically. If successful, AHELO will provide institutions with analysis to help them improve their own performance, and will provide data that will help students assess the suitability of the institution for their own needs.

Relevance is important. Is a degree programme relevant if it fails to prepare a student for the job market or demonstrates no obvious social impact? In a world increasingly nervous about shrinking job opportunities and faced with other challenges, the relevance of a degree is paramount, and the OECD conference discussed this point at some length. But while some may argue say, that natural sciences are more relevant than social history, as Robert Berdahl, President of the Association of American Universities, pointed out, the problems of contemporary society–migration, ageing, climate change, the legacy of colonialism and religious extremism–cannot be solved by the natural sciences alone. Nor is judging what is relevant just a matter for any one generation to decide. Look at the nuclear industry, which is back in favour as a source of energy, yet is faced with skill shortages because of years of unpopularity when students chose other subjects. Or what about today’s stock market crash, which appears to be rewriting the rules of economic orthodoxy by the day?

But rankings are not going to go away. After all, they can provoke useful questions, such as “why exactly are we not in the top group?”, or indeed, “how can we maintain this lead?”. Governments and universities will still use them, as fierce competition between universities induces copycat behaviour unless policy encourages diversity. Whether they serve as an accurate guide to higher education is therefore strictly academic.

Even people that do not like rankings cannot always resist them. Take this story, related by a woman from Germany at the IMHE conference. When she asked the vice-chancellor of a university in Bavaria whether he would like his university to be included in a league table, his response was a firm “no”. But then, when the woman succeeded in reassuring him that the university would not be included, he asked–just out of curiosity–where it would have been ranked compared to the others. LT

References

©OECD Observer No 269 October 2008




Economic data

E-Newsletter

Stay up-to-date with the latest news from the OECD by signing up for our e-newsletter :

Twitter feed

Suscribe now

<b>Subscribe now!</b>

To receive your exclusive print editions delivered to you directly


Online edition
Previous editions

Don't miss

  • “Nizip” refugee camp visit
    July 2016: OECD Secretary-General Angel Gurría visits the “Nizip” refugee camp, situated between Gaziantep and the Turkish-Syrian border, accompanied by Turkey’s Deputy Prime Minister Mehmet Şimşek. The camp accommodates a small number of the 2.75 million Syrians currently registered in Turkey, mostly outside the camps. In his tour of the camp, Mr Gurría visits a school, speaks with refugees and gives a short interview.
  • OECD Observer i-Sheet Series: OECD Observer i-Sheets are smart contents pages on major issues and events. Use them to find current or recent articles, video, books and working papers. To browse on paper and read on line, or simply download.
  • Queen Maxima of the Netherlands gives a speech next to Mexico's President Enrique Pena Nieto (not pictured) during the International Forum of Financial Inclusion at the National Palace in Mexico City, Mexico June 21, 2016.
  • How sustainable is the ocean as a source of economic development? The Ocean Economy in 2030 examines the risks and uncertainties surrounding the future development of ocean industries, the innovations required in science and technology to support their progress, their potential contribution to green growth and some of the implications for ocean management.
  • OECD Environment Director Simon Upton presented a talk at Imperial College London on 21 April 2016. With the world awash in surplus oil and prices languishing around US$40 per barrel, how can governments step up efforts to transform the world’s energy systems in line with the Paris Agreement?
  • Happy 10th birthday to Twitter. This 2008 OECD Observer interview with Henry Copeland said you’d do well.
  • The OECD Gender Initiative examines existing barriers to gender equality in education, employment, and entrepreneurship. The gender portal monitors the progress made by governments to promote gender equality in both OECD and non-OECD countries and provides good practices based on analytical tools and reliable data.
  • Once migrants reach Europe, countries face integration challenge: OECD's Thomas Liebig speaks to NPR's Audie Cornish.

  • Message from the International Space Station to COP21

  • COP21 Will Get Agreement With Teeth: OECD Secretary-General Angel Gurría on Bloomberg

  • The carbon clock is ticking: OECD’s Gurría on CNBC

  • If we want to reach zero net emissions by the end of the century, we must align our policies for a low-carbon economy, put a price on carbon everywhere, spend less subsidising fossil fuels and invest more in clean energy. OECD at #COP21 – OECD statement for #COP21
  • They are green and local --It’s a new generation of entrepreneurs in Kenya with big dreams of sustainable energy and the drive to see their innovative technologies throughout Africa. blogs.worldbank.org
  • Pole to Paris Project
  • In order to face global warming, Asia needs at least $40 billion per year, derived from both the public and private sector. Read how to bridge the climate financing gap on the Asian Bank of Development's website.
  • How can cities fight climate change?
    Discover projects in Denmark, Canada, Australia, Japan and Mexico.
  • Climate: What's changed, what hasn't, what we can do about it.
    Lecture by OECD Secretary-General Angel Gurría, hosted by the London School of Economics and Aviva Investors in association with ClimateWise, London, UK, 3 July 2015.

  • Climate change: “We should not disagree when scientists tell us we have a window of opportunity–10-15 years–to turn this thing around” argues Senator Bernie Sanders.

  • In the long-run, the EU benefits from migration, says OECD Head of International Migration Division Jean-Christophe Dumont.
  • Is technological progress slowing down? Is it speeding up? At the OECD, we believe the research from our Future of ‪Productivity‬ project helps to resolve this paradox.
  • Is inequality bad for growth? That redistribution boosts economies is not established by the evidence says FT economics editor Chris Giles. Read more on www.ft.com.
  • Catherine Mann, OECD Chief Economist, explains on Bloomberg why "too much bank lending can slow economic growth".
  • Interested in a career in Paris at the OECD? The OECD is a major international organisation, with a mission to build better policies for better lives. With our hub based in one of the world's global cities and offices across continents, find out more at www.oecd.org/careers .

Most Popular Articles

Poll

What issue are you most concerned about in 2016?

Unemployment
Euro crisis
International conflict
Global warming
Other

OECD Insights Blog

NOTE: All signed articles in the OECD Observer express the opinions of the authors
and do not necessarily represent the official views of OECD member countries.

All rights reserved. OECD 2016