Monday, September 11, 2006

Rankings Watch

A reader of this blog, Richard Holmes, has been keeping a close track of the latest worldwide university rankings (Shanghai Jiao Tong, Newsweek and of course, the now infamous, THES rankings). He's written some insightful stuff on his blog on the methodology used by the different organizations that compile these rankings. I'd like to highlight some of his analysis here.

Firstly, a comparison of the Newsweek and THES Top 100 finds that many of Asian universities that were present in the THES are absent or experienced significant falls in the Newsweek rankings. Ditto for many of the Australian and European universities. Why this discrepancy? From Richard's post here, we find:

"The Newsweek ranking is also notable for what it leaves out. It does not include the THES peer review which accounted for 50 per cent of the ranking in 2004 and 40 per cent in 2005 and the rating by employers which contributed 10 per cent in 2005. If we compare the top 100 universities in the THES ranking with Newsweek’s top 100, some very interesting patterns emerge. Essentially, the Newsweek ranking tells us what happens if we take the THES peer review out of the equation."

"So what is going on? Basically, it looks as though the function of the THES peer and employer reviews was to allow universities from Australia, Europe, especially France and the United Kingdom, and Asia, especially China, to do much better that they would on any other possible measure or combination of measures."

A possible underlying reason on why universities from these regions (Australia, Europe and Asia) were ranked higher on the THES ranking compared to the Newsweek ranking? Richard gives one:

"Is it entirely a coincidence that the regions that are disproportionately favoured by the peer review, the UK, France, China and Australia, are precisely those where QS, the consultants who carried the survey, have offices and are precisely those regions that are active in the production of MBAs and the lucrative globalised trade in students, teachers and researchers?"

It seems that one of the linchpins of the THES ranking system is the peer review score. QS, the organization which does the compilation of the results, have been very reluctant to release detailed information in how they obtain these peer review scores. Richard, doing some digging on this own, has unearthed some of the methods used by QS to obtain their sample size for the peer review.

He refers to two websites - one from the Special Astrophysical Observatory of the Russian Academy of Science and the other from the University of Auckland, New Zealand. Both sites tell us something more about how QS solicits and entices respondents to answer the QS survey. Richard comments:

"But the most interesting thing is the bit at the top of the Russian page. The message was addressed not to any particular person. but just to "World Scientific Subscriber" . World Scientific is an online collection of scientific journals. One wonders whether QS had any way of checking who they were getting replies from. Was it the head of the Observatory or some exploited graduate student whose job was to check the e-mail? Also, did they send the survey to all World Scientitific subscribers or just to some of them or only to those in Russia or Eastern Europe?"

"So now you know what to do if you want to get on the THES panel of peer reviwers. Subscribe to World Scientific and, perhaps, a few other online subscription services or work for an institution that does. With a bit of luck you will be recognised as a real smart person and get a chance to vote your employer and your alma mater into the Top 300 or 200."

It was interesting to read the following from the University of Auckland website:

"We note that the Times Higher Education Supplement’s ranking of The University of
Auckland rose from 67th in 2004 to 52nd in 2005. But we note also that this survey establishes its rankings by appealing to university staff, even offering financial enticements to participate (see Appendix II). Staff are likely to feel it
is in their greatest interest to rank their own institution more highly than others. This means the results of the survey and any apparent change in ranking are highly questionable, and that a high ranking has no real intrinsic value in any case. We are vehemently opposed to the evaluation of the University according to the outcome of such PR competitions."

Every ranking methodology has its own shortcomings. By understanding better how these rankings are compiled, we can better understand why some universities are present or absent in these rankings, why some of them move up, why some of them move down, how some aspects of these rankings can be 'gamed' or manipulated, and so on.

I highly doubt that our local universities will drop out of the Top 200 in the 2006 THES rankings. Given that QS representatives have made several visits to Malaysia and the region to market their services, it is likely to be in their interest to 'maintain' a least a few Malaysian universities in the top 200. Hopefully, the new VCs in UM and UKM would be less prone to the disease of 'bannertidis', which we saw in the previous UM VC.

A good university would be able to stand on its own feet, whether or not that is reflected in such worldwide rankings. That is what our local universities should be working towards, instead of short term shortcuts to 'game' the system and obtain a higher ranking.

I'd like to thank Richard Holmes for pointing me out to his website and his efforts at illuminating us as to the methodology of the THES ranking system.

9 comments:

Anonymous said...

In the end its money that counts....

Anonymous said...

THES is political too...tho their conclusions cannot be taken as absolute but there are elements of validity.

What is important is not what THES or Newsweek think or rank us...its more comparing our positions to the benchmarks of our regions such as NUS, Univ Hong Kong
Univ Tokyo. Are we at the same level with them or have we fallen?

We all know the answers..


If we have fallen, have those in authorities carried out post mortems why we have fallen and to put the blame on who? Trouble is in our country, if a scandal happens, no heads roll..No one is penalized

Anonymous said...

I think this blog should be MANDATORY readings for all those in the Ministry of Higher Education

Anonymous said...

Very nice analysis. well done to you all. I think I shall send this to the UM USM and UKM VCs. They seem slightly smarter than the others and might be able to figure out its implications.

Anonymous said...

To be totally honest, I'm kind of bored of reading about university rankings. These rankings, as subjective as they claim to be, will never be completely so and will always be prone to abuse. I spent 2 years in medicine in UM and then went abroad to do the whole course all over again in Britain, in a University not ranked highly at all in various surveys, albeit having a long and grand history. I still believe that the course taught here is much better not only in content but also in the holistic appreciation of the patients and the ability to use expertise of the different departments and the associated medical staff, something which i think is severly lacking in the curriculum back home. Also, the course structure here seems to be more didactic and not so much spoonfeeding. And all this from a university not even ranked in most surveys of universities. At the end of the day, I wouldn't pay too much attention to these ranking systems.

Anonymous said...

Isn't Richard Holmes a CNN Asia news anchor?

Anonymous said...

No! Richard Holmes is the brother of Sherlock Holmes the detective

Anonymous said...

..heard from de vine ..after some noon red vine :) that very soon colleges and universities will be graded A, B or C

..wonder what they mean

could it be...

A = all bad
B = better than worst
C = could have been better

..just a silly joke; hope not sensitive..

Unknown said...

wiht mention at above, is it UTAR recognize by malaysia MOE? because i have one sister will going enrol UTAR. i afraid that after she graduate, the certificate is not recognize and hard to get job in singapore,and there is news from newspaper, one guy who graduate from UTAR and apply for police force. and UTAR certificate is not recognize by malaysia goverment. is it true?

where is the media convince that UTAR is recognize by goverment? is stated in MOE malaysia?