OECD university rankings

Read this in a November copy of the Economist - The OECD is trying to come up with a different way of ranking universities. The results are scheduled to be out in 2010.

I'll reproduce the article in full below and then follow up with some of my own comments.

A new sort of higher education guide for very discerning customers

WORKING out exactly what students and taxpayers get for the money they spend on universities is a tricky business. Now the Organisation for Economic Co-operation and Development (OECD), a Paris-based think-tank for rich countries, is planning to make the task a bit easier, by producing the first international comparison of how successfully universities teach.

That marks a breakthrough. At the moment, just two institutions make annual attempts to compare universities round the world. Shanghai's Jiao Tong University has been doing it since 2003, and the Times Higher Education Supplement, a British weekly, started a similar exercise in 2004. But both these indices, which are closely watched by participants in a fickle and fast-expanding global education market (see chart), reflect “inputs” such as the number and quality of staff, as well as how many prizes they win and how many articles they publish. The new idea is to look at the end result—how much knowledge is really being imparted.

“Rather than assuming that because a university spends more it must be better, or using other proxy measures for quality, we will look at learning outcomes,” explains Andreas Schleicher, the OECD's head of education research. Just as the OECD assesses primary and secondary education by testing randomly chosen groups of youngsters from each country in reading and mathematics, it will sample university students to see what they have learned. Once enough universities are taking part, it may publish league tables showing where each country stands, just as it now does for compulsory education. That may produce a fairer assessment than the two established rankings, though the British one does try to broaden its inquiry by taking opinions from academics and employers.
Click here to find out more!

There is much to be said for the OECD's approach. Of course a Nobel laureate's view on where to study may be worth hearing, but dons may be so busy writing and researching that they spend little or no time teaching—a big weakness at America's famous universities. And changes in methodology can bring startling shifts. The high-flying London School of Economics, for example, tumbled from 17th to 59th in the British rankings published last week, primarily because it got less credit than in previous years for the impressive number of foreign students it had managed to attract.

The OECD plan awaits approval from an education ministers' meeting in January. The first rankings are planned by 2010. They will be of interest not just as a guide for shoppers in the global market, but also as indicators of performance in domestic markets. They will help academics wondering whether to stay put or switch jobs, students choosing where to spend their time and money, and ambitious university bosses who want a sharper competitive edge for their institution.

The task the OECD has set itself is formidable. In many subjects, such as literature and history, the syllabus varies hugely from one country, and even one campus, to another. But OECD researchers think that problem can be overcome by concentrating on the transferable skills that employers value, such as critical thinking and analysis, and testing subject knowledge only in fields like economics and engineering, with a big common core.

Moreover, says Mr Schleicher, it is a job worth doing. Today's rankings, he believes, do not help governments assess whether they get a return on the money they give universities to teach their undergraduates. Students overlook second-rank institutions in favour of big names, even though the less grand may be better at teaching. Worst of all, ranking by reputation allows famous places to coast along, while making life hard for feisty upstarts. “We will not be reflecting a university's history,” says Mr Schleicher, “but asking: what is a global employer looking for?” A fair question, even if not every single student's destiny is to work for a multinational firm.

First of all, I think it's a good alternative to the Shanghai Jiao Tong and the THES rankings in that this OECD ranking tries to measure the 'learning outcomes'. I'm generally in favor of more information being collected and published rather than less and leave it up to the 'marketplace of ideas' to sort out the implications of the different ranking systems.

Secondly, I'm sure that this ranking system will also invite its fair share of critics and criticisms as well. One immediate challenge I can think of in terms of measuring 'learning outcomes' is the difficult of measuring the 'value added' of a university. Obviously, students graduating from Harvard will, on average, know more, compared to students graduating from a community college in the US or students graduating from UM. This is because students who enter Harvard are obviously brighter and know more compared to other students in less known universities. Hence, the measure of 'learning outcomes' of a university should be the 'value added' that it provides to its graduates, rather than the absolute level of 'learning outcomes'.

I recall that Singapore used to measure (not sure if they still do) the value added of each school at the secondary level comparing the PSLE (Primary 6) scores of students entering different schools and their O level grades. This way, one can control for the fact that students going to schools such as Raffles Institution and Raffles Girls School have much higher PSLE scores compared to those students going to the 'neighborhood' schools. I'm not sure if the OECD measure is able to control for such factors. My feeling is that it probably won't and what it will end up measuring is the absolute levels rather than the 'value added' levels.

If this ranking system measures absolute rather than value added levels, I think that graduates from Malaysian public universities are at a disadvantage compared to many of its peers in other countries. Here are a couple of reasons why this is the case:

Good students in Malaysia have many options to study in non-Malaysian public universities because of scholarships (public and private), the determination of many Malaysian parents to save for and fund such studies, the relative proficiency in English of such students and the availability of options in private universities (including many 'twinning' programs). In addition, there are push factors such as the quota system in the public universities which push many good students out of public universities and leaves many weaker students in the public universities.

For most students in developed countries i.e. the OECD countries, their first option is to study in a university (mostly public, with the exception of the US) in their own country because these are seen as credible choices and also where some government funding is provided. This means that universities in such countries are able to retain their best and brightest within its own universities. Rarely would you expect many of the best and the brightest in these countries to study full time in universities in other countries. In Malaysia, the opposite is true. Rarely would you expect the best and the brightest to study in the local university. You would expect these individuals to be studying in a foreign university. Hence, the quality of students at our public universities are 'scrubbed' of many of the best and brightest students (with perhaps the exception being the law and medicine students).

For many students in Asian countries such as Japan, India, China, Taiwan and Korea, their public universities are credible options for their best and brightest to study at. Many Chinese students aspire to study in Beijing or Tsinghua. Many of the 'technical institutes' in India are a breeding ground for the best and brightest in India. Although the proportion of students from these countries going overseas is increasing (especially as China and India become richer), there is still a critical mass of good students going to their public universities, unlike in Malaysia.

A note on Singapore - Many of the best students in Singapore end up going overseas on government scholarships. But Singapore universities are also able to retain a critical mass of good local students in its public universities (apart from the law and medicine students). Let me give you some concrete examples. About half of all students who take A levels in my alma mater (and Tony's as well), Raffles Junior College (RJC), obtain 3 As or more. I think a similar cohort from Taylors or Sunway will probably have at most 20% obtaining 3As or more. Most of these 3As and above students will not get government scholarships to go overseas but go to the Singapore unis i.e. NUS, NTU and SMU. In contrast, most of the top scorers at the A levels in Malaysia ends up going overseas. A much smaller proportion of the student population takes the STPM with most of the tops scorers wanting to take up law or medicine at the local universities. Many of the top students have been 'filtered' out by STPM.

Furthermore, Singapore universities are able to attract good Malaysian students to attend. Many good Malaysians who can't get into their preferred course in Malaysian unis (because of the quota system) end up going to Singapore. I know of two Malaysian students at the PhD level at Duke who went to Singapore unis. At the same time, many of my Asean scholar friends also decide to attend Singapore unis because they don't want to or cannot afford to go overseas.

This is not to say that our public unis don't have good or even great students. The law and medicine students are clearly the exception because of the high level of competition to get into such pro grams. But the proportion is far lower than that in Singapore, not to mention the developed countries or even Asian countries such as China and India.

And I haven't even begun to discuss the learning process in our public universities. If even a sliver of some of the horror stories I've heard in regards to the learning process in our public universities - high absentee rates among lecturers, lecturers who 'dumb down' their lectures or make exams easier, etc... - then I'm quite certain that there might be a 'value decrease' in the impact of our public universities on our graduates.

I don't know if the OECD measure will include any or many of our public universities but I'm quite sure that it it does, our graduates will do worse compared to their compatriots in other Asian countries and most definitely compared to universities in the developed countries.

I think some of the obstacles I've outlined above would be too great for the current MOHE minister, To Pak, to overcome, regardless of action plans and such.