Thursday, November 08, 2007

Letter to Mkini: Criticising the MOHE rankings

Saw this letter in Malaysiakini criticizing the ARES survey carried about LAN on behalf of the MOHE. I agree with most of his criticism but I still think it's a worthwhile first step to take.

I just want to bring out two points in this letter which I don't quite agree with. I'll discuss these points below.

The Academic Reputation Survey (ARES), carried out by the National Accreditation Board (LAN), on 17 public universities (‘Public universities fail outstanding test’, New Straits Times, Nov 3), is a sad reflection of how utterly mediocre our university system is – or rather how close to that, the ARES and the LAN are as tools of public accountability.

The public has been fooled into thinking that this half-baked survey can be taken as a substitute for a ‘ranking system’. The fact of the matter is that it is not a ranking system. It was merely a perceptions survey, without the public being told who exactly were being surveyed. The ARES sent out 954 questionnaires and received 272 responses, supposedly from “higher education institutions, two Asean universities (National University of Singapore and Institut Teknologi Brunei), nine corporate bodies and 19 professional and certification bodies.”

First of all, questionnaires are only answered by respondents (or living individuals) and not inanimate bodies; only individuals perceive, not institutions. This means that we need to know who were the “persons” rather than the “institutions” who responded to the questionnaire. Did they represent the institutions and in what capacity? We need a breakdown of the profiles of these 272 respondents; such as how many were males or females, which institutions do they come from, their occupations, age, educational qualification, etc.

It would make a lot of difference to the credibility of the result (which ‘ranked’ USM as the top university), if, out of the 272 respondents, 70% were from USM itself! Furthermore, what is the breakdown of respondents from the public and the private sectors? If a majority were from the public sector then this could mean a case of government self-praise, rather than independent judgement.

Anyway, statistical validity aside (which happens to be the main problem of this survey), what about “perceptions” then? How much do these individuals know about the universities they assessed on this sophisticated-sounding, Likert-scale (which is nothing more than jottings on a scale from best to worst in answer to the questions of the questionnaire)? It really matters as to who does the “perceiving”.

To a question of whether a postgraduate programme of any university is good or bad - how could the manager of a fast-food chain company, for example, have any idea on how to rate these programmes? How could any of these individuals rate the quality of the academic staff of universities? What yardsticks are they given? I do not want to belabour the need for integrity and precision in statistical methods (or the pitfalls when abused), only to say that this is the crux of the problem of our public universities - even the guardians of their academic standards are short of standards themselves.

It may even be too late to arrest the deterioration of our universities - we are talking about the lives and future of millions of our youths, the country’s human resource, the source of its leadership. I am afraid that the education minister and the vice-chancellors are living in a make-believe world about creating an “empire of the minds” (to quote one VC in his newspaper column).

Here I am talking from actual experience. Just put a Malaysian university student alongside their counterparts from Thailand, Indonesia, Philippines, Singapore and let them all speak, and we shall know that we have done a terrible disservice to our local graduates. Not being able to speak proper English is forgivable, but not being able to articulate ideas and arguments without embarrassing themselves (if they even know of it!) is the graver cost to the reputation (and soul) of the nation.

I do not know what it takes for a wake-up call. I can only say that if it is not now, it is never. And yes, I urge the government to stop the chicanery of devising an internal ranking system. What is the point? You are only listing which of the rotten apples are less rotten, or rather which are the ones having a slower rate of decay. They are all going to get there.

Firstly, I think the writer recognizes that the identity and background of the respondents are crucial in these types of surveys. But he seems to be in two minds about who these respondents should be. On the one hand, he does not think that a majority of respondents should be from either the public universities or the public sector because this will inevitably lead to self-praise, an issue I brought up in my previous post on this issue. On the other, he doesn't think that it is appropriate for a 'manager of a fast-food chain company' to evaluate the quality of our public universities either which I interpret as not wanting too much private sector involvement (or at least private sector involvement of a certain kind).

I don't really have an easy answer for this. Presumably, one could 'solve' this problem by asking more foreigners, especially academics in other universities who are somewhat familiar with the state of research and teaching in our public universities. But this also raises the question of possible 'bias' against the newer and lesser known schools, as well as the possible accusation that we are at the mercy of 'imperialist' or 'neo-imperialist' forces that are trying to bring down the international reputation of our public universities. I frankly think that it is probably better to target a mixture of respondents - foreign academics who are familiar with Malaysia (I can think of many in ANU in Australia), some well respected local academics (Khoo Kay Kim) including some who no longer teach in Malaysia (such as Jomo and Gomez) as well as respected intellectuals and business people who are familiar with the situation in our public universities (such as Zainal Aznam Yusuf, Mohammed Arif Nun, Munir Majid). The problem will be getting enough of these people to respond.

Secondly, I think he probably under-rates our local graduates and over-rates graduates from universities in our neighboring countries (Indonesia, Singapore, Philippines, Thailand). I think if we bring together the best law students, let's say, from these countries, I don't think our local graduates will fare any worse in a mooting competition. I think our best local graduates probably have a better command of English compared to their compatriots from Thailand or Indonesia.

I also think that the average graduate from our public universities is probably no worse in terms of analytical ability compared to the average graduate from our neighboring countries, perhaps with the exception of Singapore, where standards might be more vigorously enforced. This is not to say that our local graduates are great, but rather that they are no worse. Of course, with a better university system, it would be the case that our local graduates will outshine our neighbors in all aspects, including analytical abilities, but that is a long way aways and we might not get there at all if the current situation of academic decline is not addressed.

I think it's too easy too poo-poo the efforts of the MOHE and be critical. I've been tempted many times to do this. But judging from what I see to be a sustained effort on the part of the MOHE, Tok Pa and some VCs in our universities, I will give them the benefit of the doubt for now and see the glass as half empty rather than half full.


Anonymous said...

The present VC of UKM who was the initiator of the LAN accreditation of the local universities is so busy 'praising herself and the method of assessment' even before the public can even give the feedback.

he hehe!

All of them seems to have the ' katak bawah tempurung' syndrome and member of the ' Self Admiration society'

Hidup UNMO! hehe

Anonymous said...

Survey is always the weakest part of any ranking system. It is subjective and open to manipulation, especially when respondents were asked for personal perceptions and not provided a common set of data to help in their decision making.
If Malaysia is truly serious about ranking, use an international review team of academics from overseas. This team will visit every university campuses and be allowed to observe lectures, and provided with any data they choose to see such as actual student exam papers and student grade distribution, average grades of incoming students, graduation rate, faculty profile, publication, etc. Then we will know how we compare to other universities. That process is expensive, but if Malaysia can waste RM100 M for a space tourist, why not spend some money to really gauge the quality of the universities.

Anonymous said...

agreed, the letter to malaysiakini was way too harsh. i dont think our local graduates are any worse than others in the surrounding region. if any wants to see a country that produces unemployable graduates then they should take a good look at the philipinnes. almost everyone theres has some sort of degree and almost all of them work as bus drivers, street vendors and etc. malaysia must not follow this path, our local grads lack exposure and a capacity for independent thought, not intelligence. if the government is really serious these problems can be overcome. let the suffocating control over the unis and let the student think for themselves. oh and by the way singapore unis have mush larger budgets than malaysian unis. nus which is roughly the same size as universiti malaya has a revenue in the region of 2.3 billion ringgit while universiti malaya makes do with about 10 times less. not saying that this is the only reason for the discrepancy between the two unis but money does talk.

Anonymous said...

THES ranking is out. NUS dropped to 33 and NTU dropped to 69. Anyone knows about UKM and UM?

Anonymous said...

UKM and UM are out from Top 200!!! Hahahahaha!!! no surprise at all!!
See the link below:

Anonymous said...

Both Tony and Kian Ming should have plenty to talk about the latest THES rankings.

Anonymous said...

I am waiting to hear form our lovely gov to explain it :)

Anonymous said...

In attempting the improve ranking, the govt tried to take the easy way out by admitting more foreign students and hiring foreign faculty. They got slammed instead because THES reduced the points for foreign students.

rational thinker said...

it's rather sad to know that the guiding policy and strategy to improve our universities, lies with a commercial-linked ranking all the way from our ex-colonist.

while i acknowledge the relative significance of THES ranking, it is, honestly mean nothing for us. hence the need for a INDEPENDENT and OBJECTIVE ranking for malaysia institutions.

this however was not achieved by the MOHE recent survey. it was

a) not objective
b) political tool
c) subjective-perception based
d) only about reputation
e) range of survey sample is limited
f) small number of responses
g) in summation, it's RUBBISH.

wonder how many manpower are spent to produce that shit..."all our universities passed, while 6 are good and one is "exceptional>".

i can tell you, without doing any survey that only the handful established universities are doing, some, if any, research...As for teaching, half of the rest doesn't even have proper faculty, not to mention quality education.

chandra said...

Useful resource for people they will keep coming back!
Employment Questionnaire