Letter to Mkini: Criticising the MOHE rankings

Saw this letter in Malaysiakini criticizing the ARES survey carried about LAN on behalf of the MOHE. I agree with most of his criticism but I still think it's a worthwhile first step to take.

I just want to bring out two points in this letter which I don't quite agree with. I'll discuss these points below.

The Academic Reputation Survey (ARES), carried out by the National Accreditation Board (LAN), on 17 public universities (‘Public universities fail outstanding test’, New Straits Times, Nov 3), is a sad reflection of how utterly mediocre our university system is – or rather how close to that, the ARES and the LAN are as tools of public accountability.

The public has been fooled into thinking that this half-baked survey can be taken as a substitute for a ‘ranking system’. The fact of the matter is that it is not a ranking system. It was merely a perceptions survey, without the public being told who exactly were being surveyed. The ARES sent out 954 questionnaires and received 272 responses, supposedly from “higher education institutions, two Asean universities (National University of Singapore and Institut Teknologi Brunei), nine corporate bodies and 19 professional and certification bodies.”

First of all, questionnaires are only answered by respondents (or living individuals) and not inanimate bodies; only individuals perceive, not institutions. This means that we need to know who were the “persons” rather than the “institutions” who responded to the questionnaire. Did they represent the institutions and in what capacity? We need a breakdown of the profiles of these 272 respondents; such as how many were males or females, which institutions do they come from, their occupations, age, educational qualification, etc.

It would make a lot of difference to the credibility of the result (which ‘ranked’ USM as the top university), if, out of the 272 respondents, 70% were from USM itself! Furthermore, what is the breakdown of respondents from the public and the private sectors? If a majority were from the public sector then this could mean a case of government self-praise, rather than independent judgement.

Anyway, statistical validity aside (which happens to be the main problem of this survey), what about “perceptions” then? How much do these individuals know about the universities they assessed on this sophisticated-sounding, Likert-scale (which is nothing more than jottings on a scale from best to worst in answer to the questions of the questionnaire)? It really matters as to who does the “perceiving”.

To a question of whether a postgraduate programme of any university is good or bad - how could the manager of a fast-food chain company, for example, have any idea on how to rate these programmes? How could any of these individuals rate the quality of the academic staff of universities? What yardsticks are they given? I do not want to belabour the need for integrity and precision in statistical methods (or the pitfalls when abused), only to say that this is the crux of the problem of our public universities - even the guardians of their academic standards are short of standards themselves.

It may even be too late to arrest the deterioration of our universities - we are talking about the lives and future of millions of our youths, the country’s human resource, the source of its leadership. I am afraid that the education minister and the vice-chancellors are living in a make-believe world about creating an “empire of the minds” (to quote one VC in his newspaper column).

Here I am talking from actual experience. Just put a Malaysian university student alongside their counterparts from Thailand, Indonesia, Philippines, Singapore and let them all speak, and we shall know that we have done a terrible disservice to our local graduates. Not being able to speak proper English is forgivable, but not being able to articulate ideas and arguments without embarrassing themselves (if they even know of it!) is the graver cost to the reputation (and soul) of the nation.

I do not know what it takes for a wake-up call. I can only say that if it is not now, it is never. And yes, I urge the government to stop the chicanery of devising an internal ranking system. What is the point? You are only listing which of the rotten apples are less rotten, or rather which are the ones having a slower rate of decay. They are all going to get there.

Firstly, I think the writer recognizes that the identity and background of the respondents are crucial in these types of surveys. But he seems to be in two minds about who these respondents should be. On the one hand, he does not think that a majority of respondents should be from either the public universities or the public sector because this will inevitably lead to self-praise, an issue I brought up in my previous post on this issue. On the other, he doesn't think that it is appropriate for a 'manager of a fast-food chain company' to evaluate the quality of our public universities either which I interpret as not wanting too much private sector involvement (or at least private sector involvement of a certain kind).

I don't really have an easy answer for this. Presumably, one could 'solve' this problem by asking more foreigners, especially academics in other universities who are somewhat familiar with the state of research and teaching in our public universities. But this also raises the question of possible 'bias' against the newer and lesser known schools, as well as the possible accusation that we are at the mercy of 'imperialist' or 'neo-imperialist' forces that are trying to bring down the international reputation of our public universities. I frankly think that it is probably better to target a mixture of respondents - foreign academics who are familiar with Malaysia (I can think of many in ANU in Australia), some well respected local academics (Khoo Kay Kim) including some who no longer teach in Malaysia (such as Jomo and Gomez) as well as respected intellectuals and business people who are familiar with the situation in our public universities (such as Zainal Aznam Yusuf, Mohammed Arif Nun, Munir Majid). The problem will be getting enough of these people to respond.

Secondly, I think he probably under-rates our local graduates and over-rates graduates from universities in our neighboring countries (Indonesia, Singapore, Philippines, Thailand). I think if we bring together the best law students, let's say, from these countries, I don't think our local graduates will fare any worse in a mooting competition. I think our best local graduates probably have a better command of English compared to their compatriots from Thailand or Indonesia.

I also think that the average graduate from our public universities is probably no worse in terms of analytical ability compared to the average graduate from our neighboring countries, perhaps with the exception of Singapore, where standards might be more vigorously enforced. This is not to say that our local graduates are great, but rather that they are no worse. Of course, with a better university system, it would be the case that our local graduates will outshine our neighbors in all aspects, including analytical abilities, but that is a long way aways and we might not get there at all if the current situation of academic decline is not addressed.

I think it's too easy too poo-poo the efforts of the MOHE and be critical. I've been tempted many times to do this. But judging from what I see to be a sustained effort on the part of the MOHE, Tok Pa and some VCs in our universities, I will give them the benefit of the doubt for now and see the glass as half empty rather than half full.