USM "Excellent", No "Less Satisfactory" or "Weak" Universities

Looks like the 'rankings' for the public universities are out before those for the private universities. Thanks to a friend from USM for letting me know about this NST article which came out yesterday.
17 out of the 20 public universities in Malaysia were evaluated and 'ranked'. Only 1 university made it to the "Excellent" category - USM. 6 universities made it the "Good" category and the remaining 10 were put in the "Satisfactory" category. None of the 17 universities evaluated were put in the "less satisfactory" or "weak" category.

According to the NST report,

And only one university - Universiti Sains Malaysia - made it into the second category of "Excellent"in the first ever university perception exercise in the country. It received a five-star ranking.

Universiti Malaya (UM), Universiti Kebangsaan Malaysia (UKM), Universiti Putra Malaysia (UPM), Universiti Teknologi Mara (UiTM), Universiti Teknologi Malaysia (UTM) and Universiti Islam Antarabangsa (UIA) were lumped in the "Good" category. They had four stars.

Those in the Satisfactory list are Universiti Malaysia Pahang, Universiti Malaysia Sabah, Universiti Malaysia Terengganu, Universiti Malaysia Perlis, Universiti Malaysia Sarawak, Universiti Pendidikan Sultan Idris, Universiti Sains Islam Malaysia, Universiti Teknikal Malaysia Melaka, Universiti Tun Hussein Onn Malaysia and Universiti Utara Malaysia.

None fell in the "Less Satisfactory" or "Weak" categories.
The Academic Reputation Survey (ARES), carried out by a team spearheaded by the National Accreditation Board (LAN), involved 17 of the 20 public universities.

First of all, I think it is a good first step for the MOHE to even make these rankings available to the public. I'm always in favor of having more rather than less information in the public realm. So I would commend the Minister for making this move.

Secondly, given that this is the first time that such a task has been undertaken in Malaysia, one would expect there to be a lot of flaws in such a ranking system. Even the Minister, Tok Pa, acknowledge this when he said that the results of the Setara rating system, which ranks public universities in three categories, namely, research universities, broad based universities and specialised universities, would not be released to the public as of now because of problems with the data and methodology. We would expect the methodology and the data collection process to improve over time.

These positives aside, I have to point out the flaws, of which there are many.

Firstly, the ranking system should not be based on survey data only. Some more objective data should be used in conjunction with survey data. Perhaps because the ARES specifically aims to capture public perception (or more accurately, the perception of universities among the academia and some members of the corporate world), we should not expect objective data to be included in this ranking. But, hopefully, such measures will be calculated and included in the Setara rating system.

What sort of objective measures? I can think of a few - the % of lecturers with PhDs and / or Masters, the average GPA score of students admitted, the ratio of lecturers to tutors, library book holdings, IT expenditure per student, just to name a few. The use of more objective measures should complement and balance out the measures calculated using surveys. Of course, one could still disagreement of the exact type of objective measure to use since some will obviously be more advantageous to the more established universities e.g. the average GPA score of students admitted. But I don't really have a problem with this given that these so called 'advantages' usually translate into a better academic environment in these universities. Would someone not rank let's say Harvard higher because it has a much higher average GPA entry requirement compared to let's say Diablo University?

The use of these more objective measures also forces the universities and the MOHE to collect these statistics, some of which are important to researchers as well as to interested parents, students, policy makers and so on. Wouldn't you be interested to know what the average GPA entry into UM is compared to UUM? Wouldn't you want to know how much more UKM spends on IT compared to USM?

Secondly, the high non-responses among the respondents who were sent the survey should be reduced. It was reported that the ARES questionnaire was sent to 954 respondents and but only 272 responses were received. That is a response rate of less than 30% which raises possible questions of whether the responses were biased towards any direction.

According to the report, the respondents were replied were broken down into the following categories:

242 public and higher education institutions, two Asean universities (National University of Singapore and Institut Teknologi Brunei), nine corporate bodies and 19 professional and certification bodies.

The large number of respondents from 'public and higher education institutions', which I'm assuming are institutions in Malaysia, probably explains why there were no institutions in the 'less satisfactory' and 'weak' categories.

Intuitively speaking, there should be more variance in the rankings of these universities beyond the top three categories, especially given that I didn't even recognize the names of some of the universities listed e.g. Universiti Malaysia Pahang (where is this university located?), Universiti Malaysia Terengganu (which I'm guessing used to be designated as a college), Universiti Malaysia Perlis (can Perlis support its own university?), Universiti Pendidikan Sultan Idris (I thought this was primarily a teacher training college?), Universiti Sains Islam Malaysia (never heard of), Universiti Teknikal Malaysia Melaka (no idea), Universiti Tun Hussein Onn Malaysia (is this new?). I haven't goolged these universities individually but if any of our readers are familiar with any one of these universities, please post a comment and enlighten us.

The fact that so many of the respondents were from our public universities probably meant that many of them didn't feel comfortable in giving some of these lesser known universities a 'less satisfactory' or 'weak' rating. I think even if the sample size included more corporate bodies or even overseas academics, it would run into the challenge of many respondents who cannot evaluate the universities which they have not heard of or are unfamiliar with.

The fact that many of our public universities are relatively new and unknown is probably why the following three universities were not included in the survey - University Darul Iman, Universiti Malaysia Kelantan and Universiti Pertahanan Nasional Malaysia (what?).

This being said, I think that the rankings are indicative of the relative 'quality' of our public universities. Most people would say that USM, UKM, UM are probably the top three public universities in Malaysia followed by UPM, UIA and perhaps UITM and UTM. The fact that USM was rated 'excellent' is perhaps a validation of some sorts for the more progressive policies of promotion and emphasis on research which the VC there is carrying (Dzulkifli Abdul Razak whose columns regularly appear in the NST).

What we require is a further refinement of the ranking system and more objective measures to be used in conjunction with survey responses.

I'll stop at this point and let our readers weigh in with their views.