Saturday, January 31, 2009

'Gaming' university rankings

Read this from the Duke alumnus magazine. I think it's a good reminder for our VCs - don't try to 'game' the ranking system, concentrate on improving your university in ways you see fit.

Under the GargoyleThe Rankings Game: Who's Playing Whom?
By John F. Burness

U.S. News & World Report published its first annual ranking of the nation's best colleges in 1983. In the years since, the publication has spawned a cottage industry, transformed how the public thinks about higher education, and in the process made a lot of money.

Over the past three decades, I've had ample opportunity to dissect the various rankings or discuss the validity of their methodologies in an effort to explain to a wide range of university constituencies, including the news media, why the universities where I worked—the University of Illinois at Urbana-Champaign, Cornell, and for the last seventeen years, Duke—were rated where they were. It's fun as I retire from university administration to ruminate on the absurdity of it all.

Ours is a competitive culture, and it should be no surprise that many people are interested in such external assessments of the quality of American higher education. After all, students and families spend as much as $50,000 a year to go to college, and it is reasonable for them to want a credible, independent assessment to help guide their thinking about where to make that significant investment.

That said, I don't know anyone in higher education whom I've talked to since the ratings game began who believes that the magazine rankings can capture what makes the experience offered by an individual institution unique or effective. The precision that U.S. News purports its methodologies reveal is, on the face of it, rather silly. If you look at the top ten institutions, you will see that some of them are separated by small fractions of a percent. In the Olympics, those fractions make a difference, but it's hard to understand how, in the real-life breadth of activities of a university, they make any difference at all to a student.

The rankings give considerable weight to perception and tend to be based on annual assessments, as if undergraduate-program innovations or tweakings manifest significant change in two semesters. U.S. News has artfully—in the guise of improving the veracity of its rankings—made one or more changes in its methodology every few years, which enables it to argue that there is some shift in the quality of institutions that the new methodology has captured. The cynic in me says that the changing of the methodology is more a strategy for getting different results in the rankings, which helps the publication sell more copies.

During my years at Duke, the university ranked as high as tied for third and as low as tied for eighth. The year we tied for third was my favorite. Folks at Duke were understandably elated. I recall telling university leaders, including our trustees, not to crow too much about this jump to our position of three because inevitably the methodology would change, and we would drop a few places—which, of course, is what happened.

My favorite magazine ranking experience wasn't with U.S. News but with Money magazine, which, in the 1990s, had a "Best Buys in Higher Education" issue. In that one, the public universities, almost by definition, ended up having a built-in advantage, although fifteen private institutions were listed among the top 100. Duke was not among the fifteen, much to the consternation of some of our trustees and others. So I met with the editors of Money and asked how we could be ranked in the top ten in the country in other ratings (as skeptical as I was about them) and not make the top-ten private institutions in Money's listing. They mumbled something about our library resources, and I was able to document that their numbers were wrong. The next year, Money came out with a new category: "Costly Schools That Are Worth the Price." Duke was ranked highly in that, and people at Duke were pleased. Alas, I didn't keep the pressure on the magazine, and one year later, it dropped the category.

I remember well a wonderful speech by a distinguished faculty member at my son's freshman convocation several years ago. The scholar compared the founding of that institution to Odysseus' journey, noting that both had decided not to let others define who they were. He urged the freshmen to create their own identity through the choices they made during their college years. Within a moment or two of the faculty member taking his seat, the chancellor of the university—a person I admire enormously—told the assembled freshmen and their parents that while the information was embargoed publicly until 11:59 that night, he felt comfortable telling them in confidence that the university for the first time had cracked the top ten of U.S. News rankings. The response was predictable, with students jumping up and down, and parents smiling at the thought that their investment clearly was going to be worth it. The faculty member sat there, his head bowed.

 I always said when reporters and others sought my reaction to Duke's being ranked somewhere in the top ten: "It's nice to have confirmed what we know about the quality of our students and faculty. But magazine ratings are really designed to help sell magazines. Students should visit a campus, spend real time learning about the academic programs, and determine whether or not they have the right fit with a particular institution." I still think that's very sound advice.

8 comments:

Anonymous said...

UITM is better than DUKE....

Anonymous said...

It is amazing that UM still practices recruitment by the color of your skin and is very undiplomatic in their recruitment policies. Firstly, they make you wait more than 2 hours for the interview. Secondly, when you get to meet the panel they make it prerogative to insult your skills and ability and then admit their falling ranking in the world and want you to improve upon it by tapping on your network and relationship skills. It is almost as if they saying that they would climb up the rankings by rubbing shoulders and scratching of people's backs. Finally, they wanted to know if you could encourage foreign experts to come to teach at UM to improve their rankings. Nothing to say whether you can teach or not and nothing to the effect of the package that they will pay you. Not only have they wasted your time and traveling money to make the trip for the interview but they have made it their objective to admonish and subject you to great deal of discomfort and humility. The whole interview lasted less than 15 minutes and explains really whey you have the poor perception of the institution by both global academia and by society as a whole.

Anonymous said...

It seems to me that the national institution is bias and prone to misconceptions. Further, bureaucrats exist in it with their PhDs procured through dubious means. Shall we say an exchange of commodities. The whole hierarchy is on the basis of patronage, serfdom and Machiavellian attributes. Middle Ages here we come and ranking down the drain. Further, it takes around six months for approval for the application to pursue a postgraduate program. By then your research question and problem is dated and the pertinence and nascence has disappeared into thick smoke enough to choke any Nobel prize winning dreams.

Anonymous said...

now um is recruiting more chinese (as mentioned by one anonymous -> only one color of skin) than any other races. no place for bumi anymore. go to uitm.

Anonymous said...

this blog is for loser who hate malaysia education and university, and undermined malaysian capability. banned and boycott this blog.

Anonymous said...

Kian Ming has no idea other than talking about university ranking, proud of american uni, condemn malaysia university especially um. No one will listen to what he wrote and this blog is becoming so boring.

Anonymous said...

to anon 2/01/2009 08:14:00 AM: Did you read the content of this posting??? Do you understand it?

Btw, I fully agree with what is being posted and it is fair to say that ranking contained many biases. If I am editor of Sunday Times and I graduated from let's say Sheffield. Would I want to try my best to push up the ranking of Sheffied?

Having said that, perhaps the posting on "World Class universities in Malaysia" should be removed as the posting is based on university ranking.

Anonymous said...

One thing I find about Malaysia is the need to aggrandizement itself by having its own ranking methods like the Malaysian Book of Records and the National Nobel Laureate. Unfortunately, benchmarking to the laws of a jungle will only lead to chimpanzees and grading ourselves to lower standards lead to developing the brain of an earthworm. This has been the psyche since the Malaysianization of our local national institutions of education. So really who is kidding who when it comes to upgrading of standards to whom but oneself. Benchmark to the worse you will get worse and that is what is happening to the education, sports and nearly every other industry.