How Reliable Are University Rankings?

https://arxiv.org/abs/2004.09006

by ijustlurk • 6 years ago

65 85 6 years ago

arxiv.org

eindiran• 6 years ago

Here is the code used by the paper: https://github.com/alidasdan/university-rankings

neilv3• 6 years ago

https://www.bostonmagazine.com/news/2014/08/26/how-northeast...

Freeland swept into Northeastern with a brand-new mantra: recalibrate the school to climb up the ranks. “There’s no question that the system invites gaming,” Freeland tells me. “We made a systematic effort to influence [the outcome].” He directed university researchers to break the U.S. News code and replicate its formulas.

augustt• 6 years ago

Yeah this kind of gaming is probably widespread. UChicago is known for "artificially" driving down their acceptance rate with lots of decision rounds and marketing. Cornell isn't releasing their numbers at all this year - I'd guess because they don't like them.

neilv3• 6 years ago

I've heard many allegations about established universities gaming their ranking by, e.g., soliciting applications widely -- not so much to have a larger and more diverse pool from which to choose -- but to maintain/lower their acceptance rate metric.

Two things I like about the Northeastern story are that the official demonstrated even more ambitious gaming, and it was publicized.

No criticism of Northeastern; they have many great people, doing great work. And I've heard many faculty objected to the rankings-climbing emphasis as it was happening.

nu_throwaway• 6 years ago

graduated from there. totally agree with what's in that article. at 18 I was swept away by the "allure" of their rapid ascent and aggressive re-development of campus (which has continued to this day) and went there, despite the insane cost.

faculty were generally pleasant to interact with, the education was probably average. job prospects were mediocre - alot of competition in boston and NU is in a lower tier, comparatively.

as a measure of quality, based on my experience, that ranking is worthless.

neilv3• 6 years ago

NEU has some really great people in CS, who could hold their own with their counterparts at the better-known schools in the Boston area.

I've also heard some good things about NEU cybersecurity.

I'm not familiar with the other departments.

spideymans2• 6 years ago

Why is there such a fixation on university rankings in the United States? Postsecondary rankings don't seem to matter as much in other Western countries.

notfromhere• 6 years ago

It’s what drives enrollment and tuition fees are a huge part of revenue. In other countries, there’s more state support and you don’t need to market universities

Infinitesimus• 6 years ago

A mix of trying to get a sense of educational quality and prestige.

In some specific programs, you'd want to aim for the schools known to have the more robust curriculum but outside of that, it's mostly ego/prestige. "I got to a top N school, I went to harvard", etc.

That badge signals _something_ to your peers, parents, and future employers. Think of all those companies that explicitly filter for "target schools". It works pretty well too, you'll get a call-back to an interview because you have an Ivy on your resume even if you barely passed your classes.

dudul• 6 years ago

Education is a market. To make money, universities need to attract customers. And they do that by being highly ranked.

And I disagree with your statement. Many other western countries have their "ivy league" equivalent. Not all schools are equal in the rest of the world either. But I do however agree that the "marketing" aspect is definitely more pronounced in the US.

monadic2• 6 years ago

The professional connections pay for the tuition and much more.

chasd00• 6 years ago

the importance of professional connections are understated in my opinion. You can make a career out of only knowing the right things and you can make a career out of only knowing the right people but you really excel when you know the right things and know the right people.

plorkyeran2• 6 years ago

Is there actually a big fixation on university rankings? I had no idea what my school's ranking was (apparently it's now #40) and I'm not sure I've ever had conversation where university rankings came up outside of specifically talking about where a high school student is thinking about applying. Even then it's not like it's common for people to pick which school to go to out of the ones that accepted them based solely on rankings.

People who care about Ivies and the such don't care because the Ivies have a high ranking. Someone who went to Harvard who wants to use that fact to their advantage isn't going to say that they went to a "top 3" school.

ijidak• 6 years ago

The rankings mattered to me when applying to schools because it seemed like a useful metric to judge the quality of education I would get, and the usability of that education.

When you're making one of the biggest decisions of your life, rankings can help you know how good your choices are.

Not saying rankings are correct...

But for those applying, they likely appreciate having a ranking mechanism.

It's one of the few objective measures of universities available.

How do you know, otherwise, if this university that wants to charge you $40,000 per year is really going to deliver on their promise?

You can't just go back to school again and start over.

It's a life decision, so there is a desperate desire for the applying student to have an objective way of stack ranking the choices.

(Disclosure: I attended MIT. And I basically just applied to the top 5 or 10 computer science universities -- again, using rankings. And hoped to get into one of those.)

AtlasBarfed2• 6 years ago

If you dropped 200k-300k on an education, you'll be invested in either the continued value or and improvement in your investment's reputation, which as the years go on is really its only value.

dntbnmpls• 6 years ago

I'm not sure I've ever had conversation where university rankings came up outside of specifically talking about where a high school student is thinking about applying.

Most high school students/parents/guidance counselors/etc check the rankings, reputation, average salary of graduates, etc. It even extends to majors and rankings within majors - which might matter more.

Even then it's not like it's common for people to pick which school to go to out of the ones that accepted them based solely on rankings.

Who said solely? Of course there are many other factors ( location, tuition, scholarships, etc ). But rankings/reputation/etc are a big part of a high schooler's college decision.

jldugger• 6 years ago

Is there actually a big fixation on university rankings?

I've sat in department wide meetings where deans of engineering compared their schools to 'the competition,' both in terms of US news rankings and their own preferred metrics. Usually to bemoan how unfair it is that Texas is a petrostate with 10x the population that can fund top tier engineering schools.

But by and large, the tiers are static and a uni in the #47 slot isn't making to the top 10 any time soon. Might not even make top 40. For undergraduate education I think you're right, nobody really cares. There's really three categories of schools: private schools / ivies, flagship state schools, and the rest. I'm guessing if you asked employers to rank schools, they would largely match selectivity -- how hard it is to get into the school. (or more depressingly, how good the athletics programs are doing lately)

commandlinefan• 6 years ago

Is there actually a big fixation on university rankings?

I've observed that, among people who care about university rankings in hiring, there are exactly two categories: the top five, and everything else. If you went to #200, you're the "same" as somebody who went to #25.

barry-cotter2• 6 years ago

Not true of the UK, France, Korea or Japan. I don’t know if they care about what are basically league tables but any educated citizen would find it no more difficult to reel off five to ten of their best universities than an American would to list Harvard, Yale, Princeton, Stanford, MIT etc.

I can see how it would look weird from a German, or possibly a Dutch or Nordic perspective but a deliberate effort to bring up the bottom and limited attempts to raise the top is extremely different from the US system, where there are large and growing returns to excellence.

Also the US elite just have smaller enrollments as a share of population. Oxbridge, the grandes écoles or Korea’s top three enrol 1-2% of yearly students. The top ten US universities maybe a tenth of that.

roel_v4• 6 years ago

From within Dutch universities, rankings are considered very important (more by the non-academic staff than by the academics). The general public doesn't really care all that much yet, but they do more and more each year. It's especially for international (student) recruiting that the rankings (national and international) are important, and that's seeping through into everything else for last 10 years or so now.

(not disagreeing with you, just adding some nuance about the direction of things)

arethuza2• 6 years ago

I think in the UK it is easy - both universities are ranked as excellent!

talentedcoin• 6 years ago

Canadian university board member here ...

Rankings drive enrolment, especially international enrolment, which is a major source of revenue. Also prestige helps to attract the best researchers, who are being wooed by many top schools.

ashtonkem2• 6 years ago

Well, it did. We're going to see how unsustainable our secondary education system is without the children of rich foreign families to sustain it.

Prcmaker3• 6 years ago

Australia here. Universities consistently market their national ranking here. As a staff member I've found management consistently pushes things that will raise their ranking. As a student initially at a high ranked university, I found I got a better education by switching to a much lower ranked university.

spideymans2• 6 years ago

As a student initially at a high ranked university, I found I got a better education by switching to a much lower ranked university.

That’s interesting. I’ve consistently heard the same sentiment from ex-students and alumni of certain highly ranked Canadian universities (undergrad) as well. They’ll generally complain about a lack of student support and restrictive policies at these institutions, making it needlessly difficult for students to succeed.

I suspect the higher ranked universities might be under pressure to artificially increase the difficulty of their programs, in order to distinguish their alumni from the alumni of other schools, who’s programs cover more or less the same subject matter.

Consultant32452• 6 years ago

As a general rule, you aren't allowed admittance into the Cathedral unless you've demonstrated proper indoctrination at the most "prestigious" universities.

fergie• 6 years ago

I have worked at universities in the UK and Norway, and believe me, this stuff matters. A lot.

Its great that students can make an informed choice about where to start, particularly if they do not come from an academic household. Its not a given that people have heard of places like MIT, and these lists are a somewhat neutral way of seeing which universities are the most "legit". You have to remember that a lot of low-quality institutions are marketing themselves pretty hard to students, and if it weren't for rankings it would be easy to make bad choices when choosing a place to study.

diiaann• 6 years ago

I was talking to a few European friends (1 Dutch, 1 French) and what they described was that even the most bright students typically go to their regional universities which are all "equally good". Here in the US, the rankings are almost a proxy for admissions rate. Since "top schools" are exclusive, the student population tends to be at least one the following: talented, precocious, or well-connected/rich. I attended a "top 25" university, which is a ways from the Stanford, Harvard, Yales of the world but the environment was very special. Hard to describe, but you are surrounded by passionate, ambitious people, you can't help but feel energized. Is this a fair system? That's a whole another conversation.

asaph• 6 years ago

If you only read one sentence from this article, it should be this one:

In this paper, we take a fresh look at this ranking scheme using the public College dataset; we both formally and experimentally show in multiple ways that this ranking scheme is not reliable and cannot be trusted as authoritative because it is too sensitive to weight changes and can easily be gamed.

finolex1• 6 years ago

Gaming of rankings aside, the ratings organizations themselves offer various 'services' to colleges looking for a reputation boost.

https://www.insidehighered.com/news/2013/05/29/methodology-q...

For instance, QS sells a 'star-ranking' system for schools. Coincidentally the schools who paid for a '5-star' ranking, such as the Unviersity of Bristol or the Universiti Malaya are also placed higher on the actual QS ranking than those that did not, including Georgia Tech, the University of Washington, Ecole Polytechnique and UIUC.

alexpetralia2• 6 years ago

Related to expenses..

"And there are several perverse incentives in the marketplace that make it hard for colleges to cut costs. The most basic one is that the U.S. News algorithm rewards them for spending a lot of money: Higher faculty salaries and more spending on student services lead directly to better rankings. If you reduce your expenses, your ranking will fall, which means that next year your applicant pool will probably shrink."

You'd be surprised at the burgeoning expenses of universities, and often meager profitability, despite record-high tuition rates. It seems, often like startups, a burn rate signals potential growth!

https://www.nytimes.com/interactive/2019/09/10/magazine/coll...

mattkrause• 6 years ago

I'm gonna go out on a limb and say that above-market faculty salaries are not a major issue.

Trombone12• 6 years ago

Isn't the point of an audit is that it tells you what problems to fix? It would be concerning if this audit happens only shortly before, or even after, the main ranking happens.

But if the increased ranking is done based on data from, say a half a year later, then that just seems like a normal audit effect? That it might just make the client better at gaming the rules is true for a normal audit too.

downerending2• 6 years ago

Regardless, you should think hard before accepting at the "top" university that will take you. Many times it's better to be a big fish in a small pond than vice versa.

Also, in terms of learning, it's really about the access you can get to the ten or so percent of the faculty that actually worth anything. Figure out who they are and go after them.

Also, network, network, network. Smart or not, everyone you meet in school might give you a reference/job/funding someday.

(source: hard experience)

ordx• 6 years ago

Care to elaborate? Did you have any problems after graduating from the "top" university?

downerending2• 6 years ago

More "during" than "after". Not sure if mine counts as "top", but most would say it's easily in the top 25 for my discipline. It was plenty hard for me, though, and in hindsight, I think it would have been better to be the brilliant kid in a lower-ranked school than an also-ran in a higher-ranked school (relative to the range of schools that would admit me).

That's just a guess, but I've seen a lot of others suffer from shooting too high.

jmchuster• 6 years ago

I could consider the primary advantage of a "top" university to be its network and the access it gives you to jobs. Namely, companies that do new grad hiring have a specific list of colleges that they target and will source and review resumes from. Not to say that people from other colleges can't get in, but that you'd have to apply directly and have something standout to get the recruiter to even consider you.

ramblerman• 6 years ago

This advice applies to jobs, I'm not sure it does to universities, which are more about prestige.

I.e. getting a ticket to pick any pond you want after graduation.

blendo• 6 years ago

Many times it's better to be a big fish in a small pond than vice versa.

Yes, this is seldom considered. For example, if you plan to graduate with a BS in physics, would you rather be in the top quartile at Cal Poly, or the bottom quartile at Cal Tech?

qzw3• 6 years ago

From the abstract:

*>we both formally and experimentally show in multiple ways that this ranking scheme is not reliable and cannot be trusted as authoritative because it is too sensitive to weight changes and can easily be gamed.

That's a feature, not a bug, as far as USNWR (the magazine doing the ranking) is concerned. A more reliable ranking would be more stable and therefore less interesting for the universities. Why pay attention to it if Harvard, Standford, MIT are always 1, 2, 3? It's much more "interesting" and certainly sells more copies/gets more clicks if the rankings have a certain amount of volatility. The fact that it's easily game-able also makes it more attractive to universities because they can do something proactive if they're dissatisfied with their ranking. All these factors combine to make the ranking itself seem more important and prestigious.

musicale• 6 years ago

Standford

      ^   ;-)

Not to be confused with Stamford

https://stamford.uconn.edu/

HenryKissinger• 6 years ago

The fact that it's easily game-able also makes it more attractive to universities because they can do something proactive if they're dissatisfied with their ranking

Are you saying that there is nothing lower ranked universities could do to improve their ranking in a better, more objective ranking of universities? Because it smacks of elitism.

jedberg2• 6 years ago

It's very similar to sports teams.

The tops teams don't vary all that much from year to year. They might change positions a bit, but it's rare for a new team to show up who hasn't shown up before.

They essentially have to get lucky and get someone who is super talented into a long term contract, who then attracts additional talent.

Or they have to invent a whole new way of doing things, like how the Oakland A's invented a whole new way of recruiting using math (which incidentally no longer helps them because all the top teams adopted it and are back on top again).

It's all about network effects.

qzw3• 6 years ago

Well, in a hypothetical ideal ranking system that is stable and not easily gamed, a lower ranked university would need to actually improve the quality of its education (and whatever other characteristics the ranking system measures) in order to move up the ranks. But that would likely to take multiple years or decades of committed effort and investment. It’s sadly rare to see such long term action in our world of instant gratification, whether we’re talking about education, business, government, entertainment, relationships, individual behavior...

ashtonkem2• 6 years ago

Nothing easy or quick, no.

sytelus• 6 years ago

Rankings are irrelevant for top 20 but for the rest it matters a lot. Vast majority of population ends up somewhere outside of top 20. They have to chose between A, B and C and rankings plays its part. Reliable ranking is important because it rewards universities with better enrollment (and vice versa).

skat20phys• 6 years ago

Sometime not too long ago there was a blog post or article covered on HN, about data showing student scores on some academic achievement measure over time. If I remember, all students generally improved on this measure throughout undergrad, and there was some average tendency for universities with higher average entering standardized test scores to have higher scores, but the effect seemed to be mostly driven by the students. That is, it wasn't that some schools accelerated scores more, it was that they just tended to select for higher-performing students. Also, the signal provided by school, if I remember, was relatively weak. The author was basically arguing that universities in general provide some educational value (or students just learn over time; there was no control of students who didn't go to college) but that the ranking was by and large just a signaling thing.

I can't find it now though because all the search terms I can think of just return a bunch of other stuff related to universities and students.

heyn05tradamu5• 6 years ago

Interesting. I wonder how one would measure whether an institution changes student trajectories? Or if that data is available somewhere.

covidacct2• 6 years ago

I'm mostly on board with the critique of university rankings, but I really don't buy this specific critique. It's a strawman.

Here's what they did: they took the top research universities and the top liberal arts colleges. They then observed that the rankings can't distinguish between the top institutions within each class, and that you can also flip which of those two classes is preferred.

But, in what sense is it even meaningful to compare Amherst College and Harvard [1]? Those are just enormously different types of institutions. It'd be like creating a ranking of "best cars" that includes the top 5 sedans and the top 5 trucks, and then observing that you can jiggle the rankings to get anyone on top. Is a Toyota Corolla better than an F-150? IDK. Stupid question. US News and World Report, for all its problems, does at least get this much right! they break down institutions by "type" and then rank within type.

Additionally, just because rankings are noisy and easy to game locally doesn't mean they are inaccurate or easy to game globally. Two institutions within 10-20 slots of one another are probably pretty similar and rankings aren't particularly helpful / are easy to game. But the #100 liberal arts college is probably not as good as Amherst, and the #78 National University is probably not as good as Harvard, and no amount of gaming is going to change that.

Rankings are indeed noisy and inaccurate and easy to game. But this particular article is not a compelling demonstration of that fact.

[1] For non-US readers: Amherst college (not to be confused with UMass Amherst) is in a class of peculiar institutions that are fairly unique to the USA as described here: https://en.wikipedia.org/wiki/Liberal_arts_college TL;DR: they're basically the diametric opposite of super-charged research universities like Harvard.

adas0693• 6 years ago

Thank you for your feedback. I am one of the authors of this paper.

I don't think the paper gives any hints on us dividing universities into classes and comparing. Would love to know how you reached that conclusion.

At the same time, yes, we could have divided universities into different classes: research vs liberal arts, this vs. that state, big vs. small size, etc. These are all trivial groupings but none of these would change the conclusions of this paper. For all practical purposes, we could easily have replaced the university names with labels like U1, U2, ... and still the conclusions would not change. What matters is how a weight-based composite index can be gamed and the paper does show that in multiple ways. Pls review the ILP formulations yourself and run them on the dataset of your choice.

tyingq3• 6 years ago

There is a very current thing happening where individual professors have too much power. The whole "remote teaching" thing is VERY new for some universities and professors. My son recently survived a final where > 90% of the class failed. Mostly due to poor execution of remote learning. Including simple stuff like classes overlapping meeting times, study material uploaded too late, etc.

No pressure on the professor or university, though. All on the students. There's a rant letter, from the prof, about effort where effort wasn't a factor. My kid survived solely because he has a very good memory. No accountability. I AM PISSED.

Aperocky• 6 years ago

where > 90% of the class failed.

I can't imagine this happening in a college class where the instructor get to keep his class next year.

tyingq3• 6 years ago

U.S. State college, a fairly well respected one, happening now. Chemical Engineering. They have no idea how to operate "online/remote". And that matters for classes like Fluid Dynamics, Organic Chem, Differential Analysis, etc. Is there some realistic path for me to escalate?

thinkloop• 6 years ago

Any idea what they are measuring against? For example avg. income or accolades of graduates.

dontbenebby2• 6 years ago

I think they're useful-ish, but it's important not to get too focused on specifics.

For example I did a stint in policy and I'm reminded of how everyone insisted for a long time that "top" law schools are the "top 14" or T14 for short. This is historically the measure used since Georgetown was in DC, and the children of the elite must be "top tier" regardless of if their school ranked outside top ten on every official metric such as incoming LSATs, publications, etc :D

barry-cotter2• 6 years ago

That’s not where T14 comes from.

There exists an informal category known as the "Top Fourteen" or "T14," which refers to the fourteen institutions that regularly claim the top spots in the yearly U.S. News & World Report ranking of American law schools.[8] Furthermore, only these fourteen schools have ever placed within the top ten spots in those rankings.[9] Although "T14" is not a designation used by U.S News itself, the term is "widely known in the legal community."[10] While these schools have seen their position within the top fourteen spots shift frequently, they have generally not placed outside of the top fourteen spots since the inception of the rankings.[11] There have been rare exceptions to this, however, such as UCLA School of Law appearing in the top fourteen instead of Cornell and Northwestern in 1987 and University of Texas School of Law displacing Georgetown in 2018, although the significance of these changes has been debated.

https://en.wikipedia.org/wiki/Law_school_rankings_in_the_Uni...

dontbenebby2• 6 years ago

> That’s not where T14 comes from.
There have been rare exceptions to this, however, such as UCLA School of Law appearing in the top fourteen instead of Cornell and Northwestern in 1987 and University of Texas School of Law displacing Georgetown in 2018, although the significance of these changes has been debated.

People select measures for a number of reasons. It's perfectly possible the single and only reason T14 is used is for the reasons you lay out.

OTOH I have personally had conversations with people who argued that the T14 is a measure of merit when Georgetown was in the T14 then "debated the significance" of the measure when Georgetown dropped out of the top 14 in 2018.

It was not clever or endearing.

notechback• 6 years ago

University rankings have two key issues:

1. They are significantly biased to English language (universities/research output) and bigger countries in general 2. They are totally irrelevant from the student perspective as they mostly focus on research output, which means that you'll get the richest universities and the best professors will be writing papers rather than teaching bachelor level courses.

There are some attempts to fix these issues (see eg the EU "Multirank" which allows you to choose your priorities), but it's still hit and miss.

jseliger• 6 years ago

Those are useful issues, to be sure, but there is a lot more incoherence than those two: https://www.newyorker.com/magazine/2011/02/14/the-order-of-t...

jdmg94• 6 years ago

I personally think Universities and colleges are a scam that was necessary before we had access to the bulk of human knowledge on the tip of our fingers. We should go back to an apprentice system, go through elementary and middle school and then find a profession you would want to learn OTJ, this would shift the responsibility of training to companies but the people coming out of there would be immediately productive.

tygrak• 6 years ago

Rankings are a bit dumb, especially because any non english university is going to get ranked terribly basically on every of these rankings. There's no way that the oldest university in central Europe, started in 1348 should be placed 400-500th. That's just a joke.

MR4D• 6 years ago

The only ranking that matters to the vast majority of people:

Median starting salary / median cost to attend.

Sure, there are other considerations such as. If it will get you into a better grad school or something, but for most it’s really this simple.

How Reliable Are University Rankings?

Related Stories