From Myth to Measurement: Rethinking US News and World Report College Rankings

https://anandsanwal.me/us-news-college-rankings-myth/

by chmaynard • 2 years ago

41 64 2 years ago

r00fus• 2 years ago

Modern US society seems to be layers upon layers of gamified results with little coherent vision. We blame other countries for having "stifling" regulation, "burdensome" oversight and "top-down" planning, but honestly - it seems we simply let problems precipitate for decades on end and in many cases think it's some sort of achievement.

yen223• 2 years ago

Don't think you need to single out the US here. Gamification is effective against most human beings.

screye• 2 years ago

It's the inevitable conclusion of American meritocracy.

In a world where you can achieve anything, someone without achievements must be incapable or, worse, apathetic. Pair that with routine measurement, and it creates a real compounding effect.

So parents correctly try to accumulate achievements as early as possible. To get into HFT, you must go an Ivy, for which you need a recommendation at a top high school, etc, etc....and so you need to impress the pre-K director.

Even (so-called) failure must be managed strategically. Spend 6 months as a freshman at MIT before dropping out, so everyone knows you're better than the best.

Other nations don't do much better. It's a pick your poison situation. But, other nations seem more willing to accept the poison as a necessary evil. The US (systems, not individuals) refuses to see its quirks as anything but a universal good.

Still the least bad system out there.

lordnacho• 2 years ago

College rankings are a bit like corruption rankings: they are measurements of perceptions. Not only that, perceptions are heavily influenced by previous rankings.

If your ranking doesn't show Harvard and MIT near the top, it is wrong. If your corruption index doesn't show Scandinavia and Western Europe above Africa and South America, it is wrong.

If it's wrong, people will not want to read it. If you do an "objective" ranking and it doesn't quite say what you expected, you need to weight things differently, so that your ranking has credibility.

At best you can do a bit of massage to show that you are actually doing something when compiling these rankings, and you might be able to highlight a few trends. But in the end, if you are not showing the usual suspects in the usual places, people will not believe you.

Reputation, at the end of the day, moves slowly.

doctorpangloss• 2 years ago

Show me an LLM ranking that doesn’t put an OpenAI or Anthropic product at the top.

RecycledEle• 2 years ago

I wish someone would measure the bang-for-the-buck without considering politically correct metrics.

lotsofpulp• 2 years ago

Bang for the buck exists if you get into a top 10 or 20 school where admission standards are so high that people are impressed simply by the fact that you were admitted and/or you get to network with resourceful people.

Otherwise, any state university should suffice. A few states have some state schools with very low acceptance rates and difficulty to get in that might serve to advertise/network you better than another state school.

If it’s not hard to get admitted, then just buy the education with the lower price.

BirAdam• 2 years ago

You also network with old money, the politically connected, and the global elite. The “top” universities don’t really need to be better, they just need to have the correct names and the prestige to attract those names. You may have the best professor teaching all the most useful knowledge in the middle of no where state university, but the folks attending won’t have as much opportunity despite their education because they won’t get connected to money or to power.

DoreenMichele5• 2 years ago

Viewing college through this lens is part of the problem. It's college as job training, not education.

Real education covers more than "skills and credentials I need to get a well-paid job."

WalterBright15• 2 years ago

Real education covers more than "skills and credentials I need to get a well-paid job."

I'm curious what real education would be that does not impart a skill that others are willing to pay for?

I speak as someone who selected classes based on maximizing skills that would be useful in the career I wanted.

paulpauper• 2 years ago

Even if no one published ratings, people have a general intuitive idea what the top 20 colleges are, which are fixed even if the order changes. MIT will always be top 10 for example. Harvard will always be top, too. The acceptance rate isa a good proxy for the ranking. Or success at the job market when applying.

DiscourseFan• 2 years ago

I agree mostly with the article, but this stuck out to me

Measure the proportion of graduates who pursue and are accepted into advanced degree programs. (adjusted for field of study so acceptance into medical school > masters in art history)

The only reason, however, that medical school is more competitive than an art history masters (to be fair, most paid masters programs aren't super competitive) is because the medical board has set up the system of accreditation to limit the number of practising doctors in the US in order to artificially inflate the salaries of doctors. The UK, for instance, has the opposite problem, where becoming a doctor is much easier, but for that very reason their pay is relatively low and the job is increasingly undesirable (not to mention cuts to the NHS).

This is all to say that, in just this instance, the author falls prey to precisely the same mystification that he is criticizing, by seeing something as "better" just because its more exclusive.

sealeck• 2 years ago

The UK, for instance, has the opposite problem, where becoming a doctor is much easier

According to a friend who qualified in the UK and now practices in the US the _process_ of becoming a doctor is much harder in the UK: medical schools in the UK test students holistically (i.e. big exams which test all the topics) whereas American students are tested one module at a time (so lots of cramming).

I agree that gaining acceptance is probably harder though (ish: the UK government subsidises the provision of medical teaching so places are also capped in the UK for financial and capacity reasons).

thaumasiotes2• 2 years ago

medical schools in the UK test students holistically (i.e. big exams which test all the topics)

For reference, "holistic" is the opposite of "objective", not of "specific".

ineedasername4• 2 years ago

Here's an insider's insight: Colleges hate these rankings too, including the shit eating grin they have to plaster on their face when they go out in public and say things like "We're one of the top 10 small public non-land grant research institutions in the upper middle pacific northwest region!"

But too many families have been convinced that rankings like this are useful so now the institutions have little choice but to become complicit in the ridiculous system or have their enrollment decimated and be unable to pay its bills.

toast0• 2 years ago

I dunno, my community college didn't seem to care about rankings (although they were proud of their transfer statistics), but my 4 year school seemed really proud to be in the top 10 of small private engineering schools where doctorates are not offered in the upper midwest that nobody two states away has heard of. Looking now, they've got actually some good looking rankings in some other categories.

taeric2• 2 years ago

Agreed on the skepticism. Feels safe to say that anyone not in the top 10 don't care about the ratings that heavily, but I question whether it is meaningless.

ineedasername4• 2 years ago

Community colleges care much less because the students that go there are choosing those schools for very different reasons. In some cases it’s financial, in some cases lack of options, in some cases for very specific certifications, etc. Community colleges also have less ability to chase rankings through lack of $ to plow into marketing budgets or other fluff used to inflate rankings so there’s no reason to care all that much. The issue also isn’t something a student will be too aware of after the few months of college search prior to enrolling.

fasa99• 2 years ago

For me a great example is Reed college. Smartest person I ever met, went there. Had a 4.0 GPA. This is an onery school with regards to the bell curve, C average, so a 4.0+ GPA at Reed is a huge deal. Anyway, Reed simply tells US News to fuck off. Thus they are ranked lowly but much higher in reality. Similar to that weirdo Cali desert school with like 8 students that's student run - very smart people, very poorly known to the public. Anyway, Reed is otherwise notable for having a nucular reactor and its hyperliberal political spirit, for example, instead of man and woman bafrooms they have cisgender and transgender bafrooms.

rswerve• 2 years ago

Too few people know about the US government’s own attempt to offer a better not-exactly-a-ranking system. It’s pretty good. https://collegescorecard.ed.gov/

mumblemumble2• 2 years ago

My favorite thing I've ever heard said by the principal of our kids' elementary school: "Our test scores are down, which is great, maybe that will keep some of the school shoppers away this year."

Our city has a school choice program that includes a portal where you can look up these kinds of quantitative measures, and I think I agree with him. Tiger parents slosh from school to school as they chase after rankings, and, much like ill-contained liquid cargo in ships, all that motion tends to destabilize and capsize schools.

Sadly, I don't think smaller higher education institutions can afford to take such a relaxed attitude about it. They don't get to have an enrollment backstop in the form of a semi-captive audience of parents who live nearby and aren't hyperactive enough to commit to spending upwards of an hour every weekday trucking their kids back and forth across town.

aantix2• 2 years ago

It's such a strange way to evaluate schools.

My high school was inner city. But they had AP programs, computer science classes, advanced mathematics.

It's all about your child and the individual path they'll take within the school.

And if they have or two smart friends that they enjoy hanging out with, they're golden.

gruez2• 2 years ago

And if they have or two smart friends that they enjoy hanging out with, they're golden.

The probability of that happening is greater if average test scores are higher. The "tiger parents" aren't just looking for schools that offer the best programs, they're also looking for schools that have less problematic students that disrupt the learning environment and would be bad influences on their kids.

WalterBright15• 2 years ago

My high school offered a nice selection of impressive advanced classes. The course catalog was amazing.

I took those classes, and they were all milquetoast.

It was all just a potemkin village.

In contrast, the Caltech course catalog had classes blandly labeled "introductory" when they were well known for being brutal.

jackcosgrove• 2 years ago

The OG listicle.

I do think it's a bit difficult to separate reputation from educational quality. Educational quality is affected by professors, facilities, research opportunities, etc, but these factors are saturated at a lot of research universities. Another factor of educational quality is the peer group, and in this sense reputation matters a lot. There's an "if you build it, they will come" circularity here, where if you have a good reputation, you will attract students who will burnish your reputation. Conversely institutions can get stuck in a loop of low reputation and marginal students.

If however, you are armed with the knowledge that the non-circular components of education quality are saturated at, say, the top 50 universities in the US, well that opens some doors. You can rest easy that your flagship land grant university honors program is giving you the same education you'd receive at an Ivy League school.

Personally I think student debt load and chosen major matter way more than which school you go to.

bbor2• 2 years ago

Ha, pretty funny article. Well written for sure, but it's got major "if only they let ME run things, everything would be fixed in day!" vibes. College rankings aren't broken because some data scientist made a bad decision, they're broken because they're an essential part in the contemporary American class system: sending your kid to SAT bootcamp and then a "prestigious" university is one of the few ways the top 10% can separate their children from their poorer peers. Private high school doesn't really go that far these days, other than for social conditioning and getting them into a good university where the real networking is unlocked.

All of the above applies tenfold for foreign students coming to prestigious US universities, as they pay exorbitant sums to get the name recognition, creating all sorts of weird incentives.

As someone who went to a somewhat prestigious university (Vanderbilt), fingers crossed we nationalize the whole system sometime soon... I think we can all agree that focusing on football, campus amenities, and marketing aren't where these billions should be going. Vanderbilt, to their credit, gamed the rankings a ~decade ago by offering to meet 100% of student's government-determined financial need with grants, which is probably the best outcome possible of this weird system.

WalterBright15• 2 years ago

Supplying free rides results in students who don't have "skin in the game". If one works to pay the tuition, one is incentivized to get the most value out of their classes.

Without such skin in the game, a student is more likely to do as little as possible and focus on having fun instead.

bbor2• 2 years ago

Maybe! Not sure I've seen data that supports that, but it's a sound hypothesis among some populations. That said, the students who do the very least work and learn absolutely nothing are the ones too poor to ever become undergraduate students in the first place ;)

saagarjha4• 2 years ago

My parents paid for my college tuition. This was also the case for many of my peers. They turned out just fine. Honestly they seem to have done better than those who took out loans purely because they don't have loans.

ocean_moist• 2 years ago

Here is a relevant tool that ranks colleges (and degrees) by ROI/EV: https://www.collegenpv.com/programrankings/?pcip=11&page=1&s.... Not perfect, but better.

As someone who just played the college admissions game last year, these (US News) rankings pretty accurately reflect the *perceptions* of the general, college applying, public (or maybe they are the source of those perceptions). They aren't really even good at that outside the T10.

gxonatano• 2 years ago

Instead of measuring an institution’s wealth or selectivity, it could track metrics like:
Employment Rates: Track the percentage of graduates employed in their field of study within six months to a year after graduation.
Earnings Data: Use data on median salaries adjusted for regional cost of living and industry to assess economic outcomes.

This is a toxic way of looking at education, in my opinion, like it's some kind of vocational training. By this metric, two-month training programs in high-demand fields would be the highest ranked, by far. But often, the subjects which have the deepest impact on one's thinking—and on one's curiosity, intellectual abilities, and life in general—are the ones which have no easy one-to-one mapping to a job. I've taken philosophy courses, and anthropology courses, which have changed my life completely, but I didn't get employment as a philosopher or anthropologist within six months after graduation. The value of education is in how much it can change your thinking, satisfy your curiosity, and make you into a well-rounded person. It's not just whether it can prepare you for some job. And if you reduce education to "median salaries adjusted for regional cost of living" then you're measuring education purely on a sell-out basis.

From Myth to Measurement: Rethinking US News and World Report College Rankings

Related Stories