Test scores are not irrelevant

https://dynomight.net/are-tests-irrelevant/

by colinprince • 4 years ago

281 474 4 years ago

seibelj3• 4 years ago

SAT / ACT are basically IQ tests. Grades are far more fuzzy but generally show an ability to stick with something over the long term and execute on projects.

I think both are important to being successful. Ignoring one for the other is probably bad. That said I prefer IQ over grades.

majormajor6• 4 years ago

I lost points on the SAT for mixing up 2 pi r and pi r^2 - left out the 2 for the former. (That's the one that stands out as a duh, I knew better, one - I forget what else I missed.) There were also a LOT of analogies. I've read that those have been removed, I dunno how the math section is, but both of those were much more "knowing things" than innate intelligence. Grades tend to require both "knowing things" and longer-term discipline; I could see where that would be advantageous.

Grades have a disadvantage of A's being way too common, though - no resolution at the top of the scale.

NERD_ALERT2• 4 years ago

SAT/ACT measure how good you are at following rules and how much time/money you have to go to tutors and study 1000 page textbooks outlining specific strategies to maximize your score.

majormajor6• 4 years ago

Anything you can prep for - grades included - is going to trend towards being largely influenced by "who has the most time and money."

I don't have any good ideas to get away from that.

ghaff4• 4 years ago

Most of the evidence I see suggests that working through a test book at your local library goes a long way towards erasing most of the disadvantage associated with going into the test cold. Yes, there's some time involved but I'm pretty sure declining returns set in fairly quickly. There are clearly advantages associated with recognizing the type of questions that will often be on the test. But a lot beyond that is actually knowing the vocabulary and the various geometry, etc. material.

Now, if you basically don't know or have trouble understanding the material? I wonder to what degree some tutoring and even a fair bit of time are going to make up for years of poor school outcomes (for whatever reason). Some specific strategies are 1.) pretty straightforward--if you can quickly narrow options down, maybe guess and move on--and 2.) won't move the needle a lot.

bloaf2• 4 years ago

Tutoring and extensive exam-specific prep can boost scores. They are not necessary for getting a high score.

throwawaylinux• 4 years ago

Do you have any sources for this? SAT scores aren't highly correlated with household income, there is something but it's far from everything.

In my opinion and experience of the results my peers got, these kinds of standardized tests were really pretty good. The tests and studying methods themselves didn't necessarily mimic what you would do in a real job or even a university degree and sure seemed stupid and annoying at the time when you had to do the damned things... But stepping back and thinking about it, for the most part the people who were intelligent and capable of a motivated and disciplined approach to academics and fields that typically require degrees did actually tend to score better than people who were lacking in those things.

Yes there was the occasional very intelligent person who wasn't disciplined or motivated for whatever reason who did poorly. They weren't or wouldn't have been happy or done well in college either though, for the most part. And there were a few uninspiring intellects who got extremely high marks withe the sheer grind, although I think that's an impressive attribute itself and probably very useful for a lot of fields. There certainly were not hordes of Einsteins and Newtons just slipping through the cracks because they didn't test well.

So I don't know. There is this "common wisdom" that tests are terrible, which is an attractive idea. I just don't know if that's really true. Maybe they're actually pretty good for a large proportion of people.

prepend6• 4 years ago

Which is a good predictor for succeeding in college, including college success.

WalterBright3• 4 years ago

Really? My SAT scores were average for admit to the college I attended. I never had a tutor, never looked at test strategy guides, and took the test only once, and my parents were lower middle class. The people I know who got 800s (perfect SAT scores in those days) did not game the system - 5 minutes talking to them would reveal that they were very, very smart individuals.

Want to do well on the SATs?

1. take all the math classes your high school offers

2. pay attention in those classes

3. read a lot of adult-level books (I read tons of scifi)

and you'll do fine without needing any gamesmanship.

fasthands92• 4 years ago

Compared to grades it is certainly a magnitude less dependent on how much time/money you have, how many tutors you have, and how good you are at reading textbooks.

picture2• 4 years ago

I think a major counter point is how these tests can be improved upon individually without much improvement over other subjects. The first time I took the SAT, I went in totally blind and got 1320, a few months later, with a bit more practice with pre existing precalc, I was able to get 1490. I also had many classmate report similar trends of their score increasing as the number of retakes increases.

seibelj3• 4 years ago

There’s a range you can improve on. If you are an idiot you don’t have a chance of improving that much, but mommy and daddy can buy you a 4.0 at a private school. Can’t bribe your way to a 1600 SAT anywhere near as easily as the private school game can be played. And this is another reason why I like SATs over grades - the poor can get a leg up over the rich.

ghaff4• 4 years ago

I'm not sure you're making the point you think you are. As I recall the SAT many years ago, there were definitely question forms like X:Y as A:? which rely on vocabulary to some degree but may also be hard (or at least take some extra time) if the form is unfamiliar. Also even a surface-level understanding of some areas of math you realize you sort of didn't really get can be useful. So with that fairly modest knowledge and refresh, you got what's a possibly useful but also relatively modest uptick.

bloaf2• 4 years ago

This has been studied. There is a retest effect on most "cognitive performance" tests: scores increase on the first and second re-test, then level off. (i.e. the test scores for the 3rd and 4th time you take the test end up about the same)

gloryjulio• 4 years ago

But SAT is the aptitude test, and recognizing the importance of aptitude and working on that aspect would benefit the candidate greatly. You are supposed to be able to work and improve urself and benefit from it. A person is not an static object. What happened to `bring the best out of oneself`?

ummonk• 4 years ago

It measures your ability to perform on a new subject after studying for it, plus your willingness to study. The fact that you can study for it and improve your score (but not entirely - raw abilities still matter) is a feature not a bug.

thrown_225• 4 years ago

IQ tests show the ability to sit down and do pointless tasks which an authority figure has told you are meaningful and you must guess what that authority figure wants you to do.

Extremely valuable skills for white collar drones.

Completely meaningless as a measure of actual intelligence though.

tdba3• 4 years ago

IQ is the most reliable and most valid measure of general intelligence we have been able to come up with. It is good enough for many clinical purposes, eg assessing brain damage, and also just so happens to correlate very strongly with just about every "life outcome" you can think of. Does it measure everything imaginable about human intelligence? No. But it measures what we actually know about very well. To call it completely meaningless is absurd.

prepend6• 4 years ago

I’m not sure what IQ tests you mean but that doesn’t sound like any I’ve used.

The intent is to gauge general mental ability, not rule following.

Take spatial in tests [0]. They aren’t about rule following so much as problem solving and lateral thinking.

Also iq tests are standardized and normalized. They have flaws, certainly, and are no way meaningless. It’s the best that we have until we get something better and there’s lots of correlations between iq and positive life outcomes. [1]

[0] https://en.wikipedia.org/wiki/Spatial_intelligence_(psycholo... [1] “ IQ correlates positively with family income, socioeconomic status, school and occupational performance, military training assignments, law-abidingness, healthful habits, illness, and morality. In contrast, IQ is negatively correlated with welfare, psychopathology, crime, inattentiveness, boredom, delinquency, and poverty.” https://blogs.scientificamerican.com/beautiful-minds/iq-and-...

Maursault• 4 years ago

Grades are far more fuzzy but generally show an ability to stick with something over the long term and execute on projects.

What grades show is how well the student presented what the teacher wanted returned. Generally, but not always, students that are more intelligent than their teachers and may even have a better grasp of the subject don't receive the highest grades. Grades are about obedience, not learning. The "best" students are children of teachers.

Bubble_Pop_222• 4 years ago

That said I prefer IQ over grades.

Why would any rational individual take an IQ test? IQ cannot be changed and whatever outcome doesn't change your course of action, being the fact that you have to work hard.

Low IQ: Gotta work hard

High IQ: Gotta work hard

majormajor6• 4 years ago

Have you taken one?

It certainly felt like the kind of test that could be improved with practice. Not as easily as the SAT, sure, but certain patterns seemed to appear, in a way I bet I could prep for.

I haven't tested that hypothesis, but if anyone wants to fund me to take a bunch over the next year... let me know ;)

(Why did I take it in the first place? Curiosity. Maybe also a bit of vanity. "Rationality" seems like a false idol to focus purely on in general, for decisions.)

lmm2• 4 years ago

You could change your life/career plans based on the results. Whether to go to college or not, whether to pick a job that pays well immediately or one with more potential for long-term advancement, that sounds like the kind of decision where knowing your IQ would help.

nordsieck• 4 years ago

Why would any rational individual take an IQ test? IQ cannot be changed and whatever outcome doesn't change your course of action

From what I can tell, most professions have a rough IQ floor. If you're above the floor, more IQ can always help, but other traits like conscientiousness and agreeableness tend to matter more. But if you're substantially below the floor, there's almost no chance that hard work will help you bridge the gap.

skizm• 4 years ago

Very much disagree. Lazy genius is useless. Hard working, low IQ gets stuff done. Obviously hard working genius is ideal. But without the ability to focus on longer term goals, which grades is an indicator of more so than test scores, your high IQ means nothing.

candiodari2• 4 years ago

I would argue it very, very much matters what you're trying to do. If you're harvesting apples, then sure. Hard working beats IQ. Hell, IQ is a handicap, not a plus.

If you're doing research, you have the other side of the spectrum, IQ beats everything, and hard work can actually prevent you from finding solutions.

s1artibartfast2• 4 years ago

Good thing SAT scores also measure hard work, planning, and follow up. You can improve your scores significantly with the hard work of a few hours and checking out a sample test or prep book at your local library or school

rogers184452• 4 years ago

SAT/ACT are basically poor IQ tests, because study for them results in heavily loading onto memory aspect of IQ but unloading on some others somewhat.

Ideally, you would want an IQ test and the 'Big Five' personality traits. The latter though can be annoying to get.

I have seen some firms get very good results by basically using a disguised IQ test, there are some firms that basically compose the tests, give them to 1000s of people and perform the factor analysis to extract IQ from the test, then contract the the test out for hiring departments. Some of those tests are pretty creative in that most people wouldn't know they were taking an IQ test.

beaned• 4 years ago

Before people state strong assumptions one way or the other, have there ever been any studies that measured the relationship between test scores and IQ? If so, what did they say?

ransom1538• 4 years ago

"That said I prefer IQ over grades." The evil in this is: Your IQ doesn't change. So, have the the children take an IQ test at age 10, then just setup their college admissions right there. Deny kids that don't have the right IQ. No need to bother with GPAs or "trying".

maximus-decimus3• 4 years ago

I mean, it's the logic I used to decide not to pursue a career in professional basketball as someone of below average height. Do you think encouraging short people to pursue a career in basketball is productive in any way?

If you're 10 years old and have a below average IQ, don't bother aiming for a PHD at MIT in atrophysics.

SicSemperUranus2• 4 years ago

Test scores are highly predictive

Any study in this area that doesn't control for socio-economic factors is near useless, because it mainly predicts that children of successful (ie rich) people will likely grow up to be successful (ie rich). This is almost tautological.

buscoquadnary2• 4 years ago

That's funny I grew up as one of six children and my mother was a school teacher. Guess I have to go back to my HS and tell them they need to lower my scores because I wasn't rich enough.

BoorishBears• 4 years ago

You should instead chide them for not emphasizing the fact statistics measures trends, not absolutes.

xyzzyz4• 4 years ago

You seem to be completely unfamiliar with the research in the field. It is well known that controlling for socioeconomic factors does not substantially impact predictive validity of standardized tests. See for example

https://journals.sagepub.com/doi/abs/10.1177/095679761243873...

Scores on the SAT (a test widely used in the admissions process in the United States), secondary school grades, college grades, and SES measures from 143,606 students at 110 colleges and universities were examined, and results of these analyses were compared with results obtained using a 41-school data set including scores from the prior version of the SAT and using University of California data from prior research on the role of SES. In all the data sets, the SAT showed incremental validity over secondary school grades in predicting subsequent academic performance, and this incremental relationship was not substantially affected by controlling for SES.

Or this

https://eric.ed.gov/?id=ED562860

(a) SES is related to SAT scores (r = 0.42 among the population of SAT takers), (b) SAT scores are predictive of freshman grades (r = 0.47 corrected for school-specific range restriction), and (c) statistically controlling for SES reduces the estimated SAT-grade correlation from r = 0.47 to r = 0.44.

There are more studies on this, with rather consistent results. In short, poor students do about as well as rich students when matched on SAT score.

majormajor6• 4 years ago

Why would one expect socioeconomic factors to do one or both of:

a) influence grades in primary and secondary school

b) influence standardized test scores

but not also do:

c) influence college performance (esp. freshman year when you're least removed from those same effects in HS)?

gotoeleven• 4 years ago

Since acknowledging your evidence will inhibit my ability to blame the white supremacist cis-hetereonormative ableist patriarchy for my problems I will instead respond with an anecdote about how I know a lesbian indian in a wheel chair that bombed the SATs and is now president of the united states.

garbagetime2• 4 years ago

Your comment would be equally or more effective without this sentence:

You seem to be completely unfamiliar with the research in the field.

SicSemperUranus2• 4 years ago

I am actually familiar with some of the research actually, but thank you for your condescendence.

I didn't mean to imply that the SAT or grades are not predictive of future performance, only that lower socio-economic statuses will correlate to lower SAT grades in the first place, ie your (a) proposal in the second source.

From this same source:

The SAT-grade relationship is not an artifact of common influences of SES on both test scores and grades

To substantiate my initial comment, test scores are highly predictive yes, but as you just linked, SES is highly predictive of test scores (and this is only for people who get to the damn test). This is the tautological part. High incomes make people test well which lead to jobs with high incomes. But since this correlation is not 100% but merely very high, we can pat ourselves on the back and say hey it's accurate!

Articles that defend the accuracy of the system tend to implicitly defend its legitimacy, which I think is a much more important question than "is SAT indicative of future performance" when considering whether the SAT is useful.

Onanymous• 4 years ago

well, maybe, just maybe, that's because the socio-economic status correlates positively with cognitive abilities and the kids somehow inherit the traits of their parents?

wingely• 4 years ago

Meh. Dumb rich kids end up with positive college and career outcomes regardless of their test scores. Let’s not suggest that wealth correlates to ‘better traits’.

RobLach• 4 years ago

Who's saying that test scores are uncorrelated with college graduation, which is largely dependent on passing further tests?

majormajor6• 4 years ago

There's certainly a strong whiff of "is this the right question" to the whole discussion. School and school-adjacent results predict school results. Whoo.

Tests are just leetcode for students, after all.

fasthands92• 4 years ago

Lots of schools are dropping the SAT requirement

https://meet.nyu.edu/uncategorized/test-optional-at-nyu-what...

https://www.latimes.com/california/story/2022-03-23/csu-drop...

https://wtop.com/maryland/2022/06/maryland-university-system...

I don't know how this works in practice tbh. But it is curious.

alwayslikethis• 4 years ago

It's a strategy to game the rankings.

Factors for college rankings often include acceptance rate (lower is better), and average SAT/ACT scores. By not requiring SAT/ACT scores, you do two things at once:

1. you allow more people to apply, especially those who did not take the test or received a poor score, who you can simply reject to bring down your acceptance rate

2. by making it optional, only the applicants who think their score would be an advantage (i.e. those with high scores) would send them, which brings up your "average test scores".

WillPostForFood3• 4 years ago

I don't know how this works in practice tbh.

We have some evidence from MIT that it is a bad idea to drop testing.

https://mitadmissions.org/blogs/entry/we-are-reinstating-our...

travisjungroth• 4 years ago

The post starts with four quotes on that theme.

ajwinn7• 4 years ago

I worked in Admissions at a large state school from 2005-2007. Our method for determining admissibility reminded me of the heat map chart in this article - ie GPA on one axis, SAT/ACT score on the another axis, trace your finger along both paths to see if the applicant was in or out.

I probably reviewed 20-30 applications a day. If high school transcripts were universally formatted, decisions could have been instantaneous but alas, we live in the real world and some human-in-the-loop normalizing had to be done over-and-over.

With all that reviewing, patterns emerged, namely that SAT and ACT scores strongly correlate to GPA. Now, I’m the kind of person that roots for the genius to overcome his grades and emerge a genius on the SAT/ACT. But in two years, it probably happened only twice. Before calculating a normalized GPA, I could look at the test scores and predict “admit” or “deny”.

While the author is correct to say “the irrelevance of test scores is greatly exaggerated”, in my experience, whether or not something is irrelevant has very little to do with what universities do.

I’d recommend only using test scores. Or, only go with GPAs. Only test scores is more efficient. Only GPAs looks better on press releases.

yunyu• 4 years ago

The edge cases are more common at elite institutions, which are the typical admissions mania targets. At the top ~10 university I attended, there were plenty of people (myself included) falling in the poor HS GPA bucket (mostly due to extracurriculars) with perfect or near perfect ACT/SAT scores, among other outliers.

cammikebrown• 4 years ago

I had a 3.18 GPA and a 31 ACT, is that outlying enough?

ajwinn7• 4 years ago

From a state school’s Admissions perspective, where the unit test resolves to admit/deny, there is nothing particularly outlying about your scores from what I recall.

I see 31 and think, “admit”. I see 3.18, I think “probably admit”. I check your math scores (which is where a lot of people trip up), and see you got mostly Bs- heck, you’re on Hacker Mews, probably As, and I think, “admit”. Now I do the normalizing math and yeah, Admit. Congratulations!

TotoHorner• 4 years ago

Well, I had pretty good SAT scores (2350+) and a very poor GPA (3.0/4.0).

Main reason was that I just spent all my time pursuing ECs and didn't have enough time to spend on grades.

Think you'll lose a lot of smart kids who spend time on ECs that way.

ghaff4• 4 years ago

That was probably more or less my college story for grad school. Very good GREs but lousy GPA in college. (And only somewhat less my story in grad school before going to the next grad school :-))

rootusrootus2• 4 years ago

I can see the value of the GPA. When I was applying to Oregon State way back in the day (this would have been early 90s), my SAT score was really great. Probably top 10 in my school. My grades, on the other hand, were sub-2.00. I may have had a problem keeping myself interested in school. But I found tests extremely easy.

They rightfully told me to go to community college instead, because my GPA was abysmal. I'd have failed right out of OSU. I went to that community college and failed right out of it ;-). Then I went in the USAF, and maybe a year into that experience something clicked and suddenly I felt like an adult. I felt like my priorities realigned and I knew what I wanted and how to get there, and I could stick with it.

So when I left the military, I went back to the community college, got perfect grades, convinced the admissions gal at OSU to ignore all the HS & college grades from a few years earlier, and let me in. Graduated with my bachelor's in CS with excellent grades. Ultimately went on to get my masters, though that was years later.

Anyway, all of that to say ... my SAT score would have said "admit him" but my GPA was a more accurate assessment of my grit. I think both scores are useful, but don't give up on GPA.

TimPC• 4 years ago

GPA is problematic because not all high schools have access to the elite courses that push a GPA beyond 4.0 and many top schools look for an admission barrier around 4.5 (some with exceptions for valedictorians only).

dehrmann• 4 years ago

TFA actually calls out that very high test score can be a a signal someone won't graduate.

thomasahle2• 4 years ago

Anyway, all of that to say ... my SAT score would have said "admit him" but my GPA was a more accurate assessment of my grit. I think both scores are useful, but don't give up on GPA.

Do we care to measure grit? If someone can do the courses and pass the final exams, does it matter whether they did "sufficient grit" during the process?

wodenokoto• 4 years ago

But how should they measure "I matured and now I have grit"?

jkhdigital2• 4 years ago

Basically the story of my teens and early 20s except that my grades were good enough to get into CS at UIUC... where I promptly drank/smoked my way into an academic suspension. Switched to an easier school and easier major and failed out again which prompted me to enlist in the Navy. Graduated with highest honors from my boot camp class--it's fucking amazing what a little structure can do for people who need it. 16 years later I have a BS in economics, MBA, MSCS, and now trying to beat the final boss and get a PhD before I turn 40.

svnt• 4 years ago

I was the same. Top fraction of a percent on SATs without studying but never turned in homework. Got Cs.

Got into state school in engineering, cruised through intro term, almost failed out second term when the weed-out started. Felt the same thing click into place over the summer: might as well try. No military experience required — 4.0 from then on.

I think the whole process is biased against male students because they mature later, and there is nothing difficult in high school and no real form of soft failure to wake you up.

We ought to make things harder, but support and include people when they fail the first time. We should normalize learning from failure.

So much of it is down to your family environment. My parents hardly existed as an environment.

mgkimsal2• 4 years ago

I think my HS GPA was... 2.9 or perhaps 3 (out of 4), yet I had a 29 on the ACT.

The ACT score was in the 80s before some scoring model change. It's probably changed multiple times since then, but I'd learned there was a model change in 1990(?) which makes comparisons between them a bit less useful. My siblings had slightly lower scores years later, but because the model changed, there's not a lot of comparison we could make.

balderdash• 4 years ago

Out of curiosity how did you normalize for say 3.0 GPA at a competitive admissions magnet school, and a 3.0 GPA at an underperforming high school? Did you correlate pervious students from that high school and their high school gpa and their graduating gpa?

pirocks• 4 years ago

I have long wondered about this. Note that you also need to be able to normalize grades across students who went to the same school but had different teachers. I suspect the answer is that both normalizations are hard so admissions people don't bother.

ajwinn7• 4 years ago

Great question - the short answer is that our Admissions office did not attempt to normalize non-mathematical factors. At first, this surprised me. How often do we as students somehow get the impression that extracurriculars, magnet, Montessori, AP, IB, student council, <more trophies/achievements of your choice/> are somehow positively associated with college admissions?

From a non-Admissions perspective, accounting for these non-empirical differences seems wise and fair. However, they introduce subjectivity into the equation, which increases the likelihood of a human decision maker being unfair.

So what normalization did we do? What would you do? Knowing that no equation is perfect, you must start with a goal and find the least-bad equation.

The goal: Only admit students with a high likelihood of not failing out.

Why: Failing out has consequences for most US students who take out large loans and must repay immediately if they fail out.

The realization: High schools already provide a proxy for degrees of past failure - letter grades.

Assumption: Assume that high schools use their letter grading system as a method of normalizing and displaying how hard their courses are. A B+ should represent a certain level of achievement regardless of school. The school has the autonomy to use any letter system they choose.

Solution: Assign a point system to letter grades and calculate an average. Use only classes correlated to success at your university. This is the normalized GPA.

Surprise #1 - If your high school thinks a tougher-than-average letter grading system (eg a 93% is a B+) somehow makes students more attractive to the majority of the 7000 US colleges, they are wrong. And probably causing students to lose out on scholarships.

Surprise #2 - If you believe this system is unfair, you can aim the blame at technology. Applications to most universities (until recently I’m told) increase every year as digital applications make it easy to apply to 10 schools instead of 3. To handle the increase, colleges can hire 3x more admissions people, or find ways to speed up the process.

Surprise #3 - Colleges always want more applications than ever before. By dropping SAT/ACT as criteria, it removes a point of friction in the customer experience, and removes a step in the calculation effort. (If they could drop GPA and keep SAT/ACT, this would the ideal situation from a metric and efficiency perspective, but Marketing and the public would throw a fit.)

twblalock• 4 years ago

Entrance exams are a common option in other countries, and they are a good way for students who are late bloomers to get into top schools.

hedora• 4 years ago

According to the University of California’s own internal study, there is a small group of students with terrible GPAs and good test scores. They tend to do well in college.

There used to be a set percentage of spots for students in that situation, but the admissions people decided that “Standardized tests are racist!” is a good twitter sound byte and eliminated the spots.

Of course, their own reports said that the “bad GPA, high test score” cohort were mostly from underprivileged areas (and disproportionately black or hispanic). That nuance didn’t make the 140 character cutoff, so that group is officially abandoned by UC now.

In other news, they’re now promoting a big push to make the student body composition match the general population.

However, there is only one over-represented ethnic group in the UC system. Guess which one?

Hint 1: Whites are under-represented.

Hint 2: They swore they’d never bring back UC’s old “asian quota”.

I honestly think the people running this state’s school system are malicious racists. Their results speak for themselves: California schools are currently ranked 43rd in the nation. They used to be top ten.

lokar3• 4 years ago

You jump from the UC to primary/secondary schools. They are not run by the same people.

Most of the decline is due to underfunding due to prop 13

timmg• 4 years ago

According to the University of California’s own internal study, there is a small group of students with terrible GPAs and good test scores. They tend to do well in college.

This was me, fwiw. I’m so lucky that the engineering school I got into was enamored with my math score.

Spooky232• 4 years ago

The arguments against tests, from SAT to civil service exams is that effective tests correlate to IQ. IQ=racist, because we say so, so tests=bad.

hahaxdxd1232• 4 years ago

Actually California's state school system at the UC level is 1st in the country. And 1st in the world.

concordDance• 4 years ago

However, there is only one over-represented ethnic group in the UC system. Guess which one?

Does the ethnic group that gets a quarter of Nobel prizes and was hated by Hitler not count or something?

scarface749• 4 years ago

I just posted a comment about my step son who had stellar test scores after we spent $1200+ on a private tutor. Did he get more qualified for college than the kids who had parents who couldn’t afford private tutors?

ipaddr2• 4 years ago

Yes he is more ready to be a clog in the machine. He passed the first test. He studied enough leetcode and it doesn't matter how he got there. Will the tutor be there in college and will he be able to absorb all the information college gives are bigger questions.

ajwinn7• 4 years ago

Excellent question. I’m of two minds here. As a high school student, I too struggled to get great test scores. I took the SAT four times, which finally resulted in a score that could get me scholarships.

Did I become smarter? - I sincerely doubt it.

Did I become less likely to fail out of college? …Seems unlikely, but I don’t know.

As an admissions rep, I understood that high scores, whether SAT or GPA, are certainly both proxies for a prediction: If I admit this student, are they going to fail out?

Unknown: It actually might be the case that students who get private tutoring are less likely to fail out because their parents can financially endure them struggling to repeat failed courses. I’d be interested to see a study on that.

hackinthebochs2• 4 years ago

The underlying assumption in these anecdotes is that if everyone had equal access to test prep resources they would show an equal increase in performance. But the studies[1] don't bear this out. If someone is an outlier in terms of coachability, that probably means there were gaps in knowledge that could be filled quickly to have an outsized effect on score. But this seems relevant to suitability for college admission.

[1]https://slate.com/technology/2019/04/sat-prep-courses-do-the...

club_tropical5• 4 years ago

Was there a secondary adjustment step to finalize admit/deny? Your heatmap-based method would almost certainly result in an unacceptable racial composition of the class.

ajwinn7• 4 years ago

I worked in the underrepresented student office, so I was keenly aware/curious about that possibility when I began. However, after reviewing thousands of applications, my experience, to my surprise, was that GPA to SAT/ACT scores remained highly correlated regardless of race. If memory serves (and it’s been awhile) the outliers who were more likely to have higher SAT/ACT than their GPA normally suggested, tended to come from predominantly non-white schools. I’ve seen so many reports to the contrary that this surprised me to no end, but that’s what I observed.

dghlsakjg• 4 years ago

I'm your outlier!

My ACT was 34 and I can't recall my SAT score but it was also 99th percentile. My GPA on the other hand was abysmal. I can't recall, but I wouldn't be surprised if it was well below 2.5.

Once I actually made it to college I was a pretty good student, though. Graduated with honors in 3.5 years.

vamsipk• 4 years ago

Sure, we can use only GPAs as long as all the applicants are graded on the same common platform. But today the GPA from one college is not exactly equivalent to the GPA from another one. The grading system as I heard is also not standard across US (leave alone international) with some instructors using absolute while others relative grading. So an A student from college X is not necessarily as good as a B student from college Y.

Tade0• 4 years ago

With all that reviewing, patterns emerged, namely that SAT and ACT scores strongly correlate to GPA.

My school experience was that it's easy to cheat during the year and much harder to do so in exams, so often people with generally good grades would do at best average at the end.

I always had terrible grades(except for math), but they're not part of college admission scores in my country so I had no incentive to do anything about it.

evrydayhustling• 4 years ago

Yes, nobody is dropping test scores because they are irrelevant -- although that overreach does exist in the press. They are dropping them because they add little after GPA is considered and may introduce costs, barriers and biases.

The author identifies the steel man version early on: "standardized test scores are nearly useless (at least after you know GPAs)" but then mostly ignores that parenthetical. This part that addresses it seems deliberately obtuse:

If you care about the difference in accuracy between GPA and ACT, then you should care even more about the difference between (GPA only) and (GPA plus ACT). It’s incoherent to simultaneously claim that the GPA is better than the ACT and also that the ACT doesn’t add value.

Again, the version of this that reasonable people argue isn't that the ACT doesn't add any kind of value, it's that the amount and kind of value it adds (incrementally more predictive accuracy) aren't worth the costs and biases are introduced. If you myopically value accuracy, it doesn't make folks who want to balance accuracy with other goals incoherent.

bena3• 4 years ago

I guess test scores are more efficient because GPAs are somewhat subjective and can be gamed to a degree. By both the school and the student.

Two schools with two different grading tolerances can produce students with similar GPAs but different SAT/ACT scores. As the SAT/ACT is a national standard, not a regional one.

anthlax2• 4 years ago

What do you think about GPAs being non-standard? Test scores calibrate between schools - GPAs only kind of do.

I remember many classes that were taught by two separate teachers: one teacher consistently gave students higher grades. The other teachers students consistently did better on the AP test - much more strict with homework and test scores, but it ended up positively impacting the students.

lupire5• 4 years ago

Why did't the other teachers correct their grades at the end of the term?

asimeqi• 4 years ago

Short term one get can rid of test scores or GPA since they indeed seem to be correlated. But long term:

1. If we get rid of test scores, it would cause too much upward pressure on GPA which would make GPA less reliable. There are tens of thousands of high schools in the US. Even if the GPA distribution inside every high school is correct, how do you compare with GPAs from other high schools? And from experience I can predict that, once people realize that only the GPA is important, there will be outside influences (i.e. pay the teachers for good grades) that could cause even the distribution of GPA inside one school to not be correct.

2. If we get rid of GPA, students could just study for the test with obvious disastrous effects for their education.

We need both GPA and test scores to keep both of them in check.

commandlinefan3• 4 years ago

to overcome his grades and emerge a genius on the SAT/ACT

Isn't it usually the opposite? Artificially high GPA from grade inflation and favoritism is "set straight" by a standardized test?

curiousllama• 4 years ago

To be fair, some state schools do operate on just GPA. I recall my HS had that GPA/SAT scatterplot available for past applicants: UVA had a flat cutoff at ~4.1 weighted GPA, and the SAT literally didn't matter at all.

2devnull• 4 years ago

Grades and gpas are based on tests too, but on crappy tests without any psychometric validity, often correlated with race and broken in a bunch of ways which standardized tests are not. How many high school math teachers run irt or dif analysis on their tests? Meanwhile, standardized testing companies spend a lot of money and energy trying to remove race from their tests. It is as though smart people have been working really hard on this problem for many decades and found an optimal solution. The solution is standardized tests.

d0liver• 4 years ago

My GPA was bottom 10ish at my high school. I graduated high school essentially because I already had most of the credits I needed to graduate from middle school. I got into college based on test scores alone (and I was surprised that this was possible). My GPA improved a bit early on in college and then improved much more around junior year once I started getting into harder classes. I ended up graduating with a math + CS double major in 4.5 years.

So, sometimes people are just bored. And there are a bunch of other factors here too - perhaps the biggest one being that "graduated within six years" is not a great indicator of success. Linus Torvalds did not graduate within six years but did build the first version of Linux at university.

GPA is a good indicator of how much work you're willing to put into what you're learning _now_, so it tracks well as a predictor of how much you care about GPA which is a pretty good predictor of whether you'll graduate in X years and probably not much else.

Jerrrry2• 4 years ago

28 on the ACT, no studying.

1.4 GPA.

I was eligible for the maximum academic grants for my state, just to be denied due to GPA because the string of high schools I went to were utter dystopian nightmares.

this tracks.

nemo44x• 4 years ago

Another problem with relying on GPA is it is easily gamed. Parents and students can pressure teachers and administrations since so much is reliant on it. And a 4.0 at one school may not be the same at another school.

I understand some people "test poorly" but the truth is, if you actually know the material well, you'll test just fine in most cases. We can also coach students on how to approach testing. Other ideas are to break the test into various sessions with different admins to limit cheating potential and to take an average of the scores (or some other method) for each subject.

I guess my point is - make test results the metric to remove all bias. Change how we approach testing and preparation.

matai_kolila3• 4 years ago

Your refusal to take responsibility for your part in that 1.4 GPA also tracks, heh...

garyrob• 4 years ago

My high-school GPA was mediocre because I simply was unable to pay attention in class, or to study at home, or to turn in homework that wasn't dinged over and over for "careless errors". I was born in 1956. Given today's knowledge of things like ADHD-Inattentive, I'm pretty confident that I would have been diagnosed. Some drug would probably have been prescribed, and it probably would have helped my grades (although I'm not convinced it wouldn't have hurt me in other ways).

But for some reason, when I took my SATs, I found it to be fun. I got completely immersed for the period it took to take the test, and truly gave it all I had, with real focus. I got the best SAT scores in my school, which was shocking to some people.

And it turned out that that was far more predictive of my future, including my college GPA performance, than my high school grades were.

tristor3• 4 years ago

Interestingly, I was one of those "geniuses" that had a perfect ACT score but a horrible GPA (long story). I got rejected by every college I applied to except for the local state school, so that's where I ended up going. In the end I dropped out in my final semester and just went to work, but that's besides the point. I was more than qualified to academically excel at any university I wanted to attend, and I maintained a 3.9 GPA in college up until I dropped out.

The problem with using just GPA is that GPA can easily be manipulated by school authorities who dislike you, test scores cannot.

insane_dreamer2• 4 years ago

SAT scores seem quite unfair as a single indicator of performance. GPA at least provides a view of a much wider range over which performance may naturally fluctuate.

KevinBenSmith• 4 years ago

Great & useful article. We should have more of these for “popular papers”.

jeffhwang• 4 years ago

Agreed! Although, I admittedly didn’t read the post carefully enough to evaluate how accurate its analysis of the original paper. But all else being equal, we’d live in a better world if papers that get a lot of popular attention are subjected to this scrutiny with an eye towards explaining what works/ doesn’t work for a general audience

xbar• 4 years ago

My favorite point:

"2. It might be that test scores don’t predict college graduation rates[...] is provably false."

WBrentWilliams3• 4 years ago

Cute. Disclosure, I make the data (transcripts, test scores) flow to decision makers as part of my day job at a Big 10 University. We just went "test optional".

I view "test optional" as a signaling exercise. In brief, how closely can you read the instructions and infer what you are not being told? The key is "test optional". If you sit for the test, you get more than just admissions, as merit-based financial awards are unlocked. The University Admissions does not require a test, but many of the donor contracts that fund scholarships do require those tests.

felideon• 4 years ago

Interesting, and good to know! But wouldn't you say that students are better off not submitting their scores if they scored low on the tests? Only given a sufficiently high enough score does it become a "nothing to lose" situation.

Unless the issue is students just don't want to take the test to begin with.

WBrentWilliams3• 4 years ago

I'm saying that you can be admitted without test scores. You leave money on the table when negotiating the bill when you do not take the test. Opting not to take the test could be an optimum strategy if you already know that the offered awards will not apply to you.

tqi2• 4 years ago

I view "test optional" as a signaling exercise. In brief, how closely can you read the instructions and infer what you are not being told?

Sorry, what does that mean - who is the "you" in that sentence, and what are they inferring?

It sounds like what you mean is that that your school went test optional (ostensibly to help level the playing field), but those who are lucky or privileged enough to have the secret decoder ring can decipher the instructions and find the key to optimizing their chances of admission. If so (as someone who grew up with immigrant parents who were educated but definitely would not have had the awareness to navigate this), that seems really fucked up...

WBrentWilliams3• 4 years ago

The English language has a plural "you". The word "you", in this case, means "anyone who applies for admission". I said nothing about increasing or decreasing your chances of admission.

olliej5• 4 years ago

I think there’s an issue on how these data are examined.

There is a huge amount of data backing up the claim that the school attended influences the SAT, etc.

There is a huge amount of evidence backing up the claim that SAT/ACT aren’t 100% predictive of later academic success.

But “not being 100% predictive” does not mean “is not predictive”.

The core problem the removal of SAT, etc is attempting (imo hamfistedly) to mitigate is the following chain:

(1) Student A comes from affluent family, in an affluent region.

(2) Student A goes to a private school with better teachers, etc, or because it’s an affluent region the public schools have better teachers, etc

(3) Student A’s family can afford additional SAT, etc tutors

Now imagine a somehow identical Student B, but from a poor family in a poor region. Before any other factors become involved, they don’t get the benefits of 2 or 3.

We would expect that on average 2&3 result in higher scores, and so a higher acceptance rate, and a higher rate of scholarships for Student A vs Student B. Again this is on average.

The TLDR is given two kids who have identical mental ability, the one from the richer family will, on average, have better scores. So schools say let’s drop the scores entirely, but that is also clearly nonsense: Even in this exact scenario Student A will do better in uni or what have you, because part of why they got a better score is that they have been taught and learned more - IE they absolutely do know more and are more academically proficient than Student B by this point. Moreover we know (because of yet more studies), Student A will tend to continue to do better than Student B once they’re at school, even once B “catches up”, because we know that people with fewer external stressors perform better at everything.

So even if we discard the score based entry to unis, we’ll see a strong correlation between original score and subsequent academic success.

So while I understand the desire to remove the “accident of birth” factor that benefits the lucky and penalizes the unlucky, simply removing the test requirements is fairly questionable as an approach.

This is also before asking about Students C and D who are the peers of students A and B - surely the better scoring of A and C (or B and D) is who you should prefer, but now you’re discarding theoretically ideal and unbiased data in favour of…?

faeriechangling• 4 years ago

Test scores are accurate at predicting your ability to succeed at further testing, I don't know if this is all that controversial.

What people are getting up in arms about is the validity of testing as a whole regarding real life outcomes, and this is an effective argument as it is damn hard to grade people by any metric that doesn't get gamified to death and resulting in people wasting a ton of time on playing the game.

nickelpro• 4 years ago

A student who scores highly on a Calculus exam may be generally good at Calculus, or maybe they know a couple exam-specific algorithms to succeed and got a little lucky besides.

A student who fails a Calculus exam is nearly guaranteed to be poor at Calculus.

To say exams have no predictive power is to ignore the obvious.

thethethethe7• 4 years ago

Student A that passes a Calculus exam may be be more likely to know Calculus than student B that fails but so what?

Passing a Calculus test doesn't necessarily predict if the student can apply their knowledge to production to generate new ideas and widgets which make our lives better, it just predicts that Student A can follow rules and learn something _when required by an authority_.

I think this sentiment is similar to how people feel about Leet Code interview questions. So what if an engineer can invert a binary tree if they can't formulate new, useful ideas and convince others to work on them?

Perhaps student A is more likely be create new, better widgets than student B but I think academicia/LC cannot test this directly, which is what people are getting at when they criticize tests.

Der_Einzige4• 4 years ago

I remember doing so well in calculus until the 3D graphing unit.

I think I ended up with a D or worse on that one units test. My friend was like "oh bro you just add a pi to everything". This isn't thanksgiving and it's more complicated than that!

klabb3• 4 years ago

Test scores are accurate at predicting your ability to succeed at further testing

Very well put. The whole question of "do test scores predict college graduation" is easily conflated with "do test scores predict success/ability". In reality, college is a big set of tests. It would be interesting to compare with other metrics.

club_tropical5• 4 years ago

Barring fraud, standardized tests are not gameable, unlike GPA, references, etc. You can study and improve your score, of course (just like you can study and improve GPA) but that's not "gameing" in the subversive way you mean it. Their ungameability is inconvenient for people that are looking to obfuscate the real admission criteria, which is why there is enormous effort to downplay their predictive performance.

britch4• 4 years ago

You can absolutely "game" tests.

They're taught as "test techniques." And there's plenty of tutors who teach it.

Process of elimination, read the questions then passage, common question structure, are all examples.

These are "skills" but they don't really apply outside of testing.

Hell I went into standardized tests with common question solvers programmed in my calculator. If that's not gaming I don't know what is.

TrackerFF• 4 years ago

I've only gone to school in Europe (Norway, specifically) - so I'm not sure how one can game their GPA. I'd say that a solid 80% of my degree was graded/scored on written or oral exam. Rest were home exams, or a mix of both - with 10% being the Masters thesis alone.

But then again, we don't have any standardized tests here. Not for college admission, or afterwards. The grades you get, are what you have - and everything is judged by those grades.

paraknight• 4 years ago

I can't believe I had to scroll this far down to find this comment! People are just assuming that positively correlating testing with further testing somehow implies that the way we test is indicative of future work performance or even your quality as a human (whatever that means). They may be correlated, but a person's ability to sit exams moves the needle much more significantly. More so for universities — I've taught many students who graduate with great scores that I would never ever hire.

Using the same methods to measure wildly diverse people is a tragedy that causes individuals who could otherwise contribute to society in very meaningful ways to have fewer doors open to them.

Aunche3• 4 years ago

College graduation rate is a real world outcome, which is author is referring to. You don't have to be particularly good at testing to graduate. Cs get degrees. A lot of courses are project or paper-based. If all else fails, you can take GPA boosting classes.

Of course, the far left will claim that "real world outcomes" are also systemically biased when they don't align with their priors.

rayiner6• 4 years ago

The implication of the “test scores don’t matter” argument is that we have the wrong people staffing virtually every credential-based position. There’s lots of kids with 4.0 GPAs from high school at directional state university getting education degrees. Test scores is what distinguishes those kids from the ones at Stanford. And Stanford is where everyone from Big Tech to Big Banks to Big Law to Big Consulting goes to hire. Who is working at Pfizer or Astra Zeneca doing drug development? It’s the people with high test scores. Every single Supreme Court justice in the last half century got there based on test scores. (The nominee whose test scores led her to a law school ranked outside the top 50, Harriet Miers, has to withdraw because of that.) Warren Buffett got into Wharton based on test scores. Charlie Munger got into Caltech based on his scores on the army’s intelligence test. Bill Gates got an 1590 on the SAT. Even the professors saying test scores don’t matter got where they are because of test scores.

Maybe that’s all true. But it’s quite a remarkable claim that would turn the world as we know it upside down.

karpierz2• 4 years ago

Let's steelman that a bit.

If you structured society such that only the top 0.5% of households by income could send their kids to college, you'd find that a lot of your successful entrepreneurs, judges, scientists, etc. went to college and had a high income growing up.

But I don't know that proposing something else, like standardized testing, as an alternative to this would be quite a remarkable claim. You're just saying that the scoring function produces a lot of false negatives.

rayiner6• 4 years ago

In your hypothetical, institutions would quickly realize that a bunch of non-college graduates were just as smart as college graduates. Which is actually the public perception in places where jobs are filled based on social status and connections rather than objective metrics.

What I’m saying that the current scoring function appears to match up pretty well with our intuition about who is smart enough to work on Wall Street or at Facebook. At least, these profit-seeking institutions don’t seem to think it would be better for them to shift their hiring from Stanford to kids with 4.1s and lower SATs who went to UCSB instead.

jonnybgood• 4 years ago

So a test score opens doors. It’s more of a statement about how our society and institutions are structured. That implication says nothing about the quality and acumen of a person once they step through those doors. Test scores do not determine qualities and acumen, only that you meet a standard (acumen and standard is not always one-to-one). All those you named got the opportunity to demonstrate their qualities and acumen because a test score opened a door for them to do so. So I don’t believe test scores have ever mattered to society or institutions other than that: opening a door and not a guarantee of any sort of success once you step through it.

cbruns• 4 years ago

There's no shortage of students at state universities with elite test scores.

derbOac5• 4 years ago

I think what you're pointing out is the point of the "test scores don't matter" argument actually.

masterof0• 4 years ago

I don't think test scores are a predictor of success, but I do think they are one of the best metrics we have, and actually help poor kids to "make it" to good schools. Rich kids might have access to better teachers, but Math is the same for everyone, a poor kids using materials freely available in a public library can do very well in exams, allowing him to get into an elite school, otherwise only reserved for the elites.

jklm3• 4 years ago

Test scores have already been irrelevant for certain segments for over a decade, speaking as someone of East Asian ethnicity. Personal anecdote: 3.7 GPA in average high school with 8 APs (some of which were self-study), 2400 SAT, 36 ACT, lots of state-level top 3 finishes in multiple academic competitions, varsity track, musical instruments, first generation immigrant, etc.

Ended up going to what I thought was my safety school and rejected/waitlisted at every reach school I applied to. I sometimes wish colleges mandated name/ethnicity-blind application reviews - not to sound ungrateful, but I’m still incredulous to this day that the best I could do was a public state school (still top 50 admittedly).

It ended up being a great experience in that it was a forcing function and made me realize most of these rules and expectations around admissions were meaningless. But I probably could have saved years in high school exploring things that mattered rather than optimizing for a college application process that didn’t ultimately end up feeling very fair.

testfoobar2• 4 years ago

Glad it worked out for you. Asian college applicants have it rough these days.

This is the President of Stanford speaking out of both sides of his mouth:

2022-10-12: "Stanford apologizes for admissions limits on Jewish students in the 1950s and pledges action on steps to enhance Jewish life on campus". Stanford President Marc Tessier-Lavigne apologized on behalf of the university and pledged action on recommendations in a task force report confirming Stanford limited the admission of Jewish students in the 1950s.

https://news.stanford.edu/report/2022/10/12/task-force-repor...

2022-08-02: Amicus brief filed in support of Harvard and University of North Carolina “A diverse student body enriches the academic experience for all Stanford students,” said Stanford President Marc Tessier-Lavigne. “Considering race as one part of holistic review helps us foster a diverse campus community, one in which all students have the opportunity to learn from each other’s experiences and to think critically about their own views and preconceptions. It also means that Stanford graduates from all backgrounds can go on to bring their unique insights to leadership roles in the government, business, and nonprofit sectors.”

https://news.stanford.edu/report/2022/08/02/amicus-brief-fil...

Guess which race/ethnicity Stanford is limiting now?

pannSun4• 4 years ago

pledges action on steps to enhance Jewish life on campus

It seems Jewish life is already thriving on campus - despite being just ~2% of US population, they are 7/11% of Stanford's undergraduate/graduate students [1]. To put that into perspective, and excluding from analysis the 10% international students, they are 3.9x over-represented among non-international undergraduates, relative to their US population.

In contrast, white (including Jewish) undergraduates are only 32.6% of undergraduates, but 57.8% of US population [3]. Meaning non-Jewish whites are 25.6% of undergraduates, making them 0.49x under-represented.

So despite a Jewish person having a 7.9x greater chance of getting into Stanford than a non-Jewish white, Stanford's president is apologizing that this disparity isn't even greater.

[1] https://www.hillel.org/college-guide/list/record/stanford-un...

[2] https://www.collegefactual.com/colleges/stanford-university/...

[3] https://en.wikipedia.org/wiki/Us_demographics

selestify• 4 years ago

Wow! East Asian here who didn’t do anywhere near as well as you with test scores (2200 SAT, no competition wins) but still transferred into a top 20 school. Can I ask when you applied for college? I’m class of 2012.

barry-cotter9• 4 years ago

but still transferred into a top 20 school

Columbia? A public school in a state that mandates acceptance of Community College graduates with good enough grades?

HyperSane• 4 years ago

It makes me really angry hearing Harvard defend their racist admission policy, especially when you know it was designed to limit the number of Jews admitted.

cr4nberry3• 4 years ago

Something similar happened to me. I had a perfect high school GPA, 1540/1600 SAT. Applied to several schools, all of which I was above average score wise

Only got into flagship state schools that were well below my average SAT

The one thing I've learned from this (a valuable lesson): never, ever do the self identification. You'll always get fucked. Self identifying is always a mistake

User23• 4 years ago

If you don’t self identify you just get binned as white. Putting down mixed race is the standard workaround. It’s vague enough that you’re probably not lying and it avoids being labeled white.

0x00000000• 4 years ago

I had the nearly the exact same stats and situation. Even the mf colorado school of mines rejected me

hindsightbias• 4 years ago

I wonder if a court would allow Asian teenagers to change their last name…

googlryas3• 4 years ago

As someone who applied to college too long ago to remember anything - do you self-provide your demographics? Or do they try to guess what race/ethnicity you are based on your name?

rayiner6• 4 years ago

You should vote your interests. The end goal here is a system of race-based patronage. Instead of just taking an objective test like an equal, your kids' futures will depend on breaking out sob stories to tickle the fancy of white gatekeepers. Not just for college. For jobs. For promotions. The sky's the limit.

Prop 16 failed, but that won't stop them. Asians at Meta and Alphabet look to your left and look to your right. One of you will be gone when corporate American implements the same policies as Harvard (which cuts Asian numbers in half). And who is standing in the way of that future? Republican judges.

testfoobar2• 4 years ago

There is another solution to this: inflation. As inflation eats away at income, assets and savings, true merit & productivity will be the only thing that matters. Look at big tech earnings today and the reaction in the stock market. Big tech can increase their EPS any time they want by shedding 50% of their workforce.

We're no where near such draconian decisions. But if we get there, the decisions will be based entirely on the financial contribution of an employee/team/project/department to the company's bottom line. I don't believe tribal/race/group affiliation will matter.

jancsika• 4 years ago

Personal anecdote: 3.7 GPA in average high school with 8 APs (some of which were self-study), 2400 SAT, 36 ACT, lots of state-level top 3 finishes in multiple academic competitions, varsity track, musical instruments,

I love how the GPA requires a decimal place of precision, yet musical achievement requires nothing more than listing the plural of "instrument."

readthenotes12• 4 years ago

I was shocked they didn't divulge their 400m time.

We all care about different things, I guess

qwerpy2• 4 years ago

Wow, so what does it take to get into the top schools as an Asian?

My (Asian) kids are going to face the same challenges when they apply for college. When the time comes, I'm tempted to ease off the whole tiger parent thing, let them have a more well-rounded high school experience, and go to a public state school. It's what I did. Had to spend a few years clawing my way into a FAANG compared to graduates from fancier schools, but in the end it worked out.

bcrosby95• 4 years ago

IMHO, let your kids be kids, don't push them too hard. Life gives you lots of chances if you have a support structure - my family was full of fuckups but our parents were there for us and we all eventually made it through with good careers.

The main thing I want for my kids (they're still young - 7 & under) is to find things they're passionate about - things that makes them want to work hard and overcome a challenge. That could be school, sports, music, games, or whatever.

Of course, I'm just a random internet stranger, why would anyone listen to me :)

csa13• 4 years ago

so what does it take to get into the top schools as an Asian?

The comments below on how to get into an elite university (like Harvard or Stanford) have a fairly high hit rate when all of the boxes are checked. This is a fairly detailed and actionable list, but I am sure I left some important parts out (esp. regarding edge cases and corner cases). Feel free to ask follow-up questions if you have any.

My comments/suggestions:

-Strong grades and strong test scores will get a student into most state schools. This is a very solid baseline goal, imho. These same grades and test scores will be table stakes at elite schools -- more will be required in order to be admitted.

- Make sure your kids are actually interested in going to an elite school and making the extra effort that entails. If they are not, then steer them towards state schools. The comments below assume that they want to go to an elite school and are willing to make the extra effort.

- Take a challenging curriculum and get As with maybe a B or two -- the goal is to be top 5% of the graduating class, ideally the top 1%. Note that Bs (more than two) can be overlooked very easily if the rest of the application is strong. Note that the grade criterion can absolutely suck if your kids go to a high school that thinks that AP/honors classes just means that the students should be assigned more busy work. The busy work will take time away for them to do something that is actually interesting (see below).

- Learn how to write well. This will serve the student well in the application, HS classes, college classes, and life in general. Note that students will probably be rated on their writing skills by their references, and they will need to rate extremely highly (e.g., top 1% ever for this teacher).

- Do solid on the SAT. Contrary to popular belief, you do not need a perfect score. That said, a strong score is table stakes. Specifics below.

- My main advice for folks is to prepare for the SAT, mainly by taking practice tests with recent old tests at the same time and a similar location that the student will be taking the real test. The SAT is a marathon, and most people perform below their potential due to fatigue. Getting used to the long slog and the pacing of taking the test helps most people a lot. They will also be able to find gaps in their test-taking skills, and these are usually fairly easy to address when not under time pressure to do so.

- Math -- I would definitely aim for a perfect score in the Math (since it is fairly basic), but a slightly less than perfect score in Math is fine. Engineering people and folks who apply to MIT tend to have perfect or one tick below perfect.

- Verbal -- Most people try to game this a month or two before the test. Big mistake. My suggestion is to read well-written newspapers like NYT, WaPo, and WSJ as well as "high brow" magazines like The New Yorker, The Atlantic, and/or National Review. They should make note of every word that they don't know, and they would do well to learn them. This is easy to do over three or four years and tougher to do over one or two months. Bonus points if you, as a parent, discusses these articles with them so that they are reading critically rather than just casually. The text structures are the same types of (edited and abbreviated) text structures that they will see on the SAT.

- More than grades and SAT scores, the student should be able to impress one or two teachers that they are in the top 1% of students that the teacher has ever seen. Why? This is a common question on the reference letter form. I strongly encourage the student to check out the reference letter forms early in their high school career to see the manner and extent to which they need to impress the people who write them references. The student can do this by showing interest in one or more areas and really going deep into the subject beyond what is in the high school material -- basically, have passion for the topic.

- Also note that there are schools and teachers who know understand how the admissions game is played. The two things that these folks do that help a lot are: 1) to round up their rating of the student (e.g., if top 1%, maybe rate them "best I've ever taught"), and 2) to give detailed examples of outstanding work the student has done. The recommendations give credence to whatever it is that the student writes in their application. Picking who writes recommendation letters is a very important decision that I think many applicants put very little thought into. Not only do these teachers/people need to know you well, but they need to be able to write a good recommendation letter as well (this is a rare talent, imho, unless you attend an elite university feeder school).

- Related to grades/subjects, I would personally encourage students who can do it to enroll in a joint hs/jc program. There will be much less busy work, and one or more of the professors will know how to write a good recommendation.

- Try to figure out one or more areas in which they can do something notable at a national or international level. There is a wide scope of what this can look like, but this is where most people have a substantial gap in their application. This area can be social/leadership (probably the "easiest"), sports, arts, or academics (probably the toughest). I buried this relatively deep in the list assuming most people won't read this far, but this is the one thing that really sets aside exceptional applicants from threshold applicants who have great grades and test scores.

- Examples of "notable" (social/leadership): Start a successful business (ideally in an interesting area), start a non-profit that does meaningful work, develop a community program that makes meaningful change (esp. in under-served communities), develop some ongoing development aid project abroad (often seen in church missions), amazing Eagle Scout service projects. Note that all of these are starting or leading a group to new levels of success. Just participating is not enough (anyone can do that).

- Examples of "notable" (athletics): Be a recruited athlete, be an athlete that is not recruited but would be competitive on the university varsity team and expresses an interest to play at the university, win a state championship, be chosen as "all state" in your state (preferably first team), be on an Olympic team or on the short list development squad, win a regional (or wider) event in an "Olympic sport" (e.g., figure skating, ice skating, diving, swimming, etc. that might not be covered in HS sports). On a personal level, I would probably encourage folks to look at less common sports that can be excelled in at a young age -- competitive shooting, obscure martial arts (Japanese sword drawing, anyone?), noodling, etc.

- Examples of "notable" (arts): Be a published author, have an exhibit in a gallery, win a national/international competition (this might be easier than it seems if you're clever), be a national/international touring artist. Note that this can be combined with social/leadership by doing something like creating a highly regarded arts competition, creating a successful touring music group, etc.

- Examples of "notable" (academics): high placement in a nationwide math contest, be selected for the international science fair (a series of competitions), win a Westinghouse competition, publish a research article (as primary or a significant contributor). Note that this is the toughest category to stand out in, imho. The people who apply to elite schools are absolutely crushing this category. "Placed 3rd in state with HS trivia team", while commendable, won't really look that impressive or unusual when compared to a Westinghouse winner.

- With regards to the "do something notable" point made above, this something should be referred to in one or more of the student's references, otherwise it looks like it is completely made up and will probably be discounted. Supporting documentation helps (e.g., awards, newspaper articles, published papers, etc.), and be sure to send in this documentation as "supporting materials". If your kid is doing something special, I hope that they reach out to the press to have their special thing covered -- local news outlets love stuff like this, and it looks great in an application.

- Note that there are weird edge cases in admissions. As an example, UPenn is a private school, but they have to accept a certain number of PA residents per year. While there are extremely strong students from PA at UPenn, there are also some students who are relatively weak (and it shows). Also at UPenn, each of the four schools has an independent applicant pool, so the Wharton pool (9% admission rate) is different than the Nursing pool (25% admission rate). Not to pick on UPenn, but I happen to know details about these two examples.

- When applying to a school, the applicant should have a good reason for going to that school. "Because it is #1" or whatever doesn't cut it. The access to resources at elite schools is amazing, and the applicant should show that they have at least considered how they might utilize some of those resources.

Der_Einzige4• 4 years ago

You can grind leetcode for a few months and get a FAANG role with a boot camp degree. Don't bother tiger parenting.

csa13• 4 years ago

There is something missing from your story, but I can’t exactly figure out what it is.

Some things that may have gone wrong:

- Maybe your GPA was too low for your high school. If it put you outside of the top 10% or so of your class, then that hampers your chances substantially.

- Maybe your references did not know how to write good reference letters for elite schools. I think this limits a lot of people, since tepid reference letters from people who don’t know better can kill an application.

- Conversely, maybe you went to an elite university feeder high school. If so, maybe you didn’t compare well against other folks who did get into elite schools other than test scores.

- You applied based on academic prowess, and maybe your academic prowess was not enough for elite schools. Elite schools typically want national or international levels of success in something (especially academics, which is hyper-competitive), so maybe your state competition results combined with a relatively low GPA (for elite schools) didn’t really match up to the other 100-200 or so people at each school who got in on the back of academics.

- Maybe you didn’t have much/any significant community involvement and/or leadership experience. This may seem soft, but it makes applicants stand out, imho.

Fwiw, I don’t think your ethnicity played as much of a role as you think it did. Those killer test scores you have are basically table stakes at elite schools (i.e., you need a lot of something else to get in), but they are a meal ticket at large state schools. Many/most people think that test scores and above average grades alone should be enough for elite school admission, but this hasn’t been true for many, many decades.

For reference, “top 50 state school” is something like University of Georgia or Ohio State University, both the type of school that will not slow down someone who would have fit in at Harvard or Stanford, imho.

But I probably could have saved years in high school exploring things that mattered rather than optimizing for a college application process that didn’t ultimately end up feeling very fair.

I guessing that you optimized incorrectly.

I’m curious about where you got your information on how to optimize for elite school admissions. Most people will say something like “have great test scores and great grades”, and they would be wrong for elite schools but very accurate for most state schools.

I don’t want to say that elite school admissions are perfect (or even good), but people who have never been able to see the processes from the inside (e.g., actual applicant pools) usually make very unreasonable criticisms of the processes.

Fwiw, every elite school tells applicants exactly what they want in their recruiting materials. It’s just that different people interpret those materials in different (usually inaccurate) ways.

All that said, I’m glad it worked out for you.

jklm3• 4 years ago

Yeah, in hindsight fair to say a little bit of all of the above. I agree with a lot of what you mentioned.

At the same time - if I'd gotten 300-400 points less on the SAT, I likely still would have ended up at the same school. So in that vein my gut says the test scores were close to irrelevant. Or maybe not, I guess I wouldn't know.

Separately I think it's valuable to evaluate this through another lens. It sounds like you have some experience with the admissions process, and something I've been curious about for the longest time is this: if I had been in a slightly different segment, e.g. first-generation immigrant of non-East-Asian ethnicity, how would that have affected my chances?

Is it that test scores mattered less for me in this particular case, or is it that there's generally a higher bar because of competition from peers with similar East Asian backgrounds? In both cases it feels like test scores matter less overall (even if paradoxically the bar is generally higher!).

---

For reference, “top 50 state school” is something like University of Georgia or Ohio State University, both the type of school that will not slow down someone who would have fit in at Harvard or Stanford, imho.

This was the only piece I felt differently on. There's a significant advantage to attending a top school - the alumni network and a generally stronger and more well-connected student body for starters. Going to a state school didn't necessarily prevent me from finding success later in life, but I definitely took the long way around.

Aunche3• 4 years ago

My first thought is that maybe they didn't like your college essays. The funny part is that college essays are the most obviously biased towards the privileged. Poor people can't afford touring colleges, so their "why this college" essays can't be as specific. Meanwhile, the rich can pay someone to ghost-write their essays.

csa13• 4 years ago

Poor people can't afford touring colleges, so their "why this college" essays can't be as specific.

The "why this college" essay can trivially be addressed by looking through the university website.

Most applicants don't even do this.

A simple formula for this essay is:

- $APPLICANT has a demonstrated interest in $TOPIC(S).

- $ELITESCHOOL has $RESOURCES about $TOPIC(S) that $APPLICANT would like to explore.

This can obviously be iterated, expanded, and refined, but that's the basic formula for knocking it out of the park on that sort of essay.

Touring colleges has little to no impact on this essay.

Ghost-written essays also come across as shallow unless the parents also pay off teachers to write recommendation letters that corroborate the BS. It happens, but it's not common.

trh0awayman• 4 years ago

What was your college essay about?

scarface749• 4 years ago

Let me tell two stories:

I graduated with the highest SAT scores in my school and the second highest in my city back in the early 90s. I’m not bragging, it was a relatively small town in the south with not great school systems and a total graduating class of 1500 across all of the schools.

What are the chances that happen because I was so smart or because I had a mother who was not only a high school math teacher but also spent years volunteering to teach SAT prep classes?

Second story: my step son didn’t have the ACT scores to get into the college he wanted to go to. Four months later and with the help of a private tutor who was a teacher (not in his school) that we paid $100/hour for 12 hours, he suddenly raised his scores more than enough. Did he gain aptitude in 12 weeks or did he learn test taking techniques?

hcurtiss• 4 years ago

Yeah, I hear this, but in 1995 I just bought an SAT prep book for $12 and spent a couple months of Saturdays practicing and did very well. My family lived well below the poverty line. SAT prep is not confined to the privileged.

notch656a2• 4 years ago

Selecting for people who have some intelligence and actually give a fuck is actually a great way to cull the herd.

rajin4442• 4 years ago

Don’t you think a mother able to teach those classes would be in general a better test taker than the average person, and so her children likely would be too?

And nobody is denying that test scores can be raised. That’s likely a fixed amount so just amounts to constant noise. I think you’re as likely to find a way to consistently raise a 600 SAT to 1600 as you are trying to train any couch potato into an Olympic sprinter.

scarface749• 4 years ago

So let’s say my mom had been a “good test taker”. But she didn’t spend anytime with me. Was always too busy to help me when I had a question, etc. Do you think I would have done as well?

What are the chances that I would have chosen to be software developer coming from that same small town if my parents hadn’t bought me a fully tricked out Apple //e in 1986?

Even when I was in college in 1992, most of the CS students didn’t have a computer home. By then I had a $4000 LCII set up with a $1000 laser printer and a $300 SoftPC application to run x86 compilers. My parents weren’t “rich”. My mom was a teacher and my dad was a factory worker.

lelanthran4• 4 years ago

What are the chances that happen because I was so smart or because I had a mother who was not only a high school math teacher but also spent years volunteering to teach SAT prep classes?

It sounds like you are saying that good parenting shouldn't have any effect of kids success in life.

Look, we know that poor parenting results in kids with poor outlooks, but penalising good parenting by taking away the effect of good parenting is probably not a good way to go about helping those kids with poor parenting.

scarface749• 4 years ago

I’m saying that the test isn’t indicative of “aptitude”. Well neither are GPAs though. It’s not hard getting a 3.7 GPA when the teachers know that if they make the coursework challenging or don’t give plenty of make up chances that either most of the students will fail (in the small town I grew up in) or that the Karens will complain to the principal that this one teacher will keep her little Timmy from getting into whatever prestigious school she wants him to get in (in the case of the “good schools” my step sons went to)

faangiq• 4 years ago

I’d gladly hire on test scores alone.

ramraj077• 4 years ago

So one of the best standardized tests I’ve ever taken is the Subject GRE. When I was applying for PhD programmes I and a few of my colleagues took it. It was perfect - it tested both knowledge and critical thinking skills and experimental design. It was obscure so no material was available for specific training.

In the end the scores were so accurate in predicting eventual success - a decade later, the folks who got 96% plus are all either professors or deliberately chose not to, while the rest just took industry jobs (and in my opinion because they realized that’s best for them).

Here’s the kicker - every institution explicitly said they will NOT consider these scores as part of admission process. None of the people who got in the top percentiles (4 I know) made it to a top 10 institution in the US, while a bunch of others did because they had paper authorship in their undergrad. But paper authorship in undergrad has NO correlation with your actual scientific skills! It just meant you were connected and or a hustler. It’s sad that such an awesome test is deliberately ignored by these institutions.

The irony was that these tests were also not elitist. All you needed to do was thoroughly read Lodish and Lehninger and you’re good. We did study in a semi-premium institution in India, but by no means were we privileged by any special facilities or help (at least in the context of this test). The only barrier might be the test fee itself if anything.

thomasahle2• 4 years ago

a decade later, the folks who got 96% plus are all either professors or deliberately chose not to,
All you needed to do was thoroughly read Lodish and Lehninger and you’re good.

How can these be true at the same time? If you can just read a book and get a good score, why don't everyone just get the good score?

hawk_• 4 years ago

It's clear they mean that the barrier to entry was available to anyone. That doesn't mean that anyone has the capacity to understand and apply the contents of the text.

ramraj077• 4 years ago

Lodish (a then popular cell biology book) and Lehninger (a 1500 page biochem tome) are substantial books. 101 courses would cover the first 5-9 chapters of 25. Reading back to back and understanding all of it puts you at the top of the best students in biology in the world.

I was lucky, I genuinely loved reading them. I think I went through them both once every time I had a bad breakup lol.

barry-cotter9• 4 years ago

How can these be true at the same time? If you can just read a book and get a good score, why don't everyone just get the good score?

All you need to do to get 4.0 in Calculus 1, 2 and 3 and Real Analysis is work through Stewart, doing all the questions. Just because it’s simple to describe doesn’t mean it’s easy.

ausbah2• 4 years ago

while a bunch of others did because they had paper authorship in their undergrad. But paper authorship in undergrad has NO correlation with your actual scientific skills! It just meant you were connected and or a hustler

uh, the whole point of a phd is learning how to produce scientific knowledge usually in the form of a paper? I can't think of a better measure for that then if you have experience publishing papers because that means you've contributed to the production of some scientific knowledge? sure there's tons of variability there due to circumstance, but I think my point holds

and finding connections is bad? the scientific process is one of cooperation, teams building off the work of other teams. the lone polymath is a long dead myth.

and again, having hustle is bad? I think you'd want someone who can be self motivated enough to get involved with research going on at their university

ramraj077• 4 years ago

Getting a paper in undergrad has absolutely nothing to do with your ability to be a good scientist or even a good PhD student. I’m talking about regular smart kids though, not people you could call a True Genius™. As I mentioned above, most undergrad paper authorship is peripheral where you just succeeded in establishing yourself in a lab long term and contributed to some research.

It’s great experience but I am saying again that my personal experience suggests this to be a worse indicator of success than a good standardized test.

muds2• 4 years ago

Respectfully, I call bullshit. You can't quantify success in a PhD with a single variable. There are a billion ways research can go right or wrong -- irrespective of your personal pedigree. Your ideas might be too early or late for your community to grasp, maybe you appeal to the wrong audience, maybe you're unaware of an application of your research, maybe you don't have the right set of collaborators or need a perspective that, often times, emerges out of a lucky encounters with someone. I respect your experience but I don't want it to give people the wrong idea about research success...

Paper authorship had NO correlation with research success

I think paper authorship demonstrates that you're willing to put in a non-trivial amount of work to persue a problem. That seems to be atleast one attractive skill in a PhD, wouldn't you agree?

wink• 4 years ago

I dunno, my snarky answer would probably be: Just as if they'd test for A, but you actually need to be good at B.

I've not been to any school in the US and I don't have a PhD, so all these acronyms like GPA only have a vague meaning to me. But as someone who kinda disliked school and didn't put much effort in it, I can see how school grades can be a bad proxy for university. I didn't even do that well in math in school, but I managed to go above and beyond with all my math classes for my CS degree. I'm not completely sure how the admission for PhDs work and if they look at your BSc/MSc tests... but again, I guess doing ANY work related to finishing a paper is probably closer to what you'll be doing later than e.g. your grades in database stuff when you'll be researching programming languages...

ramraj077• 4 years ago

Uhm, what exactly are you trying to get at? I said subject GRE is a very good measure of eventual success in academia, do you have a solid response to that or just a rambling tirade?

Paper authorship if the student is the first author shows grit and “gumption” I suppose? As if that’s what’s needed in academia at this moment (it’s important but not the main requirement). But almost no undergrad gets a first author paper. They get mentioned in the middle because they ran a bunch of sds gels. I wasn’t even interested in trying to become a professor and I got 10 papers before I finished my PhD, do you know how many I (or any of the folks I actually know who are now professors) had during our undergrad? Zero. And not for lack of trying. You know who actually got papers? The son of the department head.

trentnix2• 4 years ago

You can't quantify success in a PhD with a single variable.

Of course, but I'd bet the farm there's a very strong correlation!

adamsmith1434• 4 years ago

But paper authorship in undergrad has NO correlation with your actual scientific skills!

This seems like a bad take because the goal of PhDs in the US is to produce people who can publish a lot of papers, whether that's a good ideal or not. So having produced papers as an undergrad gives you a powerful signal that you are such a person.

Here’s the kicker - every institution explicitly said they will NOT consider these scores as part of admission process.

I know in my field, Physics, the subject GRE was very important for admissions and outside of the recent changes due to Covid I believe every top program explicitly requires the Physics GRE for admissions and its not until you get to much lower ranked programs that it becomes optional or not used at all.

rajin4442• 4 years ago

Does the same hold true of medical professionals? ie MCAT scores correlate with med school and/or practice performance?

derbOac5• 4 years ago

It's been some time since I read this work, but at one point there was a situation where medical schools in a state were temporarily (like a year or two) prevented from using MCATs in admission decisions due to a judge's court order. Most applicants still had MCATs due to other schools, though.

Some researchers collected data before, during, and after this time, and followed-up on the physicians later.

My fuzzy memory (which might be incorrect) is that MCATs were predictive of GPA in the first two years of medical school like 0.20 on a Pearson correlation metric, and it dropped from there as you went out increasingly further in time and included things like ratings on clinical rotations. So it was predictive, but fairly weakly so and seemed less predictive as time went forward.

cr4nberry3• 4 years ago

Why is it that the anti test score argument almost always tries to chalk it up to hiring tutors? They consistently try to make it sound like people "buy" higher test scores with tutoring in order to discredit tests like the SAT

pocketsand• 4 years ago

There is research on this. I believe it was Thomas Espenshade, social scientist at Princeton, who had data on test prep access, modeled along with SES, etc. Test prep is worth about 40 points on the SAT. Not trivial, but not a game changer.

It's true that people who are otherwise well off are more likely to have test prep, and those people's test scores are often several standard deviations higher than low-income students from bad schools without test prep. But the counterfactual of (same person) +/- test prep = about 40 SAT points.

readthenotes12• 4 years ago

Did I miss the correlation across school systems? Everything in the study was from Chicago public schools.

My older brother switched from a B student in a private high school to the public high school and slept his ways to A's

mradek• 4 years ago

I feel like standardized tests are way more fair than GPA.

I personally struggled a lot in school due to health and other issues. I also hated doing homework which is BS work, and I always had high scores on my midterms and finals which I think matter way more.

Why does busywork play such a big role? It should be exempted. If I can get a 95+ on midterm and final I obviously understand the material. Why did I need to spend hours of my day doing BS exercises when I already understood the material? Nonsense.

So I get a B at best overall because I get a F on homework? What a joke. I really think school is a waste of time when it’s done that way. Learn stuff, move on. I wish that when I have kids I can afford to send them somewhere they can actually grow and thrive, not just sit down and do a bunch of homework problems.

(Just my $0.02)

Aliabid94• 4 years ago

Most real world work is grinding through boring work as well, and it’s worthwhile to build the discipline for it.

TheHegemon• 4 years ago

Completely agree with this. I used to skip most of my classes but would get As on all of my tests.

Now I struggle immensely on going through the every day grind of adult life.

mikebenfield• 4 years ago

There are better ways to build discipline than spending a huge chunk of ages 5-18 doing otherwise pointless busy work.

hiremelocally• 4 years ago

I wish you were wrong, but you're not. Ability to work a BS system absolutely is a marker of success in society.

renewiltord2• 4 years ago

Perhaps it's worth sampling the population and seeing whether outcomes match. I'll go first. I didn't do this. My parents intentionally did my cursive writing homework for me so I could read and go out and explore the countryside.

I now run eng at a prop trading firm that you'll occasionally see in the FT.

iopq9• 4 years ago

That's why I don't do "real work" anymore. You understand not everyone has a goal of being a salaried employee in life, right?

nicoburns• 4 years ago

It was the arbitrariness of academic work that got to me. Real world work tends to be clearly directed at a goal, and if you acheive the goal it doesn't tend to matter much how you did it. Academic assessments (poorly designed ones, which happens to be most of them in my experience) expect you to do things exactly in a prescribed manner. Essentially 50% of what you're doing isn't actually useful learning, it's domain-specific learning where the domain is "what my professor happens to care about".

jimbob45• 4 years ago

It seems to be the case that GPAs are more important with scholarships than admissions, where boards are looking to find any edge to be able to differentiate students.

ninesnines• 4 years ago

I agree with many aspects of this; however, I do think learning how to grind through work is important. However, I remember being a child and a teenager trying to do the designated school and extracurriculars and then homework felt insane. I would often be doing work til 11. This does not feel like a way to inspire children or make them excited to learn. I loved school in many ways, but the homework caused so much stress when I think if I had more time to explore intellectual interests I think I would have more excited about many subjects that weren’t my favorite

brailsafe2• 4 years ago

Well, I have the opposite opinion. First of all because tests don't remotely represent the nature of doing work. It's a momentary expenditure of energy, and all of rides on perfect performance that day. Luck, rote memorization, and some skill development.

So in my view, I read it as "Why should I have to do bs when I'm lucky and tend to do well on tests?"

In my cs classes, I did terribly on tests, and got 100% on all the work, so I failed. Why? Well I have chronic sleep issues and miserable short term memory that's gotten worse over time, and the mid-term was 40%. I also just don't really care about the outcome, because it doesn't represent my ability to do any of the tasks in the real world. Oh, I was too slow in writing out the exact java syntax for a generic implementation of a priority queue implemented with an array? Tests aren't inherently virtuous. You have to believe in them to do well, and you have to feel adrenaline in the process. Usually has nothing to do with incremental progress, measurable skill, or anything valuable really. I can now recall various key bits of key events from the Byzantine empire, but did I do well on the test? No, I was a bit tired that day.

Both should be balanced, and relevant. The tests and the homework should meaningfully test whether or not I can do what they intend to indicate. Not "He couldn't sit there for 3 straight hours from 6pm to 9pm on one night of the year writing out java in pencil in a dead silent room.

iopq9• 4 years ago

WTF is lucky? You can hit all of the questions you know in one test. You can't be "lucky" on every test, that's a ridiculous claim

ht_th• 4 years ago

If I can get a 95+ on midterm and final I obviously understand the material.

That's assuming that these midterms/finals accurately and completely assess understanding of the material.

As a teacher, I need more assessment moments than one or two tests in a term to monitor, guide, and support students' learning processes, for all students in my class. This "busywork" gives me the extra input I need to adapt my teaching to my classes and students.

Of course, from my perspective, this isn't "busywork" because I try to create meaningful learning activities. However, from the perspective of some students, I can understand it feels like busywork. I try to differentiate, but that's not always possible, particularly if you're performing significantly above average.

The problem is that even if you are doing fine without the busywork, your classmates might not and need the training through busywork. Or they might need the structure and incentive to do something at all. Unfortunately, giving some students no work and other some work and other a lot of work will not go down well, particularly in larger heterogeneous groups. So we end up treating everyone the same, more or less. This can really suck for any student who's an outlier to the positive OR negative side.

Ideally, you'd get an education tailored to your needs, but in practice that's difficult to realize depending on where you live, your parents' background, etc. For example, if you're living in a smaller village with just a single high school, you don't have much choice. Similarly, if your parents don't understand what you need, they'll not go look for it. Or if your parents don't have the means to support your alternative schooling, bad luck. Etc.

jobs_throwaway• 4 years ago

A professor I had had a pretty good solution. Give homework, but only make it 5% of the score for the class. A couple of quizzes are 5% each, and the midterm + final are the remaining 85%. That way, if a student needs the structure, they get it, and on the other hand if a student is able to ace the tests without doing the busywork, they can still get their A or A+. The instructor also then gets lots of input points throughout the term.

Some students (particularly those who spent lots of time on the HW but didn't do well on the tests) found this unfair, but I think it teaches an important lesson: that you don't get credit for trying. Your performance actually matters.

derbOac5• 4 years ago

The flip side of the fairness argument is that test scores are just one time point (often just a couple of hours, maybe several hours at most, of one day) in one format with no flexibility for unusual circumstances or anything. GPA, in contrast, is reflective of repeated things over a long time period, with lots of variables accounted for.

There's issues with GPAs too, like lack of standardization, but they both have their issues with fairness.

Ancalagon2• 4 years ago

If this were the case then couldn't we just make high school diplomas, heck even university degrees, a series of midterm and final tests? Why have lectures at all when people can come in knowing the material online? What if you could start "school" in one week, test nonstop for the next month, and have your degree in CS assuming passing marks at the end?

godelski• 4 years ago

I think the author is missing some really key points here. Namely that GPA and test scores are highly correlated. I'd go so far as to say that there is a confounding variable that predicts both. I'd even say there are several upstream predictors, but I think we all know that (or I thought we did). Every single graph they show demonstrates this yet they keep missing it. The top comment right now (by ajwinn[0]) notes that they used a heatmap like the one in the article for determining admission. This is great because it builds multiple pathways to gain admission. That's honestly how we should treat variables like these: rely on either.

As to explain the confusion in the models, well when you have confounding variables like this it is easy to reduce one variable's weight nearly to zero and place the rest in the other variable. So if I'm right (might not be) I think you could also show that the inverse is true, that grades aren't a good predictor and test scores are. The breakdown suggests that this is true too tbh.

But I also disagree with some of the analysis and techniques here. It's not always easy to use models that have continuous(ish) variables (GPA/ACT/income) and discrete ones (race/gender). I bring this up because there's some hints that they aren't a stats expert. For example there's the line

The “likely-to-graduate” score becomes a probability after a sigmoid transformation.

Which just isn't true. A Softmax function[1] creates a probability distribution but a Sigmoid[2] doesn't. A Sigmoid bounds the output to [0, 1] (or [-1,1]) but being bound on that range isn't the only requirement to a probability distribution. A sigmoid does not guarantee that the sum of the elements is 1 (cumulative density), but a softmax does. In fact, it almost never will. (I mention this in my ML class a lot so make sure you all don't make this mistake too! I'll be really annoying if you do)

There's some more errors and novice approaches here and so I just want to note this because we shouldn't just take the results and conclusions as is when the modeling is bad. But I still think he makes some good (and bad) points and those are definitely up for discussion.

[0] https://news.ycombinator.com/item?id=33338809

[1] https://en.wikipedia.org/wiki/Softmax_function

[2] https://en.wikipedia.org/wiki/Sigmoid_function

MarcoSanto• 4 years ago

It'a sunken cost effect at paly, little more than that. Oh, and parents pressure, least we forget the role of family. You can test my claim empirically by asking yourself: "are all college graduates I met more intelligent than me?". Admission test measure how well you can prepare for admission tests, nothing more. There are too many confounding factors and co-factors that play in the career of a relatively fresh semi-adult aged 17-18, that reducing it to a single number is just moronic or bad-faith utilitaristic escape route. Universities only care that you graduate so they can make money, therefore they use the simplest KPI to measure that. Let's not forget Universities are staffed with people and are a business.

411111111111111• 4 years ago

It's unlikely to be a sunken cost fallacy.

You're technically right with "Admission test measure how well you can prepare for admission tests, nothing more.".

It's definitely not a silver bullet and doesn't assure success after admission. nonetheless, the ability to prepare for these tests is strongly correlated with success along this path.

If they can't be arsed to prepare for them they're unlikely to bother with studying either, making their admissions pointless too.

And if they attempted to prepare for the test but still failed... Then they're unlikely to be able to learn in the University setting either and should follow up on other avenues.

tagami• 4 years ago

“C’s get degrees”. Measuring college graduation rate is a vanity metric

Ekaros• 4 years ago

Test scores are the most objective metric available. And thus I really consider them fairest. Or we could just have entirely random lottery.

vivegi• 4 years ago

This whole concept of measuring SAT/ACT scores or GPAs and using them to "predict" graduation rates is bonkers.

Sounds a lot like the precogs in Philip K. Dick's Minority Report.

Several decades ago, I had a high-school History teacher. We had an elite private high-school in the same city as I lived (India has both private and public schools). They had an extremely rigorous selection process, including infamous interviews with parents for admission to kindergarten. Our history teacher once remarked, "They take horses and send them out as horses. We take horses, asses and everything else and send them out as horses".

The SAT/ACT/GPA filter seems to be focused on selecting horses and proudly declaring all horses graduated.

club_tropical5• 4 years ago

They take horses and send them out as horses.

Correct. Although there are huge benefits of a gate-kept horse-only environment. Competing against each other, the horses become more excellent. This is why the parents are paying the big bucks and putting up with the process. It is about gaining access to a horse-only environment.

We take horses, asses and everything else and send them out as horses.

This is just envy at the easier job of the private school teachers, not indicative of their actual transformational capabilities of the public school teacher. The best that these teachers can do (which is no small feat) is to manage the classroom such that the donkeys and the horses do not fight and the horses can stay motivated despite having fewer peers.

dirtyid• 4 years ago

If anything over indexing on solely on test scores will filter out a large % of psycho/socialpaths who gets by on personality scores.

sakex• 4 years ago

I think that when people say test scores are irrelevant, they don't mean irrelevant for studies but irrelevant as an indicator of job performance.

barry-cotter9• 4 years ago

Meta-analyzing the relationship between grades and job performance.

Roth, P. L., BeVier, C. A., Switzer, F. S. III, & Schippmann, J. S. (1996). Meta-analyzing the relationship between grades and job performance. Journal of Applied Psychology, 81(5), 548–556. https://doi.org/10.1037/0021-9010.81.5.548 Employers and academics have differing views on the value of grades for predicting job performance. Employers often believe grades are useful predictors, and they make hiring decisions that are based on them. Many academics believe that grades have little predictive validity. Past meta-analyses of the grades–performance relationship have suffered either from small sample sizes or the inability to correct observed correlations for research artifacts. This study demonstrated the observed correlation between grades and job performance was .16. Correction for research artifacts increased the correlation to the .30s. Several factors were found to moderate the relationship. The most powerful factors were the year of research publication and the time between graduation and performance measurement.

qwerty456127• 4 years ago

In my opinion discriminating against people given their past test scores is unfair. People can improve and having scored bad (which might well be a consequence of hard life circumstances at a specific moment of their life) in the past should not be allowed to ruin their future. So I believe nobody should be allowed to see one's previous test scores and judge them this way. Whenever it is legitimately necessary to assess a person's proficiency a new test should be taken.

PaulHoule4• 4 years ago

My take is that standardized tests are a road to social mobility. People can bullshit their way to higher grades but there are some rich kids who can't test their way out of a paper bag and the one way we can stop them from laundering their parents accomplishments is standardized testing.

athorax• 4 years ago

How does that stop them in any capacity from living off their parent's wealth?

ubercore• 4 years ago

I don't think that's the point the parent post is making

ouid• 4 years ago

Well, no, because schools have a profound interest in having those students parents pay them for a top notch education.

derbOac5• 4 years ago

Except you can buy your way into tutoring too, and pay for test accommodations, and so forth and so on. It might not amount to a huge effect, but if you're on the boundary area, it can make the difference. Someone might also argue that although bullshitting with grades makes a difference, it usually isn't going to turn a D into an A either (speaking as a university professor, with many colleagues at different universities, in those situations it's usually a matter of being pressured to turn an F into a D).

trentnix2• 4 years ago

Except you can buy your way into tutoring too, and pay for test accommodations, and so forth and so on.

I don't buy this. It's like saying someone has a remarkable physique or remarkable fitness only because they used PEDs when it actually requires a significant amount of effort and discipline regardless of what's in the gas.

Certainly targeted studying, tutoring, test accommodations, etc. can help. But the difference is at the margins.

PaulHoule4• 4 years ago

It's the B to B+ and B+ to A chiselers that bug me the most.

commandlinefan3• 4 years ago

you can buy your way into tutoring

You still have to listen to and understand what the tutor says, though. And it's not as if the tutor is saying anything that's not published in books or on YouTube videos - the only real benefit a private tutor could even theoretically have is that he could potentially find a few different ways to explain a concept until the student understood it.

htrp• 4 years ago

You would be surprised at the number of kids who have tutoring... and still can't test their way out of a paper bag

insane_dreamer2• 4 years ago

I see it the opposite. You can hire people to prep you for the test and pass it while having done zilch during HS; whereas bs'ing your way to higher grades over a period of several years is harder.

forgotmypw17• 4 years ago

It is not that they are irrelevant, it is that they are ableist, selecting for a narrow skill and ability combination, and creating a barrier for anyone else who lacks it, for whatever reason.

pwpw• 4 years ago

I see zero discussion of test scores and people with learning disabilities. Those who cannot be properly accommodated will be unfairly represented and often have to put up with a lot more barriers (such as frequently messed up accommodations on test day leading to anxiety). Usually these accommodations help normalize the specific testing scenario and people with disabilities do not need as drastic accommodations in real job setups. The tests are designed for the average test taker but have lots of hiccups for those that deviate from the average.

It’s a difficult problem to solve, but it’s unfair if a student gets easily distracted or anxious when taking a test around others in a time constrained setting. Those types of external factors generally shouldn’t be represented in a student’s test results yet often are. If tests are to be a good indicator for all students, I think accommodations have a ways to go.

If tests are indeed a generally good indicator of a student’s success and are continued to be used in such a manner, I feel it’s important that the experience is fair for all types of test takers. Many US tests such as the CPA exam do a great job of handling accommodations. However, many others (like the PSAT) have a lot of room for improvement.

gumby• 4 years ago

I see zero discussion of test scores and people with learning disabilities. Those who cannot be properly accommodated will be unfairly represented…

I have long wondered how these interact / can support people who are a couple of sigma out from what we have decided is the “norm”. I can see how additional test time can help someone with anxiety or distraction issues, which can help someone with other strengths shine.

But how can this work outside the academy? A trial lawyer has only so much time to work on the case before it goes to trial or before some response to a filing is required. An engineer building a rocket still has to get something designed before assembly. Arent such time accommodations are hard to implement in the “real world”?

There are nice counter examples to my question. For example dyslexic people struggle with all sorts of cases, yet technology can help many of them (e.g. new kinds of fonts), and not just current students but people already in the work force.

I can also see the school accommodation helping someone who is separately working on their anxiety issues — though mental health cost support is quite poor, at least in the US.

But at their root: do these accommodations help or do they provide an unrealistic hope to the student?

NB: I want everyone to have a fair shake, and am a fan of ADA accommodation and affirmative action and related diversity efforts. I ask this question within that context.

derbOac5• 4 years ago

... so here's the thing (speaking as someone who has taken many of these tests, been in admissions committees, and researches these tests). Take learning disabilities, and now expand it out to consider the similar factors, but with things like cultural background, life circumstances, and so forth and so on. Learning disabilities are not the same as those other things -- I don't want to equate them -- but they share some of the same issues with the test being normed on a certain standard population, and once you get outside of that, weird things happen.

Those density gradient plots in the linked blog post are interesting and useful to think about, but they're kinda hiding the fact that the vast majority of the data going into them is based on test-takers with a certain standard background, characteristics, and so forth. That in turn shapes the contours of what is error in those plots and analyses. The problem isn't necessarily that the tests are useless per se, it's that it's hard to interpret them in a way that accommodates people in nonstandard situations, or even accommodates the idiosyncracies of differences between people in general, differences that wouldn't matter in the real world.

The errors of using ACT-only, GPA-only, and so forth in the post is pretty interesting, but the author is missing the fact that that table has never really been the point of contention. The point of contention is whether that table's patterns apply equally across divisions of gender, race, SES background, age, disability status, and so forth and so on. They raise the point that the improvement in error from switching from ACT to GPA is comparable to (or better than) adding ACT to GPA, but isn't the real question whether doing so increases certain types of "predictable error variance", in the sense that you could predict the residual from things like SES, race, and so forth and so on?

I think GPA is seen as more acceptable than ACT because it's exchangeable in an important sense with the criterion being predicted. That is, if you want to know college GPA, maybe secondary school GPA is a little flawed, but at least it's ostensibly similar in terms of what it actually is. I think people have a sense that, say, you aren't using standardized tests as the criterion for college graduation, so why use it for prediction? Why have college and college GPA at all? Why not just let people take standardized tests and skip the whole degree program thing? There's reasonable arguments for doing that, but also reasonable arguments for not doing it, and many of those are the same arguments for and against using the test for admissions.

lukas099• 4 years ago

Tangent - I think that my particular disability (ADHD) helped me with my test scores. I was terrible at studying, but on the tests I would hyper-focus and do extremely well.

On the other hand, on the GRE, the writing portion was last and I could not bring myself to care. I scored in the 50th percentile, which for someone who regularly scores in the 99th percentile on standardized tests (including ones with writing portions), is pretty bad.

3vidence• 4 years ago

Maybe a hot take but I think school pedigree is irrelevant.

I went to a top university in my country after transferring in after not getting in from highschool. Turns out myself and many others would have got a lot of value from going to a less prestigious university. Ultimately most of us end up working at the same select companies. But idk I am from Canada so maybe the intensity of selection is not the same.

We put far too much value in prediction of success and not enough value in actual success. Seems like the academic version of keeping up with the Jones's

bigbacaloa• 4 years ago

Every metric has problem, but experience in a university suggests that, if one has to use a single metric to rank students, by far the best choice (the only choice really) is average grade.

awkward• 4 years ago

The full dataset isn’t available, but since we have the number of students in each ACT / GPA bin above, we can create a “pseudo” dataset, with just a small loss of precision.

Big oof. This is an example of terminal data science poisoning. The data in the paper already proved the point of the article! Using it to create some chart gore where you reproduce the same plot in uncanny valley levels of resolution doesn't add to the argument. This is using very sophisticated techniques to achieve what you could by simply blurring an image of the first chart.

Test scores are not irrelevant

Related Stories