Socioeconomic status and the relationship between the SAT and GPA (2009) [pdf]

https://files.eric.ed.gov/fulltext/ED562860.pdf

by noch • 2 years ago

27 69 2 years ago

files.eric.ed.gov

i.e. socioeconomic status is correlated with SAT performance because socioeconomic status is also correlated with stronger basic academic skills.

Amezarak2• 2 years ago

Yeah, the article ends with:

In conclusion, our work focuses on the predictive power of admissions tests and shows that this power is not an artifact of SES. There is a substantial SES–test relationship, though it is important to note that test scores reflect far more than SES. About a quarter of the variance in test scores is shared with SES, and thus there is large variability in test scores at any given level of SES. Thus, claims that tests are merely proxies for SES are unfounded

I've never understood the model of the world that people who don't agree with this. If you believe that

a) genetics exist, and traits such as intelligence are therefore hereditary; and b) socioeconomic success is correlated to intelligence

Then the inevitable conclusion is that wealthy people will be more likely to have higher SAT scores. Of course this is obviously just a correlation; lottery winners are rich but with no correlation to intelligence, children of two intelligent parents tend to regress to the mean, two average parents can produce intelligent children, families might be poor simply through bad luck or some particular emotional/mental block they have, and a million other factors. But nevertheless we'd expect test scores to be our best available wealth-blind predictor of college success and later success. Certainly it's hard to imagine children with money-poor - or maybe mildly abusive - parents having a shot without test scores. They'd be less likely to have a deep extracurricular involvement or anything else requiring parental involvement.

noodlesUK• 2 years ago

I think that the strength of point b) is the point that a lot of people question. I think for people like me who question the SES - intelligence link, the point of testing is to provide any opportunity for bright people who have lower SES to have a shot, which they would be otherwise denied due to their status.

If anything, testing is one of the things that can help create that link between SES and intelligence.

alistairSH4• 2 years ago

I've never understood the model of the world that people who don't agree with this.

My take is people question the tests because wealthy/powerful people have a history of "gaming the system" in order to retain that wealth/power. It's not unreasonable to ask "is the test predictive of anything other than the parent's ability to hire a tutor?" While a test is, in theory, easy for a poor student to pass through independent studies/library, etc, reality gets in the way - that student needs transport to the library, time to study (many work), etc.

But, based on this research, it seems the answer is "Yes, the test is actually predictive of something other than parent's SES."

cycomanic• 2 years ago

Yeah, the article ends with:
> In conclusion, our work focuses on the predictive power of admissions tests and shows that this power is not an artifact of SES. There is a substantial SES–test relationship, though it is important to note that test scores reflect far more than SES. About a quarter of the variance in test scores is shared with SES, and thus there is large variability in test scores at any given level of SES. Thus, claims that tests are merely proxies for SES are unfounded
I've never understood the model of the world that people who don't agree with this. If you believe that
a) genetics exist, and traits such as intelligence are therefore hereditary;
and
b) socioeconomic success is correlated to intelligence

That is only true up to a certain level though. The correlation plateaus and even decreases above 1%

https://www.sciencedaily.com/releases/2023/02/230208125113.h...

fallingfrog3• 2 years ago

Regression to the mean is a real thing though. The reason why is this: every gene in your body is also in the bodies of millions of other people. Your genes are not unique. It’s the combination that’s unique. And the combination is diluted 50% with each generation. So Einstein was a prodigy and a genius, but his great grandchildren are pretty ordinary. That’s just the way it goes.

underlipton• 2 years ago

It's ironic to conclude this, without skepticism, knowing that it's a study sponsored by College Board which shows that the tests which College Board administers are definitely efficacious.

NovemberWhiskey3• 2 years ago

Sure; but do you disagree with the method or the conclusions?

rayiner2• 2 years ago

Relatedly, UT Austin found the SAT highly predictive and decided to reinstate the requirement: https://news.utexas.edu/2024/03/11/ut-austin-reinstates-stan...

Those who opted in had a median SAT score of 1420, compared with a median of 1160 among those who did not.
The higher standardized scores translated on average to better collegiate academic performance. Of 9,217 first-year students enrolled in 2023, those who opted in had an estimated average GPA of 0.86 grade points higher during their first fall semester, controlling for a wide range of factors, including high school class rank and GPA.

resource_waste• 2 years ago

These kind of things are making me conservative on knowledge.

10-15 years ago 'standardized testing doesnt matter at all'
You are an idiot if you think it matters
Oops THEY were wrong. You are still an idiot and ahole for disagreeing with me 10 years ago.

Obviously there is a golden mean here, but I just remember how radical some of these teachers were.

username332211• 2 years ago

A golden mean between what? Truth and falsehoods. I hate citing 1984 in any matter such as this one, but there's a word in it for that - "doublethink".

onlyrealcuzzo• 2 years ago

The people who were saying tests didn't matter were the people that thought other things mattered and had other agendas to push...

alistairSH4• 2 years ago

But, is first-semester GPA predictive of anything? I'd rather see this result at graduation. Does the higher SAT predict higher graduation rate, better grad school or job placement, or anything else? Note, I'm not arguing it won't be - I don't know. I just know more than a few people who bombed their first semester (most with SATs well above average), myself included, who ended up with decent academic performance once they got over the "shock" of being an adult away from home.

arethuza2• 2 years ago

I did terribly in first year, scraped through second year and did really well in third and fourth years - to the point of getting a first and a class prize. I don't think it was anything to do with "shock being away from home" - but actually realising that some things require hard work.

rayiner2• 2 years ago

I just know more than a few people who bombed their first semester (most with SATs well above average), myself included, who ended up with decent academic performance once they got over the "shock" of being an adult away from home.

Individual students might underperform because they're not prepared for living outside the home, but that factor would cancel out when you're comparing two populations with thousands of students. Note that, according to the article, the 0.86 GPA point difference--which is huge, like the difference between top of the class and below median--was after adjusting for high school GPA. So the analysis should factor out differences resulting from some kids simply being more "put together" as freshman than other kids.

077sd015o462• 2 years ago

0.86 GPA points higher during their first semester. Is that really such a massive informational advantage?

cschmidt• 2 years ago

The NYTimes had an good interactive story on this same topic this week:

https://www.nytimes.com/interactive/2024/03/09/upshot/affirm...

dsq• 2 years ago

Wouldn't socioeconomic status correlate just as strongly or even more with high school AP grades and extracurriculars?

It seems that these would require more discretionary time, as well as discretionary money, when comoared to the SAT.

retskrad• 2 years ago

I can get almost all the questions correct on the SAT test without any time pressure but when I do it when the clock is ticking, I find myself unable to reason my way to the correct answer nowhere near as fast. As a result, I have to move on and skip 1/2 of the verbal and math questions because I don't have enough time. And I don't draw blank when under pressure so there must be something else going on.

I have tried to pretend that I'm an LLM and trained myself on hundreds of SAT questions on YouTUbe where people solve them. I've found that my reasoning speed under time pressure is a direct function of the amount data I've trained myself on.

Is there a quicker way to this or do people like me, with average IQ or Chat-GPT 3.5 if you will, have to train on thousands of math and verbal questions because we don't have the Chat-GPT4 like the gifted people where they ara able to be creative and don't need to rely only on past knowledge to solve novel problems?

aruss• 2 years ago

Yes, there are ways to speed up your time. When I took the SAT (2011), the main skill I developed was to quickly categorize the kind of question. Since there are only a few different kinds of questions, you could train yourself to do this formulaically/algorithmically. The type of question tells you how to eliminate a couple of the answer choices, and then you apply another question-type-specific algorithm to figure out the answer from the remaining options.

bombcar• 2 years ago

The Kaplan book (or similar) went over this. You learn to avoid “solving the problem the hard way” as much as possible.

However, the new digital SAT will require new strategies because its computer assisted, more like the LSAT/GRE (?) is.

tekla• 2 years ago

I've found that my reasoning speed under time pressure is a direct function of the amount data I've trained myself on.

That is the most pretentious way I have ever heard of saying "I did better when I studied and practiced more"

I'm actually in awe.

fallingfrog3• 2 years ago

For me, I got much, much better SAT scores than my grades would suggest, because my grades were depressed significantly due to ADHD, (forgetfulness, never remembering when tests were going to happen, going off on tangents when studying) even though I actually knew the material. So the standardized tests were a chance to prove that I wasn’t a total dummy.

And you know, I think that’s kind of the point of a standardized test- grades definitely vary based on how involved your parents are, the particular teachers you have, whether you have adhd, and yes socioeconomic status- tests in principal are blind to all that stuff.

Workaccount26• 2 years ago

1.) Intelligent people befriend other intelligent people.

2.) Intelligent people get higher paying jobs.

3.) Intelligent people live in wealthier areas.

4.) Intelligent people marry other intelligent people.

5.) Intelligent people birth intelligent children.

6.) Intelligent people use their intelligence to augment their kid's intelligence.

7.) Intelligent kids go to step 1.)

The emperor is walking around naked, and virtually all of academia is terrified to acknowledge it (namely step 5). Right there is the solution to 50 years of sociological fumbling to avoid saying "Intelligence has a strong genetic component, and we are not all born with equal mental capabilities"

Once academia can acknowledge this, then they can finally work on real solutions to close the disproportionate economic gaps it creates.

(Yes, I am aware of your anecdote that counters this. It's a broad generalization to counter the mainstream broad generalization.)

esotericimpl• 2 years ago

Except the research doesn't show this at all.

However, having wealth provides tutors, better schooling and teachers which lead to better outcomes from having better mentors, opportunities and education which leads to...

and so on and so on.

Workaccount26• 2 years ago

Right, that is step 6.

If you go over the research, you realize that they all assume the premise "everyone is born with an identical blank slate". Which leads to conclusions (either said outright or implied) like; "If you give enough resources to any given child, they can grow up to be anything."

So we end up with schools that serve low income areas getting enormous amounts of funding to teach admin assistants and tile layers to be heart surgeons. And the data on that end shows unequivocally that it is not working.

TMWNN• 2 years ago

The emperor is walking around naked, and virtually all of academia is terrified to acknowledge it (namely step 5). Right there is the solution to 50 years of sociological fumbling to avoid saying "Intelligence has a strong genetic component, and we are not all born with equal mental capabilities"

Bien-pensants will tell us to the bitter end "Who are you going to believe, me or your lying eyes?" But they are aware of the truth.

Example: From a 2018 open letter by an eminent geneticist, "How Genetics Is Changing Our Understanding of 'Race'" <https://www.nytimes.com/2018/03/23/opinion/sunday/genetics-r...>:

I am worried that well-meaning people who deny the possibility of substantial biological differences among human populations are digging themselves into an indefensible position, one that will not survive the onslaught of science. I am also worried that whatever discoveries are made — and we truly have no idea yet what they will be — will be cited as “scientific proof” that racist prejudices and agendas have been correct all along, and that those well-meaning people will not understand the science well enough to push back against these claims.

Workaccount26• 2 years ago

I have no interest in bringing race into this because of the wide genetic difference between groups that common people would label as being the same "race". Genetically there is wide diversity even within relatively small geographical areas.

silverquiet3• 2 years ago

It's worth remembering this next time you hear someone say something along the lines of, "I believe in equality of opportunity, not equality of outcome". If people are indeed born with unequal abilities, then equality of opportunity is impossible.

2devnull2• 2 years ago

You seem confused. If people are unequal then equality of outcome is impossible and not worth pursuing[1]. The only possible equality is equal opportunity which as you rightly point out isn’t really equal. But it’s the only possibility. Equality of outcome denies free will.

[1] edit to clarify, I mean “not worth pursuing” past the point of equilibrium. We can improve equality of outcome to a point but then will hit a plateau. Pretending that plateau doesn’t exist is one of humanities biggest problems. It’s a costly lie that doesn’t serve anyone but certain professional groups.

Manuel_D• 2 years ago

That's not what equality of opportunity is referring to. It's about removing barriers and discrimination that would lead someone with equal or greater abilities to do worse than someone else with the same or lower abilities.

The phrase, "I believe in equality of opportunity, not equality of outcome" explicitly acknowledges that equality of opportunity will not yield equality of outcome. In part, because of the unequal distribution of talent that you point out.

rtkwe• 2 years ago

The use of equality of opportunity there is about making sure a kid has the chance to perform as well as they can regardless of where they happen to be born and the socioeconomic status of their parents because today where you're born has a lot to do with how good of a school you get into unless your parents are well off enough to go private or have the time and energy to get you into a magnet/charter program that can bypass the geographic destiny of property and income tax levels.

ahartmetz• 2 years ago

Equality of opportunity means in a strict sense external factors, in an even stricter sense things that can be changed by policy. Otherwise, it would be asking for all people to be born with the same mental processing potential. Which is absurd, which I guess is your point. But it isn't that absurd if you define opportunity as "things society can actually do for the individual".

bglazer• 2 years ago

The problem with this is that it is utterly ahistorical. Like it completely ignores every observation you could make about how wealth was historically allocated.

Workaccount26• 2 years ago

I mean, I just laid out how you end up with wealth dynasties. And this effect is observable in any population, it's not unique to the US, much less the the West.

toyg2• 2 years ago

Ah, but <insert US exceptionalism here>.

toasterlovin• 2 years ago

The Industrial Revolution is ahistorical.

nsoonhui• 2 years ago

The book "The Bell Curve" basically expressed the same point -- with data-- as yours here. And yeah, you are right. But then everyone is terrified by it, and hence the outcryand the name calling.

RajT882• 2 years ago

Partly right:

https://theconversation.com/are-rich-people-more-intelligent...

psychlops• 2 years ago

Wow, I thought this book had been burned by zealots and its ashes buried long ago. Thank you for resurrecting it.

refurb2• 2 years ago

I think the issue is that number 5 is not true, at least not in a direct way.

Look at any family of 3+ siblings. Intelligence tends to vary significantly despite having the same parents.

RajT882• 2 years ago

Anecdata, but yes, I am not the smartest sibling in my family, but one of the wealthier ones.

The "dumbest" one is potentially the wealthiest. The smartest one (hands down) is the least wealthy.

This is besides the point though. Anecdata doesn't get at what happens on average, and that's what his post is about.

skrebbel• 2 years ago

This is the Simpson paradox in action. It’s possible for smart parents to get less smart kids, or to observe a wide variation in smartness among siblings, but neither of those say anything about whether on average, smarter parents get smarter kids.

Workaccount26• 2 years ago

Genetics is a grab bag of traits. A whole bunch of marbles mixed around in a bag.

What I am speaking to is the types of marbles and their prevalence in a given bag. Not what is actually chosen in any given drawing.

A better phrasing would be "Kids who draw from bags with higher than average amounts of green marbles, are more likely to end up with green marbles in their bag"

oarla• 2 years ago

Are standardized tests an accurate measure of intelligence? IMO, they're more a measure of knowledge and perseverance. Having said that, I don't know what is the best measure of intelligence.

There is certainly a genetic component to intelligence, but environmental factors have a big role to play as well.

Jensson4• 2 years ago

Are standardized tests an accurate measure of intelligence? IMO, they're more a measure of knowledge and perseverance.

Depends on the standardized test. Some are designed to test knowledge, others are designed to test more fundamental attributes.

ACT is more knowledge, SAT is more fundamental attributes. SAT is still a significant amount of knowledge, but not as focused on it as ACT.

fallingfrog3• 2 years ago

I would hypothesize that SES probably correlates very heavily with grades at the very bottom where you have things like abuse, abject poverty, children supporting their own parents, lack of access to quality nutrition, violence inside and outside the home, etc and even a genius would be swimming against the current. But middle class and above, I would expect the curve to flatten out. In my experience super wealthy people are more lucky than clever- it doesn’t take much brains to make more money when you’re already rich.

So, the goal here should be to identify smart kids in terrible situations, while not allowing rich kids who aren’t actually that talented to game the system. That’s the goal. And of course there will also be genuinely talented middle class kids and you don’t want to ignore them either.

077sd015o462• 2 years ago

So, I know the authors of this report personally. I sincerely have the highest respect for them, they do solid work; if I see something by them I take it seriously because I know I will disagree at my peril.

However, I also feel like findings like this get distorted and manipulated, and used to refute strawmen versions of arguments out there against use of standardized testing. Some of the recent arguments being used to reinstate standardized testing as required seem irresponsible to me, even though I don't necessarily see required standardized testing as undefensible per se (my own opinions about all of this are complex and don't fall neatly into either "side", so take this all as coming from someone who is frustrated with how things get oversimplified on both sides, not as advocating for a particular position).

Here's some things to keep in mind:

1. This is all about predicting freshman GPA, which itself is questionable as a criterion. If your sole question is "how will this individual do in our college their first year", it's reasonable. But if you are asking "how well would this individual master the skills necessary to succeed in X, Y, or Z role" it's entirely different. Freshman GPA predicts graduating GPA less well than you might think, and both of those are poor proxies for "real world" behavior. They're not unrelated to "real world behavior", but the relationship is low enough that this is the entire point of social justice advocates.

A lot of people would say neither SAT nor freshman GPA are relevant, that that's the point.

2. A correlation of 0.47 or 0.44 is something to pay attention to, but it's also far from perfect. If you want to remind yourself of how much noise is in that correlation look here:

http://www.analytictech.com/mb313/correl4.jpg

That's a ton of noise at the individual level, which is the level we care about when we're talking about admissions decisions.

The problem with a lot of this isn't with the SAT, it's that what happens is this 0.45 r inevitably gets turned into a blind metric that ignores all that real variation because it puts too much pressure on the school to take people who have all kinds of other evidence of competency, because it makes their numbers look bad.

Let me put it this way: if I was selling you a bathroom scale, and showed you that scatterplot of my scale's numbers and actual weight, would you buy it? You shouldn't, but we're making consequential life decisions based on that level of noise.

The Dartmouth report conveniently ignored that noise, when that noise is about 75% of the problem.

3. Relatedly, if you look at the individual level, things are vastly different when you start taking these kinds of population-level trends and using them to make individual decisions around a threshold. That is, it's very easy to say "0.46 correlation is pretty substantial, we should require SATs", but that 0.46 correlation is across the entire range. It includes people whose SAT scores are very very low. It's not only in the range of where decisions tend to be made, and to make use of that test as a metric for decision-making, you have to assume that what's in the college board's SES index is a perfect summary of what might be said for every applicant to your college. Maybe someone is upper class but their parents were both killed in a plane crash. Maybe they are middle class but come from an abusive family. Maybe they are applying as a much older candidate, or much younger candidate, and the meaning of the scores is really different.

In a sane world where we can have nice things, the admissions committee would look at this and make exceptions. This is the idea of having test scores optional. But when you make them required, and are required to post them, then there's pressure to take the highest scores regardless of all the other information.

4. About the other information: in Table 2, they show that HS GPA is actually less correlated with SES than SAT score. Other papers with similarly large amounts of data have shown that HS GPA is actually slightly more correlated with freshman GPA than SAT score. So is the SAT necessary? Probably not. We can argue about grade inflation etc., but the numbers are the numbers, and if it's working as an alternative that is less correlated with SES, reflects a longer sample of school behavior than just a few hours on one day, why wouldn't you prefer that?

5. Note their Figure 1b. They spend a lot of time talking about this model, and in some ways it's the focus of their paper, evaluating it against the alternative. But related to my first point, the model in Figure 1b says nothing about the extent to which a college, or anyone else for that matter, might want to select on some process represented by Figure 1b. What does it matter if test score predicts, say, some other test score composite, and both are influenced by SES?

In widely used intelligence tests, there were questions about things like 19th century European literature and classical music (I think they're still there, but I can't remember offhand; they might have been made optional now). On the one hand, yes, 19th century European literature and classical music is great, knowledge of it probably reflects some memory ability etc, but are questions about that really how you want to evaluate someone's skillset? Maybe it is, maybe it isn't, but I can tell you if you're not knowledgable about that you would not get credit on those questions.

In many ways, for many, the SAT and college freshman GPA are kind of similar. I don't feel that way, but this is the classic "book smarts" versus "real smarts" issue that always has come up since the beginning of humanity probably. Showing book smarts predicts other book smarts just isn't important in some paradigms.

Again, I'm not anti-standardized testing. I think it's useful. But I also think the way it's been used in the past, and continues to be used, is in fact broken. It's not even necessarily a problem with the tests, it's a problem with the way they get used. But to paraphrase a famous educational psychologist, if you have a thing that people tend to misuse, every single time, and there are good alternatives, maybe there is something about that thing that's a problem when you put it the hands of people, and the thing either shouldn't be used, or there should be some rules put in place with teeth to prevent it from being misused.

Socioeconomic status and the relationship between the SAT and GPA (2009) [pdf]

Related Stories