A homework question in someone’s 11th grade statistics class

https://statmodeling.stat.columbia.edu/2022/12/06/a-homework-question-in-someones-11th-grade-statistics-class/

by Tomte • 4 years ago

160 258 4 years ago

statmodeling.stat.columbia.edu

dmurray• 4 years ago

This is simultaneously a great question and a terrible multiple-choice question.

Loughla2• 4 years ago

Exactly. There needs to be a few lines for justification of your logic below the answers. Also, why was "c" missing other than to make this seem slapdash?

But to the author's point, I wish that more people understood just how "soft" statistics can be compared to other mathematics disciplines. Some folks just view statistics as gospel, regardless of methods actually applied.

PebblesRox• 4 years ago

C is missing in the screenshot because the student had already used the elimination tool to hide it.

TillE• 4 years ago

Tests and homework aren't written for someone who is just going in cold, you're supposed to take the class and read the material.

Since there's no way to judge the required sample size based on the information, it's extremely likely that option A was something specifically discussed.

tyingq2• 4 years ago

That might make sense in a different case, but the actual question doesn't have enough context to answer it.

"What is the best plan of action for the university?"

"Best" meaning what?

baandam2• 4 years ago

It is a ridiculous and intellectually embarrassing question. I just picture what someone like Feynman would think of a question like this.

I wouldn't be shocked this is a higher order effect of too much of certain 20th century French philosophers. We can't even make test questions that aren't bullshit.

There literally should be a none of the above choice because of lack of information in the question.

Jensson2• 4 years ago

But this obviously isn't a math class, which is the point. I think this post should be read in the context of "make students take statistics instead of calculus", then students will get questions like this instead of studying a topic with clear and unambiguous questions.

feet2• 4 years ago

In what way is a statistics class not a math class?

ouid2• 4 years ago

calculus is full of ambiguity. Not in a mathematical sense, but as someone who has recently TA'd a calculus class, I can confidently assert that no mathematics happens there. Mathematics is only unambiguous insofar as you can check your own work. None of those students would have been capable of that. They're just memorizing rules with many unintuitive caveats that are unintuitive either out of an absurd desire to restrict the curriculum, or because they are natural in the context of their proofs, but students are basically forbidden from seeing those.

Universities have decided that making calculus as capricious as possible serves their interests of having a weeder class.

periheli0n2• 4 years ago

Questions like this, where you must have attended the class to know the answer, are often introduced to penalise no-shows. It can also expose suspected contract cheating in online exams.

However, it is quite risky, since it probably wouldn’t stand in court, e.g. when a failed student sues the school.

trophycase2• 4 years ago

This is the kind of thing that is obvious for people who are good test takers. When you know the context of the question, then you know exactly what answer the instructor is looking for.

fancymcpoopoo• 4 years ago

The best course of action is to stop selling overpriced textbooks.

harimau777• 4 years ago

Could this be something where contacting non-responders or sending out additional questionnaires could skew the results? Maybe non-response is considered part of a poll and by fighting against that you would somehow be putting your thumb on the scale?

I'm not an expert either in statistics or in polling; so I'm just speculating.

dr_orpheus• 4 years ago

It certainly could and depends on the questions being asked. If you are sending out a questionnaire via email and some of the questions are along the lines of "how many times a day do you check your email?" or "how much time do you spend at your computer?" you could certainly get some skewed results.

baandam2• 4 years ago

Of course, it would not be a random sample if 75% get one questionnaire and 25% get two questionnaires and encouragement.

ghaff8• 4 years ago

Probably close enough and potentially closer just sending out one request.

wheelinsupial• 4 years ago

I'm not an expert either in statistics or in polling

Fortunately for us, the author is :)

The author has an outline available for a course on survey sampling for those interested in references and topics to learn more. http://www.stat.columbia.edu/~gelman/surveys.course/surveys_...

nixpulvis• 4 years ago

There is no correct answer (though there could be incorrect answers).

Perhaps you want to investigate why the 30 students didn't submit the questionnaire. In the name of the study's sanctity, you'd best convince them to complete it. But you must not coerce them, or else the results would be tainted. So you invent a time machine...

OK, I'm done.

perrygeo• 4 years ago

If you've designed your sampling properly, a 75% response rate is more than sufficient. Maybe that was choice C? This isn't really a statistics question per se; it's a question about how to design and operate a research survey. Suggesting that you need a 100% response rate is absurd and perpetuates a deep misunderstanding of research design.

tlb2• 4 years ago

My survey shows that 100% of people answer unsolicited phone calls.

Before running with missing data, you need to make the case that there's no plausible reason why non-respondents would have different answers from respondents.

Aeolun3• 4 years ago

My survey shows that 100% of people answer unsolicited phone calls.

Did you not count all the people that didn’t respond?

This is just a problem with inference, not the sampling. “100% of the _respondents_ answer unsolicited phone calls _at least some of the time_”.

dlkf• 4 years ago

If you've designed your sampling properly

You are making a much stronger assumption than this. You are assuming access to demographic information of each subject, as well as population base rates for a sufficient set of attributes to correct for non-response bias. Maybe we have that, maybe we don’t. If we don’t, option A is a good tack.

bloak• 4 years ago

Where does "75%" come from? If I were trying to estimate what proportion of the population can read, or what proportion of the population is too busy to answer a questionnaire, then I wouldn't assume that the 75% who responded are representative of the whole population. Those are extreme examples but there are lots of things that might be correlated with people's willingness to respond.

Perhaps choice C was: Start again with a new random sample of students, but this time offer FREE FOOD to everyone who responds.

ghaff8• 4 years ago

Nobody operating a survey of this type assumes they'll get 100% response rate. (Although it is very common to offer something like a gift certificate to N randomly chosen people who answer the survey.)

ouid2• 4 years ago

Without any a priori information about the similarity of the responders and the non-responders, you should assume that the non-responders have colluded to hide the effect from you as much as possible. This is not a random sample anymore.

You can still get information out of your sample of that form, but its very bad, and you should only pay attention to it if you can't afford to get anything better.

twobitshifter• 4 years ago

I mostly agree but on the subject of student life, those judging student life to be poor could be the most likely not to engage with the survey. I would go forward with the data, but the response rate should be discussed in the report.

yobbo• 4 years ago

"Did not respond" is a legitimate result of an experiment and should be included when making statistics. Increasing sample size doesn't change this.

The question itself seems confused.

spookystats• 4 years ago

Did not respond is indeed a legitimate result, however (as the blog points out) if the non-responders differ from the responders then every evaluation you do on the responders will be biased.

For example, if you ask students about their satisfaction with teaching, I'd guess that students with a bad experience are more likely to reply to your survey. Based on the data you gathered you will think that the teaching at the uni is worse than it really is.

ghaff8• 4 years ago

I'd guess that students with a bad experience are more likely to reply to your survey.

Or only those with strong feelings one way or the other answer.

Or those with strongly negative feelings fear that the survey isn't really anonymous and they worry about retribution.

A. is pretty much how this would be done in most real-world situations. Make a second attempt to get people to answer and then go with what you have assuming you did get some reasonable response rate which 75% probably is.

ltbarcly33• 4 years ago

Wanting to use data is not a valid reason to allow for using data which is not suitable to use. If you send out 120 surveys and get 90 back, you can't make assumptions about what those 30 would have said and you just have to present the data you have.

mattkrause2• 4 years ago

Yup! And the “right” way to handle that bias depends on a lot of background information/subject matter experience.

If the question were “Which musical acts do you want for the Spring Fling festival?”, it might be okay—-or even smart—-to ignore the non-responders. Including data from people unlikely to attend is probably unhelpful. If you’re asking about workloads or engagement, you certainly can’t assume that data is missing at random or the non-responses are irrelevant.

For teaching specifically, one of the smartest questions I’ve seen is “How well do you think you’re doing in this course?” The crosstabs can help address response bias.

AndrewKemendo• 4 years ago

+1 to the confused question

What are we trying to test? That isn't clear so it's impossible to actually know how to follow up.

In this case "did not respond" could be perfectly sufficient, but we couldn't possibly know because all we know is:

"The questionnaire included topics on student life, academics, and athletics."

inetknght2• 4 years ago

A large university emailed a questionnaire to a randomly selected sample of 120 students in this year’s freshman class. ... Thirty of the students emailed did not respond to the questionnaire.

I don't answer questionnaires because they're almost always gamed by the sender to cheat me out of something -- or when I do provide answers they're always false in a way directly opposite of whatever I feel like.

For a direct example, the University sent questions and then asks other students to solve why some students didn't answer. I presume there's no field for a long-form answer for when none of the canned answers provide the nuance. That's a gamed questionnaire and provides zero honest value.

Other answers might be "the student's email address is invalid" or "the student is inundated with other duties" or "the student doesn't understand the questions" or "the student disagrees that any of the answers are valid".

The best plan of action is for the University to directly contact (in-person) the students and find out how to help the student to answer the question. It is, after all, in the business of helping students to educate themselves, right?

gverrilla• 4 years ago

If you wander outside the walls of the teacher's imagination you are a criminal. They fool you into thinking you're learning let's say statistics, but you're really learning to obey, and to submit your memory and your time.

inetknght2• 4 years ago

I frequently wonder what opportunities I've missed out on having never been to high school or college. On the other hand, I'm frequently glad to have never had to deal with that sort of B.S.

jefb• 4 years ago

There's no right answer.

I suspect a few things are going on here.

1. The teacher forgot to include that the University wants 100 responses to the survey. 2. Missing answer C likely contains something like "Randomly select 14 more students to receive the survey"

That makes it an actual statistics question, you'd need to calculate the response rate from the original 120 and then compute the additional solicitations required to meet the response count goal of 100.

mindslight• 4 years ago

That still doesn't fix the problem with the question. Without knowing the goal of the survey (what it is trying to measure), you can't determine the (in)significance of the non-responses.

Each answer has a failing:

A - student didn't have much to say, answers tainted by student being annoyed and clicking through the survey to stop the spam

B - surveyor didn't like the answer they got, so they drew again. that's straightforward biased sampling

D - might not make for enough data, as you said

E - student body isn't stateless, and may have had time to talk about the survey. "Hey friends, let me know if any of you get asked to do that survey because I want to give them a piece of my mind"

Choosing between them is only possible by making some simplifying assumptions, which requires knowing what you're trying to find out.

thecopy• 4 years ago

In order to answer a question in the form of "what is the best approach" the student needs to know what they want to optimize for. There is no global optimization target, so without this the student needs to guess the underlying goal in order to say what is "the best".

Question is bad.

elsherbini• 4 years ago

As an example, maybe you want to know if freshmen from different dorms had more or less satisfaction that average, but wanted to deal with the fact that different dorms had different response rates or could even have different reasons for not responding. The fear is that the data are not "Missing Completely at Random," (MCAR) meaning the missingness of the data is correlated with some of your predictors, e.g. freshmen who are too busy having a great time at one dorm, or students who are ultra-depressed at another dorm didn't check their email.

One solution would be to impute your missing data taking into account what data you do have about the missing participants (e.g. maybe dorm, major, gender). In a bayesian context you can include this imputation as part of the model fitting, which means uncertainty gets appropriately added to the results.

This is a good primer on how to handle this using the `brms` package in R:

https://cran.r-project.org/web/packages/brms/vignettes/brms_...

zevenda2• 4 years ago

I think the best answer is "B. Start over with a new sample of 120 students from this year's freshman class.", given that you can still use the 90 answers you already have (I don't see why not). This would give the largest sample size.

tsegratis2• 4 years ago

If questionnaire's goal is to discover "Why don't students respond", then you could repeatedly random sample 120 students until, by chance, they all respond

You'd then have a lot of student data, but it is completely skewed

Solution is to stop doing statistics

zevenda2• 4 years ago

The questionnaire is about "student life, academics, and athletics".

bqmjjx0kac• 4 years ago

This seems like it introduces another flavor of selection bias. You're still selecting a sample composed of people who respond and ignoring those who don't. I don't have a better answer, though.

phamilton2• 4 years ago

ChatGPT says:

  The best plan of action for the university is to use the 90 questionnaires that were submitted as the final sample. This is because the sample of 120 students was randomly selected, which means that the 90 students who did respond to the questionnaire are representative of the entire population of students in this year's freshman class. By using the responses from these 90 students, the university will be able to obtain accurate and reliable results from the questionnaire.

ltbarcly33• 4 years ago

ChatGPT is completely incorrect, almost like it has no actual insight into the question... The 90 that did respond may or may not be representative of the rest of the students.

Example:

I send out a questionaire about plans post incarceration to 120 ex inmates. I get 90 responses that are very positive, and conclude that 100% of inmates have positive outcomes after prison. In reality 30 ex inmates didn't respond because they had committed suicide, or been murdered, or were back in prison.

ghaff8• 4 years ago

Extreme differences between those who answer a survey and those who don't is obviously always an area of concern. That said, under most circumstances, the alternative is you pretty much have to not bother with data (or do some qualitative research--e.g. interviews) or you have to go with what you have and maybe discuss potential areas of bias.

scottcodie4• 4 years ago

The topic for this questionnaire is "student life, academics, athletics", which is an important part of the question when determining if non-responses are important. In this case ChatGPT gets it correct, but explains the reason very poorly.

JasonFruit• 4 years ago

Why should we care what ChatGPT says?

soupfordummies• 4 years ago

So, option A :)

phamilton2• 4 years ago

Option D

icedchai• 4 years ago

I'd pick D because it's the only answer that requires not doing any additional work.

franciscop• 4 years ago

Since it's a "statistics class", I'd assume "best plan of action" would mean "which one of the options introduces less amount of bias" here. Like many I also struggled a bit with the question in isolation, but considering this IS a statistics class, it's probably trying to find how students would minimize bias here.

For this reason, assuming 30 people didn't answer due to some given bias*, assuming they were properly and randomly selected option B sending to another 120 students would probably hit the same bias again and we should expect around 30 students to also not answer due to this. So B and D both are hitting the same bias. Same for E, but with a smaller sample size. The only viable option is A, for which (assuming a normal bias) we should expect that some amount of the the students who did not answer initially would answer after some nudging, hence reducing the total amount of bias. We could be missing the strongest biased here and that might be relevant; but with this strategy at least we would maintain-to-reduce the amount of bias (vs maintaining it with all other options).

We could also offer some stronger incentive for those 30 to answer, either stick or carrot, but that's not an option.

*this could be anything and we are not measuring/finding it, but for some example of possible biases some students might have too many courses and biased against answering optional questionnaires, some might not be affected directly by the questionnaire so think not to answer, maybe the ones in univ. dorms tend to answer more than the ones commuting, etc.

peteradio2• 4 years ago

What really matters is if the bias of your experiment is understood/understandable by downstream consumers of your result. Question is poor in my opinion because it displays an ignorance of the basics of experiment consumption.

jrumbut3• 4 years ago

I disagree, the answer ought to be whatever was planned in advance (before they saw the initial data). Once they've peeked at the data, it's not good to change the experimental procedure as is being suggested here.

The plan should have included some accommodation for non-response (like a reminder email, a suitable reward, etc), but if it didn't it's too late now. They should analyze the data they collected and apply lessons learned to the next survey.

A single reminder email is probably not a big source of bias, but something like increased rewards really could be. To make it ridiculous, if the school emailed the 30 non-responders and offered them $100,000 cash to fill out the survey, then asked "how do you feel about your school?" they will get very different responses than they got from the students who were offered a $5 bookstore gift card.

dsfyu404ed2• 4 years ago

Depending on the value of "whatever was planned in advance" you could potentially do both.

Jiro• 4 years ago

The answer has to be A, under reasonable assumptions. If you assume that the university wants to find out the opinions of all students, and if you assume that students who didn't respond still have opinions, then the problem is that the students who didn't respond may have different opinions from the students who did, skewing the results. Only choice A tries to solve this problem.

People here are drastically overthinking it. Yes, it's true that the problem didn't state those assumptions, but you are supposed to make them, not to say "the university didn't give us its criteria for 'best plan' so all the answers are equally good". Given the way that problems like this work, it's obviously meant to test knowledge of the specific idea that the people who respond may be nonrepresentative of all the opinions. Only response A shows knowledge of this idea.

ghaff8• 4 years ago

Right, you also don't know what was covered in the class. But do the best you reasonably can to collect a random sample and then go with it is almost certainly the best answer in this case. (Although you can make an argument for D.)

tedmiston2• 4 years ago

"Has to be" is a stretch. By nagging the non-respondents you are introducing a bias that the first group of respondents were not exposed to. If you annoy a respondent, it's very reasonable to assume that can influence their responses (probably negatively).

Option D seems objectively better to me over Option A based on this bias.

cat_plus_plus2• 4 years ago

Option A is strictly superior to option D, as you are still have the data for original 90 responses. If second round responses are similar, you have increased confidence that you are not running into responder bias. If it's different, you can publish both and note uncertainties and need for further study.

silisili2• 4 years ago

I actually vote against A. My work was always sending out 'completely anonymous' surveys that I generally ignored.

Resending to tell me I hadn't done it yet was indication enough to me that it wasn't completely anonymous at all.

While the question doesn't mention anonymity specifically, I'm sure it remains a point in some percentages mind.

nh23423fefe5• 4 years ago

But I could make a table Request(email, url, submitted) and a table Response(data)

And you couldn't connect the response to the user. Thats the important part right? not the ability to resend a request

HWR_14• 4 years ago

The question never says the survey is or claims to be. completely anonymous.

tgv2• 4 years ago

I work for a company that actually runs surveys, and believe me: we don't know who the respondents are. They get a link with a randomly generated id, and that's it. For us, even their email address is hidden, because respondents register to panel at another company, which deletes this information three months after closing the survey, which usually runs for a few days. I can't vouch for the other company, of course, but I do believe them. Sometimes it would be really nice to know who these people are, because not only is non-response a problem, bad responses are an even greater problem, so we'd like to know who to exclude in advance.

So much for unbiased representation.

charcircuit• 4 years ago

The people who didn't represented when nudged can answer incorrectly because they are just trying to speed through it. It should be treated as a separate population and not merged into the rest.

ToJans2• 4 years ago

This is how we do it in reality. Missing answers are very important!

panny3• 4 years ago

The answer has to be A

We have this problem at work. You try to contact customers and they never respond. Sending repeated queries has no effect.

The real answer is: The question is poor and ambiguous, therefore the question is wrong. It reflects poorly on the academic institution asking it. I would be in search of a new school for my child if I saw many test questions like this. They didn't even enumerate the answers correctly.

It is asking for the best plan of action for the University without clearly specing out the goal and all its parameters. The goal may be "Determine if at least half of our first year students had a positive experience." and that can possibly be solved with 90 responses if at least 60 had positive experiences.

It's just like when your boss asks for a time estimate on a task he's barely defined. It depends on the spec he's asking from you. Without clear specification, the question isn't answerable.

defrost2• 4 years ago

It reflects poorly on the academic institution asking it.

Only in so far as it is posed with a multiple choice response and no opportunity to make a comment.

The question itself is about a commonplace occurrence in statistical sampling.

jandrese• 4 years ago

I have no reason to believe that students that ignored the survey the first time would respond to a second survey. There will always be non-respondants, it's just part of the domain. I'd vote for sending out 120 more surveys simply because 120 is a very small sample size to begin with.

pjscott• 4 years ago

There’s always a risk that if you increase sample size without addressing selection bias, you’ll just end up with more confidence in an incorrect conclusion. (“Dewey defeats Truman!”)

TheOtherHobbes• 4 years ago

Two fallacies:

1. The non-responders are not responding for similar reasons.

2. Those reasons are somehow related to their opinions.

Both are clearly false. You cannot assume anything about their opinions from the fact that they didn't respond. All you have is their (non)action - which could have been caused by being ill, busy, distracted by personal issues, absent, drunk, or for any other number of reasons.

None of the answers are correct. If you wanted representative data you would have to:

1. Confirm with a different sample but also...

2. Use multiple different channels and methods (phone, in-person survey on campus, maybe an email survey with some kind of benefit to encourage participation, etc) to try to eliminate the non-responders who have personally valid but statistically irrelevant reasons (away, ill, etc).

wheats• 4 years ago

As the blog post says non-responders do tend to have different opinions on average than responders. Typically they do try to demographically determine who those non-responders are so they can adjust and increase the weight of that group proportionally to their limited response. This is how most polling works in the real world and it does provide better accuracy.

tgv2• 4 years ago

Suppose the non-responders are identical to the responders. Then nothing is lost by trying to get them to participate again. But if they are not, replacing them will lose information.

ill, busy, distracted by personal issues, absent, drunk

But being ill correlates (lightly) with being older, and being drunk heavily correlates with opinions. Being absent might correlate with wealth (and not even linearly). Personal issues probably correlate with education. Add them together, and you've got a fairly large part of the population missing from your sample.

pdonis13• 4 years ago

Only choice A tries to solve this problem.

No, it doesn't. It doesn't get any information from non-responders; it can't, because they don't respond! It only gets information from some people who respond on the second try instead of the first.

next_xibalba• 4 years ago

There is nothing in the question which suggests some of them (or all) wouldn't respond to a second, third, or nth prompt. In many surveys, non-respondents receive multiple outreach attempts.

jefftk• 4 years ago

Choice A will likely get some answers from students who didn't respond the first time, increasing the representativeness of your sample. It's only futile if you conflate as "non-responders" the people who didn't respond on the first opportunity with the smaller number who wouldn't respond even given a second opportunity.

AtlasBarfed2• 4 years ago

No, the question says the original 120 were "sampling". There is no "all".

aj76• 4 years ago

Option A is absolutely the worst choice. The non-responders have already selected themselves into a non-random set, and making a request to them alters the entire experiment.

quickthrower23• 4 years ago

Surely the responders are also self selected into a non random set too?

csours3• 4 years ago

I've enrolled in online college to get my undergrad degree.

Some of my courses were related to certificates that many people list on their resume.

The questions are very bad. Very very bad.

One example:

Which of the following considerations are most important when choosing a software solution:

A. System Requirements B. Something else C. Distribution Method / Software Installation D. Something else

My answer is Distribution Method (ie, how do you install it, or is it available as a web service). After working in tech for 20 years, distribution is still the hardest problem, and the one that can take the most time, when you consider the total time by all stakeholders.

They said that is not the answer because there are software installation managers.

---

I understand why that certificate has the answer they gave, and I understand why the test is in multiple choice format, I just deeply disagree with each of those decisions. After taking that test, I feel like anyone with that cert will need to be retrained after hiring. I feel like it is worse than useless for many of it's claimed benefits. (You do learn some about hardware and standards, which is nice)

periheli0n2• 4 years ago

This is likely the result of an overworked and underpaid academic being asked to quickly draft a few questions for the online exam tomorrow. It is definitely a sign of insufficient quality control.

WirelessGigabit• 4 years ago

Not underpaid. No experience in real life software engineering. Probably thinking of this ideal world where there is planning and then a waterfall implementation.

tlhunter• 4 years ago

I suppose it depends on the questionnaire that was sent out. Sounds like survivor bias might come into play. Was the questionnaire about dying from a university buffet?

dariusj18• 4 years ago

This is sending a message that statistics is more like social science than like math

I think the real issue at hand is this question belongs in a sociology class, not a statistics class.

mharig2• 4 years ago

The question cannot be answered. What does 'best' mean? The statistical most profound? The cheapest for the university? The optimized combination of both? ... And no, assuming that the 'best' is clearly defined, because it is a statistics test, is not valid.

Apart from this, the statistical best method is to include 'did not respond' in the model, as (an)other commenter(s) has (have) already mentioned. Bayes rules, btw!

Reselecting 130 students is not different from going with the 90, if the not-/responding is a confounding variable, as you just select more from the responder group.

jareklupinski• 4 years ago

the most profound one in that case was probably C, or 'no answer'

dehrmann• 4 years ago

Yeah, they're all different types of wrong. Which is cool for an essay question, but strikes me as BS for multiple choice.

dataflow• 4 years ago

What does 'best' mean? The statistical most profound? The cheapest for the university?

It means the method producing the sample most representative of the population. It's fairly obvious they're not asking about (say) budget/cost minimization on a statistics test.

mharig2• 4 years ago

'Fairly obvious' ist not really scientific.

I may be a little bit picky, but if someone cannot formulate a clear question in such a simple case, he will fail more badly in more complicated cases.

WA• 4 years ago

Exactly. The question doesn’t even state a goal. Without a goal, there can’t be an answer to "the best action".

karmakaze2• 4 years ago

Stepping back, it’s interesting to see a homework question where there’s no unambiguously correct answer. This is sending a message that statistics is more like social science than like math, which could be valuable in itself—as long as it is made clear to students and teachers that this is the case.

I think I agree with this sentiment but not the specific wording if taken literally. There is one "unambiguously correct answer" but it happens to be problematic in practice. The others are all "unambiguously incorrect" but possibly usable for a weaker result. Also statistics is much more like math than social science where it's important to be precise and correct and not qualitative hand-wavy reasoning.

jrumbut3• 4 years ago

I think statistics is the adapter between the social sciences and math, rather than being part of or similar to either.

karmakaze2• 4 years ago

Statistics can be that certainly but it's entirely optional. Analyzing JWST data uses statistics without the social science interpretive aspect.

crmd• 4 years ago

This is a great high school-level question to test basic understanding of selection bias. A is the only answer to demonstrate that responses to web surveys are have inevitable self-selection bias.

godelski3• 4 years ago

This is the exact type of question that tricks those with more nuanced knowledge but doesn't trick those that memorize textbook answers. I'm not sure we should be encouraging this type of learning, despite it being how a lot of standardized testing questions work.

wppick• 4 years ago

Actually, that sounds like it's working as intended

throwawaysleep• 4 years ago

I would say it is a utter failure of a question if an expert in the topic is perplexed by the answer and there is (presumably) a supposedly correct answer.

Knowledge is punished? Don’t ever learn beyond the book? That’s outrageous.

gregmac• 4 years ago

I wonder how this type of question even gets written. Does the author purposely put in these "gotcha" questions to see if you read the textbook? Or are they literally unaware of how bad the question is written - ie: their knowledge is limited to the point they don't see the nuance or unambiguity?

gjadi7• 4 years ago

I used to fully agree with that feeling but not anymore.

IMO they are two kinds of tests:

1. Tests to help you retain information.

2. Tests to evaluate your knowledge.

Assuming this question belongs to the first kind (memorization) and the answer is in a textbook being studied, then this is a good question.

jolmg5• 4 years ago

So, a good question for a bad kind of test? Why shouldn't all tests be type 2?

godelski3• 4 years ago

I think we're talking about two different things. I have no problem with tests in general. But I do have a big problem with text questions that are intended to be tricky.

karaterobot• 4 years ago

Stepping back, it’s interesting to see a homework question where there’s no unambiguously correct answer. This is sending a message that statistics is more like social science than like math, which could be valuable in itself—as long as it is made clear to students and teachers that this is the case.

Well, wait a minute. Do we know whether the purpose of this question is to make students consider all the options broadly, or is it that there is a "right" answer (in the sense that all other answers will be marked wrong) and it's just poorly written? I ran into those kinds of questions all the time on standardized tests, and they never let me answer in the form of an essay explaining the pros and cons!

panarky• 4 years ago

Option C is missing, that must be the unambiguously correct answer.

bo1024• 4 years ago

What do you think it might say?

DiggyJohnson• 4 years ago

If it’s being used for evaluation, then there should be a “correct” (or “best”) answer. If it’s not used for evaluation, then I’d probably be more confused than enlightened by a multiple choice question with no best answer, and prefer it phrased as a discussion question or similar.

Just my $0.02.

Myrmornis2• 4 years ago

Since it's a multiple choice question, one assumes there is a correct answer, and presumably that is A, because there is a clear argument for it (you wish to sample from the population distribution, which can be viewed as a mixture of an initial-respondents distribution and an initial-non-respondents distribution and so you cannot claim to have sampled from the mixture until you've sampled from both mixture components, since they are probably statistically distinct).

Tangentially though, a big question here is how wise it is for the US education system to be based so heavily on multiple choice questions. There are other countries with decent education systems that do not do this.

pdonis13• 4 years ago

one assumes there is a correct answer, and presumably that is A, because there is a clear argument for it (you wish to sample from the population distribution, which can be viewed as a mixture of an initial-respondents distribution and an initial-non-respondents distribution and so you cannot claim to have sampled from the mixture until you've sampled from both mixture components, since they are probably statistically distinct)

But you can't survey non-respondents--because they don't respond!

What option A will actually result in is having a mixture of people who responded on the first try, and people who responded on the second try. In principle, neither of those will be representative of non-respondents. Whether this creates a significant problem in practice will depend on lots of other assumptions.

My own first choice would be either E or D, depending on (a) whether you have enough resources available to send 30 more emails, and (b) how much statistical power you are sacrificing with a sample of only 90 instead of 120. (If you were doing things properly, the number of emails you send out initially would be larger than the actual number you needed to get enough statistical power, by some factor that would depend on what fraction of people you expected to respond. In which case D would be the obvious option.)

tantalor• 4 years ago

how wise it is for the US education system to be based so heavily on multiple choice questions

Who said anything about "US education system"?

Where in the original article does it say "US"?

marzell• 4 years ago

This smells like a scenario where the teacher isn't looking for the "right" answer, speaking broadly or objectively.

Instead, I suspect there is a particular concept or perspective they have been trying to impress on their students recently, and in this case they are expecting the answer that most demonstrates they have paid attention to the recent lessons.

I could be wrong of course.

Xcelerate2• 4 years ago

a big question here is how wise it is for the US education system to be based so heavily on multiple choice questions

There was a great blog post I read a while back on redesigning multiple choice tests to allow the student to indicate the "confidence" of their response, with a more confident answer being rewarded/penalized more heavily than a response with low confidence. This allowed for a statistically better sample of how well the student learned the material.

I thought for sure the post was written by Scott Aaronson, but I haven't been able to find it despite extensively searching his blog, so maybe it was someone else.

abruzzi• 4 years ago

I remember taking a multiple choice question test for some scholarship 35 years ago. 20 questions with the explicit instructions that for any question all of the choices could be correct or none of them or combination, and you were supposed to check only the answers that were correct. It was a VERY hard test. In addition you could score less than zero since every correct item that you checked would add points but every incorrect item itme you checked would subtract points AND every correct item you didn't check would also subtract points. You didn't get points for not checking incorrect items.

I've never seen a test like that for school work.

Someone• 4 years ago

I would choose A, too, but if you can do A, the questionnaire isn’t anonymous.

I would go for A’: email all of them, reminding them to fill in the questionnaire, if they haven’t already done so.

AtlasBarfed2• 4 years ago

1) WHAT IS THE DESIRED SAMPLE SIZE obviously it isn't 120, because only a moron desires a sample size of 120 and sends out 120 random spam requests

2) WHAT IS THE NECESSARY CONFIDENCE INTERVAL

4) WHAT IS THE POPULATION SIZE

But in general, a 75% response rate is likely fantastic, and should be sufficient if the original "120" sample size was rationally selected. So D strikes me as overwhelmingly the best answer.

rustybolt2• 4 years ago

I remember bombing a biology exam because I spent like 60% of the available time on a single question: "Is it good or bad for digestion to drink water with dinner?"

I mentally went over all the course notes, and was sure this was never covered directly. So, I thought, something must have been covered from which I can deduce this. Again I went over all the course notes mentally, and decided that was certainly not the case. I could made up an argument why it would be good for digestion, and why it would be bad for digestion.

Since no explanation was asked, I decided to approach it as a multiple choice question. Since I could recall more evidence for it being bad for digestion, I simply wrote that down.

I got 0 points. According to the teacher's key, could be either, as long as it included an argument (it was along the lines of "water makes food soft" which was not covered in the course because it's common sense).

Still get frustrated about it. Something can't be both bad and good. The question states "X is bad or good, which is it?", not "think of an argument why X could be bad or why X could be good". It basically rewards bullshitting and punishes answering the question faithfully.

I went into STEM after that.

DwnVoteHoneyPot2• 4 years ago

Which discipline is biology included under:

A) Science

B) Technology

C) Engineering

D) Mathematics

E) None of the above

datpiff• 4 years ago

Biology isn't STEM now?

belorn• 4 years ago

On standardized tests, couldn't they be fake questions that exist only to catch cheaters?

ToJans2• 4 years ago

The only correct answer is "A: mail them" if you want to stay statistically relevant.

You cannot ignore the group that didn't answer the questionnaire, as they will most likely expose some of the behavior that you are researching (i.e. about life etc), and might have a huge impact on your results.

So, the statistical result you currently have (based on the 90/120 students) will most likely be biased, and is invalid. (25% of missing input that might heavily impact your outcome is most likely making your results useless)

Thus, the only way to make it statistically relevant is getting more answers from the no-show group.

B. If you start over and do the same thing you will most likely get similar results/no-shows. So that will not be a good solution

D. As explained before, ignoring the no-shows results in a potentially biased outcome.

E. If your initial set is most likely biased, adding another 30 subjects will not fix your initial bias.

iLoveOncall• 4 years ago

You cannot ignore the group that didn't answer the questionnaire, as they will most likely expose some of the behavior that you are researching (i.e. about life etc), and might have a huge impact on your results.

This is completely paradoxical.

You are saying that using the data from the 90 would be jumping to conclusion because you would probably ignore data that would not match those 90.

But making this claim IS jumping to conclusions, because you are making an assumption (the 30 have something in common explaining why they didn't fill the form).

scaredginger• 4 years ago

It's really not. The 30 might have something in common, which calls any findings that exclude them into question. This doesn't rely on any unreasonable assumptions

bjacokes• 4 years ago

See: non-response bias [1]

Parent comment should've said "could" instead of "will most likely", but their point is correct.

[1] https://en.wikipedia.org/wiki/Participation_bias

SoftTalker• 4 years ago

The behavior that the non-responders are exposing is that they don't like answering surveys. Factor that in how you will, but if you beg/incent/demand that they answer, you are no more likely to get honest answers.

I remember kids in school who would just answer "A" to every question because they didn't like surveys and didn't care.

Aeolun3• 4 years ago

The fact that you have to nag them will introduce bias by itself. Now 25% of your sample will be biased by the nag. It’s just a question of whether that’s worth it compared to not including them at all.

Anyway, this should be an open ended question.

pdonis13• 4 years ago

the only way to make it statistically relevant is getting more answers from the no-show group.

You can't get answers from the no-show group, because they don't respond! You are just creating three groups instead of two: those who responded on the first try, those who responded on the second try, and those who don't respond at all.

aj76• 4 years ago

Option A is absolutely the worst choice. The non-responders have already selected themselves into a non-random set, and making a request to them alters the entire experiment.

abriosi• 4 years ago

A- Asking students who haven't answered the questionnaire will likely yield a very low response rate (not a lot of entropy gained here)

B- Another shot at conducting the experiment? Great now we can do more statistics on two independent sampled sets (lots of entropy)

D- Why would we discard the chance of obtaining more information?

E- Worse than B

madrox• 4 years ago

I got an undergraduate degree in statistics, and what we were taught was that there are lots of valid ways to sample but how you sample affects how you can generalize the results.

Since this is a college survey and not an FDA trial, it's probably ok to proceed with D or E in addition to A (B is the only bad answer). That said, there most certainly is a correlation between in non-response and lower scores on the survey, so when presenting findings it's important to represent that concern. In general, surveys are pretty weak means of drawing conclusions, but sometimes it's the means.

layman51• 4 years ago

I would think "A" is a good way to proceed. This might be a silly question, but I wonder how it affects us if we send this reminder message to everyone versus just the people in our sample who we know have not responded yet. I know that maybe some of the people who did previously respond might get confused and possibly try to respond again. I would be interested in seeing if they update their answers or if they stay consistent. Any statistical insights on this?

EDIT: I thought about this some more and considered that sending out a reminder message to everyone may be akin to starting a new survey using a non-random sample. But by reaching out to just nonrespondents, maybe it’s okay.

dfee• 4 years ago

What is the best plan of action for the university?

Stop spamming the students. :p

jedberg• 4 years ago

The problem with E is that it creates a biased sample of people who are willing to respond.

actionfromafar• 4 years ago

- "I think I’ll choose C."

adenozine• 4 years ago

Perhaps:

* Take the first 90 results.

* Send the email, wait some specified time (Perhaps 75th percentile of time of how long it took to get the first 90 results)

* Apply Bayes Theorem if there are new replies

and move on.

Alternatively, having someone go physically TALK to students in quads and cafeterias will probably generate a higher quality of data. I'm guilty of it myself, but sometimes after a negative experience, I'm apt to just leave a 1-star review when it should really be a 3 or 4, out of spite. I think the internet has the inhibition-removing quality that encourages these behaviors, whereas if someone outside the store or whatever just asked me "Hey, how was your experience?" I'd have probably said, "Oh, it was alright, just wish they'd had the sauce I always order, I had to get something different."

But anyhow. It's probably C.

Aeolun3• 4 years ago

I don’t think any of the answers there are unambiguously correct.

Did you even need 120 responses in the first place? Did you truly expect every single student to respond to an optional survey? How did you train 75% of your student population to respond to a survey in the first place, 10% seems like a more reasonable estimate.

The correct answer is to account for non-response in the first place, but I guess that was option C.

scottcodie4• 4 years ago

There is often attrition when doing surveys. If the correct answer is always A then that implies there can be no attrition for completing the survey. Since the list of topics asked don't seem to correlate with non-responses (such as doing a survey about sexual harassment where non-responses may be very important), the correct answer would be likely D, to discard the subpopulation.

Archelaos• 4 years ago

Here is my answer:

The expected response rate of this year's freshman class to the questionnaire is 75 % (90 out of 120).

Option B, "Start over with a new sample of 120 students from this year’s freshman class." is on average not improving the situation, because we must assume that the most likely outcome is again a response rate of 75 %.

Option D, "Use the 90 questionnaires that were submitted as the final sample." is, of course, also not improving the situation.

Option E, "Randomly choose 30 more students from this year’s freshman class and email them the questionnaire." would give us on average 30*0.75=22.5 more returned questionnaires, which would improve the representativness of the survey in comparison to Option B and D.

Option A, "Send another email to those 30 students who did not respond encouraging them to complete the questionnaire." is the most interesting one. If we assume that the missing data from this 30 people is not random, in other words: that there is a correlation between not responding and some of the answers to the questions, it would be important to attempt to get as much of this missing data in a second call. The unknown figure is the response rate to such a second call.[1] But even if it is below 75 % it could increase the representativeness of the survey more than Option E, because it would be the only chance to represent first-time-non-respondents in the survey. However, we must also assume that the conscientiousness of the answers does not differ between first and second time respondents. Under these assumptions, I would choose Option A.

[1] This is why we cannot really be sure that Option E would not be better in a specfic case. If the second-time response rate is 0 %, Option E would trivially be better; if the second-time response rate is 75 % or more, Option A would be clearly better. Somewhere inbetween is the sweet spot were the advantage in terms of representativeness swiches from Option E to Option A. Where exactly this spot is depents on the unkown parameter of how much actually the answers of first-time-respondents and second-time-respondents differ.

raydiatian• 4 years ago

Why is this front page worthy? This is the academic equivalent of a shitpost imo.

qpwoeirut• 4 years ago

I agree; ambiguous or poorly written questions are nothing new in high school academics. I suppose it's an interesting topic to argue over, but this question isn't anything special.

celestialcheese• 4 years ago

Maybe this was a "Poll" type multiple choice question, given that it's online? Could be used as a jumping-off point for discussions next class.

curiousllama• 4 years ago

IMO, this is almost certainly a badly-written question with a single “correct” answer, A.

The common thread is that they all answers are related to “non-response bias.”

This is the type of Q that results from question-writers that have (1) no deep stats background/passion and (2) a list of textbook concepts to write multiple choice Qs for.

I used to go CRAZY in school, until I started thinking of these Qs not as “what’s the answer” and more “what answer is most likely to result in a question like this?”

scottcodie4• 4 years ago

Given the topic is "student life, academics, and athletics", would you care about the non-responses? If the topics correlate well to non-responses then you'd have a reason to care about the missing results. The best answer here is D, because there is always attrition when doing surveys and getting 100% responses isn't usually a goal for this type of experiment.

jimwalsh• 4 years ago

Given how surveys work in the real world, D would be done.

LastTrain• 4 years ago

If you send another note to the 30 that did not respond, they are no longer randomly selected, they have become another category that took two times to respond. In fact, once you choose to do anything but use the original sample you've altered your experiment.

On the other hand, it is a fucking preferences survey sent to students so do whatever...

htag• 4 years ago

If the population of students that respond after one survey request is different than the population of student that respond after two survey requests (or don't respond at all), then the original 90 responses aren't representative of the entire student population. "Altering your experiment" is a distraction because the presumed goal is to learn about the entire student population.

nashashmi• 4 years ago

The answer is C. But we don't know what it is, so we are trolled into debating which of the remaining answers is most likely correct. Answer: none.

pestatije2• 4 years ago

F. None of the above

That's definitely the correct answer.

It might have been C: (C. Deal with the 120 data you have already as is) but since it has been deleted we don't know.

wodenokoto• 4 years ago

What's the difference between your C and the option D (use the 90 samples)

If I got 120 rows, where 30 was NULL, I'd throw those 30 away.

pestatije2• 4 years ago

Those 30 are not NULL, they're "chose not to answer", usually classified as "don't know / won't answer"

s1mon• 4 years ago

FWIW, ChatGPT answered this in one go:

"It is best for the university to use the 90 questionnaires that were submitted as the final sample. Since the university selected a random sample of 120 students and only 90 responded, those 90 responses provide a representative sample of the opinions of this year’s freshman class. Starting over with a new sample or adding 30 more students to the sample could introduce bias into the results. Additionally, sending another email to the 30 students who did not respond may not be effective in increasing the response rate."

FiberBundle• 4 years ago

Not for me:

"Of the options provided, the best option would be to send another email to the 30 students who did not respond to the initial questionnaire, encouraging them to complete it. This would allow the university to increase the response rate and collect more comprehensive data. Starting over with a new sample of 120 students would be time-consuming and may not necessarily produce better results, since there is no guarantee that the new sample would have a higher response rate. Using the 90 questionnaires that were submitted as the final sample would not provide a representative sample of the entire freshman class, and choosing 30 more students at random and emailing them the questionnaire would not address the low response rate from the initial sample."

quickthrower23• 4 years ago

Convincing, confident and opposing bullshit arguments, this thing should be in politics

quickthrower23• 4 years ago

Ok now put $1 in the GPT jar

trophycase2• 4 years ago

I don't think they actually want you to respond as if you work there. I think most students probably know that. Most likely this question was in a unit about certain types of statistical biases. A is very clearly the correct answer.

kevinmmccormick• 4 years ago

William Cochran, best known for Cochran's Theorem (https://en.wikipedia.org/wiki/Cochran%27s_theorem), wrote a book about sampling techniques in 1977 (https://ia801409.us.archive.org/35/items/Cochran1977Sampling...) and dedicated an entire chapter (13) on the different approaches you can take for the cases of non-response to voluntary surveys.

One approach that might fit here is similar to what he calls "Double Sampling", which is basically to email the 30 that didn't respond, and even if you get something like 5-10 responses back, you can use that smaller sample to roughly represent the entire 30 that didn't respond. Not perfect (this will increase the error bars for your survey results), but nothing in survey design is.

A few other options are also offered, such as taking a much smaller representative sample, but somehow forcing 100% participation (possibly do-able in some circumstances) to help provide a gradient for the whole population, including those who didn't respond to the initial survey.

lukego• 4 years ago

I was contacted for the Swiss labor market survey and ignored that because I was very busy with other things (business, baby, etc.) It required multiple extended phone interviews.

They followed up and emphasised that they take the experiment very seriously and that my answers would be considered representative of 500 people in Switzerland, notionally all juggling the same priorities as me.

I was very impressed at the seriousness and made time to take the survey.

ngcc_hk2• 4 years ago

The problem is representation not response. Forcing or sample again … not sure it will give you

aj76• 4 years ago

Option A is absolutely the worst choice. The non-responders have already selected themselves into a non-random set, and making a request to them alters the entire experiment.

dusted• 4 years ago

Stepping back, it’s interesting to see a homework question where there’s no unambiguously correct answer.

Yeah, until the teacher tells you that you chose the _wrong_ answer.

DwnVoteHoneyPot2• 4 years ago

You guys are thinking too much. The correct answer is whatever the teacher said the answer is in a previous class. If the teacher said in class that the sky is black, and the exam answers are: A) blue, B) black, C) white, D) grey... then the sky is B) black.

School is not about learning or thinking. It's about absorbing info fast and spitting it back out fast.

ngcc_hk2• 4 years ago

Real life statistics would use what he/she said. Adjust the response using the population statistics. Qed.

riverdweller• 4 years ago

Am pretty sure a better answer than any of these would be:

Randomly choose *40* more students from this year’s freshman class and email them the questionnaire.

th2owaw3rfar2• 4 years ago

I think another option should to determine if those that didn't respond to the questionary belong to a certain type of alumns, for example the type of person that don't participate in social activities, anyway is important to know that some part of the population is different from the rest.

francisofascii• 4 years ago

This sounds like a good question for a political pollster. How do they evaluate all the people who don't answer the phone?

A homework question in someone’s 11th grade statistics class

Related Stories