Cheating our children: Suspicious school test scores across the nation

http://www.ajc.com/news/cheating-our-children-suspicious-1397022.html

by tokenadult • 14 years ago

50 39 14 years ago

ilaksh3• 14 years ago

Its not as simple as that. From everything I hear, the teachers and school administrators are being squeezed between a rock and a hard place.

I would guess that there is a subculture of school administrators who feel that No Child Left Behind etc. are hurting curriculum and unfair to teachers and schools.

They have to decide between being honest about their scores or firing a bunch of teachers and/or possibly having their schools shut down, and/or having class sizes they can't cope with due to the need to hold back huge numbers of students.

I'm not a teacher but I have heard a few teachers talk about this. I think its sort like what happens quite often between a manager and his employees: the manager makes some proclamations that are impractical and often unrealistic, and some significant portion of the employees decide they just have to smile and nod, but basically they are ignoring it as invalid. So in this case they feel they are forced to falsify the results.

I think that probably there are some schools in well-off neighborhoods where kids are advantaged and the administrators don't run into these issues that they consider moral dilemmas. The administrators and teachers in those schools may say that the other teachers deserved to be fired based on the poor performance of their students, but those privileged teachers aren't aware of the disadvantages that the other students are working against and therefore don't realize that students' failure to meet objectives often isn't their teacher's fault.

Personally I think that national measurements and guidelines are very important, but fine-grained command centralized control over an entire system has been proven time and again to be repressive and result in failure.

This goes for all sorts of systems, not just education. It is good to try to come up with standards, do holistic analysis, have common data formats and an egalitarian shared perspective and goals. But a system needs to be highly localized and non-homogeneous to be efficient, robust, and have the freedom to evolve and adapt to local conditions.

glesica3• 14 years ago

I think you are spot-on in suggesting that characteristics other than the quality of the schools are responsible for students' performance and learning at this point.

Seems to me that at some point we need to start focusing on parents and families instead of the schools themselves.

Clearly the educational system works reasonably well if the students have certain characteristics (wealthy, well-educated parents for one). So maybe we should be asking how can we "fake" those attributes for the students who don't have them?

It's not an easy problem to solve either way, of course. But after you've retooled the school system over and over for decades, trying a different approach seems only logical...

thret• 14 years ago

Boarding school solves the distant/absent/incompetent parent(s) problem.

nanijoe• 14 years ago

This whole "blame the parents" business is in my opinion why the school system in the USA continues to be in decline. If you teach 20 5-year olds for 6 hours a day, there is absolutely no reason why any one of them should still be unable to read at a basic level after 7 months.

For the record, I went to school in a 3rd world country, where many of my classmate's parents could neither read nor write, and that did not seem to adversely affect their performances, because our teachers actually taught.

yummyfajitas5• 14 years ago

From everything I hear, the traders and managers at failing hedge funds are being squeezed between a rock and a hard place.

They have to decide between being lying about their losses or fire a bunch of people and possibly have their hedge funds shut down.

I'm not a trader, but I have heard a few traders talk about this. I think it's sort of like what happens quite often between an investor and his fund: he makes some proclamations that are impractical and the market doesn't cooperate, and some significant portion of the employees just smile and nod but ignore it and keep collecting their base/bonus. So in this case, they feel they are forced to commit fraud and hide their losses.

I think that probably there are some desks in well-run firms where profits are high and traders don't run into these issues that they consider moral dilemmas. The managers and traders in those firms may say the other traders deserved to be fired based on their losses, but those privileged traders aren't aware of the disadvantages that the other traders are working against and don't realize the market simply moved against them.

Just thought I'd apply the same logic to another profession, albeit a profession for which most people hold few positive feelings.

Natsu• 14 years ago

The analogy breaks down when the people themselves are responsible for making their own irresponsible promises.

That said, cheating is terrible. It will create even more unrealistic expectations and the more unrealistic things become, the more honest people will be weeded out.

mokus2• 14 years ago

You left out the part parallel to "I would guess that there is a subculture of school administrators who feel that No Child Left Behind etc. are hurting curriculum and unfair to teachers and schools."

The standards schools are being measured against (and rewarded or punished for their performance relative to) are far less objective than "made or lost money". You have not applied the same logic to a different profession, because it misses out on the fundamental point of the comment you are parodying - that the standard is, in many cases, judging the schools wrongly.

The dilemma is not whether or not to lie about failing. The dilemma is whether or not to lie about a shitty metric that _says_ you are failing when you can see clearly that you are not.

I'm not going to make a claim one way or another about what I think of that argument. I'm just pointing out that the argument you are parodying is not the one you appear to think it is.

saulrh6• 14 years ago

Ensuring that scores are reported correctly won't even start to fix our educational system. We don't need to cut down on cheating; rather, we need to realize that more data is not better data, that teachers know a lot more about teaching than we do, that the teachers were doing a hell of a lot better before we started measuring them, and that we should just let them do their jobs.

ilaksh3• 14 years ago

I agree about more data is not better data. I get the impression that its not just that they are doing standardized tests. I mean I remember doing quite a few California tests when I was growing up (I was born in 1977 so this was before No Child Left Behind).

But from what I hear, they have gone completely overboard with the standardized tests so that it takes up all of the time.

I don't see anything wrong with having a few simple standard tests to cover some important things. I think it is actually important to have some kind of holistic measurement and analysis and guidelines.

But clearly there are too many tests, too comprehensive, and the attempt to mandate actions based on the tests is repressive and even destructive rather than helpful.

mackyinc• 14 years ago

I agree to you, just let the teachers do their job.

einhverfr• 14 years ago

I think, as you seem to point out, the real problem is trying to reduce teachers to factory work, where these tests are sort of quality assurance after the fact. It's important to get back to the idea of hand crafted education, rather than trying to mass produce it, and work on hiring good teachers, and turning them loose.

tomjen32• 14 years ago

How do we know the were doing a good job before? Heck most of the teachers i had in school were okay, but there were a significant minority who could have been replaced with a doorknob and improved how much we learned.

saulrh6• 14 years ago

I've had bad teachers before, and I've had good teachers before. What I've observed is that the tests tend to prevent the good teachers from being good without really detecting the bad teachers. A few of the best teachers I ever had were forced to waste a month ticking boxes on the "get ready for the test" list, a month that they could have spent teaching things that were actually useful. Meanwhile, the bad teachers went on doing exactly the same thing they were doing before, because even a doorknob could have checked off "students must have studied the following important people from the civil rights movement".

viraptor• 14 years ago

This reminds me of the standard problem off measuring programmers'' productivity. If some high level manager gets to do what they think would work, you end up with something like counting lines, or bug fixes per day without any regard for the bug size. You'll get measured that way against the other teams/members. Isn't that pretty close to what you're describing? Maybe it's a similar issue? Maybe people are just fooling themselves that you can reduce this problem to any meaningful single dimension?

yummyfajitas5• 14 years ago

...teachers know a lot more about teaching than we do, that the teachers were doing a hell of a lot better before we started measuring them...

How do you know this? Even in principle, how could you know this?

To me, it sounds like "I was such a good government contractor before they started auditing me and making sure I actually delivered what was promised."

saulrh6• 14 years ago

There are ways to evaluate teachers other than running tests. In particular, I was in about ninth grade when NCLB hit, and its effects started rolling in in 11th grade. I was sufficiently aware of the situation to observe. It was a lot more like "The government contractors building my bridge were meeting all their deadlines until their managers started demanding documentation in triplicate for every nail they hammered in."

DanBC4• 14 years ago

> that teachers know a lot more about teaching than we do

Teachers think they know what's right, but without good quality research (large samples and preferably some blinding) they cannot know what methods are more effective.

nullflux• 14 years ago

The problem with education is that these tests treat students as a commodity. "Standardized" tests tend not to be even remotely useful in diagnosing the positive social capability of a human being.

It is a travesty that American teachers and administrators are put in situations where they feel that they have to falsify results because their students are not reaching them. The learning capabilities and specialties of students are as diverse as the rest of human nature, and the best way to create proper incentives for students and teachers is to build a pervasive culture that rewards learning and meaningful contribution to society. Perhaps American education needs to edit its message: aligning extracurricular American cultural values to highlight the positive effects of learning on success will likely increase test scores more than the next set of standardized whatevers that get thrown at them by an overwhelmed, out-of-touch committee.

droithomme2• 14 years ago

OK, so they analyzed 69,000 schools by comparing test scores to a model they created predicting what the scores should be, and 200 out of the 69,000, or 0.3%, didn't match the model. (Note: main article says 200 districts, but the associated article showing the data says 200 schools http://www.ajc.com/news/cheating-our-children-1393866.html)

It's amazing that 99.7% of schools matched their model. Must be a very good model.

It's ridiculous that they are using this to say there is evidence of cheating though since, even though their model is 99.7% accurate, it is not 100% accurate, and that's to be expected.

The methodololgy also relies on questionable assumptions. They look at 3 year slices to find cases where test scores jumped up one year and then back down the next. They consider these evidence of systematic, district wide cheating. That is unproven. If it was cheating, why would the scores go back down. Is there a correlation then between years of upswing and schools being on the list of underperforming schools about to be forcibly shut down by the feds? That might be evidence of cheating if so. But if that was so it would surely be claimed in the study. Also, district wide cheating means a mass conspiracy in which hundreds of teachers managed to, under direction of a coordinated central authority, successfully cheat for one year but not the next, and to keep the secret, and did so at hundreds of schools. That's quite a conspiracy.

I am sure cheating takes place from time to time, but it is far more likely to happen at the individual teacher level than as something happening district wide and thus coming from a central source because the larger a conspiracy gets, the less likely it is that everyone can keep the secret. Checking for schools that have a single year with higher scores and finding a small number that do is poor evidence of a cheating conspiracy.

tokenadult2• 14 years ago

I appreciate your thoughtful comments, droithomme, on many of the HN threads on education reform. You are correct that the statistical model used by the newspaper report needs more validation. On the other hand, as the story relates, the model is a generalization of a model developed during reporting by the same newspaper about known cheating incidents that could be independently verified by other means, including witness reports of overt conduct that breached test security or altered student-submitted answers after the test booklets were submitted to teachers or school principals. The earlier reporting by the same newspaper revealed that sometimes agreements ("conspiracies") to cheat on tests were rather widespread in entire schools or even entire school districts, and stayed secret for a time (at least) because all the school officials involved were united in having an interest in presenting fake test scores.

I think that the scores bouncing up but then back down is, again, an empirical observation from the known cases of cheating in the area the newspaper reported on most closely a few years ago. As the submitted article here suggests, it is certainly notionally possible that a group of students might spurt forward suddenly in academic performance if they enjoy the instruction of a particularly good teacher. But, if genuine learning had taken place, it is notionally much harder to find the same group of students fall back the next year to a level lower than the level they reached the previous school year. It is much more likely that students might progress in grades from a teacher who is eager to look good, even if it means cheating, to a teacher who is not in on the cheating plot, or who at least is less effective at cheating in a manner that massively boosts scores.

I think some of the other claims that you suggest would be made by the newspaper, if the statistical model were definitely backed up by other evidence of cheating, could be made if journalists in the various localities involved look for other sources of information. Not in all cases, as you correctly point out, but in many cases, additional reporting leg work could probably find direct evidence of violation of test security before the test resulting in exact "teaching to the test" item by item, or alteration of student item responses after students submit their test booklets to school officials for scoring. That is currently an open question, but based on known previous examples in various parts of the United States, there is enough smoke here to prompt journalists to look for sources of fire.

droithomme2• 14 years ago

Interesting, I didn't catch from the article that this three-year bump model had been tested on multiple incidents and found to correlate. It mentions in the third paragraph their methodology used "a pattern that, in Atlanta, indicated cheating in multiple schools". It seems Atlanta was the only place they are claiming there was a correlation, but even of this single data point they offer no specific evidence beyond the red dot in their graph. I did find it interesting that the biggest recent case where there was systematic corruption within a system was in Atlanta, and the newspaper here trying to show that cheating is common throughout the rest of the US and not just something special in Atlanta is Atlanta's leading newspaper.

Recent analyses of data on 18,000 teachers in New Yock City has shown that year to year testing results are not as consistent as one would assume.

http://news.firedoglake.com/2012/03/07/data-analysis-of-valu...

> Gary Rubinstein has been analyzing the data at his site, and he’s found some incredible results. First, he found that there’s almost no correlation among teachers year-over-year in the data. A teacher is likely to be judged as effective one year and ineffective the next. The average change in performance was a relatively large 25 points, and it did not fit with commonly accepted beliefs that teachers improve performance over time, particularly between the first and second year.

http://garyrubinstein.teachforus.org/2012/02/26/analyzing-re...

> Looking over the data, I found that 50% of the teachers had a 21 point ‘swing’ one way or the other. There were even teachers who had gone up or down as much as 80 points. The average change was 25 points. I also noticed that 49% of the teachers got lower value-added in 2010 than they did in 2009, contrary to my experience that most teachers improve from year to year.

> With a correlation coefficient of .35, the scatter plot shows that teachers are not consistent from year to year ... nor do a good number of them go up

These sorts of results where the data is available to the public for analysis makes me skeptical that testing results are actually consistent from year to year and that inconsistencies are evidence of cheating.

I contrast these findings to the AJC article's claim that "Experts say student learning doesn’t typically jump backwards." The article does not state which experts they refer to, what their qualifications are, or what exactly they have said. That is suspicious to me, especially since the article has many quotes where they are no so shy about identifying their sources. Shouldn't it be even more critical to identify specific sources when using an appeal to authority argument citing "experts"? Especially when in cases where we have actual data, such as the study I cite above, scores did in fact jump backwards 49% of the time. This difference between empirical data and vague assertions by unnamed experts is suspicious. I must wonder who are these experts.

Mind you, I think there is cheating. I am highly skeptical that the AJC's method has been shown to be a valid way to identify where it is happening.

glesica3• 14 years ago

I didn't look too closely at this, but didn't you state that this was actually reported for schools, not districts? So then it could actually be that a single teacher could cause the "bump", especially in rural areas. I might be misinterpreting you though...

rsanchez1• 14 years ago

When you tie incentives and benefits to test score performance, something's gonna give. You can either institute auditing of schools to guard against cheating, or have less incentives riding on school test scores.

If you think about it, the schools with poor test scores are the schools that need more help, not less. The schools performing well are doing great with what they have, they don't need more. Only if a school consistently performs poorly with more help should some punishment take place.

Cheating our children: Suspicious school test scores across the nation

Related Stories