How Coaches and the NYT 4th Down Bot Compare

http://www.nytimes.com/newsgraphics/2013/11/28/fourth-downs/post.html

by trevin • 13 years ago

50 44 13 years ago

Uhhrrr2• 13 years ago

Hm, so the data points for likelihood of getting another first down is based on data from the fourth downs that coaches actually attempted? It may be that the fourth downs they didn't attempt were not as propitious for some reason - wind (which the authors mention), injuries on key players, lateral positioning which doesn't favor one formation or another, etc.

jhermsmeyer• 13 years ago

Agreed. The sample is very likely tainted by factors that made success much more likely than average if we believe that NFL coaches are both conservative (they forgo punts only when the odds are much in their favor) and rational (they are win maximizing).

Still, there is probably a case to be made that coaches should be a bit more aggressive, but that edge could be small enough that other factors (like perceived incompetence) dominate it.

cheald• 13 years ago

The choice to go for a 4th down conversion is more likely than not primarily decided by the current score and time left in the game. However, the success of a 4th down conversion probably isn't heavily influenced by those factors (other than perhaps player fatigue), which would suggest that coaches are indeed too conservative.

001sky• 13 years ago

is more likely than not primarily

This seems pretty loose. I'm not trying to be pedantic, but the theoretic flaws in the data are such that you would want a pretty tight logic in your model to have any level of comfort with it. The proportion of the state-space that is unobserved seems ~large and critically relevant.

zeroonetwothree• 13 years ago

No, it's based on third down data. The likelihood of converting 3rd and 1 is probably similar to that of converting 4th and 1.

Uhhrrr2• 13 years ago

Thanks! I read poorly the first time.

timfrietas• 13 years ago

This does not take into account an important factor that is somewhat intangible, but real to the psychology of the game: point swings. If you go for it on 4th and 1 at your own 1 and lose you are definitely down 3 points and probably 7. This puts pressure on your offense--which just failed on a 4th and 1--to somehow make it all the way down the field next time. That can be demoralizing for that offense, paralyzing in fact. That is much harder to model, but an important part of a correct model, I believe.

aero1422• 13 years ago

This actually seems measurable. Do teams that are behind perform worse on average? Do teams perform worse than normal after failing a 4th down attempt?

twoodfin3• 13 years ago

The same statistics wonk has done a lot of work on "momentum" in games. His conclusion is that there is no such thing:

http://www.advancednflstats.com/2013/11/momentum-1.html

That jibes with my own anecdotal experience watching a lot of football, despite how often announcers like to cite it as a factor.

Zimahl2• 13 years ago

Why would you ever go for it on 4th and 1 inside your own 10 yard line? You would put yourself at a severe disadvantage if you failed.

Since the data from this is from previous plays in the last 10 or so seasons it is flawed. There probably isn't very little data of a team going for 4th and 1 and that deep in their own territory. The only time this has probably happened is in extraordinary circumstances - specifically in a game-winning drive scenario with time expiring, where the defense is in 'prevent'. In these cases it is highly successful in that situation but you don't have the time to get the score.

thejteam3• 13 years ago

In place of 4th and 1 data AdvancedNFlStats would substitute 3rd and 1 stats. And I would agree that this substitution is suspect. Just a bit shy of bogus in fact.

Zimahl2• 13 years ago

Well, getting a yard in the NFL isn't typically a big problem and 3rd and 1 vs 4th and 1 are feasibly no different. But turning the ball over inside 30 yards is at least 3 points for the other team with only rare exceptions.

vacri2• 13 years ago

I was thinking the same thing about 4th&short in your own red zone - generally the data points you'll get are from desperate late-game plays where the defence is also not playing to their standard playbook.

altoz2• 13 years ago

I think statistically, if you punt from the 9 yard line, the other team is going to get excellent field position (around the 50 yard line). The difference between failing on 4th and 1 from your own 9 and the other team getting the ball around the 50 yard line is 41 yards of field position.

That may sound like a lot, but I'd imagine the actual point difference is on the order of 2 points.

The more interesting thing to consider would be whether taking a safety there is ever a positive.

swang2• 13 years ago

No. Because you cost yourself 2 points and you give the ball to them. Essentially you gave them 2 points at the cost of being able to "punt" the ball at the 35.

chatmasta• 13 years ago

If you turn the ball over before reaching your own 30, you give the other team at least 3 points, because a field goal at that distance is trivial.

This bot also requires both coaches to follow it's recommendations, because otherwise the comparison between what they call "point values" of the two teams becomes meaningless.

thejteam3• 13 years ago

This was discussed some in https://news.ycombinator.com/item?id=6728821

I've thought about it some since then and I'm also not convinced that expected points is a good metric. That will maximize the expected point differential per season but I am not convinced (although I could be wrong... I haven't put pencil to paper to make my thoughts rigorous) that there are enough possessions in the game on average to make this metric useful even in the first half.

gerad2• 13 years ago

The best defense is keeping the other team's offense off the field. Going for it on fourth down has a much higher probability of doing that than a punt does.

kosei2• 13 years ago

Tough to take everything into account, but I find it troubling that all non-success/failure outcomes are not part of the calculation (ex: 8-yard gain on a 4th and 10; a fumble on a 4th and 2 run, etc). It simplifies the formula to the layman, but I have to imagine that it adds enough additional variance to matter, especailly considering we're talking in variances of less than half a point in their stated example.

Also (and to their point), there is a ton of variation between offenses, defenses, punters, running backs, etc. Using this as a rubric is a nice idea for a rule of thumb for an armchair quarterback, but I'd strongly disagree with using it to accurately criticize any decision within a couple points of expected value.

altoz2• 13 years ago

This is especially true of 4th down plays. A surprise play-action pass on 4th and 1 can go for an especially long gain and thus should have higher expected value. There's also the fake-punt scenario, which I suspect tends to be more successful than a regular 4th down.

kosei2• 13 years ago

Absolutely. And I'm sure if coaches started trying to run more 4th down plays, that would dramatically change the algorithm as well.

slg2• 13 years ago

These type of articles never take into consideration the reason why coaches don't go for it as often as a computer might. The coaches face a reverse moral hazard when taking risks because the negative outcomes are multiplied for them. Even if the statistics support their decisions, a coach will be fired for taking too many risks and failing. Is gaining an extra 2% (or whatever) in expected returns worth the risk of losing everything if the low(er) percentage outcome does actually happen? It is the same idea behind the whole "no one gets fired for buying IBM/Microsoft".

Until you get a coach's motivations more in line with the motivations of the team, you won't get a coach that is as aggressive as these computer simulations suggest they should be.

symbaton• 13 years ago

People always say this, but it is actually true? Has any coach ever been fired for missing too many fourth downs?

slg2• 13 years ago

I can't think of one off the top of my head. Mike Smith was under a lot of pressure a little while back for his aggressive decisions on 4th down (that were statistically correct IIRC) but wasn't actually fired.

However, the obvious counter to your question is how many coaches really go for it frequently on fourth downs? Most coaches just follow the conventional wisdom unless they are already in a desperate situation like Ron Riviera or they have built up a lot of good will like Bill Belichick.

judk2• 13 years ago

Not even missing too many. Just missing more visibly, compared to coaches with equivalent records.

spinlock2• 13 years ago

Ask Nick Saban how trying to kick a field goal with 1 second left in the Baylor game is working out for him. There are plenty of people saying that any other coach would be fired.

gerad2• 13 years ago

I think there's selection bias in the underlying data. The dependent variable (realized success on attempted 4th down conversions) is conditioned on 4th down attempts by coaches.

Since coaches are conservative, they're likely to go for fourth down conversions only in cases when they are likely to succeed (due to defensive mismatches, etc).

Hence the model may over-estimate the benefit of going for it on fourth down.

foobarqux• 13 years ago

Moreoever, the probabilities are likely to change if behavior changes. For example fakes or trick plays become less successful if you do them often.

vacri2• 13 years ago

There's an interesting hole in the data at 4 & 10 on the opponent's 36. In a sea of punting and kicking, this one data point has actual coaches going for it.

Also, looking at the overall pattern of the data, my gut tells me that the optimal rubric is somewhere in-between the 'bot and the meatcoach. There are some strange holes as well - at 4th&3, you should go for if you're on your own 8 or 9, but punt if you're on your own 7 or 10?

judk2• 13 years ago

Without confidence levels, the differences are meaningless. Maybe that's the flip from 49% to 51%

thejteam3• 13 years ago

That makes sense. At the 36, it is a 53 yard or so field goal. At the outer range of most kickers and pretty low percentage, so you give the ball to the opponent at the 40 something yardline with high probability. On the other hand, its also close enough to be very difficult for a punter to pin in inside the 5 or 10, so even if you fail you only give the opponent 16 yards over a punt.

But the holes at the your own end you mentioned, that makes no sense at all.

reddog• 13 years ago

Here is a high school coach from Little Rock that never punts, and always onside kicks. His reasoning is sound and it seems to work out for him.

http://www.youtube.com/watch?v=AGDaOJAYHfo

dfan• 13 years ago

It is interesting what an emotional reaction people instinctively have to 4th down. Going for it on 4th down and failing feels like a catastrophe. But punting would have given the ball away anyway, so all you've done is give the opponent an extra 40 yards or so. Which is bad, sure, but it's a hell of a lot less bad than, say, throwing an interception on 1st and 10. But somehow giving up the ball on downs has come to be seen as a massive failure, on par with an regular turnover.

snowwrestler2• 13 years ago

Turning it over on downs is a turnover, as sure as fumbling is. It's not accounted that way in the box score, but in terms of impact on the game it is the same.

An interception is better or worse than turning it over on downs by the field position delta. If it is returned past the line of scrimmage, then it's worse. If it is not returned, it's better. A deep interception with no return is like a punt in terms of impact on the game.

snowwrestler2• 13 years ago

The fundamental flaw with this analysis is that statistics are descriptive, not predictive.

You can't reliably predict the future with statistics alone. If you could, then technical analysis of the stock market would be the most successful investing strategy. It is not.

Likewise, a lot of folks seem to think that taking an historical statistical trend and extending it into the future is a scientific approach to football coaching. It's not.

Scientists use statistics to understand evidence, but they formulate theories in terms of root causes. In other words it's not enough to know that dropping a hammer has a high correlation with acceleration of 10 meters per second squared. If you want to predict how fast other objects in the future will fall, you need a theory that describes why and how anything falls.

It doesn't look to me like the 4th down bot embodies or implements any cohesive theory of football success. It just finds trends and dots the line into the future.

nova22033• 13 years ago

It's the same thing with penalty kicks in soccer. Statistically, you are more likely to score if you kick the ball down the middle. The goalie is going to go one way or another and there's a 50% chance he'll guess right. So why don't players kick it down the middle..because if they kick it down the middle and the goalie manages to block the kick, they'll come off looking like idiots and it would look real bad on the highlight shows.

How Coaches and the NYT 4th Down Bot Compare

Related Stories