Lessons from Moneyball

http://37signals.com/svn/posts/3098-lessons-from-moneyball

by themcgruff • 14 years ago

65 52 14 years ago

funkah• 14 years ago

A/B testing and the like will only guide you to local maxima. Human thought and intuition, ie design, will retain its importance as the impetus for original ideas which can then be optimized through testing. And users will continue to evaluate software as they do today, "with their own eyes and emotions." Sorry, nerds.

asianmack• 14 years ago

Why test when you can wait for 37signals to publish their successful results? Then just copy that.

rmc• 14 years ago

Because your audience might not be the same as 37signal's audience, and hence might react differently to different things

noahnoahnoah3• 14 years ago

Sure, small A/B testing will probably only get you to a local maxima. I've never understood why most people think A/B testing is just about changing the color of a button -- it's that, but it's so much more. It's about testing any idea, no matter how big a change it is.

Most people don't lack original ideas, just like there's no shortage of people who would like to be major league baseball players. What people do lack is the ability (or perhaps just the will) to evaluate their ideas objectively.

And while users may evaluate software with their own eyes and emotions, that behavior manifests itself in objective, quantifiable outcomes that have some impact on your business. If measurable user behavior doesn't impact your business, I'd guess you aren't really in a user-driven business.

patio11• 14 years ago

Suffice it to say that if my clients included somebody into boating they could purchase a more-than-modest yacht with the gains from A/B testing. If it ever happens I'll suggest Local Maxima for the name.

mbyrne• 14 years ago

Even Better: Dharmesh Shah's Startup Lessons From 17 Hard-Hitting Quotes In "Moneyball"

http://onstartups.com/tabid/3339/bid/76799/Startup-Lessons-F...

dshah• 14 years ago

Thanks for linking to my article. Glad to see my friends at 37signals are fans of Moneyball too.

sachinag2• 14 years ago

Billy Beane's shit doesn't work in the playoffs, the book/movie ignored the Big Three, and information advantages go away.

The startup corollaries: small-bore inefficiencies can't overcome luck and strong competition, don't attribute success to A when it's really the obvious B (i.e. what differentiates your product doesn't mean it's actually better - Gowalla and foursquare is my favorite example here), and people will steal your landing pages because they work better.

dionidium• 14 years ago

Billy Beane's shit doesn't work in the playoffs

What is this supposed to mean? The season gives you a 162-game sample. The playoffs are a coin-flip, by comparison. It's just too small of a sample to draw any conclusions, which is why both old- and new-school baseball guys have an endless array of cliches about how anybody who makes it in to the playoffs has a chance to win.

sachinag2• 14 years ago

It's a reference to a famous essay by Nate Silver (you may know him from FiveThirtyEight at the NYTimes) and Dayn Perry: http://sports.espn.go.com/espn/page2/story?page=betweenthenu...

In short, the A's roster was actually constructed to succeed over 162 games but the compromises he had to make on defense and power pitching to get extra value are particularly fraught in the playoffs, where you're battling the better offensive teams in the league. Of course, the playoffs are a crapshoot (the Cardinals had no business being in, let alone winning in 2011), but the compromises Beane had a direct effect on their heartbreaking playoff losses in the early 2000s.

ssharp2• 14 years ago

I agree. Even a 162-game sample isn't enough to get any great level of confidence, but it is certainly better than a 3-5 game sample of the first round or 4-7 of subsequent rounds.

Beane and the A's were mostly focusing on offense, because that's were they could find value. It's significantly harder to find value in pitchers, and pitchers can help damper some of the "luck" factors in the postseason, though it's very much open to randomness.

Modern baseball management is focused on building your team to win enough regular seasons to make the postseason, and then hope luck falls in your direction. If you can couple that with great starting pitching, you can reduce the amount of luck you need.

I'm not sure how relevant the techniques in Moneyball still are. At this point, every team is looking at Sabermetrics, and some of the high-value stuff isn't there any more. For example, getting on base isn't worth as many runs in 2011 as it was in 2002 because there are less home runs hit.

durandal1• 14 years ago

Mandatory XKCD: http://xkcd.com/904/

loceng• 14 years ago

Luckily with software you can earn or raise more money to 'change your team' to then be fit for the playoffs.

prophetjohn3• 14 years ago

This doesn't make any sense. Billy Beane's "shit" either works or it doesn't. No one is claiming that good analytics guarantee success in small samples, but if properly evaluating talent improves chances of success in a 162-game sample, it improves the chances of success in a best of 5 or best of 7 series.

Maybe you're arguing that the amount of luck influencing the outcome of the playoffs is much higher than the amount of luck influencing the regular season; this is obvious. What it's not is a good argument for not using all available tools to put the best talent on the field.

Duff• 14 years ago

Not necessarily. For the season as a whole, you don't really care about individual contests -- you're just looking at beating the averages.

For a playoff, things are different... you're in a match-up against a single team that is better than the average team. I would argue that money and luck plays a much bigger role in the playoffs for a smaller team. If you're playing the Red Sox or Yankees, chances are that all of the key players qualify as "best talent", while your budget forced you to choose your investments more carefully or choose players with less-known ability.

floppydisk• 14 years ago

Actually, the A's did have a streak of playoff appearances 6+ years ago under Beane's system[1].

Reading the book, Moneyball, I came away with the conclusion that the success of Beane's system boiled down the fact that they started measuring everything and making empirical decisions based on data rather than emotion or "gut" checks. In turn, this let them find value where others saw failure. In a competitive environment with limited resources (players Beane's instance) this system will work up to the point where few, if any, try it--meaning lots of overlooked value exists.

One point at the end of the book was that the Red Sox, when they hired Theo Eptstein as their GM, brought in one of Beane's assistants to implement the system the A's were using. As more teams start using it, the quantity of undervalued players decreases because everyone starts valuing the same thing.

Start up takeaway: Measure everything and make decisions based on empirical results instead of "gut" checks. That's really what it boils down to. I'm not just talking about the front-end stuff like landing pages, conversion rates, clicks, etc. Measure it all, response times, database operation times, network transmission time, average request size, whatever you can get your hands on. Having all that data will allow you to make informed decisions and identify places for improvement to increase value.

That's my takeaway.

[1] http://en.wikipedia.org/wiki/Billy_Beane

gerggerg• 14 years ago

The only real lesson in Moneyball is do what you love and don't let others tell you not to.

What I find most interesting about the story is the different effect it has on people. Business types see it as "Take a great risk and it'll pay off (literally)." Science types see it as "Math works".

The reality is, you just have to do what you want to do and not care if other people tell you you're crazy. Was a situation like Moneyball an exception to the rule? Hell yes. But with out those 1 in 1000 exceptions to the rule we'd have very little to show for 50 thousand years on the planet.

andyakb• 14 years ago

There are many many many more time where "doing what you love" is incorrect than times where following math is incorrect. The more we focus on math and mathematical relationships between different variables, the more accurate we get.

Do not listen to detractors IF YOU HAVE EVIDENCE SUPPORTING YOUR CAUSE, but do not blindly pursue something because of an unsupportable "gut" feeling

its_so_on3• 14 years ago

(below may include spoiler)

I thought the real lesson was "massively outperform the limits of your shitty resources beyond anyone's expectations, by thinking outside the box and concentrating on algorithms and data modeling; and still not succeed. Then if someone is impressed and offers you more personal compensation than you can imagine, to do the same thing but for them, while actually being given enough resources to actually have a chance with it, then you go and fucking do it."

The startup analogy is easy: if you are solely responsible for running a free service with a million users on $300/month hosting budget, and you almost do it by inventing amazing algorithms, but the service still lags and sucks, but is almost there, then when Amazon makes you an offer to come and do the same thing for them you fucking take it.

idoh• 14 years ago

At the end of the movie he was offered millions, but he turned it down to stay close to his family. The lesson from the movie (that deals with putting a price on everything) is that some things are priceless.

brandnewlow• 14 years ago

That whole scene there at the end really killed me. It was not made clear to me why the protagonist made the choice he made. I wonder how he feels watching that scene in the film every time he attends a screening.

kaonashi• 14 years ago

This reads disturbingly like a Thomas Friedman editorial.

brudgers2• 14 years ago

The problem with Moneyball as an analogy for web software development is that software development is not a game with a finite pie outcome model, i.e. there isn't a winner for every loser or vice versa.

For web software development there isn't even a clear way to determine winners - e.g. market share or brand awareness may trump revenue as a measure of success depending on the purpose for which web development is undertaken.

If Bean's goal had been to put butts in seats, it is unlikely that his methods would have worked.

akashshah• 14 years ago

The term you are looking for is 'zero-sum game'

InclinedPlane• 14 years ago

The way to get butts in seats is to have compelling games. The way to get compelling games is to have a competitive team. The way to get a competitive team is to find high quality players. I doubt any of this was not in the minds of the team's owners and management.

brudgers2• 14 years ago

Winning often correlates with butts in seats.

However, it is unlikely that a strategy optimizing dollars/win is the same as dollars/season. Merchandising and TV contracts play too big a role (not to mention ticket prices). Stars sell merchandise and seats even for losing teams.

startupfounder• 14 years ago

The lesson of Moneyball is to 1) do what you love/what you are good at and 2) purchase undervalued assets.

This is the same strategy that Warren Buffet has been using for decades.

The perception is high risk because you are going against the status quo, but in reality you are in a low risk senario because you are doing what you love and you see an undervalued asset.

In the startup world it is the same thing. If you are doing what you truly love/what you are good at and you can identify undervalued assets then you will win in the long term.

For employees this is finding a company that you can calculate that it will succeed, then getting stock in that company by "taking the risk" and it all works because it is something you deeply care about.

For founders this is finding an area that is ripe for disruption with a problem that you face in your life. Starting a company is a way to realize that value that other's are not capturing.

Really successful companies, founders, employees and investors do what they love, love the ones around them, pay respect to everyone and can brutally identify undervalued assets.

akkartik• 14 years ago

"The perception is high risk because you are going against the status quo, but in reality you are in a low risk senario because you are doing what you love and you see an undervalued asset."

No, it is high-risk. The risk is that the undervalued asset you see may not really be one.

But staying with the status quo is high-risk as well.

Risk is a red herring; people optimize for regret-minimization. There's less regret if you go with the crowd because there's other people to blame -- and regrets are less apparent because everyone suffers together. But if you go against the status quo and your life sucks, you have nobody else to blame, no fellow sufferers.

This isn't a rational thought process. But you have to see the enemy clearly before it can be defeated.

anamax• 14 years ago

> The lesson of Moneyball is to 1) do what you love/what you are good at and 2) purchase undervalued assets.

> This is the same strategy that Warren Buffet has been using for decades.

Buffet has been using govt and others to produce undervalued assets that he then buys. (Did your CitiGroup investment have a govt guarantee?)

He also pushes tax policy that directly benefits him. (Hint - the "millionaire's tax" that he's pushing doesn't apply to him. Same with the estate tax - virtually none of his estate will be taxed. He sells insurance to other folks who will have to payit.)

Guess who benefits from delaying/killing Keystone XL? (Buffet's railroads aren't nearly as efficient as pipelines.)

pron5• 14 years ago

I realize this comment may not be popular, but I gotta ask this. How come every burp that comes out of 37signals lands on the front page of HN?

(Conformist... Yeah... http://www.youtube.com/watch?v=V4CggHUypjY)

EDIT: I guess I'm just allergic to cults.

billpatrianakos2• 14 years ago

At least your comment is popular with me. The reason for this is most likely because at some point in time 37Signals proved themselves to have very interesting, important advice to give. We could all learn a lot from them. Once an entity establishes themselves as a credible source like 37Signals has there's this trend where people stop thinking critically about what they're currently saying and just assume everything they say and do is top quality because they've already established themselves.

This is true of any popular figure/entity that has earned some authority in an area. PG, for example, could bust in here and tell us the world is flat and a decent amount of people will support that statement (nothing against him, just a hypothetical example). Now, I really really respect 37Signals, I adored Rework, and a lot of their posts are really great but lately I've started to notice the same thing as you, pron. Not every post of theirs is a gem worthy f this front page and I've started to notice a smugness in them that I don't really like but I'm just one guy who has not earned the same respect and authority as them so I suppose I don't have much room to talk here.

noahnoahnoah3• 14 years ago

(I'm the author of the original post, and obviously I work at 37signals, so feel free to take what I say with a grain of salt.)

Why do posts make it to the front page? Because enough people upvote them, so there must be something that appeals to at least some people. If you have comments on how I could have made this post more interesting or relevant to you, or improvements in my writing style, I'd certainly appreciate hearing them. You can reach me at noah@37signals.com.

billpatrianakos2• 14 years ago

Hey Noah, I can't speak for the OP but I will speak for myself here as I share a similar sentiment. You can't please everyone all the time and I wouldn't expect anyone to start taking advice on how to make a post more relevant to me personally. I would simply find another blog that fits my interests. There are plenty. You know the audience and you did a good job of engaging them. Your writing style is great too. You totally captured the 37Signals voice while still making it your own.

Now, what I was talking about is a pattern I see where posts from particular sources end up here very frequently. When I see that happen I start to wonder if they're here because they truly are great articles or just because they came from a particular source with a lot of clout. There's nothing wrong with this in and of itself, it just makes me wonder if we're starting to give particular sources a free pass, allowing them to ride the wave of their previous successes without really evaluating if the latest article truly is worthy of so much recognition. For me, the test would be to put the same article on an unknown blog and see if people still feel the same way about it.

Please know, that there's nothing about the content or style of your post that I object to at all. My mind just goes off in that direction after seeing any source on the front page very frequently in a short period of time. And you're right that this on the front page because the majority found it appealing. We have a great community here and I trust them to choose what's good. I want to thank you for posting this comment here because it forced me to be honest with myself and ask, "what made me react so negatively/too critically to this post being here?" and while the reason I mentioned earlier is still true I have to admit that there are two less than desirable traits or attitudes that I let slip out.

The first was plain old jealousy. The "I wish I was successful and cool like 37Signals" type jealousy. The second was negativity for negativity's sake. It was like I just had to go against the grain here for the sake of going against the grain. I'm also in a bad mood today which doesn't help. I think I have to admit I was partly just being petty and that's not cool. Rather than delete my previous comment I'll just man up and admit I was letting my inner asshole/troll slip out and maybe others will be able to learn from me and catch themselves when theirs slip out. Anyway, it wasn't a bad post at all and I'm surprised you even took the time to respond to this unwarranted negativity.

pron5• 14 years ago

I'm sorry if I have offended you, that was not my intention. There is nothing terribly wrong with the post. It's fine. That is not the point. My point is that there seems to be a certain cult following to several persons/companies in the industry among HN readers, which doesn't entirely surprise me, but does disappoint me. I think this community should be more free-thinking and not drink anyone's KoolAid. In fact, I'd like to see posts coming from from certain figures given a much more critical treatment than a reverent one. HN is full of useful, insightful critique, but there could be more of it, and we can do away with the unnecessary reverence.

harpastum• 14 years ago

I appreciate your sentiment, and I think 37signals does get more attention than they deserve on many occasions. However, the superior tone of your comment really rubs me the wrong way. Calling people who find value in something you don't doesn't immediately make them 'cultists'.

ojbyrne• 14 years ago

For me, the lessons would be:

1. Always question the conventional wisdom. That's what needs to be disrupted.

2. Every great innovation that makes someone famous or rich always involves some guy who's been plugging away for years on his own without any notoriety or wealth (In this case it was Bill James, who invented "sabermetrics" while working as a night watchman).

tristan_juricek• 14 years ago

If you haven't read "Thinking, Fast and Slow" it really summarizes the problems in a lot of decision making in a very clear way.

I'm still reading through, but, you have two major actors in your brain. A "fast" side, and a "slow" side. The fast side makes gut decisions, and the slow side, well, applies more complex judgements and can include a more unbiased view of the world. There's a ton to this process, but importantly, the fast side is very happy to jump to conclusions when presented with missing information.

The lesson of Moneyball, then, was that the "fast side" thinking clouded judgements of baseball prospects by filling out ridiculous statements like "he's got a hot girlfriend, must be able to have the confidence we need on the field".

I'm not sure how you can use the baseball metaphor in terms of software design, though.

But, you could easily say that we're presented with an incomplete amount of information when building software. (What's going to happen when it runs? Who knows! Just build a nice API and continue.) So we definitely all probably have a lot of "quick learning" techniques and processes we just adopt to. For example, Googling before you just stop and think. Or using the UI techniques of other applications, because "that's how the big profit-making companies do it".

sambeau• 14 years ago

Where I find Moneyball (and a lot of A/B testing) falls down is in the complexity around the player (or mechanism).

Teams often contain players whose value is off-pitch or subtle: leaders, morale-boosters, supporters.

The system that Moneyball demonstrates would struggle to find the a manager that would uses a Moneyball-type system. Where are the stats for the team-picker?

While I'm definitely not saying that there is magic in sports (nor any successful website that ignores A/B testing) I am saying that sometimes a talented spotter (or a talented designer) can produce a fit that a spreadsheet cannot.

(yet.)

Lessons from Moneyball

Related Stories