Lead Bullets (2011)

http://bhorowitz.com/2011/10/26/lead-bullets/

by zbravo • 13 years ago

123 46 13 years ago

norswap• 13 years ago

Perhaps I am not getting the point, but it seemed that the silver bullet was to fix the performance issue.

kcorbitt• 13 years ago

I think the connotation of "silver bullet" in this post is an extraordinary idea that solves the problem definitively by changing the rules. A lead bullet, by contrast, acknowledges the current field of play and just plays it better. Less glamorous, but still plenty deadly.

mturmon• 13 years ago

These elemental analogies are reminding me of the Manhattan project.

One silver bullet (E = m c^2 and Einstein's letter to Roosevelt) and some lead bullets (the long slog of building huge industrial plants to purify the uranium). Maybe sometimes you need both.

narag• 13 years ago

The alledged siver bullets, as I understood it, where the set of partnerships and acquisitions that would broaden the product line and surround the web server with enough functionality that we would be able survive the attack.

I wonder if this story is related:

http://www.jwz.org/doc/groupware.html

saalweachter• 13 years ago

The point was that there was no easy fix to the performance issue (a silver bullet). They just had to buckle down and apply a myriad of optimizations, innovations, and hacks to slowly, incrementally get the performance where it needed to be (lots of lead bullets).

obviouslygreen• 13 years ago

This seems like a couple of nice anecdotes thrown together due to an unfortunate preoccupation with a barely-related cliche.

The way I'm reading it, the point is to concentrate on your product and making it, in reality, better than the product(s) of your competitor(s) rather than... pivoting? Building a different product? Making your product into something else? Changing your target audience? I guess the only real lesson I'm taking away here is that "if you have product, sell it; if it sucks, make it better." Not exactly groundbreaking, though certainly true.

If the desire was to communicate something more general, taking it out of the context of existing product-based businesses with market share and competitors would probably be necessary (but would have probably killed the anecdotes... which is why I'm guessing the article didn't really get off the ground in this regard).

So yeah, it's mostly true... but where it is, it's sort of trivially so.

tomblomfield• 13 years ago

As you suggest, I think the point was that there's often no brilliant idea that will provide a magical solution to a hard business problem - be that a product pivot or a new sales/marketing gimic.

Instead, you often know what you need to do - it's a long, hard slog, and it's about sticking at it longer than your competitors.

It's a recurring theme in Ben's blog:

>> Mediocre CEOs point to their brilliant strategic moves or their intuitive business sense or a variety of other self-congratulatory explanations. The great CEOs tend to be remarkably consistent in their answers. They all say: “I didn’t quit.”

http://bhorowitz.com/2011/04/01/what%E2%80%99s-the-most-diff...

aresant• 13 years ago

Reminds me of one of my favorite quotes from Admiral Stockdale:

"You must never confuse faith that you will prevail in the end, which you can never afford to lose, with the discipline to confront the most brutal facts of your current reality, whatever they might be."

saurik• 13 years ago

wilfra• 13 years ago

Sounds like the Microsoft strategy (Bing, Windows Phone etc) - and now what G+ is trying to do to Facebook.

What are some examples of startups doing it (or not doing it)?

SatvikBeri2• 13 years ago

Perhaps duckduckgo?

orangethirty3• 13 years ago

Not to take a cheap shot at ddg ( user), but they are a more consumer focused company. Google on the other hand is more b2b.

lifeisstillgood• 13 years ago

  The customers were buying; 
  they just weren’t buying our product.
  This was not a time to pivot.

Carve this into the tabletops in every Starbucks in San Francisco

malkia• 13 years ago

silver bullets are actually much less efficient than lead (vampires or not)

RandallBrown• 13 years ago

but if it's a werewolf, it's the only way to kill them.

aaronbrethorst• 13 years ago

Too many people are unwilling to do the hard thing. Probably because their "goal [is] business for the sake of business (http://al3x.net/2013/05/23/letter-to-a-young-programmer.html)

coffeemug• 13 years ago

That's a great post.

There is also the opposite problem (one much more common in newbie founders IMO) -- seing lead bullets everywhere when a silver bullet would do. I used to believe I can just outthink and outwork other people by sheer force of will. I still believe I can, but I would have saved myself (and my team) a lot of effort if I learned to look for silver bullets when they fit the problem.

Telling lead from silver requires some learnin'.

cl8ton• 13 years ago

Netscape towards the end was pitching to business and not consumer. Consumers like touchy-feely-hip things (Silver Bullets), business wants things that work. Price, Performance and Support are all that counts at the end of the day. (Lead Bullets)

artsrc• 13 years ago

Our business spent a fortune on a very poor performing, piece of junk, software. Maybe the support is OK, but the real world stability sucks too.

Sometimes hookers, expensive lunches, tickets to sporting events and being golf buddies with managers are what counts. And if those are what counts, then those are the lead bullets.

TheBiv3• 13 years ago

This was a lessons-learned post, but I would LOVE to read a "How we made those lead bullets" post.

Because his post goes like "guy says lets use lead bullets." -> "We hunker down and use lead bullets" -> magic -> "Boom. $1.6 billion company"

marcosdumay• 13 years ago

> "How we made those lead bullets"

Optimize something, reduce the running time by a few ms, optimize something else, reduce by some more ms, repeat 8 hours a day, with several people in paralel for a few months.

The concept of a "lead bullet" implies that the history won't be interesting to read, just some repetitive boring hard work.

TheBiv3• 13 years ago

> "Optimize something"

I may be the only one, but knowing what the 'something' was is incredibly valuable bc we all, as people, pick 'something' and not everything in any sort of bullet is created equal.

leftnode• 13 years ago

I agree completely. The geek in me wants to know what they executed on. If I'm an entrepreneur facing the same situation, I'd like to read exactly what others did. Even if it can't help me directly, it gives me some motivation that if others did it, I can do it as well.

Even if it was boring or rote work, I'd love to know what it was.

TheBiv3• 13 years ago

I feel the exact same way! The geek in me would like to know what they optimized and what they brute forced!

hobs• 13 years ago

Knowing exactly how they did it probably wouldn't even work for someone else. And that is not the point, instead of facing their problems head on they were avoiding them, and that almost destroyed them. It's almost a parable.

graycat5• 13 years ago

I see some lessons in the OP, but not the ones Ben intended.

Ben is saying that at times need to work very hard? Okay.

Ben is saying that if a competitor has a much better product that is selling well and threatening own business, then likely just must do the work needed for a competitive product. Okay.

But after that I lose it with Ben: His "lead bullets" are defeatist, close to luddite, look like he is trying to degrade himself and his company, to fight by getting down, dirty, simplistic, nose to the grindstone, shoulder to the wheel, and ear to the ground and trying to be successful in that position, to suffer first and count on that being the path to success. Not good. Indeed, one of the things we're supposed to be doing is innovating, which likely also means being creative, original, maybe even mathematical or scientific.

Ben reminds me of Edison sending yet another team to the Amazon jungle to look for more plants that he could bake into carbon and try for a light bulb filament that wouldn't burn out so soon. The real solution was to borrow the idea of a tungsten filament from the chemist Swan in England and to use a really good vacuum pump from a guy in Germany. Edison could have had his teams slogging through jungles for 1000 years without finding anything. But, eventually Edison did go with tungsten and the German vacuum pump.

For Windows' first version of IIS being as much as five times faster, tough to believe that just a 'lead bullet' approach would yield what was missing from what Ben had. Instead, look at the architecture and think a little about what the heck the poor computer is being asked to do. Then look at where the computer resources are going. Likely then, tweak the architecture: So, if are spending time walking down arrays or linked lists in O(n), then consider using an AVL tree in O( ln(n) ). If executing the code for a page when know that the output will be the same as the last 10 times, then cache the output and don't run the code again. If caching the output in main memory, i.e., virtual memory with page faults, then consider just using disk file for each page to be cached and save on page faults. If are connecting to a database, then have a pool of connections. Etc.

I.e., think a little. Looks like Ben doesn't like thinking, more likely has, really, a low regard for his people and doesn't trust them to be productive if asked to think. Ben wants to measure hard work by sweat, long hours, and suffering and not good work with good results.

Here's a lesson I draw: I agree with Ben that a business that has been going along well might suddenly encounter some problems, e.g., a five times faster competitive product given away for free.

First, such problems should have been expected. If Ben had inefficient software, then he should have known that and not had to learn it from a Microsoft product. Or, how and why did the Microsoft people do a much better job? Sure: They wanted something fast, at least not wildly inefficient, thought some about the architecture, and did a decently efficient implementation. Why? Because they cared enough. Then before Microsoft's work, Ben hadn't cared enough.

Second, Ben should have done much better trying for, expecting, counting on, working toward silver bullets. I sense that Ben doesn't much believe in silver bullets, suspects that any efforts toward silver bullets will be just time and effort wasted. Bluntly, I seriously doubt if Ben knows how to direct a productive, creative, effective little applied research project. Ben reminds me of the remark in the movie about the auto guy Tucker "A well run company doesn't waste time on research". Yes, some research is close to intellectual self-abuse, but it's the job of a good CEO and a good Board to know that this is not always true (e.g., SR-71, 22 nm line widths) and how to make applied research productive, not just throw out the baby and drink the bathwater.

Third, any entrepreneur with a good chance of doing well has to ask himself at least 10^10th times if he wants anyone like Ben on his Board of Directors. Why? Sure: The CEO and his team have seen an area where they might do better. With the CTO, the CEO outlines an applied research project to get the coveted better results. So, for the next Board meeting the CEO includes some foils on the applied research project. Then in the Board meeting Ben wakes up and starts asking questions: How long will the project take? How much will it cost? How good will the results be? What will be the milestones during the project? What is the project schedule, step by step? Or if an experienced auto mechanic can do a brake job on a Ford F150 truck in the time in the book, why not this 'research' project?

So, the CEO has to tell Ben and the Board that for all of the questions, at present he is comfortable with the situation, will continue to review the situation during the work, but otherwise really has no answers at all for the more specific questions. Presto: Ben gets torqued, brings out his fiduciary powers, fires the CEO, before his stock is vested, and 'gets the company back on a business like, bottom line basis' and just blows it.

Even worse, maybe the applied research has already been successful, and at the Board meeting the CEO introduces the CTO who explains the work and, then, the next step, converting the applied research to production software. Again, just over the software work, Ben does an upchuck.

I just don't see Ben being able to work effectively with things that are new, powerful, valuable, innovative, etc. NSF, NIH, and DARPA problem sponsors can. Ph.D. supervisors can. Successful Ph.D. students can. Research professors can. But I don't see Ben as able.

SatvikBeri2• 13 years ago

I just don't see Ben being able to work effectively with things that are new, powerful, valuable, innovative, etc.

As long as you don't consider app.net, Skype, Meteor, MixPanel, lyft, ifttt, FourSquare, FaceBook, github, factual, crowdtilt, cohere, bump, box, or airbnb to be new, valuable, or innovative[0].

[0] http://a16z.com/portfolio/portfolio-venture-growth/

graycat5• 13 years ago

Typically when A16Z funds a company, the applied research and software development have already been done. And the evaluation of the company by a large venture firm is usually based on 'traction', not core, technical internals that might be new, innovative, with silver bullets, etc.

The question relevant to the OP is what happens after the company is going and suddenly encounters a product from a competitor five times faster and for free. The present A16Z portfolio does not answer that question.

Moreover, what you are pointing to as 'innovative' companies in their portfolio is not much like the 'innovation' of a "silver bullet". That is, the 'innovation' in your list from their portfolio is, on the face of it, just innovation in the business 'idea' or business model and not core, internal, technical innovation such as needed to make some software five times faster as in the OP. E.g., you mentioned AirBnb. Where is the core, technical silver bullet in that business? Maybe they have some such bullets in their server farm now, but when they got venture capital likely they had recently still been thinking air mattresses. Similarly for Facebook: Now they have one heck of a server farm with, maybe some silver bullets, but early on they were something from Zuck's dorm room typing with no bullets visible, silver or lead.

I've seen some silver bullets and created several in my career and simply don't see anything in Ben's background or his many blog posts that suggests he knows how to invent or direct the invention of silver bullets; indeed, what I've seen is that, from his thinking, in any company where he was on the Board, likely he would ruin any efforts he saw to create silver bullets. Ben's technical background is just routine; otherwise, he was a 'business manager' who pushed others to work with "lead bullets".

Learn a lesson: (A) No way, not a chance, will any usual, US, red-blooded, no-nonsense, bottom line oriented business manager want anything to do with any subordinate doing anything the manger would have to think about longer than 10 seconds to understand. They still want it like the Ford factory 100 years ago where the supervisor had all the knowledge and the subordinate was there just to add routine muscle to the knowledge of the supervisor. Such a manager does not want to bet even 1% of his career on any work he does not personally understand. Period. (B) In information technology, venture capital Board members are much like such managers -- they want nothing to do with pursuing silver bullets. (C) Actually, the fraction of the population at all good at creating technical silver bullets is quite small, necessarily so since in this context 'silver' means both valuable and rare. A problem is that if the company, once it's up and going, needs a silver bullet, a Board that doesn't understand how silver bullets are made won't back a silver bullet effort, even if the CEO sees his way clear to success, and will insist on lead bullet approaches.

Again, I just don't see Ben as a silver bullet kind of guy.

Net, the silver bullet business is fairly well understood, but nearly all the venture partners are among the worst at working with that business. Again, good work on silver bullets includes the SR-71, 22 nm line width, and most of the funded projects of NSF, NIH, and DARPA.

In particular, the crucial, core technology of my startup is some silver bullets I created in some original applied math based on some advanced prerequisites in pure and applied math. I've converted that math to software, but to evaluate the real promise of the company just must trace the math. A16H has already written me that there is no way they can consider the original applied math. Okay. But once my company is up and running and needed a silver bullet, and I already know one place it will, if an A16H partner were on my Board then I can see it right in front of me like a freight train 50 feet away at 80 MPH coming right toward me, as soon as I tell the Board about the little project for the next silver bullet, they will in unison upchuck on the Board table and fire me. No way. Not a chance do I want any such person on my Board. I'll let those guys stay busy with another local, mobile, social, photo-sharing app, like Instagram except for video clips or some such.

I'm not used to working with people who are afraid to work with silver bullets. Instead I've been working with silver bullets right along in all my career in applied math and computing for problems in US national security and business, in my Ph.D. work, in likely the world's best computer science lab, etc. Again, from my experience, Ben doesn't look like a silver bullet kind of guy. Neither does A16Z.

jongraehl• 13 years ago

I want to agree with you (everything you say is good), except that there's no evidence that Ben is actually anti-silver-bullet. It seems like he focused efforts on improving the company's own product/tech in-house instead of looking for an alternative to doing that, because the alternative ideas were low-quality. Presumably lots of thinking was involved on all sides.

graycat5• 13 years ago

Maybe. But an entrepreneur who has a history of being successful with silver bullets and wants their advantages in his start up and contacts A16Z needs to know if an A16Z guy on his Board will be able to work effectively with silver bullets.

For Ben, his background doesn't look very technical, i.e., like someone able to draw from the research library, add some new ideas, and come up with a silver bullet in, what, two months, two weeks, over a weekend, in the shower after a 2 mile run? Next, in the Bayesian sense, there is a high 'prior' probability that a guy with Ben's background would resent technical work under him that he didn't understand.

An entrepreneur needs to know, to be quite sure he has a Board able to work effectively with silver bullets. I know many such people although only a few in business, and all of those people had good advanced technical backgrounds. Net, finding such a person in a VC firm looks tough.

I will add a point from my project: Back there many moons ago, I wanted some info from the Internet and couldn't get it. So, I put my feet up, popped open a cold can of Diet Pepsi, reviewed the problem and some of my best graduate school math, and had some new ideas. For both mathematical reasons and just simple judgment, the technical ideas look at least comparatively good as the crucial, core technology needed for a business to let users get more info from the Internet. Then I noticed that my ideas solved much more in getting info from the Internet than my original problem; such a situation is common because commonly a solution does not use every detail of the original practical problem and, thus, also applies to other problems that share the details that were used. It's like someone cooked up the Pythagorean theorem to help survey farm land in Egypt after the flood waters receded but use nothing about farm land. So, the Pythagorean theorem also works for carpentry, stone cutting, navigation, astronomy, and with some generalizations, much of multivariate statistics, Fourier theory, Hilbert space theory, and, thus, quantum mechanics -- my ideas are not nearly that powerful, just have utility nicely beyond my original question.

So, I wrote the corresponding, core software. Then I needed to put a Web site in front of my core software. Then, given my thin checkbook, reality came between me and my keyboard and typing, and I got delayed. In addition, I ran into computer security problems with Windows XP SP3, some Windows system administration problems with SQL Server, etc. -- more delays.

So, I'm a pre-revenue project that needs just a few more Web pages, one moderately involved and the rest dirt simple, some initial data, and some routine things like a domain name to go live. For where my techniques for getting info from the Internet work, if Internet users, in the US and around the world, like my work, judging from the latest version of Mary Meeker's (KPCB) report, then the project will be able to grow to be a big thing. Right: For a large fraction of what's on the Internet, a much better way to get that info for the US and around the world, 2+ billion people, so, sure, $400 billion. My project didn't start out with any such ambitions; I just wanted something fairly routine, but the usual Web sites were not much help.

It does look like even with my thin checkbook I can (continue to) bootstrap this thing, still, if only for a source of what aviation calls "reserve fuel", I contact venture firms.

I've learned that apparently most of the firms have agreements with their limited partners that they will fund only projects with good 'traction', that is, usage and/or revenue significantly high and growing rapidly. And, then, the venture firms will 'play with the Web site', look at the user interface and user experience, and try to estimate how 2+ billion Internet users, desktops to smart phones, would like the site. Okay. Fine.

But the key to the work was some silver bullets. That is, the market need is easy enough to see; the issue is how the heck to get data and write software to manipulate that data and get the results for the users. From efforts from others, clearly cooking up how to do that ain't easy. For that, my answer is some original applied math -- then the core software just implements the math.

Well, not a single venture firm has shown any interest in the crucial, core silver bullets. A16Z essentially deliberately laughed at any possible role for applied math (NSF and DARPA don't). Okay.

Fine with me to continue on with my thin checkbook, go live, and get some traction. Shouldn't be much more work for me assuming that people like the site at all (even if my math works as well in practice as I hope, still people might not like my site).

But, then, I come to the idea of what might come after a venture check? Okay, there will be a Board. If A16Z were to write the check, then Ben or one of his colleagues would be on the Board. Setting aside A16Z, there would be one or more venture partners on the Board.

Then once the initial crunch after going live dies down, and if the usage is growing quickly I've hired a COO to help address that, it will be time for another silver bullet, in this case, for some quite different approaches to ad targeting. I want especially effective ad targeting but only from my own data and with next to no concerns about invasion of user privacy. I have some ideas how to proceed, but they are silver bullet ideas. If I put up the applied math in a Board meeting, could count with shoes on all the venture partners in the US with educational backgrounds to understand.

So I will be asking my Board to back a silver bullet project for ad targeting. And there's a lead bullet approach -- mostly just use ad networks with their pros and cons. And I can see Congress getting all concerned about privacy and severely constraining what ad networks can do. Tilt; my revenue could be at risk.

Sure, some VCs emphasize how much freedom they give their CEOs. Fred Wilson and Brad Feld do. But a lot of VCs also emphasize how they are 'hands on'. Either way, the Board can fire the CEO, before the stock is vested, and that's me. So, net, I want a Board that can work effectively with projects to generate silver bullets (or to use Mother Goose, golden eggs). For more in Mother Goose, the story of Little Red Hen is too close for comfort.

So, how do I know if a VC can work with silver bullets? Sure, contact them and see what questions they ask and how they react to my answers. Do they ask good questions? Or if they can't evaluate the silver bullets I've already got in production quality software, what the heck are they going to do with a silver bullet ad targeting project that starts from much less? More generally, what would they do with other silver bullet projects?

The US DoD got the scram jet working, after only a few failures. Good for the US DoD. They've been nicely successful with silver bullet projects since early in WWII -- drag free satellites, GPS, etc. So, it is possible to work effectively with silver bullet projects. There are lots of people who know how, and I've worked with some of them in my career in DoD work, in grad school, and in computer science research. And if I have to have a Board, then I want such people on my Board. When I look at VCs, I don't see that they have such backgrounds. I'm concerned.

Sharlin• 13 years ago

Instead, look at the architecture and think a little about what the heck the poor computer is being asked to do. Then look at where the computer resources are going. Likely then, tweak the architecture: So, if are spending time walking down arrays or linked lists in O(n), then consider using an AVL tree in O( ln(n) ). If executing the code for a page when know that the output will be the same as the last 10 times, then cache the output and don't run the code again. If caching the output in main memory, i.e., virtual memory with page faults, then consider just using disk file for each page to be cached and save on page faults. If are connecting to a database, then have a pool of connections. Etc.

I read the original article pretty differently. As far as I can see, these all are exactly the kind of lead bullets the article meant. No miracles, nothing that doesn't directly involve making your product better, just some good old-fashioned programming work. The author's point was that if you have an inferior product, you roll up your sleeves and make the product better.

graycat5• 13 years ago

Yes, the AVL trees are like you describe, but the caching may not be because it's not clear to me (and I'm writing for ASP.NET and IIS although have yet to look into caching) if knowing when could cache needs just a lead bullet or a silver one. For more, I'm not deep enough into the internals of ASP.NET, the Windows 'global assembly cache', just how SQL Server connection pools work, etc. to know where the silver bullets would be in such work. So, my examples of bullets are too close to lead bullets unless the price of silver has declined a lot!

mathattack• 13 years ago

With all due respect, I think his message is something different. It's, "Are we looking for a new strategy (silver bullets) or do we need to execute our current strategy better (lead bullets)."

Many times people are all over the place trying to over-pivot, when what they really need to do is focus at the job at hand. If execution is lacking, a new strategy won't help.

At a prior firm, a few of the sales people would come up with new flavors of the month. "We need to Open Source. That's why we lost." "We need to be SaaS. That's why we lost." "Not enough functionality." "Too expensive." The reality is there were buyers in our market, we just weren't winning.

This isn't to say there is never a time for a new strategy - pivoting is vital. It's just to do it at the right time, and have the guts to stick with execution when it's needed. Of course knowing when that right time is doesn't come easy.

graycat5• 13 years ago

I took the silver bullets as mostly being just technical, e.g., for how to get the Web server software at least five times faster. But, yes, some of the 'options' considered did not want to attack the software performance so directly and wanted to do something like 'pivot'. Maybe an idea for a pivot could be a silver bullet -- maybe. That is, I didn't see pivots, strategy, or routine business execution or building a fire under the sales and service groups, although these can be crucial, as candidates for silver bullets.

On "guts", for a lot of entrepreneurs, they know as they start that if they fail it will cost them their savings, house, car, credit rating, likely damage their career, and maybe hurt their marriage and health. So such a person can't be short on "guts". And unexpected disasters, such as Microsoft's five times faster IIS, can pop up needing guts.

Still, one major source of needing guts is vast projects with half-vast planning. Or just poor projects. Hopefully with good projects with good planning, with some silver bullets in the crucial, core technology, the need for guts and "to stick with execution when it's needed" will be much lower than usually assumed. On "guts", entrepreneurs short on guts would not even have started.

CurtMonash• 13 years ago

I think I can summarize that post in three words:

"Thou shalt execute".

rubinelli• 13 years ago

Here is another lesson buried in the post: in most cases, faster is better than better. It doesn't matter if you have a much more stable product, with hot swaps, integrated monitoring, auto-balancing, the whole works. If your competitor is five times faster, you are toast.

If you look at the past decade and compare the technologies that "won", you will see that in almost every case, they were faster than the alternatives, and had performance as a major selling point. The notable exception is Ruby on Rails, but it was sold so well (the original Rails screencast was ground-breaking) and it solved such a huge pain point (XML hell) that, in this case, it didn't matter (though even they had to answer the question "but does it scale?" at some point).

6d0debc071• 13 years ago

Let's be realistic here - worst case that you can do anything about: it's possible the other side just hopelessly outmatches you in some area, and likely always will (they've better programmers, they're allowed to use more powerful software, they've better connections, they have enough money to put you out of business by undercutting) - and 'maybe you don't deserve to exist' is not a good answer to 'perhaps we should focus somewhere where we've a competitive advantage.' Maybe you'd be really good at something else, or maybe not - but you've the resources there and you may as well make a go of it if you look to be totally screwed otherwise.

You need to have some realistic assessment of how badly you're losing. Gungho "Yay effort!" is not a worthwhile risk analysis.

mikescoffield• 13 years ago

This reminds me of a product features framework I saw once. It mapped features to a 2x2, which categorized features into 4 buckets. The lead bullets were considered "table stakes" (core features that were necessary and everyone had) and silver bullets are probably "fool's gold" (features that are distinctive, but don't necessarily drive adoption).

Lead Bullets (2011)

Related Stories