Why I, Jeff Bezos, Keep Spending Billions On Amazon R&D

http://www.sfgate.com/cgi-bin/article.cgi?f=/g/a/2011/04/27/businessinsider-why-i-jeff-bezos-keep-spending-billions-on-amazon-rd-2011-4.DTL

by rrrgggrrr • 15 years ago

326 124 15 years ago

sfgate.com

revorad• 15 years ago

PR damage control?

matthavener• 15 years ago

I think so, but not for the reddit/EBS outage.

mfringel• 15 years ago

Exactly. This is all about their earnings. Given a general trend towards decreasing or outsourcing R&D, Amazon is still doing it in-house, because it's their competitive advantage. Explaining that to a bunch of panicky stockholders who just saw a Big Ugly show up on the cashflow statement is mandatory, no matter how much they care about the long-term.

protomyth• 15 years ago

Interestingly, the number of sites affected might have been a revelation to Wall Street on the importance of Amazon cloud services.

prpon• 15 years ago

That letter is so geeky, I doubt if 90% of amazon investors would get the things he was explaining. As a geek though, I love it.

michaelochurch• 15 years ago

My feeling about it was that it was written to appeal to top programmers, not investors except so far as "You can read this letter, in which I prove that I know the language one needs to speak to hire smart people".

athst2• 15 years ago

Good point - maybe this is just the greatest recruiting tactic ever.

jordanb2• 15 years ago

Judging from things Steve Yegge has written about Amazon, and from friends of mine who have worked there, I don't get the impression that Amazon is a company dedicated to technical excellence.

In particular I seem to recall something about 40 million lines of C++ code in their core product, and about mediocre engineering talent and high levels of burnout due to the technical debt they're required to deal with...

In fact, ever-increasing engineering costs would be just as indicative of mounting technical debt as it would of an increasing commitment to R&D.

petercooper• 15 years ago

In my experience, their tech job listings list tech degrees as a must-have. That seems to be a common attribute of companies that care more about HR box ticking than technical excellence, alas.

mikle• 15 years ago

More than that, it is very easy for only one programmer or researcher doing a bad job to create technical debt that will take disproportional time to pay off.

daeken2• 15 years ago

I thought about going to Amazon when I decided to get out of the startup world about 6 months ago. I went through the standard application process, and promptly closed the tab when I found that they would only accept my r�sum� in MS Word format -- rejected PDFs, HTML, and even plaintext. It may seem like a silly reason not to work somewhere, but I think that the hiring process says more about a company than just about anything else.

mlinsey• 15 years ago

I was hired with no college degree, when Microsoft and Google would not even consider me on that basis.

Also, for Amazon and other companies, there is a world of difference going through their public-facing recruiting process and getting to know someone on an interesting team through your network, and getting hired through them.

petrilli• 15 years ago

I get the feeling that you're both right and wrong. Right in that a lot of the legacy Amazon code is probably burdened with huge technical debt. I also get the feeling they're pushing the boundaries in other areas. A bit of the "chase the shiny coin" syndrome.

One of the issues is that none of us really know what goes on inside Amazon. They are notoriously secretive about most everything.

billswift• 15 years ago

"Secrecy is the beginning of stupidity." It both reduces feedback, often dramatically, and wastes your time and energy in maintaining the secrecy.

chrismealy• 15 years ago

Yegge left a long time ago.

pkteison3• 15 years ago

I've seen some negative stories about working at Amazon. I don't recall seeing anybody write that it's a great place to program. Anybody have any stories or links about the upside?

jmelloy• 15 years ago

I know a lot of people who work/have worked there, and it seems to be a very love it or hate it company. They work people hard, but the people who like that kind of environment thrive. They seem to be pretty ruthless about improving in certain areas and are very metric-focused. I'm not sure it's a great place to program, but it's certainly a great place to learn about programming.

aphexairlines3• 15 years ago

The upside is that the company has excellent developer tools and infrastructure (I mean aside from AWS), very accessible principal engineers, a huge amount of paying customers so your work has real impact, and it seems like most teams have autonomy to work and deliver in their own way. Within a week or two of joining I was given ownership of an important internal product while also getting up to speed on the main event (the retail website). The technical challenges are there for the taking.

I also mentioned some of this at http://news.ycombinator.com/item?id=2488980

haberman• 15 years ago

I worked at Amazon from 2004-2007. On one hand, Amazon does some really excellent and novel work like Dynamo and AWS (S3, EC2, EBS, etc). They organize developer conferences and bring in really good people to lecture. They hire really solid distributed systems people like Pat Helland and give them reign to work with engineering teams to improve their designs.

On the other hand, when you're in the trenches of a product team at Amazon, good engineering takes a back seat to features, deadlines, and keeping everything running. There was an incredible amount of pressure from the top to maintain high levels of service, but not a lot of support for taking the time to refactor or rework those systems to address the underlying causes of service disruption. Everything was always an emergency, but we always had to make the shallow fixes.

Overall, I'm much happier at Google where there is a lot of support for good engineering at all levels of the organization, not just as a means to an end, but as an end in itself.

puredemo• 15 years ago

Too bad that Google doesn't touch the customer service standards at Amazon.

chaddeshon• 15 years ago

Good engineering can't take a backseat to features, deadlines, and keeping things running. Good engineering IS features, deadlines, and keeping things running.

If you aren't making features and meeting deadlines you aren't engineering anything. You're just messing around. Features and deadlines are the difference between science and engineering. There's never enough time, that's what makes it hard.

Of course, Amazon might be pushing time tables too far. I have no idea, and it seems like you do.

thejbf• 15 years ago

There is a balance between shipping a good product with good technology. If you are constantly pushed by deadlines and make things just work, it breaks stuff in the future. Any engineer that is continuously working under pressure of tight deadlines cant ship anything creative, solid, simple and open to change.

Good engineers also have to be good analysts. What often makes a programmer better is his/her opinion to say yes and no to products people/designers. But yet again there is a balance. haberman mentioned Google but I personally don't like the over-engineered Google culture. It's too left-wing, if tight deadlines are too right.

tmarthal• 15 years ago

Maybe not working at Amazon, but their suite of companies are still technically innovating : their http://www.a2z.com subsidiary is doing some amazing stuff, with top-tier talent.

dustinupdyke• 15 years ago

I don't see technical excellence having a lot to do with technical debt in terms of if you are excellent, you have little or no debt.

Business needs change too quickly for a system as large as Amazon; they simply cannot maintain net zero technical debt and not cripple their reaction time in the market.

The key is to know what level of debt is necessary and acceptable to take on and deal with going forward.

tybris2• 15 years ago

I would say the opposite is true. Amazon is a collection of largely independent product teams that make their own decisions and share very little code. Infrastructure services are used based on its merit. Stuff that works stays, things that don't die out. Unlike most tech. companies that depend on large, common infrastructure, Amazon is carefully designed to evolve.

Main problem is that a new family of species called AWS is so well-adapted and successful that its population is rapidly outgrowing some of the older, less dynamic species on which it depends. You can call that technical debt. I'd call it progress.

aphexairlines3• 15 years ago

> 40 million lines of C++ code in their core product

That was a long time ago. As Jeff Bezos confirms in the article, Amazon is a constellation of services now.

px• 15 years ago

Investors want higher margins but obviously Amazon is investing substantially in itself. So far, though, Bezos seems to be winning the tug of war. Many analysts are glad to see higher revenue and feel that Amazon is cementing its position at the top of the ecommerce world. AMZN shares are up over 5% despite the earnings miss.

czhiddy• 15 years ago

Interesting how AMZN jumps 5% after missing earnings, yet GOOG drops nearly 10% with nearly the exact same results (increased hiring and expenses resulting in lower than expected profit).

meterplech• 15 years ago

I think that is because investors believe in Jeff Bezos more as CEO than Larry Page. Bezos comes from Wall Street, so Wall Streeters probably have more confidence in him. Plus he has been CEO for a long time, whereas Page comes from academia and just started the role (again).

brown9-22• 15 years ago

I think it's really silly to try to draw out any meaning, or even assume any meaning exists in the first place, in single-day movements or reactions like these. What happens in a few hours of trading after earnings announcements means nothing after a few days.

Eliezer• 15 years ago

Stocks don't react to how results in announcements compare to last year, stocks react to how results in announcements compare with how Wall Street priced the stock yesterday based on their guess to what the results would be. Efficient markets 101 - otherwise you could make money by buying, one day in advance, stocks that obviously probably did well compared to last year. If this is not obvious to you then you should never invest in anything besides index funds (seriously, I've seen people do really badly by not understanding elementary implications of the inexploitable markets hypothesis).

anatari• 15 years ago

Um, missing earnings means earnings was less than what the wall street analysts expected, it doesn't mean it was less than last year.

tybris2• 15 years ago

Simple:

"Google made a 10% across-the-board salary increase, which is driving up expenses this year."

"[Amazon] spent ferociously to build more fulfillment centers and expand its technology offerings."

mkramlich2• 15 years ago

As a general rule, short term movements on Wall Street are "dumb" because there's a lot of emotion driving trading on that scale, plus lots of weird automated/hedge fund trading going on. To give just one example of something that probably contributes to noise: naive trading programs that see an uptick on "Hathaway" in the news streams, assume that's a signal indicating some new item about Buffet's company, so they buy up BH -- but in reality the spike was caused by a wave of news about the actress Anne Hathaway. I'm sure there's a lot of dumb/naive algorithms like that out there. Plus who knows how many pre-scheduled trades or triggered trades. In the short term, lots of dumb noise and froth. I think overall movements on Wall Street are the most meaningful and valid in the longer term, and across a large class of things.

sahillavingia• 15 years ago

$AMZN stock is flying right now: http://www.google.com/finance?client=ob&q=NASDAQ:AMZN

ZackOfAllTrades• 15 years ago

Why?

Seth_Kriticos• 15 years ago

They just announced their quarterly results: http://www.americanbankingnews.com/2011/04/27/amazon-com-inc...

JabavuAdams2• 15 years ago

Lol. Sold a bunch of AMZN last week to buy AAPL, just before AAPL announced earnings. Looks like I should have kept AMZN.

As a Canadian, the whole currency conversion thing adds another source of risk.

3pt14159• 15 years ago

I was mulling over that exact trade, but figured the relatively small amount of AMZN I had didn't justify getting off my tush. Sold a couple hours ago with a solid 60% up in 10 months (usd -> usd) and 41% in CAD terms. I'm fairly bullish on AAPL, but I already own 200 shares (selling medium term call options, hence the 100s multiple) so buying some more probably isn't worth it. Just going to keep it in cash until I see something.

Do you have any experience in hedging your US investments by shorting the USD? I'm wondering if the additional fees are worth it.

javanix• 15 years ago

The end result of all this behind-the-scenes software? Fast, accurate search results that help you find what you want.

Has Bezos ever actually used Amazon's search functionality?

I know it isn't the easiest problem in the world to solve, but their search is by far the worst of any of the major shopping sites that I use on a regular basis.

pkteison3• 15 years ago

Specific examples? It seems pretty good to me. Which shopping site has a better search, and what makes it better?

Exact keyword match is obviously fine. "harry potter and the sorcerer's stone" gives me various options for buying that book and a bunch of other harry potter related books, and i can sort by some useful options like reviews or bestselling.

Harder problems, Google shopping vs amazon: Misspellings: Hairy Potter - Amazon suggests 'did you mean Harry Potter?', but does not initially put that book in front of me. When I check again later they are now providing a second list, "Results for "harry potter" (corrected from "hairy potter")"

Google shopping shows me the Did you Mean question, search results are the lego sets and one movie.

Concepts/descriptions instead of exact thing: "Boy Wizard Series" - both fail to suggest harry potter to me, but they do start with "Harry, a History" which might be enough to jog my memory. Still, no real points here, but this is a hard problem, and I don't know any shopping site that does it well. Even Google's regular web search is not fantastic at this, hitting mostly news articles and not reading my mind and popping out with "I bet you want to see information about Harry Potter..."

So where's the trouble?

Splines• 15 years ago

In particular, searches for computer equipment is generally frustrating. I've also seen sorts result in a large number of irrelevant outdated used items getting filtered to the top.[1]

I'm willing to cut them slack since it's good enough, but look at Newegg to see how it should be done. It's not a completely fair comparison (since Newegg specializes in this), but they do it so much better.[2]

[1] http://www.amazon.com/gp/search/ref=sr_nr_i_0?rh=k%3A16gb+sd...

[2] http://www.newegg.com/Product/ProductList.aspx?Submit=ENE...

metageek• 15 years ago

I agree that it's mostly pretty good, but here's one flaw that always annoys me: when I tell it I want something under $50, and I only want stuff sold by Amazon, they'll include an item even if Amazon's price is $60, as long as at least one seller has it for under $50.

hga• 15 years ago

You can say that again. By far the worst; quite frequently I have to make several tries and occasionally resort to Google. I used to play with this technology in the early-mid '90s when it was a document imagine adjunct and I'm truly impressed by how bad it is.

I would be very surprised if they were not losing a significant amount of business from this deficit, especially outside of the well cataloged area of books.

neuroelectronic5• 15 years ago

There is probably over a million products on Amazon. Is there any store in the world that offers a bigger selection? The only problem I have with the search is that they need something more NewEgg-like for searching for computer stuff.

ZoFreX3• 15 years ago

A million is really not a big number, especially in the context of a distributed database system.

frouaix• 15 years ago

You're off by two orders of magnitude.

spinchange• 15 years ago

This is instructive. Google's stock might not have tanked as hard the other week if LP had come out and placated the street with some jargon like this, re: Google's also ballooning expenses.

athst2• 15 years ago

Bezos explicitly said it in this letter - he expects all of this stuff to make the company gobs of money in the future. For a lot of the stuff Google spends money on, I don't think investors are necessarily very confident that it will have the same impact on future profits.

TWADDLE2• 15 years ago

Its funny how people seem to be whining about Amazon.

In fact Amazon is one of the best places in terms of innovation. In addition to having a strong core business, they also have AWS, kindle, android app store.

Who cares if their legacy codebase is 40 milllions lines of C++ and not your hip Ruby or LIPS nonsense. The truth is that they get work done. AWS is one of the best cloud computing system out there. Nothing else comes even closer. Dropbox, Heroku and Reddit all rely on them. Who cares if they take resume in word format, they pay well.

Its sad to see HN turning into a language fanboi and apple fanboi club.

The AWS EC2 outage did not even affect 0.01% of their revenue.

jacques_chester• 15 years ago

My understanding is that the original code for Amazon was written in hip Lisp nonsense with some ironic-retro C thrown in.

vaidhy2• 15 years ago

Disclaimer:I work at Amazon as a dev manager.

Working at Amazon does mean that you need to work hard, long and smart. There is a lot of technical debt to deal with too. But the technical debt did not come from bad design, but from the way business grows and changes. There is a 20% growth YoY. Inventories run into millions of products across multiple warehouses. We ship through multiple means and we ensure the product shipped reaches the customer on the promised date at over 95% of the time.

In real life, this means that you are going to have trade-offs and some of the trade-offs do turn into technical debt. What we do with the debt is more important and amongst my groups, we are decreasing the debt by leaps and bounds.

If you want to see how it is, give me a shout. We are hiring, as usual :) I can take resumes in any format too :)

senthil_rajasek• 15 years ago

How long have you worked at Amazon and where are you based?

vaidhy2• 15 years ago

I have been there for about a year and based out of Hyderabad.

cmelbye2• 15 years ago

This is only slightly on-topic, but regarding the service-oriented architecture that he mentions at the beginning of the article, how would one implement this internal? Specifically, how would you communicate with your internal services? Just an HTTP API? Is that fast enough? Another thing I'm wondering is what's the best way to handle internal authentication between your frontend and some backend service for example? Sorry if this is too specific for this discussion, but I've just always wondered this and I figure that HN knows.

adpowers• 15 years ago

I don't think it covers authentication, but take a look at Thrift:

http://incubator.apache.org/thrift/

It provides an RPC framework and IDL.

iam• 15 years ago

If only thrift had some sort of signal/event support :(.

shykes2• 15 years ago

DotCloud [1] is built from the ground up around distributed services. We use a custom rpc mechanism called ZeroRPC. It uses ZeroMQ [2] for transport, msgpack [3] for serialization, and some extra glue for RPC mechanics, discovery, event dispatching, etc.

Among other neat tricks, it allows us to handle synchronous RPC calls and asynchronous message passing in a unified way. It also maps transparently to any ZeroMQ topology.

Combine this with the recent release of gevent support for zeromq [4] and you get a fun, fun playground :)

[1] http://www.dotcloud.com

[2] http://www.zeromq.org

[3] http://msgpack.org

[4] https://github.com/zllak/gevent-zeromq

cmelbye2• 15 years ago

Oh, awesome, thank you!. I never thought to use ZeroMQ, but that would work perfectly. Ironically, it doesn't seem like DotCloud supports these sorts of architectures very well yet, any plans to fix that? ;)

aphexairlines3• 15 years ago

I wish the company would reveal some information on the internal systems he outlined in his letter. Amazon's architecture is impressive and its infrastructure teams build great stuff to help developers get their work done & deployed.

Exposing these things would be a powerful recruiting tool.

Why I, Jeff Bezos, Keep Spending Billions On Amazon R&D

Related Stories