Two Decades of Recommender Systems at Amazon.com

https://www.computer.org/csdl/mags/ic/2017/03/mic2017030012.html

by chwolfe • 9 years ago

230 74 9 years ago

computer.org

aezell• 9 years ago

20 years and they still think I need more than one vacuum.

ht: https://twitter.com/kibblesmith/status/724817086309142529

wlesieutre3• 9 years ago

Same experience. I ordered a pair of speaker stands from Amazon on Monday.

This morning they sent me an email saying "Amazon.com has new recommendations for you based on your browsing history." It was a bunch of other speaker stands.

stult• 9 years ago

Well, do you happen to know how many other people are repeat buyers of similar products? Maybe a large number of buyers end up buying a second set of speaker stands because they don't like the ones they just bought or because they're enthusiasts who gift them or have multiple audio setups. Just because it seems unlikely someone would be a repeat buyer in a given product category doesn't mean it isn't profitable to target every buyer as if they might be.

saosebastiao2• 9 years ago

In some senses, their reasoning is rational. Some models work for some situations better than others...that recommender model might work exceptionally for someone trying out different brands of the same consumable product (like Natural Deodorants, for example). For other products, an accessory product recommender model might work better (like for Vacuums and vacuum attachments). And since the number of scenarios is infinite, you might as well try as many as possible and stick with all of the ones that show promise.

Which is fine until 90% of your browsing consists of scrolling past different recommended product widgets, which seems to be the case at Amazon now. At some point, taking different working models and trying to work on converging their benefits into a more monolithic model would be a huge benefit.

boomzilla• 9 years ago

Don't think there is much rationale behind all these models. It's more like P(would buy a vacuum | bought a vacuum) > P(would buy X | bought a vacuum) where X is a single product. Now P(would buy a vacuum | bought a vacuum) < sum(P (would buy X | bought a vacuum)) for X that is not a vacuum, but what would be the recommendation? Hey, you bought a vacuum, come back and buy some non-vacuum stuff?

For most recommendation UIs, you would need a hero item that make people want to click on. It might turn out that another vacuum is probably the best item for some people to click on, and go on to buy other stuff once they are on the site.

baddox3• 9 years ago

Why would you need different models like "accessory recommender"? If they have enough data to conclude that purchasers of itemA are highly likely to purchase itemB, why not just recommend itemB, even if itemA and itemB are both vacuums and a few people will complain about that on technology forums?

ryanworl• 9 years ago

I can certainly imagine a world in which you, having just purchased a vacuum, are more likely to purchase a different vacuum than any random person they just chose to show that recommendation to.

For example, you might return your first one and then want to buy a different one. Instead of letting you forget about it and then months later potentially buying elsewhere, they preempt you and offer new ones soon after.

This is obviously speculation, but I think Amazon could come up with something better if they felt it pressing i.e. recommending the same product category was no better than random products.

blacksmith_tb2• 9 years ago

My instinct is that they are willing to live with showing some comparatively weak recommendations occasionally if that means not adding layers and layers of exceptions on top of their basic recommendation engine. And/or they have projected that the ROI isn't there for building in more sophisticated recommendations.

seanwilson• 9 years ago

This is obviously speculation, but I think Amazon could come up with something better if they felt it pressing i.e. recommending the same product category was no better than random products.

You'd think they'd have so much data on buyer behaviour that they could recommend you e.g. other cleaning products that vacuum cleaner buyers tend to buy like mops, dusters and sprays. I can't imagine how recommending you the same thing you just bought over and over that you're not likely to get again for years is going to yield the best results and even manually selected recommendations would do better than that but willing to be convinced as obviously it's in their best interest to be smart here. You'd think you'd even explicitly build in a rule in your recommender system not to recommend something that has been recently bought.

saurik• 9 years ago

But there are so many other things that make more sense: just imagine what a salesman would try to do, and he answer is not "another vacuum". They could try to get me to buy bags to go with the vacuum. They might think I am into cleaning right now and try to sell me a feather duster. It just makes so little sense that their reaction to "bought a thing" is to run ads showing me "same thing you just bought" instead of "things that go well with the thing you just bought"... particularly as they know what those things are as before you leave Amazon's website they are all over "people who bought this tend to buy this also". When I buy a camera they should try to sell me a lens cap. When I buy exercise equipment they should try to sell me weights or towels or mats. When I buy a DVD player they should try to sell me DVDs. Yet for the next two weeks they insist on spending as money showing me the thing I just bought as if I want to buy it again: what the hell? :/

linkmotif2• 9 years ago

Yep, this. I look at something and it recommends it to me for months. I buy something and same thing. It's quite obtuse. Maybe there's something to it, maybe it does a good job growing revenue, but there's noting elegant about it.

baddox3• 9 years ago

Maybe there's something to it, maybe it does a good job growing revenue, but there's noting elegant about it.

It almost certainly does a good job growing revenue, and they almost certainly care more about revenue than about the perception of "elegance." You say that as if it's unusual or unexpected.

teej• 9 years ago

Given that Amazon's system can be described as: "... for every item i1, we want every item i2 that was purchased with unusually high frequency by people who bought i1", I'd say it's completely rational that their system recommends you vacuum cleaners if you've already bought one.

fs111• 9 years ago

I bought an expensive camera in March, they still insist I need 5 more from different vendors.

hammock• 9 years ago

We always had three vacuums in the house growing up: a main canister vacuum (kept in a closet), an upright for the hard kitchen floor (kept handy), and a dustbuster (also kept handy).

Three tools with different uses.

dahdum• 9 years ago

Reminds me of Netflix recommendations. They keep recommending I "continue watching" the ~45 seconds of credits I missed from the last dozen shows I watched. Most lists have movies/shows I've already seen at the top still.

I'm sure there's some reason behind it all, whether technological, user behavior, or obscuring a small catalog. I wish I knew what it was.

bradjohnson2• 9 years ago

Or if I watch a couple episodes of a show and decide I don't like it and click the "Stop recommending this show" button, it'll still show up in the "Continue Watching" list. What's the point of the button?

martijn_himself• 9 years ago

Or Netflix decides to 'hide' a show I want to continue watching, and I have to search for it.

foobiekr• 9 years ago

my personal ultimate example of this was I bought a lawnmower. and then for two months, all of my amazon advertisements, the ones following me around the web, were for lawn mowers.

even a vacuum I could see some weird "yes, maybe I could buy one as a gift." but a lawnmower? no.

arkitaip• 9 years ago

Just for months? Amazon keeps recommending me the same 90s Radiohead albums based on stuff I bought a decade+ ago. Retargeting can be so on point these days so I don't get why Amazon's recommender systems have been horrible for years. All they have to do is to peek at my cart, which I use as a wishlist, to figure out what I'm currently interested in.

surement• 9 years ago

The suggestions are probably based on browsing history rather than on buying history. It's not that surprising; you likely browse for multiple items, which helps make inferences about what you might want to buy. You only buy a small fraction of the items you browse, much of them unrelated. Taking one of those points (your purchase) and guessing that it belongs to a cluster of points that shouldn't be suggestions is not obvious. Further, basing suggestions on purchases might give terrible results given that your purchases are mostly unrelated.

DonaldFisk3• 9 years ago

That sucks.

The problem is that the algorithm seems to be unable to distinguish interchangeable items (e.g. vacuum cleaners, lawnmowers, Swiss Army knives) which you need exactly one of, similar items (e.g. different death metal albums, or Hammer horror movies) which if you bought one of, you'll probably want some others, and connected items (e.g. different parts of a course) where you'll probably want all the others.

icebraining• 9 years ago

It's weird, though; seems like an "average time between purchases" associated with each category would be an obvious metric to use in the algo.

andrewingram2• 9 years ago

Some fungibility factor attached to each product wouldn't be amiss here

TheSpiceIsLife• 9 years ago

Maybe they've worked out that showing people more of what they just bought makes people go on and on about Amazon's recommendation system all the time.

And when people are complaining about something they endlessly repeat "Amazon Amazon Amazon Amazon".

When you have nothing to complain about you're less likely to evangelise.

biztos• 9 years ago

To be fair, I myself have two vacuums. Maybe buy another and see what happens.

roselan• 9 years ago

Well, if you bought some particular brand I won't name, you will need one per year.

dcminter• 9 years ago

It's certainly an annoying behaviour - but my personal suspicion is that it arose from the system's origin as the recommender for a bookstore.

I mean if I read a book by P.G.Wodehouse it in fact IS likely that I want to read another similar (but not identical) book by P.G.Wodehouse.

Quite how they managed to hang onto this quirk is a different question that I can't even fathom the answer to (I don't believe they haven't noticed and I find it very hard to believe this is actually the optimum sales technique for vacuum cleaners!)

mad44• 9 years ago

I had read this paper, last week and took notes. Here is my summary of the paper. It was a fun read. http://muratbuffalo.blogspot.com/2017/07/paper-summary-two-d...

ignoramous• 9 years ago

Thanks for these and other posts and generally putting in effort and making the commentary publicly available.

/Offtopic

ernsheong• 9 years ago

initially I was worried I wouldn't understand or enjoy the article

Surprised to read this coming from an engineering professor. I guess we mortals have more in common with professors than we think :) thanks for the humility.

firasd• 9 years ago

Related: Deep Neural Networks for YouTube Recommendations https://research.google.com/pubs/pub45530.html (Discussion: https://news.ycombinator.com/item?id=12426064)

ekarulf2• 9 years ago

Just in the event someone out there is looking to work on these types of problems, Amazon's Personalization teams are absolutely hiring.

Homepage: https://www.amazon.jobs/en/teams/personalization-and-recomme...

Applied Science: https://www.amazon.jobs/jobs/372996

Software Engineering: http://www.amazon.jobs/jobs/549950 http://www.amazon.jobs/jobs/402127 http://www.amazon.jobs/jobs/430623

Software Managers: http://www.amazon.jobs/jobs/385902 http://www.amazon.jobs/jobs/437405 http://www.amazon.jobs/jobs/489879

If one of those don't strike your fancy but you are passionate about working in this space, feel free to send me an email: ${HN_USERNAME}@amazon.com.

mabbo• 9 years ago

Just think: all those things you can't stand the Amazon's recommendation engine does, here's your chance to fix it for everyone.

nine_k• 9 years ago

Are they hiring a product manager?

johntb86• 9 years ago

Given how long some of these problems have been going on, it may be a chance to learn why it's not possible to fix them within their current organizational structure.

decafbad• 9 years ago

I remember this one: https://lh3.googleusercontent.com/beyxJM_pM6jH8AKDU0ITxXn9W1...

Not very old.

fooker• 9 years ago

Yep, I am definitely going to buy a $800 camera the day after I bought a slightly better one!

losteverything• 9 years ago

I use the amazon app to price match at another retailer.

The last scan or search always is somewhere in a recommendation (& one of my game apps)

npstr• 9 years ago

I investigated different algorithms for recommendation systems once and was astonished to find that the best one was also the most simple one: Jaccard similarity. No other similarities nor a proprietary custom built algorithm (which was the actual target of the investigation) could beat it.

DonaldFisk3• 9 years ago

This implies that you know all the relevant attributes of the items you're recommending. For something like music, books, or movies, there might literally be thousands of them, most of which are unidentifiable, any of which might differ in importance from one item to another.

I built a collaborative filtering system (http://web.onetel.com/~hibou/morse/MORSE.html) and didn't have to worry about who directed a film, when it was released, where it was filmed, its budget, who starred in it, or what its genre or plot elements were. All the relevant information was implicit in how the people who saw it rated it out of ten.

NumberCruncher2• 9 years ago

Could be interesting for you: https://en.wikipedia.org/wiki/Netflix_Prize

scj• 9 years ago

Thank you for the link, I intend on studying it carefully this weekend. I may want to implement something similar using board game data from boardgamegeek.com (I've already been working on a recommendation system). BGG has a 1-10 point rating scale (using floating point numbers), so I suspect the method described will fit.

My only concern is the number of mutually rated titles between users in board gaming is probably lower than movies. Which I suspect will reduce confidence rates.

The current approach (with a Jaccard system), requires a degree of human intervention and only works on some users. It meets the goal of recommending titles for me, but it'd be nice to expose the system externally.

NumberCruncher2• 9 years ago

Funny fact: 4 years ago I was looking for something what I would call today a "transitive similarity index" to measure similarity of shopping baskets of brick-and-mortar stores. Because I didn't find anything I "invented" my own algorithm. This year I implemented the same algorithm in a system which is live since yesterday. Googling up the Jaccard similarity I just realized that I re-invented a transitive version of it. Now my baby has a name. :)

gcb0• 9 years ago

anecdotal, but I've been their customer for all those years, and besides the few books early on, absolutely one hundred percent of their recommendations was utter useless.

bane• 9 years ago

Sometimes it works surprisingly well, I think I remember a presentation where somebody demonstrated that searching for components of a meth lab on amazon would have amazon recommending to you other components to help you finish the lab.

nn3• 9 years ago

If you want to see a truly terrible recommendation system try aliexpress.

geomark• 9 years ago

Indeed. But then I end up looking at things I would never have searched for. Today I was looking for a UV sensor module and searched for "8511", a portion of the part number for a commong one. First 4 results were UV sensors. Next one was " Women Police Costume Uniform Secductive Role Play Sexy Lingerie Sex Outfit Babydoll Fancy Halloween Costume Sexy Cop Outfit", followed by "Synthetic Druzy Mineral Stone Double Flared Saddle Ear Gauge Wood Flesh Tunnel Plug Piercing Body Jewelry Expanders". To be fair, I should have included "uv sensor" in my search phrase. But then I would have missed that cop outfit.

sundvor• 9 years ago

20 years, and I still can't see a chronological listing of the other books in a series when shopping for my Kindle.

Traubenfuchs• 9 years ago

I bought some embedded system accessories two years ago for college and I am still getting offered embedded system accessories.

Amazon's recommender system is garbage (for me).

Two Decades of Recommender Systems at Amazon.com

Related Stories