Machine Learning-Powered Search Ranking of Airbnb Experiences

https://medium.com/airbnb-engineering/machine-learning-powered-search-ranking-of-airbnb-experiences-110b4b1a0789

by jamesjue • 7 years ago

41 8 7 years ago

syspec• 7 years ago

I find the searches on Airbnb to be pretty ineffective. Often times I’m trying to find a listing I was previously viewing and it’s nearly impossible to find it (referring only to the cases where I eventually do find it, not when it’s no longer available)

pheug• 7 years ago

It seems to be working as designed:

"experiences that were clicked but not booked (which we treated as negative labels)"

moritzmeister• 7 years ago

Too bad that they don't provide any reasoning for this labelling. Also it isn't clear to me over which time span they were looking at individuals to make the label decision, i.e. are they considering sessions or just single clicks with subsequent bookings?

s_Hogg• 7 years ago

The headline screams red flag to me. Calling something "machine learning-powered" isn't a guarantee of improved performance at all. Their baseline is already a fairly complex model, so there's no way to gauge how useful having a model at all is.

The partial dependency plots they show have gigantically wide error bars on them - they talk as if they offer a really nice confirmation of intuition, but the real relationship could be anywhere in those intervals, so it's probably weak evidence at best. The only one of the three where you could say with confidence that there's a relationship is the higher price -> lower score one. Add it all up, and you've discovered that people prefer cheaper stuff, with maybe a whisper of it being more complicated than that. In which case, their stage 1 model likely amounts to "show them the cheap stuff first", without the developers involved realising it. This is one of the problems we have with modelling - the model could be doing something simpler than we realise.

This gets more worrisome when you consider the fact that they're doing A/B testing to check for performance improvements. How are they able to claim that they improved bookings by 13% with their stage 1 model? Improved relative to what? More importantly, how do they know that this isn't a dataset-specific effect given that the data used for the stage 1 model is "small"?

I didn't continue reading after this, the whole article looks like confirmation bias to me.

orasis• 7 years ago

The A/B test was against a random ranking.

benjaminwootton• 7 years ago

Fantastic that they share this information and an enjoyable read.

This said, users can’t currently order AirBNB search results by price ascending or descending.

https://community.withairbnb.com/t5/Hosting/Sorting-listing-...

I think we should remember to empower the humans a bit more than the machines?

dwoozle• 7 years ago

Look at the authors. China is really going to put the screws to the US and it’s going to be sooner rather than later.

sbmthakur• 7 years ago

Machine Learning-Powered Search Ranking of Airbnb Experiences

Related Stories