Why We Abandoned Crowdsourcing

https://flightfox.com/community/why-we-abandoned-crowdsourcing

by todsul • 12 years ago

111 54 12 years ago

Good to see that crowdsourcing, one of the darlings of web commerce, is getting exposed for what it is: Cheap and crappy. The web capitalists loved the idea of labor competing against itself, maximizing value for the customer and minimizing labor costs. I.e. free work by 90% of the labor.

Good luck with the new system! Although you're doing it for practical reasons, I still applaud you moving away from this practice.

todsul14• 12 years ago

Hey aridiculous, we really wanted the economics of the competitive format to be fair. We got to the point where our experienced experts were winning 30-50%, so for them it wasn't so bad. Customers sometimes complained about "only" getting 2 or 3 experts, but we quickly realized it was better for everyone, including the customer.

It's a bit like being a software consultant. Most clients don't pay enough for you to do your absolute best work. That's what happened in the contest format. The best experts didn't make enough to do their best work. With the current format, we see them really sinking their teeth into big trips and having fun.

bduerst• 12 years ago

The problem lay in that some see it as a way to exploit the labor of others.

The real value is in setting it up so that crowdsourcing benefits both parties involved - i.e. CAPTCHAs for human verification and textual image analysis.

nlh2• 12 years ago

I've been a huge FlightFox fan and used them for several big (and complicated) trips - they've gotten me spectacular deals each time. This was all in the competitive phase, btw - haven't tried the new format yet.

One thing I just saw -- they dramatically raised the price if you're doing a "miles flight" -- it's now $99. I suppose I understand the thought process -- if you're not spending $$ on the flight, you're more willing to spend $$ on the consultation?

But honestly it makes me less-likely to use them. I use miles on some flights precisely because I DON'T want to spend any cash. Adding $100 in cash to the cost of the trip feels counter-productive.

Anyway, still a fan of the service, even if I've been priced out of using it :/

todsul14• 12 years ago

Thanks nlh. First, thanks for the feedback on the pricing. We're talking about the miles product right now and will update it very soon. The high price was to account for most miles trips being fairly complex, but we'll segment a little more and offer you much better pricing.

UPDATE: just spoke to the experts and made it $69 for now. We'll change the pricing to be +$19 to use miles, but that'll take a little time. Reason it's more is that the experts have a few tricks that take considerable time.

Great to hear you received value with the competitive format. We still have the top 0.3% of experts from the previous system, they just get paid more and can spend a lot more time on your trip now. But we'll keep analyzing and make sure it's working well.

nlh2• 12 years ago

Sounds totally reasonable. +$19 to use miles feels completely fair. I'm back in baby! ;)

(but seriously, thanks for the update. I know pricing is an imperfect science and takes time to optimize. Keep on kicking butt.)

jdcryans• 12 years ago

(not a flightfox user but my 2 cents)

Booking an award trip can be much harder to do unless you have an easy route like a domestic non-stop. The ITA matrix won't show you those flights, expertflyer.com used to be really useful but right now you can't get access to the United (and most of *A) data, some airlines have vague routing rules that may or may not play in your favor depending on which agent you talk to on the phone (looking at you US Airways), etc.

I think 99$ is steal.

misterbwong• 12 years ago

Being a bit of a miles guy myself, I agree with jdcryans. Most other award booking services charge ~150 per person so $69 (or even $99) is a great price.

That being said, the website doesn't really address the quality aspect of their award bookers. Extensive knowledge of routes/partners/tricks is essential to building a (good) award itinerary. I'd be hesitant to go with Flightfox even if it's a bit less expensive because a knowledgable award booker can mean the difference between getting an award and not going on the trip at all.

cocoflunchy2• 12 years ago

Has anyone tried FlightFox? I wonder if it's worth using for simple flights like NYC-Paris or NYC-SF.

todsul14• 12 years ago

hi coco,

It's definitely better for longer trips, but NYC-Paris is a good one for us because I know our experts have a few tricks up their sleeves with this route.

I often fly Montreal-SF so can also comment on NYC-SF. If you just want the cheapest economy flight AND you have checked bags, no, don't use Flightfox. However, if you have carry-on, then our experts can help.

It gets much more interesting if you want to fly this route in business. I've done it a few times in business for about economy prices (usually a little less). It involves a few little tricks and a slightly complex booking, but for those who do it often, it can reduce the pain of 12 hours in econ. I usually only go to SF for a day or two, so being more relaxed has a noticeable affect on my time there and proceeding. It's not always available, but I'm usually pretty flexible.

cocoflunchy2• 12 years ago

It's too bad I already booked my flights for NYC-SF (carry-on) on the 15th of this month... I would have tried it. Do you think you would have been able to find better than the $318 roundtrip I got? Or business for the same price?

kingnight2• 12 years ago

I used it a 1.5 years ago, during the competitive phase, for a two destination trip on a different continent. I came away not entirely thrilled by the results of the experts. After paying the fee, what they found was slightly cheaper than what I was able to do independently, but had a funkier itinerary that didn't justify the slight price reduction (that may have been actually offset by the cost of using the service).

That said, I did receive tremendous feedback from Todd. Reading this post has me reassured that he and his team are on the right track.

My wishes—

I wish I had the money and time to seek out such types of fares/trips more often so that I could try the service again. I may have one such trip coming up but I still feel reluctant paying a fee up front for what might not turn into anything.

I wish FlightFox/someone would offer services to deal with the airlines' kerfuffle on multi-destination, multi-carrier flights. I swear, if you have to deal with any inconveniences in a situation like this, it is 100% "Hi, this is ABC Airlines, you'll have to call XYZ Airlines as this leg of the trip is under their control" and vice-versa.

todsul14• 12 years ago

Hi kingnight, our experts will refund you if they can't help or you're not happy.

In the old system, there was always a lot of tension when we'd refund customers, because the experts would have already spent a lot of time. But in the new system, we made it so we (admin) can't actually refund you, only the experts can. This has worked wonders because they now use refunding as a form of segmentation. Often they'll refund after 30 seconds if it's clear they can't help, which is a far cry from the amount of effort they'd previous expend before a refund.

nomurrcy• 12 years ago

I've used it twice. Once for a Seattle -> Berlin -> Munich -> Seattle trip. Another time for my honeymoon (Seattle -> Krabi -> Tokyo -> Seattle)

Both times I found business class tickets for not a crazy spread over coach retail and significantly below the business class prices I was finding on my own.

I've had great experiences both times. For complex / long haul flights where you want to find a cheap business class seat, I'd recommend them without question. I'm a big fan of flightfox.

prawn2• 12 years ago

Yes, I was big on Flightfox when I heard about it and tried it for an international trip last year (Australia-Malaysia-France-Morocco-Italy-Switzerland-London-France-Malaysia-Australia). Paid for the top tier service but didn't end up using any of the results because I found a better and cheaper series of flights myself. I was a bit disappointed actually and went from telling everyone about it to not considering it since for other trips.

coldcode2• 12 years ago

How does FlightFox makes money? Given that both the expert and FF needs a cut, how does this scale?

jusben1369• 12 years ago

how many airline tickets are sold a day?

coldcode2• 12 years ago

Airlines pay us at most $10 per ticket no matter the price. Clearly this isn't what make FF money because it's not enough to scale if the profit is shared.

Aloha2• 12 years ago

I don't understand the pricing.

Is it say 29 dollars to consult for me, then another unknown dollars for the fare, or is that 29 dollars all inclusive or per leg?

todsul14• 12 years ago

Hi Aloha, thanks for the feedback. We'll try to make this more obvious asap.

To explain, our pricing is "from" $29. That's for the simplest trip. It goes up with complexity because the experts must work harder. This doesn't include the fare, this is the fee for consulting.

As you can imagine, this doesn't make sense for short flights for say $100. But our experts really start to work wonders on trips over $300-500. For multi-continent or business class, the savings often get into the $1000s.

If you click on each product, you'll see tweets from real customers talking about different results.

Aloha2• 12 years ago

I'd suggest giving ranges for total out the door pricing based on mileage or some other formula. I know this is a hard thing to guesstimate because of how complex airfare pricing is.

I personally love the idea of paying for a travel consultant to take care of my trip for me - and more importantly, be there to take care of me if something goes awry - its like a modern day travel agent, but with more transparency in pricing. Showing what someone would have paid for said trip thru kayak or whatever would also show the added value - if you can figure out a way to give semi accurate numbers of the savings on the front side based on mileage and actual savings when the trip consultant had set up the trip for you would go a long way to getting return customers.

Offering travel insurance that actually does something (versus what most of the common carriers offer) might worthwhile, especially for complex fare hacking trips split across several carriers.

The biggest issue with the travel industry is a lack of transparency - none of the existing online services really help with that.

bretthopper• 12 years ago

Your pricing/trips page (https://flightfox.com/trips) is really lacking some direction and information. I can easily see new Flightfox users being really confused. I was slightly confused and I've used Flightfox twice before.

I would suggest carrying over the 3 steps graphic/info from the main page to the top of the /trips page. But more importantly, you really need a better breakdown of exactly what those prices are. Even the "Guaranteed" keyword seems out of place and confusing. What exactly is the guarantee? That you'll charge me $149 no matter what?

Your create trips page is also lacking a bit of focus to me. Would be useful to have some more tangible examples of trips and example trip descriptions. I'm sure if you provided a simple example/template, you'd reduce some initial friction on first contact with an expert.

todsul14• 12 years ago

Hi brett, we're working madly on this now. We built the new version from scratch (very little in common), so we're a few-week-old startup all over again.

We can imagine these products being their own page with details, testimonials, examples, etc. Something like http://elto.com, whereas our current page is more like the 99-designs one.

Just wanted to mention, "clarifying" our products/pricing is priority #1.

prawn2• 12 years ago

Should that be something you work on before announcing the change? Seems quite critical to me.

Tarrosion• 12 years ago

I am glad to hear that. If I understand your product I'm a prime target: once or twice a year I plan travel where my flexibility in terms of airport, days, etc. is far too complicated for even the ITA matrix (the best flight search I know of). And I'm also about to plan international travel for an anniversary, and would love to get some help and a deal on that.

But I'm terribly confused what category I fit into. For the first case, am I multicity or within North America? For the second, am I a group, ultimate Euro trip, super cheap business, ...?

esteb_li• 12 years ago

I think crowdsourcing's issue is with the experience. For a customer, crowdsourcing requires more effort than plan google searching for example. You need to submit your issue, come back, edit/comment. Bad experience is much more frequent than good ones. And a neutral experience is closer to a bad experience which might result in the user not going through crowdsourcing again. Good luck with the new direction

todsul14• 12 years ago

I think this is spot on. We used to underplay the experience issue, but then we realized a poor user experience often hinders access to the best savings.

What made this worse was the animosity between customers and experts. It was expected between competing experts, but the concept of the "award" (and by necessity "switching award") caused tension in all directions. We still had many great results, but not to the level needed to build a great business. The difference with the current system is night and day. Everyone is on the same side; it's even a little erie since we're so used to the previous animosity.

encoderer• 12 years ago

Congratulations. You invented travel agents.

Modernized them, I suppose.

Not exactly an exciting and thrilling turn of events but it seems the metrics all point toward this as a better solution so I suppose "exciting" is overrated.

todsul14• 12 years ago

Hi encoderer, we were saying the same thing before testing this format.

However, unlike travel agents, our guys are genuine flight experts. If you travel wide, fly a lot, or use miles, you'll understand why this matters.

Behind the scenes, it's all about commissions. Agents work on commissions and most flights don't pay commissions. So it makes little sense for agents to spend hours on your flights.

This manifests in the wide skill gap between (most) agents and our experts. Most agents won't save you a cent, they'll just be there for good service. But our guys get insane deals you won't find anywhere else. Often they involve shorter flights with fewer stops too.

eddmc• 12 years ago

A lot of travel agents are genuine flight experts. If you don't believe that then you haven't spent much time with good travel agents.

Also, it's not about the commissions. That went out with the '90's

Source: I've worked in this field for 13 years now and sell a SAAS product to travel agents.

emhart2• 12 years ago

Great write up. It left me wondering - do you think you could have started here? Or did the competitive format bring you the talented experts & public attention you needed to make consultation work?

Congrats on the re-launch.

todsul14• 12 years ago

hi emhart, interesting question. The contest format was certainly novel and helped garner a lot of early attention from press. So there's that. But I think you may be right that the contests helped us a) find experts, but also b) learn what was possible.

That somewhat fits with the idea that crowdsourcing is best when you have less info. Once you have more info, then (at least we think) it's best to work with fewer specialists. Although this is mostly referring to the customer, maybe we can apply the thinking to our product lifecycle too.

emhart2• 12 years ago

Thanks for the thoughtful reply.

bhuga• 12 years ago

I used the new 'consultant' model last week and loved it. I really liked that the consultant could help me get what I wanted, not what I asked for ("I'm cold, put me in South America." "Buenos Aires is expensive. How do you feel about La Paz or Lima?").

I hope they standardize on it. The rest of the site seems to reflect the old models. For just one consultant, why do I need 27 different pricing options for 'kinds of flights'? I don't know what I want. Just charge me one price and be done with it. And the ratings system needs a field akin to 'money saved vs kayak' in addition to the 1-10 scale, because I'm no longer sure I got the lowest price as opposed to a consultation.

But I loved it all the same and will be back. Flightfox is doing a great job.

katowulf• 12 years ago

Great, insightful article for techies; very interesting insight into your business strategy and R&D. Thanks for sharing!

RyanZAG• 12 years ago

Honestly the pricing is the real unnerving part here and it's definitely going to be a deal breaker for much of your audience.

Personally I'd prefer something like this: I enter my current flight details and the price I was going to pay (your experts should be able to verify this price as correct pretty quickly?), and then your experts would give me a better flight plan and I'd pay you 10% of the difference or something, and nothing if your experts don't find a better plan. This way I'd always feel like I'm 'winning' by using your service and not taking an expensive shot in the dark.

todsul14• 12 years ago

Hi Ryan, segmenting is our greatest challenge and (planned improvements aside) your experience was intended. It's the old startup adage: better to have 1,000 customers who love you than 1,000,000 who don't care.

What you're suggesting is what we call "gambling against Kayak". That's not our target market because it's not the market we can help most. We often say internally, it's not whether we can possibly help a customer, but whether we can likely help them.

Our idea to create "products" was to help segment customers into those we can help most. For example, complex trips, flying premium, using frequent flyer miles, traveling with pets, euro-city hops, etc. These are the people we can "likely" make happy and will help us grow to then target a wider audience.

OoTheNigerian• 12 years ago

So I am traveling to 2 continents in the next month (Including the US). Most likely a minimum of 3 cities.

I have always wanted to use FlightFox but what is holding me back is the uncertainty of what I will get AFTER I have committed to pay.

What happens if I want to change my flight? Cancel? With the existing travel agent model, I am certain stuff like this will be taken care of.

When is the relationship over after the purchase or after the journey.

PS: Like Brett said, your pricing is really confusing. Is

London ---> New York international or Multi city?

How about,

London---> New York ----> San Francisco ----> Accra International or Multi City or both?

Do i pay different prices?

I plan to book my flight over the next few days. I am really tempted to prove I am an early adopter :)

todsul14• 12 years ago

Hi nigerian, apologies for the confusing pricing, we're working on that now. Anything more than one destination is multi-city. So your first example is just international, but the second eg is multi. Your second itinerary should be perfect for our guys.

chx2• 12 years ago

Disclaimer: I was a Flightfox expert. It was one of the best computer games I ever played and it occasionally paid me to play. (In a similar vein, Drupal development is the best MMO I could imagine -- the graphics surely leaves something to be desired but the social aspect is literally life changing and the quests keep you in flow.)

I am not sure whether flight search in a crowdsource format could or should arise again but Flightfox made the (deliberate?) blunder of revealing all expert email addresses to the other experts in an email announcing big changes coming on January 14. If this would be a bigger industry, that'd been quite some news... I know at least one person who is thinking on continuing -- thanks Flightfox for keeping the community alive.

todsul14• 12 years ago

Hey CHX, apologies for the blunder with the emails. It was an honest mistake of not BCC'ing and certainly not deliberate. As you saw, a handful of people replied to the whole group and talked about cloning our product, which wasn't really in our interests. We don't get worried about that stuff, we were just a little sad we messed up for you guys.

chx2• 12 years ago

I honestly say I liked this. I would've agreed to disclosing my address to the others to keep the experts together.

jdp23• 12 years ago

Very interesting learnings, and well-presented too!

I have to say, I don't see this as abandoning crowdsourcing -- the "crowd" of experts is continuing to work together. To me it looks more like investigating various approaches to getting good results from the crowd, and discovering a better alternative for the particular problem. Terminology aside, it's very interesting to think about what other situations this pattern could work in.

DorintheFlora• 12 years ago

It is a pleasure to see this analysis of the business strategy, what worked, and why. Tweeted. :-)

bartkappenburg• 12 years ago

Very interesting read! Especially when we are almost ready (as in next week) to launch our own crowdsourced travel advice platform on http://www.voyando.com.

Let me share what we found during our test period.

We have no 'application process' for experts (flightfox has) which makes the 'crowd' somewhat smaller. In our experience this removal of the process increases the competition and the chances of a better outcome. Take stackoverflow for example: limiting it only to top-programmers would harm the concept. (i see SO as a crowdsourced programming advice platform :-))

Our platform supports more than just flights. We have hotels and complete trips as well. The philosophy of our platform is more on the convenient side of things instead of the 'hacker' mentality that FF has (no pun intended, FF is great for HN crowd I think). Our service could be used by your mom and dad.

Given the fact that we have no selection procedure of experts, a broader range of travel types and aiming to be a convenience service, we are convinced, also looking at our test phase, that crowdsourcing IS the right way to go for us. But hey, only time will tell and maybe you'll see a blogpost about our pivot in 18 months as well!

Congrats to FF for this pivot and the really great and open blogpost.

slaven• 12 years ago

I just wanted to say how slick their service is, I just used it earlier today and was paired with a very competent agent. I'm now waiting for the day Flightfox can take care of the whole thing, including booking a flight. There is a definite opportunity there!

fibbery• 12 years ago

I tried flightfox when I was planning a vacation last year. I had thought it was more like taskrabbit where people would compete for the lowest price and was confused/disappointed when it told me I had to pick someone. Unfortunately none of the results seemed particularly promising.

In the end, either I should be working for flightfox or the service isn't as good as it should be, because the "expert" results were worse than what I found on my own... and I had the impression that the lack of competition was part of it.

todsul14• 12 years ago

hi fibbery, we made this change exactly for this reason: so customers would get great results more of the time. With the new format, not only do you get one of the best experts, but that expert can spend more time just on you. That last part is key.

This idea that "one person digging deeper" is better than "more people competing" first came to us at an in-office hackathon (flight hacking, not software hacking). We noticed with more "iterations", experts could hack even the most mundane trips. In this particular example, they got NYC<->Berlin (round-trip) for about $400 with rigid dates. The rigid dates part is key because it simulates the real world (we've seen < $300 with flex).

Once we worked closer with some of our experts, we realized the top ones are miles more analytical. They either write their own software or compile huge spreadsheets of their findings.

So this change of format was to put you in touch with one of them, not with someone who does this on the side. We wish we could have created a new income stream for the average person, but to ensure customer value, we had to go this way.

mgkimsal• 12 years ago

Wish I'd known about this change a few weeks ago - would have tried the new format - wife just booked a flight. I used flightfox once before, and while the idea was novel, the execution was... a bit weird.

Something that wasn't addressed in this article write up was how customers felt. I felt bad for having 4 or 5 people doing work on my behalf, giving them feedback ("change these days, find it cheaper", etc) then only 'awarding' one of them. I did not enjoy that experience - it made me feel like I was abusing their time.

Additionally, I never quite got the price my FF person found me - perhaps I waited too long, or something else happened. I think he'd sent me a screenshot showing that he was being shown that price as available, but when I went to the airline (delta, iirc) I wasn't getting that price. I was still getting a better price than I'd been aware of before, but not quite as good as what the guide claimed to have found.

Again, the contest format was not just bad for the reasons listed - users (well, at least this one) felt uncomfortable using it.

Good luck with the new format!

danbmil99• 12 years ago

Sounds like we're orbiting back to the travel agent model...

Why We Abandoned Crowdsourcing

Related Stories