Benchmarking the Major Cloud Vision AutoML Tools

https://blog.roboflow.ai/automl-vs-rekognition-vs-custom-vision/

by rocauc • 6 years ago

29 10 6 years ago

blog.roboflow.ai

yeldarb2• 6 years ago

Biggest surprise for me was that they can’t train on COCO. Would have thought their performance would be a major part of the marketing considering how prominent that benchmark is in the research community.

junipertea• 6 years ago

I guess research community would not use commercial products? The number of classes and datapoints is quite extreme for a “common” use case, usually you would want to train it by yourself.

kuzee• 6 years ago

Just learned a lot about the different hosted ML options, thanks for the write-up with backing data. How much did this cost you overall?

rocauc2• 6 years ago

The 'hard costs' were the training time for each model ($125) and conducting inference on the valid set ($15) for each.

Now, we had a few failed experiments like trying to run the COCO dataset through each tool. And none of this includes the implicit cost of the computer vision engineering time.

RcouF1uZ4gsC• 6 years ago

This is amazing content marketing, similar to Backblaze hard drive stats.

It was well-written and very informative. I enjoyed reading it and learned something new and potentially useful. And it brought attention to a potential pain point (trying to train and infer models on multiple platforms) and suggested their product to help(roboflow). It got the marketing message across without being irritating or obtrusive. A win-win situation for both reader and company.

Again, great writeup. I wish more content marketing would be this engaging and useful.

rocauc2• 6 years ago

Really appreciate it; we had fun writing it. More of this to come.

mlthoughts20182• 6 years ago

The thing with evaluating these services is that business people or product managers don’t understand that accuracy on benchmarks doesn’t map well to accuracy on your problem. If you have a wacky data distribution (hint: you do), say like a collection of photos all taken inside the same building, you can’t generalize to consider benchmark stats.

On top of this, these services charge you for usage, not for accurate usage. This is a major issue that so many people overlook.

What does it matter if it’s cheap per request? That only helps you if accuracy per request is very high. Otherwise you’re paying for cheap garbage, or in some use cases you must grow requests much larger to overcome errors per request, like if you’re trying to get a bulk of labeled data via these services.

Most use cases are still better off hiring ML engineers who understand how to evaluate accuracy for the unique business use case, fine tune or train a model, and do it in house (or at least give you deep assurances of the rare cases when the big cloud services actually are cost effective to use).

For most use cases, you’re wasting your money trying for ML-as-a-service like this.

yeldarb2• 6 years ago

There's a huge gap between paying a couple hundred dollars for a pretty good model and investing hundreds of thousands (or millions) in hiring a team.

Certainly if you're building something mission-critical, like self-driving cars, you want to get the best possible performance (regardless of price).

But there's a whole other set of use-cases where "pretty good" is good enough (eg finding people sharing your company's logo in social media posts).

mlthoughts20182• 6 years ago

“finding people sharing your company's logo in social media posts”

that seems very different than your comment describes.

I agree hobbyists may use these services. If a business is looking for spending “a couple hundred dollars” on a significant ML problem, that business is living in a fantasy and needs a reality check.

There is really not a market for low end cost but high specific task accuracy business problems.

Also your comparison with hiring a team is a very false dichotomy. Hiring a team may cost you $1MM but it’s amortized over all the work and projects they do. It’s not $1MM just for the in-house equivalent of this one AWK Rekognition task.

jorams• 6 years ago

I think there's a number missing under Google Cloud AutoML Vision Inference Cost:

Our tests yielded x predictions per second

Benchmarking the Major Cloud Vision AutoML Tools

Related Stories