Show HN: I built an AI that recognizes food

https://www.infino.me/hungrybot

by subcosmos • 10 years ago

202 127 10 years ago

subcosmos41• 10 years ago

Built using those newfangled convolutional neural network things. Trained caffe on a set of about a million food related images. Can't believe how accurate the classifier is...

UI needs work. Would love feedback on how to make food logging more intuitive.

teLeopardthy• 10 years ago

Any info on the network itself (number of layers, input resolution, pooling, dropout etc)?

subcosmos41• 10 years ago

Stock Cafenet, trained from scratch.

sandGorgon3• 10 years ago

why caffe ? why not something like h2o or tensorflow ? just curious.

subcosmos41• 10 years ago

because Im a newb. Looking forward to expand my repitoire!

cemsazara• 10 years ago

Is not Caffe actually GoogLeNet? So technically it is Google architecture.

AJRF4• 10 years ago

You could probably outsource enough of the UI work to Fiverr to make a really decent looking app.

Things like Main screen, App Icon and colour palette.

subcosmos41• 10 years ago

My color palette was carefully engineered to induce vomiting and weight loss! :p

swah5• 10 years ago

Your app kinda sucks now, but the detection is great. You could also do food identification as a service...

I'm trying to use fatsecret/myfitnesspal for counting calories. Something like this could make it much easier!

To me, a better workflow would be: take pictures of everything you eat and classify later (at night, maybe on the PC). Of course, this software would simplify the classification, the user would only fix any errors and adjust sizes.

This workflow would be well suited to your app, which operates on a server? (my understanding).

Congrats, I understand the relevance of AI for simple apps now ;)

subcosmos41• 10 years ago

Ive been thinking of that 'classify later' workflow. I think of it kind of like facebooks photo tagging, and it might even be better as web app that you get email reminders for to complete entries.

It would also work nicely for background geolocation services, since if the photograph was taken near a restaurant the service could extrapolate those things.

Anyways, I might make this project opensource and build a nice community around it. Depends on whether or not that model would best support my research goals.

Keats• 10 years ago

I would love a blog post about that!

imacomputer2• 10 years ago

krashidov2• 10 years ago

How much experience with machine learning did you have before you started this project?

subcosmos41• 10 years ago

Cut my teeth on the netflix prize several years back. Took ML in my graduate coursework, and have dabbled with it on and off since. I love the whole theory of matrix factorization and its myriad uses.

zopf• 10 years ago

Where'd you get your images? Food-101 plus something else?

We're working on a similar component for our commercial behavioral-economics-driven app suite, targeted toward patients with chronic diseases.

Do you use location as a way to filter the set of possible foods, as in Google's im2calories project/paper last year?

They do some pretty awesome depth calculation stuff too: https://www.google.com/?ion=1&espv=2#q=im2calories+type:pdf

Edit: oops, should have read further. I see that you pulled appropriate terms from WordNet and used ImageNet to gather training images from Flickr. Cool!

amelius• 10 years ago

How does it measure the glucose content of that sauce?

zo1• 10 years ago

Probably gets it from FatSecret?

subcosmos41• 10 years ago

It does! They've been gracious enough to let me use their data since I'm running a nonprofit experiment.

AJRF4• 10 years ago

Something like this would be a fantastic addition to MyFitnessPal.

I find myself sometimes not logging food on that app because I couldn't be bothered to search for it.

On the other hand if the product has a barcode I always scan it in because that is convenient.

subcosmos41• 10 years ago

Same - its one of the reasons I wanted to make my own. That and data export functionality, which is requisite for research.

swah5• 10 years ago

I feel like its easy to make a better app if you don't try to please all use cases - just try to make the food logging in less taps. Logging in those apps takes me around 5-10 taps plus typing the food name. Ify ou could do it in 2 or 3, it would be great.

jcoffland4• 10 years ago

If you're eatting a lot of food with barcodes on it then your doing it wrong.

lj33• 10 years ago

I wonder if somebody would buy a "highly opinionated calorie tracker" that only has non-barcode foods stored in it and mocks you when you try to add bad food to its database. If you miss your macros for the day, it would then text everyone in your phone that you're a fat weenie with no willpower.

SolaceQuantum• 10 years ago

I don't understand this sentiment. My grocery store allows us to print barcodes that describe the weight and price of any produce or bulk order we purchase. This ranges from quinoa, oats, nuts, seeds, tea, along with the usual fruits and veggies.

Also, don't bulk packages of vegetables tend to have barcodes on them inherently?

subcosmos41• 10 years ago

I see you work for the folding@home guys. Excellent work!

dimino2• 10 years ago

All food has bar codes.

aembleton• 10 years ago

This is not compatible with my device which is an LeTV X500, running Android 5.0.2. Is there any reason that it might not work? I'm in the UK, if that might be the reason.

subcosmos41• 10 years ago

Im new to cordova (the toolkit I used to build the app), and I did notice that the number of supported devices is kind of low for my build. I may have included a plugin that narrows down the pool. I need to dig into this.

Sorry about this!

rymate1234• 10 years ago

This happening for me as well with a Moto X Play running Android 6.0.2

Looking forward to trying the app once I can install it though, looks neat

subcosmos41• 10 years ago

Found my bug! Try again. Way more droids are supported now.

lj33• 10 years ago

This is happening for me too. Samsung Galaxy S3 on US ATT running Android 4.4.4 Cyanogenmod 11.

subcosmos41• 10 years ago

I think I resolved this problem. Try again!

devstack• 10 years ago

That's very interesting indeed. Are you using any API?

subcosmos41• 10 years ago

Fatsecret API for nutritional information and Foursquare for checkins, but installed and trained the neural network on my AWS instance for the prediction stuff.

tdkl• 10 years ago

Is it possible to implement the myfitnesspal API in the way user can scan the food and it gets entered in their food log on MFP ?

aantix2• 10 years ago

I remember the latest Android camera app being able to extrapolate distances in the app (thus simulating photos with large apertures/shallow depth of field).

I wonder if you could use this visual distance algorithm to somehow judge portion sizes/plate sizes?

subcosmos41• 10 years ago

Haven't thought of that! What I have considered is letting users put something like a quarter or ID card in the field of view for that purpose.

Aside from that though, I figured Id do portion estimation down the road should this preliminary version take off.

swah5• 10 years ago

Its not that bad to let the user estimate it too.

aantix2• 10 years ago

Good thinking. Assuming you don't have the portion size data, you may be able to run your data set through Mechanical Turk or hire an oDesk worker, and you could build a training set..

zaroth• 10 years ago

Please do blog about the technical details, if you could post a walkthrough of setting up and training the net it would be great!

subcosmos41• 10 years ago

Someday! Right now I need to put the fire out in my server room. Didn't quite prepare for HN traffic ...

misiti37802• 10 years ago

where did you get the annotated training data ?

subcosmos41• 10 years ago

wordnet: https://wordnet.princeton.edu/wordnet/

aaronfc2• 10 years ago

I can't find anything related to images... Could you expand on this?

franciscop• 10 years ago

Isn't it strange that in [1] the beginning of the "Normal" BMI has more risk of death than the beginning of the "Obese"? I'd even venture to say that the "Overweight" range has, in average, lower Mortality Risk than the "Normal".

PS, another plus for the Metric System. The Imperial System form includes Feet+Inches, while the metric one only needs Centimeters for the height.

[1] https://www.infino.me/join/

subcosmos41• 10 years ago

Its been contested in a few papers now that overweight people can outlive normal weight, according to these BMI subcategories. But I am not surprised that things can be fuzzy, since ultimately its a continuum.

bjterry• 10 years ago

The finding that people with an "Overweight" BMI have a lower rate of all-cause mortality is consistent and has been called the "obesity paradox." A very interesting study came out this month that addressed it.[0] Basically, they did a prospective study of 55,000 (mostly elderly women) where they measured their body fat percentage directly using DEXA scans. It turns out that when you control for body fat percentage, it's actually better for all cause mortality to have the highest BMI category and the lowest body fat percentage. Of course, the measure "BMI after controlling for body fat percentage" is kind of unintuitive, since BMI is supposed to be a crappy proxy for body fat percentage. But basically, as long as your body fat percentage is low, being bigger is healthier than being smaller.

0: http://annals.org/article.aspx?articleid=2499472 (this paper used to be available, because I read the whole thing, but I guess it's paywalled now)

subcosmos41• 10 years ago

Thanks for posting this! I wonder if part of this paradox is also explainable because people with high muscle mass will tip towards the high end of the scale.

Very cool!

yomly2• 10 years ago

BMI is a terrible metric - it's just a function of weight and height and a terrible indicator for health - most professional atheletes would have high BMIs due to a high muscle to fat ratio (muscle tissue being denser than fat tissue). It's not abnormal to be "overweight" - "obese" at 10% body fat depending on what sport you're doing. (long distance atheletes are normally quite lean, sprinters are bulkier)

rcar• 10 years ago

It's really not a terrible indicator for health despite this argument getting trotted out every time the subject of BMI comes up on the internet. Obviously, as you pointed out, there's individuals for whom BMI is inaccurate but they represent a very small portion of the whole population. At a population level, it tracks very nicely with health outcomes.

If you're a powerlifter or a sprinter, don't change your habits just because your BMI is high. On the flip side, if you sit behind a desk all day, don't blame a high BMI on the metric being inaccurate.

DanBC• 10 years ago

The low end of normal includes people with eating disorder (anorexia used to have very high mortality rates) and some people with drug addiction.

Note that for women the low end of normal is probably sub-optimal.

jcoffland4• 10 years ago

Most people in Europe measure their height in meters plus centimetres.

zeveb2• 10 years ago

PS, another plus for the Metric System. The Imperial System form includes Feet+Inches, while the metric one only needs Centimeters for the height.

Don't be absurd. There's no reason one couldn't choose to input inches only, or decimal feet, or metres, decimetres and centimetres, or decimal kilometres.

sandGorgon3• 10 years ago

Where did you get your training data set? I'm trying to build a network to recognize faces from ID cards and match them to faces/selfies. Onenof the hard things has been to get a training data set.

subcosmos41• 10 years ago

Check out wordnet! I think they have face categories.

zodPod2• 10 years ago

This sounds like it could be used for the wrong purpose for sure.. Are you working for some nosy employer?

sandGorgon3• 10 years ago

Employer? No - we are automating paper applications for financial institutions. This includes a photo ID and a photo.

That is checked manually today.

virtualritz• 10 years ago

First thing I tried: logging /two/ Granny Smith apples manually.

I had do the entire process twice because there is no (obvious) way to change the serving size to 2.

subcosmos41• 10 years ago

If you click the 'no barcode' button to pull down a fatsecret nutrition record, there is a servings pulldown that is displayed. Its kinda hard to see. Working on the CSS

npalli• 10 years ago

The classification looks good, congrats!. Where did you get the million labeled images to train your classifier?

subcosmos41• 10 years ago

Wordnet! Got a list of URLs from there and then hit flikr with some python.

micro_cam• 10 years ago

Nice work! Another big user group of stuff like this is diabetics who need carb estimates to adjust insulin doses.

subcosmos41• 10 years ago

I earned my PhD in diabetes research at the Mayo clinic. My whole goal for this project is to cure that damned illness.

Aim high ...

natch• 10 years ago

Is there a writeup about how you approached this? All I see are a couple of app store links.

subcosmos41• 10 years ago

Ive been meaning to make a blog for some time now, since my nonprofit also has some preliminary research results with the genetic basis of obesity. Just havent had the time really. It will happen though. These neural networks are amazing and worth sharing.

mrfusion• 10 years ago

I think something like this could get really good at estimating calories too. I'm saying within +\- 50 calories.

subcosmos41• 10 years ago

I've heard that Google is working on that problem. They have way more resources than my one-man part time nonprofit operation, so I look forward to how awesome their thing will be when they finish it.

Will probably opensource this, since I want to grow up a nice research community around quantified self fanatics.

mark_l_watson• 10 years ago

That is very cool subcosmos! I am also very interested in both cooking+nutrition and AI (see for example my web site cookingspace.com). My contact information is in my HN profile - shoot me an email if you want to talk about AI, food technology and business, etc.

gchristen• 10 years ago

Hey Subcomos, great work! We've been working on something pretty similar, an app called Logameal (only available in the UK). We have run into (are running into) a lot of those UI issues too and would be glad to help. Let me know if you'd like to chat.

subcosmos41• 10 years ago

Your UI looks great!

koolba• 10 years ago

Someone should combine this AI with something like Google glass (so you don't have to pull out your phone), a basic calorie/nutrition filter, and a shock bracelet[1].

That way when I look at a Krispy Kreme donut I get immediate feedback!

[1] http://pavlok.com/hello.php

subcosmos41• 10 years ago

I absolutely LOVE this idea .... :)

misiti37802• 10 years ago

google glass is dead though right ?

subcosmos41• 10 years ago

In some niche areas it's really useful. Some people I know recently raised 17M to build software for surgeons. Looking forward to seeing more uses like this.

nacs• 10 years ago

It changes month to month.

internaut2• 10 years ago

I'm dying here. Sign me up for your startup.

Let's call it: "American Dystopia"

subcosmos41• 10 years ago

There's a trump joke in there somewhere.

Zolomon• 10 years ago

This should be in use at those self-registry machines at food markets. I imagine that you place the food item on the scanner and it will automatically register it or ask for human input when something unrecognized is found.

Would decrease cheating perhaps?

solaxun• 10 years ago

I've been thinking of this exact thing for a few years now, good to see others are interested too. I think there is HUGE potential for an app that can accurately (within a reasonable error range) count macro-nutrients through pictures of food, but I think that is WAY more difficult than you would imagine. How can you train a classifier to tell how much butter is in a dish? How can you account for food that is literally underneath other food?

Maybe a combination of spectrometer, weight, and photos would be a future solution. I certainly don't have an idea of how it would be implemented, but I think using data from all 3 of those sources could be a good step.

subcosmos41• 10 years ago

Another angle is post-consumption. Ive always wondered if precisely monitoring body temperature might tell you something about consumption, since there is this 'thermic effect of food'. Would be hard to get right though ...

In the lab, we can use doubly-labelled water to precisely measure calorie intake and burn, but its so damned expensive. If only it were commercially viable.

hokkos• 10 years ago

About post-consumption could a turd-shot help to compute the calories in/out ?

nickpsecurity2• 10 years ago

You have a business opportunity here if you train that on fresh vegetables. Many people have trouble with these issues:

1. Knowing what the hell a (example) rhubarb is or looks like.

2. Knowing if it looks fresh or about to go bad.

Being able to aim a camera slowly across an isle with it getting highlighted would solve first problem. Spinning vegetable around in front of the camera would could solve second. So, try that and make it a paid app.

subcosmos41• 10 years ago

You know, its funny. being a former fat guy, Im not entirely familiar with all of my vegetables. I took this app to a farmers market recently and it was actually helping me identify a few things ...

Had never seen a beet before un-cut. Ashamed of my ignorance ;)

nickpsecurity2• 10 years ago

I live in a semi-rural area. Gives me a bit more experience. I still don't know what all these things are in the supermarket, though. So, I can't judge. ;)

That said, you should try training it on different kinds of produce that look very similar. One example I recall is Zucchini vs Cucumber as the stem at one side is the giveaway. Another is Nappa Cabbage vs Romaine Lettuce. Only two examples I could think of off top of head that would confuse people.

nathan_f77• 10 years ago

I would pay something like $5 per day for this if it was a real person who looks at my photos, categorizes them accurately, and enters them into MyFitnessPal.

I won't trust an AI until it has a massive database and a million users. Until then, it's just not going to work well enough. You also need to convince users why they should use your app, instead of MyFitnessPal (which already has a massive database, including barcodes.)

subcosmos41• 10 years ago

I have considered something like this as a business model, although I was thinking of nutrition coaching instead of mere logging itself, but thats actually a really great idea.

jqueryin• 10 years ago

There's nothing stopping you from having a human aspect do the training and verification behind the scenes.

It's how companies such as Expensify were able to handle their receipt OCR gracefully for such a wide variety of invoices and receipts.

I'd err on the side of saying this is a more palpable sales approach to selling it as AI. Letting your customers know that while AI is running the show, things are being monitored closely and fixed by real people when misclassifications arise.

kbenson• 10 years ago

I wouldn't pay $5 a day. I would pay $30 a month. Mechanical turk, with a note that the picture needs to contain some sort of scale (and a request to manually scale the item if you didn't include one and it's not obvious), would probably take you pretty far.

dvirsky• 10 years ago

I remember there was an app like that a couple of years ago, that didn't really take off. Don't remember the name or story.

HamSession• 10 years ago

I agree with your suggestions for human assessment because of problem posed by food calories varying significantly due to different preparations.

Take for example soda, if I gave you a picture of soda in a glass could you tell it was diet or regular? You might scoff at such an edge case but it quickly becomes more common when looking into food perpetration techniques. This is why caloric estimation is a really difficult and causes restaurants to not list their calories as the calories of a meal do not equal the sum of it's parts.

All these solutions are common as people want a signal to tell them to stop eating, but these are insufficient as people will simply ignore it due to hunger cravings (as happens on diets). Any nutritionist service in addition to detailing calories would need to incentive the patient to recognize the need to lose weight or setup a helpline.

tryitnow• 10 years ago

How is that " the calories of a meal do not equal the sum of it's parts"?

That seems to violate the laws of thermodynamics.

gohrt2• 10 years ago

causes restaurants to not list their calories as the calories of a meal do not equal the sum of it's parts.

In my area, restaurants list the calories for each menu item, exactly to the extent required by law (chain restaurants are required to public calorie counts), plus some promotional "under 500 calories" or what-have-you for diet-targeting places.

logn• 10 years ago

It would be difficult to account properly for ingredients such as butter and oil which are hard to see in a finished dish and drastically affect the calories.

But people would probably still pay as long as it's a good faith attempt, since they can't necessarily do better themselves, so it would be just as accurate and save time.

And with geolocation, you could start to figure out where people are and then you'd know exactly the calories if they're at a chain restaurant.

subcosmos41• 10 years ago

Yup! The app as it stands does geolocation and factors this into account for nutrition lookups, but I could certainly do this automatically without requiring a check in.

a_small_island• 10 years ago

Impossible. How can someone tell whether you cooked with butter and oil, for instance?

wingerlang• 10 years ago

Sending notes with the image maybe? Combined with a clear estimate-based response it might be okay.

For me I would be ok with knowing weather the meal was in the 500 or 800 range.

KaoruAoiShiho• 10 years ago

How much hardware and how much time was used in training?

subcosmos41• 10 years ago

Took a single AWS instance about a month. Not too bad.

zeveb2• 10 years ago

Pretty nifty, but please consider allowing creation of an account with a simple email address, or even anonymously. I really don't want to have to connect my Facebook or other social-networking information to your app in order to use it.

Thanks for allowing one to try it out without logging in via another service; it'd be great to be able to actually have an account.

subcosmos41• 10 years ago

Email accounts are definitely on the long todo list.

doppenhe• 10 years ago

This is great. Where are you hosting your caffe model? Just AWS with API Gateway?

subcosmos41• 10 years ago

Its on the instance right now, but I could certainly post it to github at some point. I think I need to retrain it though and swap a few categories out. Theres a few animals in there ;)

Also, been needing to train on non-food items to weed out abuse.

holografix• 10 years ago

This has HUGE potential. Some of the comments here disregard the fact that AI can do a better job than humans. An expert chef looking at dishes will make an educated guess on the ingredients and quantities that went into a dish and will have a better idea of the caloric content. The AI could learn a similar skill, identifying whether a dish contains pastry, buttery sauces, fatty meat etc etc.

johnPullman• 10 years ago

Another app that is similar to this is called tuingle http://tuingle.com

halonewstza• 10 years ago

halonewstza.blogspot.com

Show HN: I built an AI that recognizes food

Related Stories