Automating breast cancer detection with deep learning

https://blog.insightdatascience.com/automating-breast-cancer-detection-with-deep-learning-d8b49da17950

by megandias • 9 years ago

65 21 9 years ago

blog.insightdatascience.com

opportune• 9 years ago

So basically they took a simpler version of software used in hundreds of medical imaging labs around the world and put it in an app?

I've worked in labs that do this exact same thing. It's cool that they're putting this in a consumer-facing format but the technology (re: accuracy in ML models) still isn't there yet, and there are a lot more performant and sophisticated models than the relatively simple network architectures the authors describe. At risk of being overly critical of the author, to me this just seems like a first, rather naive implementation of ML to be shown as a proof of being able to do ML related things, with the actual performance or usefulness of the model being secondary to that.

In short, putting this into practice at a consumer level: good. Actual modelling: maybe not so good.

jessriedel2• 9 years ago

So basically they took a simpler version of software used in hundreds of medical imaging labs around the world and put it in an app?
I've worked in labs that do this exact same thing. It's cool that they're putting this in a consumer-facing format but the technology (re: accuracy in ML models) still isn't there yet

Don't these observations explain each other? The whole point of have a consumer product is to reduce office visits and/or increase the monitoring frequency. So reduced accuracy is acceptable since any positive results would be followed up by the standard, more sophisticated tests.

They aren't trying to mimic the state of the art in a phone.

daveguy• 9 years ago

Medical diagnostics have to be held to a high standard. False negatives (the app says you're fine when you have cancer) and false positives (the app says you have cancer when you're fine) both have significant consequences.

False negative (24%) means you are not getting the treatment you need as soon as you could. Imagine people delaying a real mammogram for a few years because they have an app that says "all good!".

False positive (35%) means unneeded tests and doctors visits. This is less concerning as a bad outcome, but it does mean more stress , more expense and longer wait times to schedule a doctor visit.

This is an interesting exercise in ML, but there is little chance this app would get approval as a diagnostic tool.

hprotagonist4• 9 years ago

Sensitivity/Specificity of 0.65/0.76 are laughably horrible for a diagnostic.

I note they didn't give a ROC curve or d'.

killjoywashere• 9 years ago

Yeah, considering one presumes this is a screening test, the sensitivity is a catastrophe: a negative result means you have no idea if you have cancer or not. You spent money that could have been better spent on just about anything else (composting the dollars, for instance).

ska3• 9 years ago

There is a reason mammography is the standard for screening, not ultrasound.

Honestly the more interesting case is probably this + mammo, as there is research suggesting sensitivity increases with US + mammo, over mammo alone.

hprotagonist4• 9 years ago

I don't see why a screen you cannot trust should be a precursor to a mammogram. It just makes things worse.

wlamond• 9 years ago

The data should have been split into train, validation, and test sets before augmentation was applied. As it is now, there are likely examples in the validation and test sets that are highly similar to examples in the training set. The goal of validation should be to measure performance on previously unseen patients, not previously unseen scans of patients you've already trained on.

ntenenz• 9 years ago

Holy data leakage Batman!

Because the data augmentation is performed prior to the random train/val/test split, nearly identical instances may be found in each set. In other words, a 1 degree rotation of training image X may be found in the validation or test set which would artificially inflate performance.

amelius• 9 years ago

Is there a public dataset available somewhere?

theprop• 9 years ago

Getting image recognition to work to "replace" radiologists is both a game-changer and a HUGE money-saver and thus a GIGANTIC opportunity. Radiology is the most common specialty and one of the highest-paid in just the US -- 40,000+ radiologists earning $350k/year (so maybe $15 bn in salaries). Radiologists and thus some of the smartest & most capable people in the country and they spend the whole day just looking at images...computers should be doing this!! Not only that, computers will soon better...for many skin tumors and others, even the leading human experts differ in their opinion, so humans today out-perform computers but I don't foresee that being the case for long.

A game-changer in cancer treatment would be "early detection". Once it gets to be inexpensive to analyze images, then we can have vastly earlier cancer detection...you could work into a scanning machine in a local pharmacy, get scanned & get the analysis right there...you could go in for monthly scans. Theranos had something of that vision with a drop of blood in pharmacies which could also provide very early detection of cancer...but they didn't execute technically among other issues.

mikekij• 9 years ago

I've been looking at this space for a few years. I've come to the conclusion that one of the biggest obstacles to ML replacing/supplementing radiologists is the malpractice ambiguity. If a doctor misses a lesion on a chest Xray, the lawyers know who to sue. What if a ML algorithm that no human understands misses this. Who is liable? It's another sad example of technological progress in healthcare being stifled by bureaucracy / liability concerns.

avvakum• 9 years ago

Liability is of a radiologist is ultimately offset by malpractice insurance. Similarly, the lawyers would go for ML algorithm developers. Now I can see ML insurance business with premiums tied to ML clinical performance ... Obviously there has to be a legislation in place for this.

wst_• 9 years ago

How is that possible? Are these images so obvious as 2+2=4? Legitimate question, I am layman in this area.

swengzju• 9 years ago

hey I'm the author, sorry for any confusion, I did separate the hold out sets prior to the image augmentation. Sorry for not making that clear in the post.

lostlogin• 9 years ago

Pigeons are good for this problem too.

https://www.google.co.nz/amp/s/www.washingtonpost.com/amphtm...

Automating breast cancer detection with deep learning

Related Stories