Deep learning algorithm diagnoses skin cancer as well as seasoned dermatologists

http://news.stanford.edu/2017/01/25/artificial-intelligence-used-identify-skin-cancer/

596 279 9 years ago

news.stanford.edu

lucidrains4• 9 years ago

This is why we need a platform for these models asap. I would totally download this app today and use it regardless of what the FDA thinks.

wakkaflokka• 9 years ago

I'd love that. I'm still irritated that the FDA forced 23andme to remove the Alzheimer's/Parkinson's report (though I completely understand why). Now I have to run all my 23andme data through promethease, which is significantly more complicated to understand.

komali22• 9 years ago

Are you sure about that? Playing devil's advocate here, we have plenty of examples of scientists jumping the gun without being peer reviewed or going through a rigorous follow up testing progress, especially when it comes to medicine. The alzheimer's 60FPS blinking light example is pretty good - some scientists got it working on mice, but we don't know any potential side effects it could have on humans. Maybe none and that'd be great! Maybe it causes schizophrenia, who knows? We have no way of knowing yet! Just very very educated guesses.

I say when it comes to medicine, err on the side of caution. Obviously a diagnosis app isn't too dangerous - worse case scenario the app gives you a positive diagnosis, so you go into the doctor's, they take a sample, and find the growth to not be cancerous. No harm, no foul. But other ideas could be more dangerous.

fpgaminer• 9 years ago

Worst case scenario would be giving a false negative, wouldn't it? And that's the danger, not a false positive (though false positives may lead to higher health costs due to increased doctor visits).

crusso• 9 years ago

As you say, it's a diagnosis app. It shouldn't be judged on the same slippery slope as a treatment technique.

doesnotexist3• 9 years ago

Eric Topol puts this up there as the most impressive AI/medicine publication to date. https://twitter.com/EricTopol/status/824318469873111040

The paper ends with "deep learning is agnostic to the type of image data used and could be adapted to other specialties, including ophthalmology, otolaryngology, radiology and pathology."

romaniv3• 9 years ago

Systems that outperform doctors in some specific area of diagnostics aren't new. One of the earliest examples of such systems is Mycin [1], which also was developed at Stanford, but around forty-something years ago. Never went to production because of practical issues that have nothing to do with its accuracy. It's interesting that all of those "practical issues" are no longer relevant, and yet we don't see a widespread use of similar software.

[1] https://en.wikipedia.org/wiki/Mycin

lucidrains4• 9 years ago

Comparing apples and oranges here. Mycin is an expert system dealing with changing rulesets with A LOT of manual teaching. The current papers deals with computers discerning visual patterns by itself.

throwaway7294• 9 years ago

I disagree. Data isn't magic fairy dust, and romaniv is correct -- plenty of expert systems-based approaches did perform well in practice.

Many of the barriers standing in the way of wide-spread use diagnostic software are not technological in nature.

Ignore those barriers at your own risk.

brandonb6• 9 years ago

I wrote a bit about the systematic issues preventing promising AI research from moving to production in medicine, including a very brief history of MYCIN: https://blog.cardiogr.am/three-challenges-for-artificial-int...

I think now really is different. Part of that is algorithmic advances like deep learning, as shown in this Nature paper.

An even larger part of it is that the financial incentives are flipping due to value-based care. In 1979, a hospital that implemented an expert system for accurate diagnosis may, paradoxically, see its revenue fall. Nowadays, with ACOs, risk-based contracting, and bundled payments, the financial incentives create tailwinds rather than headwinds for large-scale adoption of AI in medicine.

Contrary to popular belief, the medical system can absorb new techniques very quickly--when incentives are aligned. And they are now becoming aligned.

leereeves3• 9 years ago

I hope someday soon we'll develop systems that allow us to "ask" a ML algorithm what factors led to a decision (diagnosis in this case).

It would be interesting to compare that with the current state of the art in the field, and see if ML can contribute new scientific/medical theory as well.

derefr• 9 years ago

I'm not so sure we could ask the algorithm itself, in any literal sense. An algorithm trained to introspect might actually be wrong about its own "memories" or "motives"—just like a human might! (Though, likely, without the penchant toward political rationalizing.)

This is most simply because, whatever the algorithm is trained to do, it's certainly trained better to do that thing than to introspect. Introspection is a separate skill!

But there's also a more insidious element: introspection (in humans, at least) tends to result in the creation of a lot of "personal concepts" that don't map to well-known common concepts. An introspection on one mind must necessarily result in a taxonomy that contains terms for the tiny, unique features that only that mind has—which makes it very, very hard to communicate one's personal introspections to others. (You might call this a kind of overfitting: the introspection capability becomes optimized for that one mind, but ceases to translate well to features in other minds—like human minds.)

I'd place a much stronger bet on our ability to train one AI to "stare at the brain" of other AIs as they make decisions [tons of them, as its training data], with the expected output being a general theory on common AI features responsible for the given calculation step. A computer psychologist, of sorts. :)

Of course, you could include such a pre-trained model as a "module" alongside the AI itself, and call the combined system "one AI" if you like.

romaniv3• 9 years ago

Well, there are ML algorithms that can be interrogated about their reasoning. One type is symbolic systems that use data to generate or "tune" their rules. Another one is stuff like this: https://www.youtube.com/watch?v=UqPcq0n59rQ (it's a kind of ensemble system that uses a linear combination of human-interpretable classifiers). Both kinds are pretty interesting, because you can manually inspect and tune them after training.

kharms• 9 years ago

Depending on the algorithm, there is this: https://homes.cs.washington.edu/~marcotcr/blog/lime/

It's a neat approach that I've used with some random forest classifiers.

iamleppert• 9 years ago

Honestly, I can't wait for deep learning and computational methods to dethrone doctors and upend the medical profession. In the next five years, expect a computer to be able to predict most diseases a lot better than doctors can -- and with none of the attitude, high cost, or inconvenience.

Mind you I'm not talking about researchers, who will always have a job. I'm talking about practitioners. I've had a medical condition from birth and I've had to deal with my share of doctors. Outside of the insurance system, they are easily the most unpleasant part of the whole ordeal to deal with. There are some gems, but most you will encounter are pompous, arrogant, and "commanding" -- when they enter a room, they are flanked by "residents", "assistants" and generally give off this air of superiority which is really just because of their route experience. The whole thing comes off more as a performance than anything else. Worse, they often get mad when you question them or ask them to explain themselves, or how they arrived at a conclusion.

Good luck finding work when an algorithm can do your job better than you. It's only a matter of time.

doesnotexist3• 9 years ago

Not so sure we are going to see an automation completely replace doctors. Just look at all the problems both technical as well as social in the self-driving cars arena. There lot more people are qualified to be drivers than to practice medicine.

Instead, I think we'll see more powerful diagnostic tools at the disposal for physicians to use. Doctors will still play an important role in treating their patients and will be more effective because they'll have powerful tools assisting them.

But to your point, will technology help patients feel more empowered in their medical encounters? Or to get more value out of their interactions with their doctors? https://www.remedymedical.com/ seems to think their platform will do just that for primary care / telemedicine visits.

dave_sullivan2• 9 years ago

Doctors will still play an important role in treating their patients and will be more effective because they'll have powerful tools assisting them.

I get into this argument with people a lot.

I say something like, "Self driving cars will replace all the transport jobs" and the pedant says, "Well they won't replace ALL the transport jobs, so argument null and void." But no shit they won't replace ALL the transport jobs, if they replace even 30% of them, that's huge. Highest unemployment during "The Great Depression" was ~25%. I think in the case of transportation it will be more like 80% fewer transportation jobs in 10 years, something like that.

With doctors and hospitals, if there were better incentives in place promoting any level of efficiency, we would have less need for doctors (and hospital staff in general). I agree that we won't see 100% automation. But a hospital finding it can do more and better work with 50% less staff? I could see that being a possibility in our near future, with the caveat that it won't happen in the US due to the regulatory environment and incentives in healthcare.

Countries outside of the US might be more open to the idea of efficiency in healthcare.

bluGill• 9 years ago

I think it will be both. They will start with software for doctors - who will use it because the computer never forgets to ask some critical question that could discover a 1 in a million disease and not the common cold (with similar symptoms) that the everyone else in the office today has.

Eventually urgent cares and ER will install them for people directly. No reason the machine can't prescribe antibiotics for the latest form of strep that is going around. That leaves the doctor for the hard cases where the machine doesn't know.

komali22• 9 years ago

I never thought doctors would be hit so soon by the automation/AI crisis, but this article challenges that thought. However, due to the state of robotics at the moment, surgeons for example aren't going anywhere for a good decade I'd estimate, and it's not like people can, in the mole example, self-remove a chunk of it for biopsy with 99% accuracy. Then there's treatment that has to be done at a hospital under monitor of professionals, etc.

_delirium• 9 years ago

I never thought doctors would be hit so soon by the automation/AI crisis

I think most people in AI are more surprised that it took this long. The tech has been there for decades for a pretty large percentage of routine diagnostics, especially carefully defined clinical-sample type diagnostics. People were pretty sure in the '80s that it'd all be automated soon, but it never managed to make it into actual hospitals so funding dried up. Mixture of bureaucracy, legal issues, patients not liking the idea of computer diagnosis, doctors not liking the idea of computer diagnosis, incentives, etc.

I think changes in all those "environmental" factors are likely to be the biggest boost to something like this getting deployed in practice. Tech advances are good as well, and will expand the range of diagnostics that can be automated, but there is already enough low-hanging fruit in medicine that is easy enough for any of a half-dozen AI techniques to do it, that I don't think tech is actually the bottleneck.

eva19844• 9 years ago

Well, with DL powered algorithm/device, people can pretty much practice self-diagnosis, and only under circumstances, where the model is predicting a higher risk, they can go to clinic to get the human treatment.

Overall AI will ease the demand for human doctors, maybe not overnight, but gradually yet quickly. Cheaper healthcare for all, thinner salary for doctors, I will say not a bad deal overall.

sremani• 9 years ago

There are not that many doctors in the pipeline to meet the demands of the future. Especially, with longer lifespans and demographics spurts in places like Africa. These are good complementary tools for medical care with the doctor "quarterbacking" while all the "blocking and tackling" is automated.

supersaiyanverx• 9 years ago

I don't have much experience with specialists, but my experience with GPs has been awful, for every good one you meet, you have to deal with 5 other terrible ones. I will never give up my GP until he retires or an AI replaces him because he is probably the first doctor since my childhood with an actual air of competency around him.

I feel bad for the rest of the people who visit my clinic and have to deal with any of the other garbage practitioners who usually fall into one of two buckets. Foreign (mainly Indian) hacks with zero medical knowledge and bedside manners, and greedy yuppie strivers with a knack for memorization but terrible analytical ability.

Most doctors don't deserve their inflated salaries or social status, and I hope they are soon brought back down to Earth by technology, they have been able to skid by for way too long.

majkinetor2• 9 years ago

This is exactly my experience. In some medical domains its even worse - I think I have never seen a good dentist. Can you even build up passion about staring at random human mouths ?

mhb• 9 years ago

Another perspective: The Heroism of Incremental Care – Atul Gawande https://news.ycombinator.com/item?id=13485354

zwieback2• 9 years ago

I feel the exact opposite - in my treatment for prostate cancer the human interaction with doctors were a hugely positive experience for me. Interpretation of biopsies and inspection of cancer images were part of that process and I'm sure machine vision algorithms could help in this area. However, even if the classification of the cancer cells improves, the role of the doctor guiding the patient through the right treatment process still remains something I would not want to turn over to an algorithm.

I have also encountered doctors I did not like but fortunately for me I had a choice where to go. Maybe machine learning should focus on weeding out unpopular practitioners instead.

transcranial2• 9 years ago

I think medicine will over time morph from a field generally perceived as an intellectual one, to a largely humanistic one, like nursing. Most people, especially the HN crowd, vastly underestimate the importance of the human touch. A doctor who possesses this can make a huge difference, but sadly the are outnumbered by those who do not.

Broken_Hippo2• 9 years ago

Ok, lets look at this stuff.

Would it have mattered to you if a machine interpreted the results and a nurse, fully versed in the specialty, had a consultation with you and guided you through the treatment process?

It seems like the bigger issue here is having some warmth of some sort in the situation - which I think is where medicine is really going, at least in the way we experience it. Computers and machines to do the hard diagnosis, with educated humans guiding other humans through. I believe we'll have less unpopular practitioners because we'll be able to spend more time teaching bedside manner and communication skills.

nilkn• 9 years ago

Actually I think moving diagnostics and other highly technical decisions into ML systems only opens up room for making sure that the humans still involved in the process are maximally empathetic towards their patients.

Right now, a really rude doctor will be tolerated within the system because of their technical knowledge, diagnostic skills, surgical skills, etc. But if all that other stuff is performed by machines, and we just have nurses, etc., there to tend to patients, explain things to them, and generally provide a human touch, I'd expect that very rude human employees would no longer be tolerated.

AndrewKemendo• 9 years ago

Holding aside for a moment current capabilities, would you prefer: Human interaction with more fault or no human interaction and less fault?

The obvious rebuttal which is common is: "why not have a compassionate human read the machine results/analysis/interpretation/prescription?"

To which I respond, how much are you willing to pay to have someone read to you?

j8m88• 9 years ago

I've been harbouring similar thoughts to you for quite some time. I live in Canada, where healthcare is supposedly "free" (it's not, you get taxed like crazy here!). The treatment you receive here is subpar in my opinion. The declining quality of treatment, I'm guessing is most likely attributed to an increasing population and the limited number of quality doctors available to treat these people.

I would assume these challenges aren't unique to Canada and from an outsider's perspective the medical system in the US seems worse (maybe not if you're rich)

amelius• 9 years ago

You should always ask your doctor for a treatment plan, i.e., a structured approach to curing your condition. Make them plan a few steps ahead. And question that plan.

eatbitseveryday• 9 years ago

an algorithm can do your job better than you

What I find people miss here is that computers will not help you heal your boo-boos. Say you get stabbed in the face with a knife. You need a doctor to help you. Or to give birth.

Gatsky2• 9 years ago

Honest question: if you have had a bad experience with a healthcare system that doesn't deliver the sort of care that you want, what makes you think that some machine learning based implementation with a human spokesperson is going to be better? How will you question the results of the algorithm?

eva19844• 9 years ago

Cheaper, more cost effective for sure.

adamnemecek2• 9 years ago

Basic income can't come soon enough.

treehau5_• 9 years ago

Basic income is a pipe dream. We need a new economy.

adamnemecek2• 9 years ago

You need an economy resilient to rapid economy changes.

nojvek2• 9 years ago

Problem with basic income is its kind of like inflation. If everyone has money and supply is same then the prices go up and we are at square one.

What we need is to dramatically increase supply and make the unit cost much lower.

Phones, cars and most commodities depreciate because next version is usually cheaper and better. Land doesn't grow so their prices go up and up.

Biggest living costs are housing, transport, education, healthcare.

kazinator• 9 years ago

Only as well? Not faster and cheaper?

calebgilbert• 9 years ago

This is not hard to imagine at all. I know that there must be some absolutely excellent doctors out there, but I don't trust the bottom 80% of doctors much at all, and honestly would rather have an algorithm most of the time, especially starting off. The lack of robust consumer level 'medical doctor apps' is one of the biggest mysteries to me.

brandonb6• 9 years ago

This is the second major study applying deep learning to medicine, after Google Brain's paper in JAMA in December, and there are several more in the pipeline.

If you've developed expertise in deep learning and want to apply your skills to healthcare in a startup... please email me: brandon@cardiogr.am. My co-founder and I are ex-Google machine learning engineers, and we've published work at a NIPS workshop showing you can detect abnormal heart rhythms, high blood pressure, and even diabetes from wearable data alone. We're working on medical journal publications now based on an N=10,000 study with UCSF Cardiology.

Your skills can really make a difference in people's lives. The time is now.

kevinalexbrown7• 9 years ago

Before neural networks got deep, there was a lot of very impressive work applying neural networks to medicine. Example:

http://suzukilab.uchicago.edu/research.htm

IIRC they were outperforming the average radiologist on some tasks 10 years ago.

selestify• 9 years ago

Why haven't they replaced the average radiologist yet then?

StavrosK• 9 years ago

If you've developed expertise in deep learning and want to apply your skills to healthcare in a startup

What if we haven't but we do?

brandonb6• 9 years ago

If you're a strong software engineer but without any particular expertise in deep learning, feel free to email me as well. brandon@cardiogr.am :)

We (Cardiogram) in particular are hiring for: mobile engineers (building an Apple Watch app used by 100,000 active users), storage and data infrastructure engineering (how are you going to store 10 billion sensor measurements those users are producing?), and machine learning engineers (now that we have the data, what can we do with it?).

In practice, many people fit multiple categories. There's a tight relationship between data infrastructure and machine learning, for instance, since new infrastructure often enables new algorithms. Likewise, building a new feature in Cardiogram for Apple Watch may give us 10x more labeled data, and therefore make the deep learning algorithm perform better. Interdisciplinary is good.

rpedela2• 9 years ago

Very interesting! What is Cardiogram's business model?

brandonb6• 9 years ago

The app is free to you, but if we can detect (say) diabetes early and route you to the right medical care, we save the healthcare system money. We've built a platform called Cardiogram Connect which optionally lets you share aggregated data in exchange (usually) for financial rewards. You'd choose to connect with whoever pays for your healthcare--depending on your situation, that could be a health system, accountable care organization, insurer, or employer--and that entity is the customer that pays Cardiogram.

nickpsecurity3• 9 years ago

I heard before that a problem with heart monitoring for ML is that most of the samples out there are with abnormal hearts rather than normal and false alarms. As in, there's not enough published data to establish a baseline for high accuracy across general population. Most claims like this cite rules protecting medical records/data. Never got to ask a specialist in the field for confirmation or rejection of that claim.

So, how true or false is it?

brandonb6• 9 years ago

True -- the available data sets (like MIT-BIH arrhythmia database) are quite small. In our case, we launched a study with UCSF Cardiology which has recruited about 10,000 people so far, and that's how we get a data set large enough for deep learning.

ksolanki• 9 years ago

Indeed, I second that the time is now. There are several imaging modalities/organs that provide diagnostic information about a variety of human functions or diseases. For example, retina is a unique organ that is amenable to imaging of central nervous system, cardio-vascular system, and microvasculature without any incision. This allows us to detect, screen, monitor, and predict risk for diseases such as diabetic retinopathy, macular degeneration, glaucoma, and even cardio-vascular risk, Alzheimer's, and stroke.

If this interests you and you have developed the expertise, there is another startup opportunity to explore -- please email me at solanki@eyenuk.com. We are bunch of machine learning PhDs, developing/publishing and commercializing deep learning algorithms for disease/risk identification from retinal photography. We play with millions of retinal images, and it's a lot of fun!

the_watcher3• 9 years ago

Telemedicine has a lot of regulatory hurdles to get to market, but initiatives like this are extremely exciting, since they can likely be taken to market in a way that explicitly clarifies that it's not a diagnostic, it's simply a low barrier way to actually get that mole you've got looked at. If you don't have health insurance, you could actually get an idea of how critical it is to get in to see a doctor. That said, the obvious concern would be the extreme cost of a false negative (although the evidence suggests that the algorithm is no more likely to provide one than a doctor, the concern over single accidents caused by self-driving cars, even when the overall rates are far lower makes it pretty clear that the bar for success to the public for non-humans is substantially higher than it is for humans)

leereeves3• 9 years ago

Are people allowed to practice medicine without a license, if they "explicitly clarify that it's not a diagnostic, it's simply a low barrier way to actually get that mole you've got looked at."

I think the same, whatever the law regarding that is, would apply to a person or company providing this service via deep learning.

the_watcher3• 9 years ago

WebMD essentially does this, except you diagnose your own symptoms.

eva19844• 9 years ago

That said, the obvious concern would be the extreme cost of a false negative

Probably not. People won't go to doctor unless they sense something wrong with their body. So it is actually filling the void here.

On the other hand, false positives will cause a bigger problem, because swarm of people will get triggered by the fear of cancer, and hospital might not handle the sudden surge of traffic for treatment.

the_watcher3• 9 years ago

I think I agree with you from a policy perspective. However, the cost of a false negative has major PR implications (just wait for the first "I tried this algorithm and it misdiagnosed what turned out to be cancer") story. While I totally agree that those stories would be entirely unfair when looking at rates, that's not how it would play out in media response (and unfortunately, regulatory response).

rawnlq• 9 years ago

There's an app used by over a million doctors called "Figure 1" that allows them to share medical images for crowdsourced diagnosis and treatment of rare cases.

I wonder when we will get to a point where machine learning can help there?

[1]https://figure1.com/medical-cases

EternalData• 9 years ago

This is going to be part of a greater trend of automation starting to affect fields considered to be white collar and paths to prosperity. I think the same is going to happen with financial analysts, entry-level lawyers etc. It'll be interesting to see the political response, especially given how charged the atmosphere has become around "preserving" jobs.

btilly• 9 years ago

This reminds me of a talk that I saw about wavelet based algorithms in the 1990s for detecting tumors in mammograms.

The algorithms found most of the tumors that humans had missed, with similar false positive rates. BUT humans refused to work with the software!

The problem was that the software was very, very good at catching tumors in the easy to read areas of the breast, and had lots of false positives in more complicated areas. Humans spent most of their effort on the more complicated areas. Every tumor that the software found that the human didn't simply felt like the human hadn't paid attention - it was obvious once you looked at it. The mistakes felt like stupid typos do to a programmer. But the software constantly screwed up where you needed skill. The result is that humans learned quickly to not trust the software.

antognini2• 9 years ago

This is very true and directly related to my research. (I work at a company developing software to interpret EEG data.) There's a huge difference between an algorithm with a low error rate that makes mistakes seemingly at random vs. an algorithm with a somewhat higher error rate whose mistakes are at least comprehensible. A doctor is much more likely to trust the latter than the former. Almost as important as developing a detector with a low false positive rate is developing a detector that can figure out when the problem is too hard so it knows not to even try. (And it seems that this problem is just has hard.)

One of the things we do is perform a Turing test of sorts where we test if the performance of our detector is statistically indistinguishable from a human. (In fact, we actually have a contest running right now where we give you 10 EEG records, some marked by humans, some marked by our software, and if you can figure out which were marked by which we'll donate $1000 to the American Epilepsy Society.)

kafkaesq• 9 years ago

A significant finding, to be sure. But like the paper itself says:

Here we demonstrate classification of skin lesions using a single CNN, trained end-to-end from images directly, using only pixels and disease labels as inputs.

What they achieved was algorithm to classify skin lesions - not perform a "diagnosis" of the overarching pathology, i.e. skin cancer.

kevinalexbrown7• 9 years ago

One major, major advantage that medical imaging has for deep learning is the similarity of each data point, especially the 'background data.' For instance, human brains typically look very similar across individuals (up to scanning parameter differences), except in the abnormalities - which are often precisely what you want to highlight.

As an example, I recently trained a neural neural network to perform a useful task for our lab using 3 (!) hand-labeled brains.

jpgvm• 9 years ago

It's insane that you were able to get reasonable results with such a tiny dataset.

I am learning machine learning right now and I find working with datasets with fewer than 100 examples to be quite difficult.

It seems counter intuitive when you first think about it but having way more data actually makes the task of fitting the model much easier as there is granularity that can be used to get feedback on adjustments to the structure of the model.

kevinalexbrown7• 9 years ago

It was an image segmentation task, and the features were similar across data sets. The other thing that made it work well was heavy use of data augmentation that captured ways in which different data points could reasonably differ.

There was a really cool medical imaging paper recently that literally just labeled several 2D slices in a 3D dataset consisting of 3 images and performed a reasonable segmentation:

https://arxiv.org/abs/1606.06650

lscholten• 9 years ago

Great results! Deep learning has been gaining track in other medicine areas as well.

One such task is lung cancer nodule detection from CT scans. A paper I recently co-authored applied many different architectures to this detection and achieved very good results. (https://arxiv.org/pdf/1612.08012.pdf)

The best combination of systems detected cancer nodules which were not even found by four experienced thoracic radiologists.

michaf• 9 years ago

Do you participate in the current Data Science Bowl regarding CT lung cancer detection [0]? The prize pool of $1,000,000 seems quite attractive, especially if you recently developed new state-of-the-art CT lung cancer detection ML models. The only somewhat strange aspect of this competition (at least to me) is to not include locality annotations. They only provide labels of cancer/no cancer per patient...

[0] https://www.kaggle.com/c/data-science-bowl-2017

bluenose69• 9 years ago

What about 3D aspects? The word "bump" is used in most descriptions I've seen online, although I don't know if that is something the doctors consider or just something that's enough to suggest a visit to the doctor.

zxcvvcxz• 9 years ago

Quick, someone tell me why doctors won't be obsolete in 20 years!

Geoffrey Hinton believe that we should stop training radiologists now:

https://twitter.com/withfries2/status/791720748624797697?lan...

sedachv6• 9 years ago

That is the kind of prediction that people will look back on and say "I can't believe the hubris." MYCIN had better diagnostic performance than infectious disease experts by 1979 (https://jamanetwork.com/journals/jama/article-abstract/36660...) and in the 1980s the question was "How soon will expert systems replace human doctors?" Putting a number like "5 years!" is asking for disappointment.

transcranial2• 9 years ago

Unfortunately the paper is in Nature, paywalled, instead of Arxiv, and data/code/model/weights inaccessible. While publishing in Science/Nature/NEJM/JAMA is definitely the right approach for deep learning to gain validity in the medical community, faster progress could be gained by having a more open platform, with constant and real-time validation with more data, more medical centers and clinics. The reason progress in DL has been so breathtaking is in no small part due to the culture of openness and sharing.

rikelmens• 9 years ago

sci-hub.cc is your friend.

alexmlamb2• 9 years ago

That's true - but the principle still matters and scihub may not be around forever.

drfritznunkie• 9 years ago

Coming from a family of people in the medical professions, they've all seen reports of how _everything_ is going change in their fields because some new computer program can do X...

To which my father usually mutters something like: "Why fuck are they wasting their time with that? Can't they fix the fucking medical billing system instead?"

Most of the medical professionals I know echo similar sentiments.

majkinetor2• 9 years ago

Meh, seems majority of older folks say the same BS.

rscho4• 9 years ago

As MDs, I think it is very clear that all of us who understand even the slightest about computers and tech see that machine learning is the way to go. Medicine is ideally suited to ML, and in time, it will absolutely shine in that domain.

Now for people eagerly awaiting the MDs downfall, I think you are precipitating things a bit. We all tend to believe in what we do, and I concur in saying that expert systems will replace doctor judgement in well-defined, selected applications in the decade to come. But thinking that the whole profession will be impacted as hard as factory workers, with lower wages and supervision-only roles, is not realistic. What will be lacking is the automation of data collection, because you seem to underestimate by far the technical, legal, and ethical difficulties in getting the appropriate feedback to make ML appliances efficient. I firmly believe in reinforcement learning, and as long as the feedback system will be insufficient, doctors will prevail, highly-paid jerks or not.

I myself am an anesthesiologist, a profession most people think of as a perfect use case for those techs (as I do), and wonder why we haven't been replaced already. The reality is that the job is currently far beyond what an isolated system could do. We already have trouble in making cars follow the right lane in non-standard settings. I hope people realize that in each and every medical field, the number and complexity of factors to control is far greater than driving in the right lane.

People who drive the medical system have no sense of technology. They cannot even envision the requirements for machines to become efficient in medicine. That is why we are seeing quite a lot of efficient isolated systems pop up, but we won't be seeing fully integrated, doctor-replacement systems for a long time. This will require a new generation of clinical practitioners, who will understand how to make the field truly available to machine efficiency.

dev1n• 9 years ago

After a procedure (unrelated to my heart), the anesthesiologist overseeing me suggested I go see a cardiologist. They noticed some tell-tale signs of a specific heart disease while they were monitoring the EKG readouts. I'm happy to say that anesthesiologist made my life a hell of a lot better because of that suggestion.

Not something I would have expected a machine to tell me after a procedure completely unrelated to my heart.

rifung• 9 years ago

But with sufficient compute power, couldn't you just have the machine equivalents of a cardiologist, anesthesiologist, neurologist, etc all looking at your information at the same time?

It seems easier to scale compute power than to scale doctors considering how long doctors take to train.

Of course, being able to get a machine to a doctor's level is not easy!

newyankee• 9 years ago

Why do you think so, if you have the right data with many different algorithms running on it (even the one you are not interested in) i am sure it will uncover those. I am not doubting the hunch your Doctor felt due to his expertise was useless, but with continuous data ingestion and analyses coupled with as granular as the features can get this can only improve

In fact complicated algorithms with so many streams of data that are probably not directly shared by specialists until it is needed may uncover new things even before traditional Doctors spot it

monkmartinez2• 9 years ago

A paramedic could see the same things on the EKG. It doesn't take much training to see bad rhythms on the monitor.

autokad• 9 years ago

a benefit to ML is that they can check for everything. equipment isn't here for that yet either, but I imagine it will be.

I'm assuming the 'data' for your condition was apparently in the EKG, thus there is no reason in the world an ML shouldnt have picked that up. Where as with humans, maybe you got a tired, new, or bad anesthesiologist.

if you said they made the determination by something other than the EKG, I would be more convinced the 'machine' would have missed it

digler9992• 9 years ago

but it should (unless the medical / insurance industry pulls their shenanigans - which is a real risk) be a lot cheaper to have a machine examine you. just like we have blood pressure machines at the drugstore. we could have checkup-supplement kiosks that could check your vitals cheap/free and tell you the same thing.

what I'm excited for with AI + medicine is how many poor people will be able to get _some_ level of healthcare where they were previously getting none. These programs aren't free, but once they exist they are cheap to run. so people in 3rd world countries can get their films read, or have an AI dematology program look at their skin.

cynically though, I worry that once these AI programs get perfected they will only exist behind the worlds toughest paywall and insurance will just charge you the same fee to see a real doctor. they'll not let it run films on poor african villagers because they have to "protect their IP" or some bs.

Otp1111• 9 years ago

ECG machines already print likely diagnoses.

NegatioN• 9 years ago

I quite rarely see people get flamed for providing a well-formulated opinion on HN, so I don't think you need to preface your piece with that.

I mostly agree with your post, ML has great potential to augment and assist doctors in making the right calls. Paraphrasing Jeremy Howard, "Doctors will be skeptical of these tools coming to take over their jobs at first, but will learn to trust them more and more when they see the kind of predictions they make, thereby slowly but steadily leading to their adoption (as an enhancement or tool) in many areas."

nextos2• 9 years ago

The problem with deep learning right now is that there's not a great deal of methods outputting a nice estimation of uncertainty in predictions. But this will become less of an issue as they eventually converge with Bayesian methods, or at least if probability is introduced in a more principled way.

tma-1• 9 years ago

This might be a dumb question, but why not use the classification accuracy score as a measure of uncertainty?

dfabulich2• 9 years ago

What will be lacking is the automation of data collection, because you seem to underestimate by far the technical, legal, and ethical difficulties in getting the appropriate feedback to make ML appliances efficient. I firmly believe in reinforcement learning, and as long as the feedback system will be insufficient, doctors will prevail, highly-paid jerks or not.

We're already seeing a significant rise in the role of nurse practitioners at the front line of medicine. Today, they gather the data and hand it to an MD, so handing it off to an ML system would be straightforward.

rscho4• 9 years ago

You overestimate the data gathering capabilities of the medical system by a huge margin. Even in seemingly objective parameters, you will see: missing data, reporting details dependent on the gatherer, a substantial amount of inaccuracy, etc. etc. Plus, you are describing people who barely know how to use a web browser. And you should also take into account that people driving the system don't even see the value of expert, scientific educated consultancy in those matters. For example, I was just asked to perform "data mining" on a dataset of 150 observations, 400 variables, and ~20% missing data by a renown professor. He told me I was bound to find interesting things in view of his ~400 back-to-back t-tests.

No, I am quite confident in that ML will really take off when data fed to those systems will be automatically gathered.

revelation• 9 years ago

The difficulty is not "following the right lane", it's the huge vision problem. Analog computers can follow a lane perfectly fine.

If your profession only relies on reading a bunch of discrete values from monitors, it's a prime candidate for automation regardless of the complex process that derives the values.

rscho4• 9 years ago

I think I do understand machine vision, since I just published a paper using it. If my figure-of-speech does not convince you, I encourage you to step foot in an operating room if you get the occasion. I guarantee 100% that you will be unable to implement a system as efficient as the attending doc. Because numbers are just part of the job. You already see systems advertised as more efficient than anesthesiologists. Until we see those systems tested in a real, usual OR setting (meaning, hopelessly chaotic), you should take those efficiency statements with a grain of salt.

zdean2• 9 years ago

MDs also have the benefit of an artificial monopoly (vs factory workers) that I imagine will fight hard to keep its members employed.

https://www.cato.org/publications/policy-analysis/medical-mo...

abakker• 9 years ago

Calling medical education and licensing a monopoly is pretty intellectually dishonest. To borrow the OP's analogy, that is like saying drivers with licenses have a monopoly on driving. Seems like a pretty strange argument.

Certainly, the medical industry does not face the same competitive pressure from outside, but that is the nature of industry in general.

Rather than calling medicine a monopoly, it is more accurate to accuse heal care providers and practitioners of colluding to ensure prices are high. Monopolies by definition cannot collude since there would be nobody to collude with.

In the grand scheme of things, I believe medical costs are accurately priced and it is people's wages that are not adjusting correctly.

bluetwo3• 9 years ago

I agree with both sides here. AMA protectionist? Yes. Absolutely. That is their #1 goal even if they don't write that one down.

But I'm not sure opening up the gates is exactly what we want either. There are countless examples of charlatans trying to push the line here, and as a consumer, it would be too difficult, especially while fighting a serious disease, to tell who can really help me and who just wants my money.

There are better ways to make healthcare more affordable and more effective than to throw away the idea of proper certification.

For instance, some states (like the one in which I live) have granted more powers to nurse practitioners. I support this move.

delegate• 9 years ago

but we won't be seeing fully integrated, doctor-replacement systems for a long time.

I see your point and agree - if you look at ML and computing in general as a tool useful within the current context of medicine.

But I think the whole field of medicine (or a big part of it) may become obsolete - because the change will be driven by technology, not medicine.

By monitoring people's vitals in real time - think wrist bands which can do blood tests (+ urine, sweat, sperm, etc), feed that real time data to ML algorithms, train it on billions of people and you can accurately tell ahead of time when a person is getting sick. Next day the drone brings the medicine.

Doctors will still be involved of course, but their role will be mostly to confirm whatever the algorithm has decided..

So I think we'll be seeing a move to preventive medicine / real-time monitoring during the next decade or so and within our lifetime people might not be involved in medicine at all.

That is, if we have peace and things go according to plan... Which is a long bet of course.

apathy4• 9 years ago

Nice theory but acute medical emergencies are often stochastic. Let me know if you get to the point where you can predict a pulmonary embolism in members of the general public with enough accuracy to dispatch an ambulance.

Preventable conditions are often a combination of genes plus environment plus luck. Trouble is, the "luck" part dominates more often than we'd like. This is why preventive medicine is often accused of being tremendously arrogant.

We've got a long ways to go, and it would help to focus on the highest marginal benefit items. (We were supposed to have flying cars, and all I got was the sum total of Hunan knowledge on my phone... not a bad trade, on balance)

rdlecler1• 9 years ago

If anything it could put more MDs to work as problems are identified at home and escalated to an MD when there is an issue.

smrtinsert• 9 years ago

I don't think anyone is eagerly awaiting it (unless they were horribly misdiagnosed which is sad reality), but there is an inevitability to it.

It's not a matter of if, its a matter of when. Greed will be the impetus, and eventually quality catching up will be the nail in the coffin for traditional medicine.

apathy4• 9 years ago

You present this as binary: doctors OR ai.

I propose that the doctors who use ai to make themselves damned near infallible (by catching the corner cases where the ai is poorly trained) will make out like bandits, just like top lawyers. The ABA is a monopoly, too, and legal cases are perfect fodder for RNNs. Yet good corporate lawyers still make money faster than they can count it.

Meanwhile, shitty lawyers are out of a job.

Lather, rinse, repeat, but with a million times more regulation and inertia.

agumonkey• 9 years ago

I'm not for death of jobs, but I'd love to have a tad more information, a tad sooner, rather than relying on luck with which doctor is experienced or not, or wait until the best specialists are available.

If ML unleash a global increase in diagnostics for all MDs than it's even better.

bluetwo3• 9 years ago

(In my opinion) Doctors have done a good job of protecting them selves from technological disruption by only allowing it to be used where it is profitable to them (billing) and not where it would threaten them (diagnosis).

Even the concept of electronic medical records has been resisted, and is currently poorly implemented. The gov't had to literally bribe doctors to convert to electronic systems and most seemed to go out and get the least poorly built products on the market.

I'm not trying to paint them as evil. Heck, even NASA scientists were suspicious of computers taking their jobs. I see this in other fields all the time.

Seeing the potential and seeing the current state of affairs kind of makes me sad. I want to live in the world where a balance has been reached. Where my medical record is a salient form of AI that checks if I lost those 10 pounds I promised and loops in the doctor when my flu symptoms linger too long. I want it to eliminate the hassle for doctors, hospitals, insurance companies as well as patients, and I think in the process it can drive costs down and raise quality of life.

jrowley2• 9 years ago

Doctors aren't the ones making these decisions - hospital administrators generally are choosing what to invest in. Most MDs I know are excited about research and not necessarily afraid entertaining some amount of change.

lostdog• 9 years ago

Doctors protect themselves from technology changes by forcing themselves to be included in any technology change. Get blood tests? You can't get the results until a doctor looks at it. MRI? Same. ML will provide another batch of tests for them to order, and if they work better, that's great for the patient, but you'll still need to make an appointment with the doctor for ordering test, and another for diagnosis.

Whether this is a good thing or not is open to interpretation. Doctors make mistakes, but people often catastrophize their test results.

jazoom12• 9 years ago

I think this is way off. Doctors generally actually just have no idea about technology. I'm a programmer and a doctor so all I ever see is ways we can improve things with technology.

I've never seen a doctor shun technology to keep themselves relevant (I've no doubt it will become a thing more in the future).

georgeecollins• 9 years ago

I don't think that is fair. I think Doctors would love to use a technology to help them with diagnosis, particularly if it can corroborate an uncertain diagnosis. But I think there are a lot of liability problems with systems that try to provide a definitive diagnosis, as well as issues of patient trust.

deepGem2• 9 years ago

I had come to similar conclusions about doctors but after interacting with a few doctors closely, my perspectives changed. This is in India though where the regulatory environment is a lot different.

Per my interaction, the best doctors will do whatever it takes to improve their efficiency. Their biggest problem is the sheer volume of patients. Everyone wants to go to the best doc, and word spreads pretty fast. If you start doing good, very soon you are swarmed with patients 24/7. So any tool that can speed up their patient handling time is welcome. This is not a scheduling problem. People built a ton of scheduling tools and those tools have done next to nothing to move the efficiency needle.

What's needed is, what you said, a diagnostic tool with automated feedback from patient's activities and dramatic physiological changes. The best doctors that I know will take it up in a heart beat.

danso• 9 years ago

I hope computational thought somehow makes it into a MD's curriculum. It always depresses me when people see automation as a zero sum game against humans, instead of seeing automation as something to empower human practitioners.

scarmig• 9 years ago

My issue with doctors:

Recently, my dad was sick with a pretty bad cough. Like, so bad that he couldn't speak without coughing. He fainted twice from minute long coughing fits, one of those times hitting his head his head on the stove on the way down, leaving a deep cut and blood everywhere.

He went to at least three different doctors. He got a scan of his chest. Everything looked clear, and all of the doctors were stumped. Things were pretty bad.

I mentioned this to a UCSF resident friend, and her immediate response was "Oh, is he on <some blood pressure medication I forget the name of>?" I was like, uh, let me see. Called my mom, she checked, and, lo and behold, he was on it. So his doctors took him off it and within a week he was better.

This coughing wasn't some obscure side effect of the medication she knew through sheer brilliance: it's a side effect that's been widely known since the 1970's. Hell, it was on the drug's Wikipedia page.

So there's a couple morals you could take from this. One would be, wow, doctors are smart to be able to diagnose an issue based on a single symptom and some reasonable assumptions about a patient's background! The other is that the median doctor is pretty worthless; spending tens of thousands of dollars gives you no guarantee you'll see someone competent; and that a medical system that relies on you grabbing drinks with a UCSF resident to get good results is fundamentally broken.

Machine learning and expert systems don't have to be as awesome as the best doctors to be valuable. They don't need to be better than competent doctors, even. They just need to provide a bare level of competence to provide a huge amount of value.

cdelsolar• 9 years ago

Atenolol? I'm not a doctor but I'm reasonably sure it's this.

inopinatus• 9 years ago

You didn't say whether the medication was declared at the initial admission. Nonetheless the thing I took away most from your remark is a reminder to be aware of side-effects. Or more broadly, a reminder that patients are participants in their own treatment, and that the existence of doctors doesn't excuse us from responsibility toward ourselves & those we care for.

jazoom12• 9 years ago

That's quite a lot you've taken away from one anecdote.

ACE-inhibitors are well-known to cause a cough but not like the one you're describing.

If I was to guess I'd say he had a respiratory tract infection that magically disappeared (as they do) not long after ceasing the medication. This is a common type of scenario where lay people get confused about correlation and causation, and is one of the reasons you need doctors to help you.

Even if it was the medication, like I said, this sounds like a somewhat unusual case.

You'll probably spend the rest of your life thinking the doctors didn't know what they were talking about, but in my experience with doctors (I have a lot of that since I'm a doctor myself), even the "median" ones are far better at diagnosing things than lay people.

cheez• 9 years ago

It's not doctors. I am working on software for the OR right now and I had never taken a biology course. The amount of stuff you need to know is IMPOSSIBLE. No, it is literally impossible. The human body is beyond ridiculously complicated and this is why most doctors pattern match for everything they've seen before.

This is why, if I were an MD, I would be latching on to ML as quickly as I possibly could. There is a TON of money to be made here, sadly, I do not have the domain expertise to make it, though I am picking it up pretty quickly.

kevinalexbrown7• 9 years ago

the median doctor is pretty worthless

This seems fairly disrespectful. I suppose you mean that the median doctor isn't perfect, but do you have any idea how many hours residents put in? You might think carefully before calling the median one worthless, and ask whether they're really, truly wasting their lives in the service of others.

Less emotionally, I find it hard to believe that half of doctors proved essentially no benefit.

it's a side effect that's been widely known since the 1970's

Guess how many side effects have been widely known since the 1970's?

For clarity: I am 100% in favor of thinking critically about a doctor's advice, having received contradictory diagnoses before.

marcell• 9 years ago

A recent MD friend of mine mentioned that the standards in medical school are much higher today than 20 or 30 years ago. While this is pretty anecdotal, you may have better luck with doctors who graduated more recently. Perhaps someone with experience from medical school can comment.

Separate from that, I've had similar experiences. The median doctor I've interacted with isn't able to put in effort to really understand an individual case, and it's very easy to get overlooked or ignored. I've run into a few exceptions but that is the general rule. Source: wife had pregnancy with complications, young children.

colomon• 9 years ago

Was it Lisinopril? A cough is a recognized side effect of it, but I think something as severe as you are describing would be fairly rare?

While I think you might be overreacting a bit here, I have a similar story. I had an irritated, gunky eye for the better part of a year. Was bounced around multiple doctors, including one specialist, and had all sorts of really annoying treatments I was supposed to do. Finally got bounced to a specialist at a major teaching hospital. Took him five minutes to correctly diagnose me, and the major part of the fix involved instantly stopping all of the things recommended by the other doctors. (The rest of the fix was simply to make sure my eye was protected while sleeping.)

So yeah, I think there is a lot of value that might be added by an expert system that is familiar with the slightly more exotic things that can go wrong with your health.

naveen99• 9 years ago

Angiotensin receptor blockers can give you a cough. Almost all drugs have side effects. You got to read the drug information yourself and watch for known side effects. Start one new medicine at a time so you know which one is giving new side effects. By the way, reducing salt intake is amazing for lowering blood pressure.

Itsdijital• 9 years ago

I just want to add something really important onto this.

ALWAYS READ EVERYTHING YOU CAN ABOUT DRUGS YOU ARE PRESCRIBED!

Sorry for all caps, but it is super important. Not that your dad is in the wrong, lots of people have justified (to a degree) trust in their doctors. However doctors are people and just by that alone aren't perfect.

A few years ago my doctor prescribed me an antibiotic for an ongoing illness I had. I read the entire pamphlet for it and did some reading online about it. All before taking it. Turns out it can cause seizures if it interacts with polypropylene glycol, one of the main ingredients in e-cig juice, which I use daily. I had told my doctor I use an e-cig.

Really I cannot stress how important it is to be knowledgeable about the drugs you are taking.

Shinkei2• 9 years ago

Really happy you wrote this anecdote because it illustrates a point that physicians are arguing about a lot right now--how to measure quality care. Is it readmission rates, is it 30-day mortality, is it getting all the required screening exams for each patient... we don't know. In the US, the CMS is trying to mandate many of these and it's really hard to measure quality of care.

For example, I am a radiologist. Probably 90% of my cases are mundane and are either normal or have 'easy' pathology that I can readily detect and quickly report. Another 9% is a mixed bag of things that take a lot more time--something I need to reference or think a bit more about or possibly show a colleague. And then there's the 1% that is truly a 'make' or 'break' case. My sub-specialty training and experience can really shine in these situations and I can easily dispense a diagnosis or make a 'tough call' where another might equivocate or defer to more imaging or a follow-up. I like to think I'm not being paid for that 90%... I'm being paid for the other 9+1% That's where I truly add value in the system. But how would I measure my 'success' rate when even the best eventually make mistakes given a long enough time and there is often not a gold standard for diagnosis or even long term follow-up/resolution to many of the tough cases I've seen.

Studies have been done where experts review a sampling of cases and that's essentially what we do now for quality control--we randomly review a few prior exams for a case we are reading and then submit feedback based on our opinion on the same case. In these studies, trained radiologists do very well and make clinically significant mistakes only rarely.

But here's the problem. Let's say we use machine learning to interpret a scan, something like a CT of the chest to look for a pulmonary embolus. On that scan, the machine may see an incidental pulmonary nodule. Well that's fine, we have good data on how to follow-up on those and what to recommend. Well, what about an anterior mediastinal lesion? Now it's not as clear. The differential is wide and dependent on age, sex, symptoms, history, etc. Let's say we build that logic tree, let the machine learn from 1000 anterior mediastinal mass cases with tissue diagnosis. Well guess what, we do those studies all the time and it turns out that NOTHING is 100% sensitive AND 100% specific. You have to sacrifice one for the other... it's a balance. So, you would need to build ROC curves for every possible finding on every scan and decide that 1% of the time you will miss a cancer to save having to biopsy an extra 20 people... or maybe you want to miss cancer only 0.1% of the time, but that means you'll have to biopsy an extra 200 people and one will have to be hospitalized from complications. You think they will be happy knowing you made that decision? Guess what, we are already doing that with Mammography. The screening guidelines that the different societies and agencies argue about are this very rationale... how many women should be allowed to die to prevent those extra biopsies, false positive workups and all the things that come with that. I welcome machine learning into medicine. Guess what, it's destined to be mired in the same ethical dilemmas we face everyday.

I have a feeling I'll make it to retirement.

copperx• 9 years ago

In the doctors's defense, most people know that blood pressure medication may cause coughing. At least most people with hypertension.

Your dad's case is like complaining about constant nosebleeds while taking Aspirin every day. It's a bit obvious, to be honest.

dbbolton3• 9 years ago

"Oh, is he on <some blood pressure medication I forget the name of>?"

I'll bet the farm it was lisinopril, or another ACE inhibitor. The ACEI cough is notorious enough that any 1st year medical student probably should have been able to piece this together.

The problem is not that those doctors were ignorant. It's more likely that they did not ask the right questions to get a proper history. You should ask every patient to list their medical problems and medications unless it is a routine follow-up when nothing has changed.

Something else that can happen is patient ignorance (and I am not insinuating your dad was guilty of this, or making a value judgment on people who are).

Often times you can't get a proper past medical history or medication list unless the patient brings all of their prescription bottles to the office. E.g.:

"What medications are you on?"
"Well, I'm on a sugar pill, a water pill, a cholesterol pill, a stomach pill, and some allergy medicines."
"Ok, do you know the name of that sugar pill?"
"It's a little white pill. I think it starts with an 'F'. No wait, I'm thinking of my water pill."

"Do you have any medical problems?"
"No."
[glances at meds list] "Ok, so why do you take metformin, hydrochlorothiazide, atorvastatin, omeprazole, cetirizine..."

throwaway7294• 9 years ago

Anesthesiology is one of those things that you'd think even classical control theory would be able to do given enough effort. You'd think... but not so! Apparently jobs that people go to med school to learn how to do require tremendous amounts of effort to automate. Who would've thought? ;-)

ZoeZoeBee• 9 years ago

Anesthesiology is one of those things that have already been automated, even if you're unaware. The problem with anesthesia via robot is not a technical one at this point, rather it is an example of the current practitioners doing all that they can to preserve their way of life.

https://www.technologyreview.com/s/601141/automated-anesthes...

https://www.washingtonpost.com/business/economy/new-machine-...

nickpsecurity3• 9 years ago

For a counterpoint, see the answers on this Quora:

https://www.quora.com/Why-is-machine-learning-not-more-widel...

Medical diagnosis in the general case, esp with mis- or disinformation from patients, is quite complex. The data sets available aren't that good given the built-in biases & missing data. What we're seeing is deep learning can help when it focuses on one, little thing with a ton of good data available while ignoring everything else. That's what MYCIN did back when this concept started. That's not enough to replace MD's any time soon.

What you'll find is we can at best supplement the decision-making practices of MD's by running their data through a bunch of ML systems in parallel to try to suggest things they might miss. This data will overtime feed into the ML systems to improve them. Augmented Intelligence, not Artificial Intelligence, will remain the best way to do things due to all the stuff in doctors' brains from their professional experience that's not in machine learning datasets.

shamino• 9 years ago

I was about to comment about this topic, but you said everything I wanted to! Have an up vote!

BaronSamedi• 9 years ago

One of the things the old expert systems like MYCIN could do was answer the question "Why", as in, why did you come to that conclusion? And it would show its reasoning. I may be mistaken but I don't think that the new, neural net based systems can do that. Interesting regression if true.

adamsbriscoe• 9 years ago

Pretty soon, what happened to anesthesiology (one physician managing multiple extenders managing lots of technology in multiple places) is going to happen to the rest of medicine.

Physicians will adapt by carving out even narrower niches ("I only specialize in the LEFT eye..." ;). But I really don't see a they-took-our-jobs moment until people are allowed to file lawsuits against AI. Companies will never put out an AI M.D. in the wild unless there's a flesh and blood counterpart to shoulder the liability (+1 job right there).

Don't get me wrong though, I eagerly anticipate the day when going to the doctor/machine is not such a miserable soul-sucking experience... (I'm a physician.)

pokemon-trainer• 9 years ago

I hate all physicians, except the one who saved my life.

ta_donk_gt• 9 years ago

In my opinion, the AMA and the FDA are the main blockers of machines replacing doctors (outside of surgery). Nurses can do, and to a large extent already are doing the majority of the data collection. Beyond that, the doctor's main differentiable skill is pattern-matching and decision making based on experience. Machines with mountains of data generally perform this function much better than humans.

That said, I do agree that we likely won't see the full potential of replacing doctors with machines in the near term simply due to the political and regulatory hurdles, as well as the need to overcome the fear of the public and address the very real and significant privacy issues.

As to anesthesiology, I've actually done some work for a startup in that space working on a system that could very easily move anesthesiologists to a more supervisory role where they would watch (say) 50 surgeries at once (surgeries being handled by software tied into all monitors, as well as EMR systems for history and physical data, and able to dispense chemicals as needed) and step in if and when there was a problem. Technically, there is not much really stopping it, but politics and regulations present significant impediments (and probably rightly so, for now).

slv77• 9 years ago

The majority of anesthesia is routine sedation of healthy people for routine things like colon screenings. Those types of procedures are ripe for automation with systems like J&J's Sedasys cutting costs from $2000 to $200 a procedure. It was pulled off the market last year due to anesthesiologist opposition rather than problems with they system.

It seems that the barriers to automation may be more the result of strong professional associations rather than specific complexities of the job.

Thoughts?

oddstorms• 9 years ago

Actually, machines will never replace human judgment. It's absurd to even think machines are capable of such a thing. Machines are tools for humans to use. If your job is adding numbers or stacking boxes or something, sure. But anything beyond the basics is science fiction or delusion. A machine will never even approach an understanding of psychic continuity, for example.

Now, I'm absolutely not saying that we can't engineer solutions in the medical field. You know more than I do about what diagnostic evaluation takes and how procedures are performed, so I don't pretend to understand that aspect perfectly. What I do know for sure is that a machine cannot and will never replicate human thought and decision making and cannot replace a doctor ever until the end of time.

Also, you bring up a good point about salaries. Doctors are important and do great work but deserve to make no more than double or triple what the lowest paid person in their society makes. It's time to bring lawyers, doctors, accountants, developers, engineers—the professional class—back down to reality with the rest of us.

sAvi0r• 9 years ago

i honestly can't tell if this comment is a parody of something/copypasta or not tbh

Moshe_Silnorin• 9 years ago

Yes, you are safe because of licensure and regulatory capture. Hardly seems fair to the factory workers.

qwrusz• 9 years ago

Don't worry Doc, your job is safe. Machines replacing high-touch specialized professions like doctors is not the risk here. The bigger immediate risk may be from machines making these jobs much much shittier, and then there's fallout. You won't get fired, but you may want to quit.

MDs are not factory workers. First you have the AMA protecting you. Second, there appears to be a very strong preference by humans for medical care to involve another human being there providing the care - in person.

There are other professions where machines are already much better than humans and the need for humans to even be there in person is debatable. Airline pilots for example. Of course, while a computer can fly a plane better than a human and planes can be flown remotely, it will take much longer for passengers to accept the idea and get on a plane without a human pilot in the cockpit.

The effect of machines for pilots already highlights risks to come. Humans still need to be there in person, but they almost never need to do anything but sit there. The job is to be a warm body and tell people to look out the window while they fly over the Rockies (seriously, take off to landing no human is really needed to fly anymore - pilots do nothing 95% of the time).

How did machines taking over the cockpit affect how the pay, benefits, competency and job satisfaction of pilots took a nosedive? Did you see the movie "Sully"? Where a human did step in over the machine, he almost got crucified even though it was the right move he only just barely was able to convince others of it.

Is it possible machines won't replace doctors very soon but could soon start to lead to lower pay, less trust, less prestige (here come the nurse practitioners at least), less need to use your brains at work and less meaningful interaction with patients...? And when a machine can't do something in those rare cases, will the doctor be ready for that? Think the best and brightest still will want to do this? Doctors already are not fans of that thing where patients come in having looked up their symptoms online, well they are about to do that 10x and be more right than you.

Total guess here: how likely is it that we see the best and brightest increasingly go into medical research instead of practice medicine? As the corporate pharma kickbacks you guys get can't make up for the low pay and shitty shit job conditions machines have created for doctors, while nurses and B-team quality doctors take over the actual physician working with patients stuff?

sekou• 9 years ago

Even though diagnosis is only one piece of the puzzle, what I would hope is that this becomes part of the answer to the high cost of healthcare.

jwtadvice• 9 years ago

In my opinion the way to stage these technologies is not to blitz toward a fully cyborg doctor replacement, but to bolster the capabilities of the doctor with new technology - similar to how calculators did not replace mathematicians (despite historical headlines suggesting this would happen).

Giving a doctor the ability to get a "second opinion" fast and cheaply to a patient is a large boon to medicine, and shouldn't be underestimated. It allows the doctor to deal with all the nuance limited automated tools can not, and gives the MD the ability to check themselves against the computer. If the MD finds themselves disagreeing on something like a skin condition, the feedback can both improve the doctor's service and provide bug information for the code and databases used to train the AI.

hughdbrown• 9 years ago

Can someone clarify for me how the training and testing sets were constructed? One problem is that cancerous and benign skin are unbalanced in a representative population. How was this imbalance handled in testing? How was the testing set constructed? And so on.

fantispug• 9 years ago

For each of the 3 tests the training sets were classified with a biopsy, images were randomly seleced then blurry images were filtered out by a separate dermatologist. The ratios Benign:Malignant were 70:65, 97:33, and 40:71 respectively.

These close-to-even ratios make for a more powerful test of classification. I would assume that these test samples have biopsy data means that some dermatologist thought that they might be malignant (unnecessary medical operations are unethical). This might lead to some bias towards samples that are difficult for humans to diagnose.

Separating these into binary classifications of specific tumor types makes it easier to classify than out of every possible tumor type (as a dermatologist does).

Still the claims this paper makes are very promising. A lot of the training data was classified by dermatologists, not biopsy. Using more biopsy data could lead to even better classification, as well as improvements to the model.

james_niro• 9 years ago

I truly believe with smart algorithms and big data we can change the way we live. Smart medicine, proper diagnoses and early detection of disease we can improve our lives

nbmh• 9 years ago

This is interesting and impressive work, however, I noticed that they compared the algorithm's performance to dermatologists looking at a photo of a skin lesion. This seems like a straw man comparison because any dermatologist would presumably be looking directly at a patient and would benefit from a 3D view, touch, pain reception etc. I realize that this was the only feasible way to conduct this study, but still suggests that an algorithm looking at a photo cannot match the performance of a dermatologist looking directly at a patient.

dsmithn• 9 years ago

Respectfully disagree. Telemedicine is going to be an important aspect of medicine, Dermatology in particular.

Rural and underdeveloped areas are going to be the largest market IMO. Everyone can access a smartphone but not everyone has the luxury of seeing a Doctor in person, and if they do the time/travel costs can be significant.

Disclosure, I work for an EHR startup with a Telemedicine product.

yalogin• 9 years ago

These new methods appear to best suited to be used in the pet world sooner as the ethical and legal issues will be a bit less stringent than in a human context. May be that is where things will start to change.

kumarski• 9 years ago

Skin conditions are one of the few modalities where ML make deep sense as a diagnostic.

I think pharmacovigilance is the other area based on my interaction with the industry of folks at pharma and healthcare provider companies who work in ML.

Disclaimer: i run mlweekly.com and help at semantic.md

trhway• 9 years ago

so, basically while i'm taking shower the HAL ... err ... Google Home cameras in the shower would check for moles development, blood O2 from the color of skin, vascular health from the reaction to the water temperature, pulse from visible pulsations, mental and other conditions from the eyes movements, etc...

WalterBright• 9 years ago

My (old) dermatologist could spot skin cancer from across the room. I asked him how he could do that, he said he's seen a million of them. It's the same idea as "deep learning".

habosa• 9 years ago

Diagnosis based on image recognition is something machines are already very good at, even without recent deep learning techniques (although I am sure they will help).

For instance in college I worked with a radiologist to write an image-recognition program to identify osteoporosis from 3D MRI data. We used some super-basic image segmentation algorithms to identify the bounds of the bone layer that we cared about. From there a model was able to determine mechanical properties of the bone and therefore make an assessment with much more granularity than the human eye.

This was a first-year grad student class and I was coming at this totally naive with some Matlab scripts, and we managed to get usable results in weeks.

Here's a sample of that professor's research: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2926228/

While I am not in the camp of "machines will replace doctors", I think radiology and other similar fields are in for a sea-change in technique and a large reduction in the use of human judgement.

ThomPete• 9 years ago

As someone with two melanomas under my belt (and more than a 1000 moles) what I really want is the ability to do a mass scan of my body also further down at the cellular level not just looking at the moles on the surface.

I am lucky enough to have Sloan Memorial as my hospital and no other than Dr. Marghoob one of the leading experts and I actually have a scan of my body made with 50 or so High Definition Cameras (I am litterally a 3d model in blue speedos and with a white net on my head).

They have a new system where they can look at the cell level without doing a biopsy and actually found my melanoma before they did the biopsy (i.e. they knew it was melanoma before they did biopsy) but it's really a cumbersome process and I had 6 experts studying and working to position that laser properly.

So the real challenge today is how do we get the data into the system.

arikrak• 9 years ago

On a related note, does anyone know how IBM's Watson health is doing? They've been developing it for years but I haven't heard much about their results.

caycep• 9 years ago

I wouldn't be surprised at tasks that involve image recognition - these include dermatology (visual inspection) and pathology. In fact, I wouldn't be surprised if CNN's were better at pathology as every time I looked at microscope slides, there is so much "visual clutter" in a typical tissue specimen that I'm sure I was missing a ton of information on the slide.

ChuckMcM• 9 years ago

I read the headline and wondered how ml could train the difference between a new dermatologist and a seasoned one. Cancer I get, looks totally different than non-cancerous skin :)

That said, pulling this is one of the best ML applications to date. Recognizing cats or scenery doesn't seem nearly as useful

sungam• 9 years ago

Dermatologist here. Most skin cancer diagnosis is relatively straightforward and if the lesion is suspicious will require a biopsy to establish the subtype of the cancer and plan further treatment. There is no reason why this initial visual diagnosis cannot be performed at the same level as a dermatologist by a machine or indeed by a non-doctor trained intensively for a relative short period to interpret photographs.

The difficulty is two-fold. Firstly liability - a dermatologist aims not to miss a single case of melanoma in the tens of thousands of patients seen over their career, if this algorithm is used widely in millions of patients then either the sensitivity will have to be higher and more biopsies performed or there will have to be an acceptable rate of missed diagnosis for melanoma.

Secondly, in edge cases such as moles that are slightly atypical. In these scenarios there is no way that I would be comfortable making an assessment from a photograph. Now of course, a machine could also gather further information via methods such as in vivo confocal microscopy but in this case the cost savings are likely to be negligible.

Deep learning algorithm diagnoses skin cancer as well as seasoned dermatologists

Related Stories