Darts: Differentiable architecture search for convolutional, recurrent networks

by kumaranvpl • 8 years ago

51 8 8 years ago

I'm pleasantly surprised by how well it works! No offense to Barret Zoph, Quoc Le et al but the "let's throw 10000 GPUs and TPUs at the problem" approach was frankly absurd.

And a huge +1 for the inclusion of random architectures in the results.

osipov• 8 years ago

Why do you think that throwing compute at the problem is absurd? Arguably that's the approach that produced the resurgence of interest in CNNs.

smaddox• 8 years ago

I'm not the original commenter, but I think it's fair to say that throwing compute at the problem is absurd from an academic point of view (where the goal is understanding), but not from a commercial point of view. Although there's still some academic value in throwing compute at a problem, just to see what's possible.

chadmeister• 8 years ago

Yes I think the whole point of Zoph and Le's research is to empirically demonstrate the desired trade off of compute cycles vs feature engineering

NhanH• 8 years ago

Can you link the paper of Zoph,Le that did the “1000 GPUs” thing? Thanks! I tried searching for it but apparently there was quite a few papers on parameter search.

kripke• 8 years ago

Not GP, but you are probably looking for the "Neural Architecture Search" series [1] [2] [3]. First one uses something like 1k GPUs for a month, next one is a bit more reasonable, and the last one actually has a reasonable training time, and has comparable performance to DARTS (see ENAS in comparison tables).

1: https://arxiv.org/abs/1611.01578 2: https://arxiv.org/abs/1707.07012 3: https://arxiv.org/abs/1802.03268

ashelmire• 8 years ago

Cool, was thinking about this subject earlier today.

Abstract:

This paper addresses the scalability challenge of architecture search by formulating the task in a differentiable manner. Unlike conventional approaches of applying evolution or reinforcement learning over a discrete and non-differentiable search space, our method is based on the continuous relaxation of the architecture representation, allowing efficient search of the architecture using gradient descent. Extensive experiments on CIFAR-10, ImageNet, Penn Treebank and WikiText-2 show that our algorithm excels in discovering high-performance convolutional architectures for image classification and recurrent architectures for language modeling, while being orders of magnitude faster than state-of-the-art non-differentiable techniques.

Zoph's paper on architecture search: https://arxiv.org/abs/1611.01578

Concept of architecture search from ages ago: https://static1.squarespace.com/static/58e2a71bf7e0ab3ba886c...

aabajian• 8 years ago

Not to get existential here, but the analogy to human neural networks is compelling. The algorithm efficiently redesigns itself to fit the data, just like neural synapses reconnect one another as knowledge is learned, to optimize the storage and retrieval of the information.

Darts: Differentiable architecture search for convolutional, recurrent networks

Related Stories