HN user
There’s lots of work on distillation, smaller models, approximations, etc. People already have simpler forms of these running on smartphones. Models seem to be growing faster than we can make them small though :D
A DGX-2 will do just fine.
Quite a few of my colleagues came back Monday the 27th. I imagine many stay longer since the trip is onerous.
http://nsaphra.github.io/post/hands/
This one perhaps
fp16 saves a lot of memory and is worth doing. I've not had trouble fine tuning all these models with fp16.
Zipf’s law
Links to papers:
https://arxiv.org/abs/1811.00207 - Towards Empathetic Open-domain Conversation Models: a New Benchmark and Dataset
https://arxiv.org/abs/1811.01241 - Wizard of Wikipedia: Knowledge-Powered Conversational agents
https://arxiv.org/abs/1810.10665 - Engaging Image Captioning Via Personality
https://arxiv.org/abs/1902.08654 - What makes a good conversation? How controllable attributes affect human judgments
And a demo of a bot that they've produced: https://www.facebook.com/Beat-The-Bot-212188996195556/
From the paper:
"$50,000 was divided among the human participants based on their performance to incentivize them to play their best. Each player was guaranteed a minimum of $0.40 per hand for participating, but this could increase to as much as $1.60 per hand based on performance."
So the humans weren't betting their own money, but they still made more money if they won.
That should tell you how grad students are paid.
I mean, that’s the thing though. Undergrad in state tuition plus R&B for NCSU and UTexas (programs I got in state tuition for) is at 9k and 10k per year with 5% interest for undergraduate, and more like 6-8% for MS students.
You can be employed in a high paying sector like tech after going to an in-state engineering school, and still l end up with $2k tacked on in your first year of employment. So then what is more rational: pay off the loan as fast as possible, or build up an emergency fund with a generous 2% ROI.
Indeed it may somewhat be a result of financial illiteracy, but today's young people are significantly better educated than previous generations (https://en.wikipedia.org/wiki/Educational_attainment_in_the_...).
With the cost of education, healthcare, and housing all rising significantly faster than inflation, it's possibly just harder for many to save up that emergency fund. Indeed, it's particularly difficult to justify having an emergency fund if you're also facing five figure 8% interest student loans.
That is the actual plot of the SheWork episode in Broad City.
Yann LeCun did not, otherwise he’d be a coauthor. As it is, this was a collaboration between NYU and Facebook AI Research, with multiple authors working at both institutions.
I don’t know man, 20% of your annual income would be seen as a sizeable fine. That’s 20% of their yearly profits, and it wipes out most of their earnings for Q1.
And replaced by residual connections in transformers, which are absolutely dominating LSTMs now.
Interestingly, we have 3 in North America: East, West and Texas.
It’s indeed a very high trials to success ratio. Again though, there’s enough papers preceding this one that you could have good confidence in the effort. Another thing that helps is orgs like OpenAI have their own servers, rather than renting ec2 instances.
You also don’t just launch that many things and them ignore it. You monitor it to make sure nothing is going terribly wrong.
But yeah there’s also the fact that if you’re Google, throwing $2m worth of compute at something becomes worth it for some reason (eg Starcraft)
There’s a natural way to parallelize these models so that using 128 GPUs is the same as a 128x batch size. You can similarly simulate 128x batch size by accumulated gradients before backpropping. So you can test on just one or a few GPUs before you run the full thing.
By that point you know it’s going to work, it’s just a matter of how well and whether you could’ve done nominally better with different tuning.
There’s been enough research leading up to this paper to suspect that just scaling larger would play out.
Top of the document says use a ballpoint pen, which doesn't smear and can't be erased like a pencil.
An infinitely sized 2 layer NN is universal in the same way a Turing machine is universal — sure you can write any program; God help you if you try.
So Russia ays they were forced to develop it after Trump withdrew from the nuclear treaty. 2 months is some impressive R&D time, huh?
An easier way to understand it is in the context morphology: word prefixes and suffixes mean things, and words have common roots.
For example, polymorphism could be decomposed into poly-morph-ism. Antidisestablishmentarianism, which is unlikely to appear much in the corpus, becomes anti-dis-establish-ment-arian-ism. Now the system can learn how to reuse "anti-" or "establish" from other examples more easily than trying to learn the full word's meaning from the one or two examples it might see in the corpus.
BPE is a clever way to induce these sort of decompositions automatically without any linguistic annotation, making them useful in multilingual settings. Other languages are much more morphologically rich than English, and there it really benefits.
Picking which items to show you in your newsfeed; making recommendations for places based on your friends; automatically generating blind-friendly captions for uploaded photos (read the hover text for some of your friends’ photos); suggesting events you might be interested in; friend suggestions; ads.
Buy a dumb terminal. A very cheap, lightweight laptop that can handle a web browser, ssh, and ideally drive an external monitor or two. Then rent time on a cloud service of your choice.
Deep learning is very power hungry. Pretending you can do it on a battery is a fool’s game.
Either this was a distraction/rogue American sympathizer, or the Trump Administration has jumped the shark in its “America first” behaviors.
For the record, good old fashioned bag of words representations (tf-idf, LDA, LSA) still provide useful representations for documents. Obviously we hope to do better, but recently people act like there's no way of turning a document into a vector.
My understanding was that Amazon, until recently, intentionally hasn’t turned any profit. You don’t have to pay taxes if you don’t make any money...
This is a subfield called anomaly detection.
Wait but, the batch size is 8x bigger for the TPU? That's not a fair comparison; increasing batch size always speeds things up...
Alas, the "secret sauce" is already freely available: http://waifu2x.udp.jp/. The only real difference is the (very, very appreciable) effort to make it commercialized software. Anybody who cares enough to disassemble it for reuse could build the core tech themselves.