plusepsilon

Oh wow, how did I not know about this :) Looks super useful.

How would ELMo work if a neural network needs to be run?

Ya it seems like just another deep learning course.

I can vouch for fast.ai though. They're the best if you're just starting.

I don't think (some) people understand; a slick data annotation tool like this is vastly more useful than the 20th variant of GAN that DeepMind produces :)

Hi jo_,

We have - 1 ML/NLP engineer (me) - 1 CEO w/ a linguistics background - 1 founder is an English professor at Berkeley - 2 with previous experience teaching English - 1 Berkeley PhD in Deep Learning / NLP advising us - 1 Berkeley PhD in English helping us categorize writing issues

We're working on assessing writing quality too. Get in touch!

Not sure how to edit but should be $!

WriteLab | ML Engineer | Berkeley, CA | ONSITE, SALARY: 100K-130K

We at WriteLab (writelab.com) are building ML tools to give immediate writing feedback for students and English language learners. There is plenty of room to impact the product by designing and implementing new features, usually starting with data collection. We use all the good stuff in deep learning and NLP including: SpaCy, scikit-learn, TensorFlow, etc.

Strong background in machine learning and experience deploying ML models in production is a must. NLP and DL experience is a strong plus.

Interview process: initial video call with NLP engineer onsite interview to discuss previous experience and go through an NLP / ML problem lunch with CEO

Apply at Angelist (https://angel.co/writelab/jobs/243578-machine-learning-nlp-e...) or email me at motoki@writelab.com.

I think spaCy uses perceptrons (essentially a shallow neural network) so it should be faster. Accuracy is pretty similar with SyntaxNet at least on the training data but I'm guessing SyntaxNet works better on long range dependencies.

I wonder if the spaCy update will go deep :)

Cool, I feel like the days of the LSTM as the de facto strong baseline is ending as these new sequence models come out.

I don't understand the math completely but it looks like dropout can be derived from a Gaussian prior (approximating the Bernoulli) in a Bayesian context.

One useful tidbit is that you can get prediction intervals from deep learning models by running it forward N times with dropout and take the mean and variance of that distribution (plus another precision term).

I actually like the keyboard. I tend to glide over the keyboard to find the right keys, minimizing wrist movement. Combination of feel, sound, flatness are somehow very satisfying.

But! The butterfly keys are stiffer for smaller keys so ironically(?), the Fn keys and arrow keys are hard to find and press. I actually use the arrow keys so I find using the Ctrl shortcuts more often for moving the cursor around.

I feel like a touch screen on the trackpad would've made more sense. Since it's so large there's room to be creative with on-screen shortcuts, dragging sliders, choosing an emoji :), etc. You can still keep all the keyboard shortcuts you need and not need to look at 3 things at once (screen, keyboard/trackpad, touch bar).

We (foxtype) actually have a dev tool that does exactly this.

If we publish it as an online tool do you think people will find it useful?

We have multiple corpora, some language models built in neural networks, etc.

Oops, accidently sent.

We're currently building an online editor that:

1 checks for compatibility of words in a sentence (essentially popularity) 2 give example sentences for a certain word 3 word suggestions depending on context.

Language models would be a decent way to check popularity though it would be noisy. Sentence level rewrites would be hard unless you make it template driven.

Thanks for the mention above (foxtype.com).

We're currently building an online editor checks

Reddit and Hacker News comments are surprisingly good data. They cover a wide array of topics and writing styles, generally written better than Facebook comments or Twitter, easier to process than Common Crawl or ukWac, and less rigid than newspaper writing.

Thanks for the background info.

If the layers are learned locally, can it take into account interactions between layers?

Any hints on how to integrate the whole document for summarization? ;)

I've seen copynet, where you do seq2seq but also have a copy mechanism to copy rare words from the source sentence to the target sentence.

I agree Keras is the easiest way to get into RNNs and deep learning in general.

It learns word representations and sentiment classification at the same time.

Traditionally, word representations are learned by looking at surrounding words. So "good" and "bad" will have similar word representations.

By training on sentiment, similar sentiment words should be clustered together.

If it uses stochastic gradient descent (it should) to train using batches of data, you can apply that to online learning.

Cool stuff! It wasn't described in the paper (afaict), but did you have ConceptNet embeddings as well? I can certainly see a way of creating embeddings by using all the links as context. (e.g. http://conceptnet5.media.mit.edu/web/c/en/knowledge)

edit: actually looks like you "retrofit" existing word embeddings by re-weighing based on strengths of the links.

Looks really good.

How is this different from Crowdflower? https://www.crowdflower.com

A lot of commoner food (like ketchup and soy sauce) would be revolutionary if they were discovered today. Ketchup is the perfect blend of umami (tomato), salt and vinegar. Soy sauce goes well with most anything. Many of the fancy sauces you pay bank are not as good as soy sauce or ketchup. They become common because they are sooo good but also a little boring once you get used to it.

Combining orange and chicken is simple but powerful because they are favorite foods for many people and they have completely different flavor profiles (of course orange chicken could be executed better than Panda Express!).

It's really interesting how English -> Emoji doesn't get criticized as much as Emoji -> English.

Compare to Dango: https://news.ycombinator.com/item?id=11870283

The meaning of emojis are generally pretty ambiguous so not all sentences will make sense because the emojis themselves don't make sense.

There's definitely some syntactic constraints (picks one emoji for the predicate?) when decoding.

In any case, I think it's fun.

-> The loud and alarming infant drives a police car. -> Santa takes a dump on Santa. -> The ogre feeds cheese to the evil creature. -> The prawns become gluten-free. 🇯🇵🇰🇷 -> Japan infuriates South Korea.

*edit: darn, emojis don't get parsed well here.

The daily email digest was really good. Gave me links to interesting stories that I would have never seen in my information circle which strongly skews tech-y.

Author calls it "Model-Based" in place for Bayesian.

I transitioned from using Bayesian models in academia to using machine learning models in industry. One of the core differences in the two paradigms is the "feel" when constructing models. For a Bayesian model, you feel like you're constructing the model from first principles. You set your conditional probabilities and priors and see if it fits the data. I'm sure probabilistic programming languages facilitated that feeling. For machine learning models, it feels like you're starting from the loss function and working back to get the best configuration.

Much of the underlying machinery behind Bayesian vs. machine learning models is the same. Hidden Markov Models are Hidden Markov Models whether they have a prior or not. But this difference in feel influences how you build models and hence, the results.

Now that optimization algos for Bayesian models are catching up, Bayesian ML might become a thing.

Cool stuff.

In many cases the simple ensemble is more fragile because you have to keep track of multiple things at once. The winning solution for the Netflix Kaggle competition was an intractable ensemble which never got used in production. Also when you update your models you'll have to tune them individually, then manually tune the ensemble weights.

Another advantage of joint learning (which the authors mentioned) is that the individual models need not be as big when trained independently since they complement each other. Though the joint model will surely be bigger than each of the individual models.

PyCharm is unparalleled in its understanding of code and it's great for building codebases. It is a programmer's tool first and foremost. I find PyCharm's interactive features clunky and have to do extra work to see the data.

RStudio / Rodeo provides an interactive data analysis environment where multiple "views" are presented right in front of the user. A view could be a plot, a data frame or interactions between the code editor and the terminal. As a data analysis person it really helps to put the mental strain of code far away as possible and just explore the data.

Jupyter Notebook are nice but it can get overwhelming (too much scrolling) when things get complicated. Great teaching tool, however.

I think each of these tools have different use cases and it's great that Python is getting more user-friendly with the data science workflow.

There is Beaker notebooks which is similar to Jupyter. Haven't tried it but you can integrate multiple languages in one notebook.

http://beakernotebook.com/