moshe

I've never used PyTorch or Chainer, so maybe I'm wildly wrong, but here goes..

Dynamic batching requires you to predefine a set of "batch-wise" operations (node types for computation graphs). This works fine for eager execution as well, because the eager code is inside a function definition. Of course if you want to do dynamic batching you need to introduce a layer of abstraction, e.g. instead of saying a+b you create a dynamic batching operation for addition and add edges to an execution graph, this is not different than in our TF implementation.

Thanks, insightful questions.

You're absolutely right about combinators and long-range dependencies. We have a special block type in the high-level API (https://github.com/tensorflow/fold/blob/master/tensorflow_fo...) for accumulating results here without having to explicitly shuttle them around; not perfect but very handy in many cases.

Regarding your second question, the equivalent of padding in dynamic batching is "pass through" do-nothing ops, which are introduced transparently but worth being aware of if you want to understand the machinery. The worst case scenario here is chains of wildly varying lengths, where we need to add pass-throughs to the shorter chains to match the length of the longest chain.

Please note that the GitHub page says "not an official Google product", rather than "project". An official Google product would be something like gmail.

TensorFlow Fold provides a TensorFlow implementation of the dynamic batching algorithm (described in detail in our paper [1]). Dynamic batching is an execution strategy for computation graphs, you could also implement it in PyTorch or Chainer or any other framework.

Our particular implementation of dynamic batching uses the TF while loop, which means that you don't need to make run-time modification to the actual TF computation graph. At runtime, we essentially encode the computation graph for (let's say) a parse tree as a serialized protocol buffer (tf.string), so instead of varying the computation graph itself we vary the input to a static computation graph instead. This particular implementation strategy is very much a byproduct of how TensorFlow works (static computation graph, heavy lifting happens in ops implemented in C++).

[1] DEEP LEARNING WITH DYNAMIC COMPUTATION GRAPHS, https://openreview.net/pdf?id=ryrGawqex

I have gotten plenty of great applicants - thanks guys! Please stop sending me your application info at this point...

It is, actually.

"Currently pursuing a BS, MS or PhD in computer science or a related technical field." is verbatim from the intern job listing (http://www.google.com/support/jobs/bin/answer.py?answer=1262...), and I suppose it is there because the target audience for Google Engineering internships is students who are aiming to work some place like Google Engineering after graduating - not exactly the same category as "technically qualified students". So if you'd consider working at some place Google Engineering post graduation, then you should definitely apply.

For the record, I would consider economics to be a related technical field (cf. John von Neumann, cf. "Evolution of Cooperative Problem-Solving in an Artificial Economy" - http://www.whatisthought.com/hayek32000.pdf)...

> Intelligence is general and is capable of coping with > novel problems. AI therefore invovles building machines > which have that level of generality. > There is a bias when you think of "programs" to think of programs that a human would actually write or consider useful. But in fact, this is only a tiny (and nontrivial to specify) fraction of the space of "possible programs" - see for example _Foundations of Genetic Programming_, by Langdon and Poli, for some next explorations of general program space. Most possible programs are completely useless and uninteresting to humans (same with proofs, grammars, etc. etc.).

To do AI you have to realize that in the sentences: "Humans have general intelligence and can solve problems in many domains" and "Breadth-first search is a problem-general technique that will always find a solution if one exists" the word "general" does not mean the same thing at all! (cf. the huge literature on inductive bias in human cognition).

For a nice list of program induction problems solvable by search (with a system called ADATE), see http://www-ia.hiof.no/~rolando/Examples/index.html .

a) Not everyone in Google Research has a PhD, though of course there is a bit of a correlation (also, plenty of "research" takes place in general engineering) b) You do generally have to be very familiar with one of the Big 4 to get hired (C++ in my case) c) Plop is kind of a weird edge case insofar as it is very long-term research, and really cries out to be implemented in a language where you have the same features at compile-time and run-time, and can conveniently represent and manipulate code as data (i.e. a Lisp). For doing plop in any other language, your first task would be to implement these features, which is rather a lot of work (believe me, I've tried ;->). For most anything else one of the Big 4 would be used, for the reasons Steve Yegge outlines...

Well, the difficulty with this is that program induction is in fact too general to be very interesting - to create a competition you would need to publish some test problems, or describe a special distribution of "natural programs" that you especially cared about...

Well, sort of. Genetic programming with probabilistic modeling and sampling instead of crossover/mutation (i.e. it an estimation-of-distribution algorithm), where programs are represented in reduced (normal) form and which variables to use in the probabilistic models is determined adaptively and changes as the search progresses... ;->

Thanks for the pointer. I was actually just looking at ACL the other day - I have a friend here at Google who is investigating integrating it with plop as a 20% project. I'm very excited that your work combines theorem-proving with learning, kudos! Do you do anything probabilistic to decide which generalizations to try?

Heh, thanks. moshe at google dot com was already taken, it was my second choice...

None taken. Thanks!

But it is! I really am lucky enough to get to do machine learning research at Google and code in Common Lisp!

See http://research.google.com/. Another way you can tell that this is an official Google project is the 'Google' label on the right-hand side of http://code.google.com/p/plop/, which is only added to code developed at Google that has been open-sourced.

Cheers!

Eventually, learn arbitrary programs from data, e.g.

Input: (learn 'fib '(x) '(((1) 1) ((2) 1) ((3) 2) ((4) 3))) Output (defun fib (n) (if (< n 3) 1 (+ (fib (1- n)) (fib (- n 2)))))

as well as standard machine-learning tasks such as supervised classification.

For what it does right now, see the examples at the bottom of the quick start guide:

http://code.google.com/p/plop/wiki/QuickStart

For more technical background see e.g. http://metacog.org/main.pdf (my dissertation). I will also add a list of relevant publications to the wiki...

Actually, I work full-time for Google Research (and have written all of the plop code, so far). Google happens to have sponsored a bunch of summer-of-code students through opencog.org, one of whom I supervised, but he coded in C++.

I think that the idea of a Lisp system that learns Lisp programs via probabilistic modeling is intrinsically interesting regardless of who funds it, but that could be personal bias ;->.