Statistical Learning as the Ultimate Agile Development Tool (Peter Norvig)

http://videolectures.net/cikm08_norvig_slatuad/

by jbr • 17 years ago

62 8 17 years ago

It's a very interesting talk - but the title is misleading (title is the same in the submission as in the source). Norvig focuses on statistical learning and how that can be used to infer the meaning of words and the appropriate results in searches, particularly image searches. How these approaches tie into agile development, or any software development really, is given at best a cursory pass with a couple mentions that I, for one, wasn't able to follow. I kept waiting for the other shoe to drop and it never did.

I'd love to be told that I'm wrong or that I'm missing something here, so if anyone wants to enlighten me on the connection, please go for it.

jbr2• 17 years ago

My take: What I extracted from the talk regarding agile software development is his point about keeping specifics out of the code. If the operational specifics of your application are kept in "data form" instead of precise a priori knowledge, it is easier to be "agile" (adaptive). Agile is an incredibly loaded word, but I think he was referring to lightweight, working software that is easily adaptable.

As far as how it's software development: It's an architectural choice whether you're going to a) hardcode domain specifics into your code, b) keep some of the domain specifics out of the code, but neatly organized and structured, or c) leave the data messy but extract meaning from a large corpus statistically.

andrewpbrett2• 17 years ago

That makes a bit more sense - I'll buy it. Thanks.

jbr2• 17 years ago

I'm particularly interested in the idea of distribution/statistical-test driven software testing. Instead of concrete assertions (truth assertions, equality assertions), we would make assertions about descriptive statistics or other tests. Or just assert that if we do something stochastic N times, we should expect some truth/equality test to pass at least M times.

I'm not sure what to do with that idea, but that's actually the core of why I posted this. Any other hackers out there excited by this idea?

tristan_juricek• 17 years ago

Yes; I'm a lot weaker in my statistical understanding, but it seems like it would be a fantastic way of slicing up the testing pie.

Unit tests are kind of handy if you've got some sort of "fencepost" style error where you need to figure out a piece of logic that was overlooked. But that's about it.

This sort of testing would way more relevant, because it's a way of focusing on "what (data) do we 'well' right now"?

I'm thinking that such sort of systems might be useful as a basis for performance or concurrency testing for example.

pgbovine• 17 years ago

some research projects that might be relevant (although not quite what you were thinking about):

statistical bug isolation: http://pages.cs.wisc.edu/~liblit/pldi-2005/

CHESS - automated finding of concurrency bugs: http://research.microsoft.com/en-us/projects/chess/

you could start from there and traverse thru related work

jwr• 17 years ago

The videolectures.net site is frustrating. On the one hand, it is a great collection of interesting talks. On the other hand, its annoying use of Windows Media means that I have to watch the talks in the browser, while I would much rather download them for watching on my iPhone.

Does anybody have a solution that would allow me to download these, convert (possibly with VisualHub) and watch on an iPhone/iPod?

macmac• 17 years ago

VLC will capture the stream and convert it to h.264 at the same time.

Statistical Learning as the Ultimate Agile Development Tool (Peter Norvig)

Related Stories