An "AI Breakthrough" on Systematic Generalization in Language?

https://aiguide.substack.com/p/an-ai-breakthrough-on-systematic

by picometer • 3 years ago

47 4 3 years ago

aiguide.substack.com

shermantanktop2• 3 years ago

1) https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headline...

2) Study used 25 people

3) via Mechanical Turk

Interesting topic but I don't see a breakthrough.

viraptor• 3 years ago

Why would it matter? They wanted data generated for comparison and got it. They wanted a common type of human error and got it too. What's the criticism exactly?

shermantanktop2• 3 years ago

Yes, they got some data, and got some errors. But they are using 25 samples to characterize "human reasoning". Sure, it's a place to start.

The bigger issue is that this pattern - small samples, via Mechanical Turk - is a frequent flier in papers that make claims that end up failing further scrutiny. It's more common in sociology and psychology than AI research, but I think we know this isn't a solid foundation to build a lot of extrapolation on.

sgt101• 3 years ago

Very interesting write up and analysis of a significant paper. The blog's author has herself made fascinating contributions in AI (especially on abstraction). I find it very helpful to read these kinds of in depth reviews.

An "AI Breakthrough" on Systematic Generalization in Language?

Related Stories