A test which predicts ability to program before the start of training

http://www.eis.mdx.ac.uk/research/PhDArea/saeed/

by ColinWright • 15 years ago

77 36 15 years ago

eis.mdx.ac.uk

wccrawford• 15 years ago

If I'm reading that right, it doesn't test the ability to get the 'correct' answer, but rather that they are able to come up with a system that makes sense (at least to themselves), and follow it faithfully and repeatedly.

I've met programmers who can't do this, and they were really bad... Suitable only for mindless tasks where they were given all the logic and just had to key it in and test it.

masnick2• 15 years ago

This is fascinating.

I observed the "double hump" in an intro computer science class I took in college. Out of 20 people in the class, only 3 had prior programming experience. As the semester progressed, some were able to learn basic Java while others were not; aptitude seemed to be fairly independent from both effort put into this class and performance in other courses.

Since then, I've wondered why this was the case -- it seemed like there was a "programming gene" (not literally) that gave some people innate ability, and without this you were sunk.

It seems like there would be more research on phenomena, but this is the first work I've seen that hypothesizes why this happens.

hasenj• 15 years ago

It's probably related to abstract thinking. Some people (the majority?) have a preference for concrete thinking, and they get stuck at trying to just grasp the very concept of computattion.

DavidMcLaughlin• 15 years ago

Using Java as the language to detect if someone has the "programming gene" does not make any sense. There is so much cognitive overhead to writing a simple program in Java compared to, say, Python that there will be many other factors involved in why a student would end up performing badly.

In my own case I disliked Java so much in my first year of University because I couldn't understand why there was so much effort just to get this little black screen to print "Hello, World."

I would go on but I actually already wrote about this a couple years ago on Reddit: http://www.reddit.com/r/programming/comments/84c90/i_fear_as...

masnick2• 15 years ago

I absolutely agree.

This was in 2009, so today's obvious alternatives to Java (Ruby or Python) were not as popular.

However, it should be noted that for the course I mentioned, the friction of getting started in Java was not an issue: there were only 20 people in the class and we all lived in the same dorm so people could get help when they needed it.

bane3• 15 years ago

I think much of it has to do with a natural ability of certain people to functionally break down problems into discrete solvable atomic sub-components.

I've observed that most people do not demonstrate this ability. An example, while doing a consulting gig for a large agro-business a few years ago they wanted to solve a single question

"should we brand our new product with verbiage about it being all natural?"

Helping them resolve this question involved coaching a dozen very well educated people through a simple process of functionally breaking this question down into solvable sub-components.

"how do consumers feel about the term 'all natural'?" ->

("how can you find out?", "are there equivalent valid terms we can test?") ->

(("do you have data that represents a consumer voice?", "is there a way to process this data to find sentiment?"), ("who in your company might know equivalent industry terms?", "can we ensure the regulations around these equivalent terms allows us to use them in this case?")) ->

((("lets get data from site A, B and provider C", "let's use these tools and techniques to extract terms, timeline the usage and score for sentiment"), ("let's meet with these folks and collect there terms and use them in the above process", "send the top 3 results to legal for regulation confirmation")

Basically just keep breaking down the questions become so trivial they basically solve themselves.

This was a revolution to them. The only method they were aware of to answer the question above was to blanket shopping malls with market survey teams at a very high cost. People in general tend to think of establish procedure not in functional decomposition and problem solving.

I've observed that this holds true for nearly everybody I've encountered who couldn't learn to program. This kind of problem solving technique was something that they simply hadn't learned to do and it's exactly what you have to do to learn to program.

"I need to write a program to do X, how do I do X?" they need to learn to break that problem down into smaller and smaller pieces until those pieces become orthogonal with the language they need to use. X might be made up of 1000 sub-components, and arriving at those is simply a matter of dividing X down in a series of phases, asking "can I answer this with the language I know yet?"

Even people who do fine in other classwork, say they breeze through calculus, aren't taught this and as far as I know it's simply not taught in academic situations outside of some of the more formal maths (e.g. any math where you spend more time with proofs than mechanical problem solving) because schools teach via composition instead of decomposition (or synthesis instead of analysis).

What I mean is that schools start from first principles, in the maths it'll be axioms and mechanics, in history it's ancient history, in language it'll be simple reading and writing skills, etc. and build up to some standard. But at no point are people taught to take a complex problem and break it down into more simply solvable sub-components. This is exactly the kind of skill that people need to learn to program and IMHO should be the entire first year of any degree program that requires programming of any sort.

The beauty of course is that if you can't break something down further, and you don't know how to solve it, it's probably solvable with a technique you simple don't know at that point...and that's where education can slot in.

Daniel_Newby3• 15 years ago

Are there any existing courses that do this? Systems analysis? Operations research?

Jabbles• 15 years ago

This paper is from 2006. Its results have since been shown to be not as impressive as first thought:

http://crpit.com/confpapers/CRPITV78Bornat.pdf

We now report that after six experiments, involving more than 500 students at six institutions in three countries, the predictive effect of our test has failed to live up to that early promise.

ColinWright2• 15 years ago

Towards the bottom of the lunk-to page:

  Meta-analysis of the effect of consistency on success
  in early learning of programming is our fourth paper
  that was presented to Psychology Programming Interested
  Group (PPIG) Annual workshop on July 2009.  

  Abstract: ... The test does not very accurately predict
                levels of performance, but by combining the
                result of six replications of the experiment,
                five in UK and one in Australia. We show
                that consistency does have a strong effect
                on success in early learning to program but
                background programming experience, on the
                other hand, has little or no effect.

http://www.eis.mdx.ac.uk/research/PhDArea/saeed/SD_PPIG_2009...

In short, the jury is still out, but there is some sort of effect showing up, and it's going to need careful experiments to find out what's really going on. It does seem to show promise, though, of finding a predictive effect.

pmb• 15 years ago

It is superb that they have done the replication and followup and are reporting the null hypothesis.

mikerg87• 15 years ago

There is a test for midi-chlorians now?!??!

skimbrel• 15 years ago

Told that there were three groups and how they were distinguished, but not told their relative sizes, we have found that computer scientists and other programmers have almost all predicted that the blank group would be more successful in the course exam than the others: “they had the sense to refuse to answer questions which they couldn’t understand” is a typical explanation.

I almost cracked up at this. As much as we'd like to say that's how we behave, I'd say just about all of us are more likely to try to answer anyway.

tokenadult• 15 years ago

Previous HN discussion of paper by same author:

http://news.ycombinator.com/item?id=1145654

Edit after kind reply by Colin:

I think the earlier discussion was especially valuable because of a checklist of factors to look at in further research offered by one of the HN participants who commented there.

http://news.ycombinator.com/item?id=1146044

Here in this discussion that began today, in another reply, a participant posted

it seemed like there was a "programming gene" (not literally) that gave some people innate ability, and without this you were sunk.

Some kind of "nature" explanation is what the human mind jumps to as an explanation of individual differences, by innate human cognitive biases. To tease out causation in such a matter would take what Robert Plomin and other behavioral geneticists call a "genetically sensitive" study design, which is lacking here. But there is such a huge body of human behavioral genetics research already, that I can confidently predict what the finding would be for anything properly called "programming ability":

1) Programming ability will be found to be heritable, that is that it shows tighter correlations among closely related persons than among distantly related persons, because all human behavioral characteristics are heritable,

and

2) programming ability is still exquisitely sensitive to environmental influences (which perhaps are not well understood), especially influences of "nonshared environment," the kind of environmental variation that results in siblings brought up in the same household differing from one another.

A reference for both general propositions that is well worth reading is

http://people.virginia.edu/~ent3c/papers2/Articles%20for%20O...

Johnson, W., Turkheimer, E., Gottesman, I. I., & Bouchard, T. J. (2009). Beyond heritability: Twin studies in behavioral research. Current Directions in Psychological Science, 18, 217-220.

ColinWright2• 15 years ago

Thanks for that.

Some might cheer and say "Ha! Auto-hoist, petard-wise," but this latest submission points to the page that documents the continuing saga. They have addressed many of the criticisms leveled at the paper in the submission wo which you refer, and carried out both further studies and a meta-analysis.

Hence I thought it worth re-visiting.

Having said that, the discussion on that submission is definitely worth reading.

jamesrcole• 15 years ago

Though this is only relative to current techniques for teaching programming, and we shouldn't assume we know all there is to know about how to teach programming.

There's reasons to think there's a lot of scope for improvement.

For a very general reason: that we don't really know that much about how our brains and thinking work. Once we do we are likely to be much better at identifying precisely why some people are poor at certain tasks, and at knowing how to train people to improve their skills at those tasks.

[EDIT: wording]

jwingy• 15 years ago

I'm of the school of thought that anything is trainable, including programming, barring actual physical limitations. I think up to a point, even a low point, programming is something almost any average person can do. I think the brain has proven time and time again how adaptable and elastic it is.

mkrecny• 15 years ago

I agree that the brain is "adaptable and elastic". But it is still subject to physical limitations. Probably to a much lesser extent than physique.

It's not a very motivational message but I really believe some types of thinking are beyond the reach of certain people, due to the physical (read genetically determined) constraints on their brain.

xutopia• 15 years ago

"Untitled Document"... some people don't know how to do web.

Atropos• 15 years ago

I maybe took 3 weeks of programming in high school 8 years ago, so I just took the test for fun and here is what I don't understand: I picked the mental model that the left variable ALWAYS takes the value of the right variable and changes its own value permanently.

Of course this mental model makes all questions trivially easy, for example:

"Question 12": int a = 5; int b = 3; int c = 7; a = c; [a becomes the value 7 permanently] c = b; [c becomes the value 3 permanently] b = a; [b becomes the value 7 permanently] a = 7; b = 7; c = 3

If I had picked a different mental model by chance that was more complicated for my working memory, I find it quite likely that I would have made some careless mistakes. So maybe some participants were not incapable of forming a consistent mental model, but picked a very hard mental model to start with?

MostAwesomeDude• 15 years ago

It has been suggested that the three core concepts of assignment, encapsulation, and inheritance are unteachable/unlearnable and have to be present a priori in order to become a programmer with any competence.

I've seen people not be able to answer these questions, even after a term of CS. :c

orillian• 15 years ago

I would tend to agree. I have seen over the last couple years a number of people who were passionate and dedicated to learning programming fail, and fail hard.

Two years ago I was tutoring a young lady that had a serious passion for programming, it was what she talked about constantly, and she was in college taking a number of beginning courses in programming. She studied day in and day out, working countless hours every night trying to "practice being a programmer". She did every exercise given in the text books, and even had a number of other textbooks on VB.NET (the language being taught in the course.) and she studied them for hours every night. At the end of two semesters she still could not program anything if the steps to the goal were not outlined for her. She could recite to you the syntax of VB.NET and many of its companion (.NET) libraries by heart, she could tell you what a piece of code did when she looked at it (she was strangely adept at reading code and knowing what it did) but to ask her to come up with the required steps to sort an array of numbers from highest to lowest and her brain simply could not break down the logic to a point where she could write it herself.

I also tutored a young fellow last fall that was in a similar predicament. He was trying to pass a basic programming course so he could get an IT diploma and this one beginner programming course was a requirement for this diploma. He was driven to study, and was willing to put forth the effort to get through this course, and he most likely would have (given multiple choice options been) able to see the pattern. On the other hand ask him to assign the value of “b” to ”a” and he would start to get lost. Ask him to assign “b” to “a” and “a” to “c” and “d” to ”b” and he completely shuts down. No matter how many examples he was given, no matter how hard I tried to show him alternate ways of looking at it, the act of assigning values to variables and then interacting with those variables to manipulate the values just would not stick.

My opinion after dealing with a number of others like these two students is that some people lack the ability to develop patterns. Not necessarily to see patterns, but to come up with their own. They tend to have poor visualization skills and they tend to find it hard to understand the pattern links when given something like an analogy. I would say that the ideas behind encapsulation and inheritance stem from assignment but they look at larger assignment patterns than a simple a = b. Thus, if the individual lacks the basic ability to conceive the pattern of assignment required to get to the goal of the code they are not able to learn the ability to program.

ctdonath2• 15 years ago

There is a huge difference between "= means assignment" vs. "= means equality". One is an act of change, the other is observation of factual truth. HS algebra is the latter; using that would have left you very confused ("wait, b=a is false!").

suivix• 15 years ago

Even if the test were accurate, an average but motivated guy would focus well and quickly become better than the naturally gifted guy who slacks off reading websites all day.

SoftwareMaven• 15 years ago

I do wonder if there is a pedagogical solution to teach programming to people in the "bottom hump". If you could identify them, you could take them down a path that teaches the required thinking (problem decomposition, linear behaviors, etc) without using any code at all.

Of course, ideally, this would be something everybody comes out of high school with, and most of the top hump have probably intuited this from math classes. Looking at my kids' curricula, though, there is nothing explicitly trying to teach these concepts.

PedroCandeias• 15 years ago

I take issue with pretty much all tests of potential. In my humble view, all they really do is measure how your experience, up to the point when the test is taken, has prepared you for whatever it is you're testing for. A sufficiently intelligent person should be able, given enough training, to become proficient in any area of expertise.

I have no hard data to back this up, save for my own observations.

esrauch• 15 years ago

I think the two are not as mutually exclusive as you present it to be.

Consider something like language acquisition for example; if you had a 30 year old that already knew 3 languages they would almost certainly perform better on an exam to test potential to learn new languages than a 30 year old that knew one language.

If I am reading you correctly you are saying "that's not actually a legitimate potential predictor; it just indicates what experience you have, not a true measure of your capability going forward", but in reality if you have only learned one language by the time you are 30, you are at a point where your gene pool is not a great predictor. By the time you are 30, your lack of previous experience has actually had a physical effect on your brain that prevents you from being as good at language acquisition going forward.

It's not merely the fact that you have experience in a particular area that improves your direct ability to answer questions; lacking experience actually makes it more difficult to acquire the same levels of experience.

In other words, having experience makes you more able to gain experience; a test that measures your experience is a legitimate indicator of future learning potential. It's exactly along the line of colleges teaching you "how to learn" and any practical skills that you gain along the way are largely incidental.

jmmcd• 15 years ago

But according to the article (and it's also pasted [here](http://news.ycombinator.com/item?id=2903166)), they've identified something independent of experience and preparation.

Daniel_Newby3• 15 years ago

A sufficiently intelligent person should be able, given enough training, to become proficient in any area of expertise.

Indeed! Combine several tests of mental potential into an omnibus test, and run a large group of people through the omnibus. If you run the results through common factor analysis, you will find that getting a good score on one subtest is highly correlated to getting a good score on each of the other subtests. Psychologists have named the common underlying cause g, the general intelligence factor. (Scaled for age and population variation, g becomes an IQ score.)

tomlin• 15 years ago

Not to be a dick, but broken graphic links and "Untitled Document" and this test is to do what? Filter the herd?

thurn• 15 years ago

If anything, this test illustrates that assignment is a very confusing concept, even more so if you pick a silly symbol like = for it. I wonder if they tried to teach any of these students a more functional language that doesn't rely on the idea of assignment.

ctdonath2• 15 years ago

I've noticed a few key factors impacting ability to learn programming:

Hard work. Too many students just won't do the work. They'll show up, go thru the motions, rattle something passable into place, and wonder why they didn't pass (or almost didn't). The work has to go beyond just what is required into self-motivated learning beyond.

Grokking variables. Many don't comprehend the nature of a variable, and thus cannot succeed in using them. They know the right things to say about variables, and may achieve passable results using them, but just don't get the concept.

Task decomposition. Many can't get past "just do it", viewing the term as an end instead of a beginning.

Strict adherence to rules. Whatever the rules of syntax are, that's what you have to work with, and deviation just doesn't work.

Semantics. One must be capable of realizing that certain phrasing influences perception.

Abstraction. Many can't.

Get those down and you'll learn. Miss one or more and it will be very hard.

NHQ• 15 years ago

I don't believe that some people just can't learn to program. Certain types need a certain type of teaching method. I failed my first programer course in college (Java), and totally left the idea behind. Almost a decade later I learned to program. I taught myself, with the help of the internet. In other words, the internet supplied me a methodology for learning to program.

A test which predicts ability to program before the start of training

Related Stories