Stanford study shows success of different programming styles in CS class

https://ed.stanford.edu/news/stanford-study-shows-success-different-learning-styles-computer-science-class

by kornish • 11 years ago

156 51 11 years ago

ed.stanford.edu

mellavora• 11 years ago

Fantastic! Good steps towards "artificial wisdom". Define wisdom-- skill in a craft (here: knowing how to think). Using ML to identify "wise" programming mindsets. Then (presumably) developing educational programs which teach these.

“Educational data mining should not be used to reinforce the existing ineffective forms of assessment, but to reimagine it completely,” Blikstein said. “Pre- and post-assessment is a black-box. We do not know much about what is happening in between.

“Ultimately, this work may help us better understand some forms of human cognition because you can see what people are doing and how they are thinking in real time.”

partition3• 11 years ago

What was interesting is that the higher-performing group (Alpha) was stuck less often that the lower-performing group (Gamma). Is this the next programming motto after "Don't Repeat Yourself"?

Don't Get Stuck!

Now how does one avoid getting stuck while programming? I still get stuck a lot, but these really helped:

- Test what you write as quickly as you can.

- Hold it in your head! Or you won't have a clue what all your code, together, is doing.

- To get started, ask yourself, what is the simplest thing that could possibly work?

BenderV• 11 years ago

- To get started, ask yourself, what is the simplest thing that could possibly work?

I'm not sure I would go for what appear to be the simplest now ; I try to find the implementation which will have the simplest design to use/modify/delete in the future.

danieltillett• 11 years ago

Overcoming getting stuck is the difference between a highly productive dev and the average dev. This is especially the case if the problem being worked on is hard. I am a very average dev in regards coding skills, but due to my background I have learned strategies to “unstick” myself after ending up in the middle of a massive mud pit. This has allowed me to solve problems better devs have not been able to accomplish.

VMG• 11 years ago

What was interesting is that the higher-performing group (Alpha) was stuck less often that the lower-performing group (Gamma).

Which may just be correlated with high performance, not necessarily the cause.

If you make "don't get stuck" a motto, you can easily imagine people skipping over difficult problems and not thinking about a good solution.

tarpherder• 11 years ago

I share your concern that this will result in a lot of sloppy solution to lots of little problems which in turn results in one big mess of sloppy-but-it-works solutions which can be a big problem (but doesn't have to be). However, even when working out a good solution I think it's important not to get stuck. Make sure you are making some sort of progress; it's okay that it takes time but it is not okay if someone is just moving in circles instead of inching closer to the solution. Basically if something takes "too long" to get working, try a different approach or come back to it at a later time to try again. I quite like the "Don't get stuck" motto if we look at it that way.

bjwbell• 11 years ago

"don't get stuck" reminds me of the feynman method of problem solving:

1. Write down the problem. 2. Think very hard. 3. Write down the answer.

Rich Hickey's Hammock Driven Development, https://www.youtube.com/watch?v=f84n5oFoZBc, is IMO very good for actually not getting stuck in a bad place problem solving.

mironathetin• 11 years ago

"If you make "don't get stuck" a motto, you can easily imagine people skipping over difficult problems and not thinking about a good solution."

It is also important to define, what "getting stuck" means. For the research the alpha performers where the ones who fulfilled the goal quickest. If you switch the point of view from the teachers to the learning student, instead of fulfilling the requested goal quickest, you might follow an interesting problem that you discovered on the way. Is that getting stuck? No! It is following your own path of learning which is a very effective method. It does not look best to the teacher, but it can be very good for the learning student.

Asking your own questions and answering them is an important part of a learning process. It also makes you independent, so you don't simply do most efficiently what someone told you to do.

diminoten• 11 years ago

Each of your three bullet points has a related developer concept:

* Test early, test often test automatically [0]

* Remember the big picture [0]

* The minimum viable product [1]

    [0] - https://pragprog.com/the-pragmatic-programmer/extracts/tips
    [1] - http://practicetrumpstheory.com/minimum-viable-product/

pjc50• 11 years ago

I'd offer two bits of marketable wisdom from this.

- think back. Students who change style do better. This implies that those students are reflecting on their own style and trying out techniques (performing experiments) to find out what works best for them.

- think ahead. Use lookahead to avoid backtracking. Some amount of top-down planning will make your life easier and avoid traps, even if your natural style is bottom-up.

datenwolf2• 11 years ago

Test what you write as quickly as you can.

Until you run into that one problem where you can't test between the small steps, because you need the whole thing to be up (to some degree) and working for any test to work.

Operating system kernels would be an example of that: The best test for a *nix OS kernel is, if it can run a shell. You need all the essential syscalls to do something sensible and if any of the required parts doesn't work the whole thing fails.

Another example would be refactoring a complex library into something more manageable. If you keep working in small, testable steps you can move only along the gradient, bordered by "can execute" and "doesn't execute". So you'll be able to reach only a local extremum. Now if you're in the fog and don't know where to go, that's fine. And for most development this is exactly how it happens. But sometimes you can see that summit on the other side of that rift and you know you have to take a leap to get over there.

I spent the past 3 weeks doing exactly that, refactoring a code base. I knew exactly where I wanted to go, but eventually it meant working for about a week on code without being able to compile, not even think of testing it, because everything was moving around and getting reorganized. However now I'm enjoying the fruits of that week; a much cleaner codebase, easier to work with and I even managed to eliminate some Voodoo code nobody knew why it was there, except that it made things work and things broke if you touched it.

- Hold it in your head! Or you won't have a clue what all your code, together, is doing.

Or, sometimes it's important to get it out of your head, take a week or two off and look at it again with a fresh mind and from a different angle. Often problems seem only hard because you're approaching them from that one angle and you're so stuck with wanting to get it done, that you don't see the better way.

Instead you should write code in a way that it's easy to get back into it.

- To get started, ask yourself, what is the simplest thing that could possibly work?

And then wonder: What would it take to make this simple thing break. Make things as simple as necessary but not simpler.

Dewie• 11 years ago

Operating system kernels would be an example of that: The best test for a *nix OS kernel is, if it can run a shell. You need all the essential syscalls to do something sensible and if any of the required parts doesn't work the whole thing fails.

How do you test kernel? Run a virtualised kernel? I think at least DragonflyBSD supports that.

lmm2• 11 years ago

Operating system kernels would be an example of that: The best test for a *nix OS kernel is, if it can run a shell. You need all the essential syscalls to do something sensible and if any of the required parts doesn't work the whole thing fails.

So start with something simpler. Start by making a kernel that can run /bin/true, that never reclaims memory, that only boots on whichever VM you're using for testing. You absolutely can start with a kernel that's simple enough to write in a week, maybe even a day or hour, and work up from there. See http://www.hokstad.com/compiler for a good example of doing something in small pieces that you might think had to be written all at once before you could test any of it.

I spent the past 3 weeks doing exactly that, refactoring a code base. I knew exactly where I wanted to go, but eventually it meant working for about a week on code without being able to compile, not even think of testing it, because everything was moving around and getting reorganized. However now I'm enjoying the fruits of that week; a much cleaner codebase, easier to work with and I even managed to eliminate some Voodoo code nobody knew why it was there, except that it made things work and things broke if you touched it.

Which is great until you put it back together and it doesn't work. Then what do you do? I've watched literally this happen at a previous job, and been called in to help fix it. It was a painful and terrifying experience that I never want to go through again.

In my experience with a little more thought you can do these things while keeping it working at the intermediate stage. It might mean writing a bit more code, writing shims and adapters and scaffolds that you know you're going to delete in a couple of weeks. But it's absolutely worth it.

lfowles• 11 years ago

I think as you get more experienced, what is considered a "small step" changes as you're able to keep more context in your head (subconsciously of course). For the complex library example, that would be keeping the new vs the old architecture at the forefront of your mind instead of switching all thought to "const? maybe? what does that mean in this context?" and "how can I get rid of this reinterpret_cast?"

Absolutely agree with "Instead you should write code in a way that it's easy to get back into it" though!

VLM• 11 years ago

My interpretation of that part of the article was "refactor early and reimplement often"

That grabs in the concept of switching between planning and experimenting modes with the end result of not getting stuck.

I may be biased in having learned this stuff a long time ago so when I get stuck its from getting hopelessly backed into a corner at which point the best solution is pretty much start over differently, and first semester programmers might get trapped a little more easily, like they don't even know common ancient anti-patterns so they don't even know where to start looking when debugging.

Reading between the lines it almost sounded like the A level programmers wrote their tests first (which is usually easy) then wrote code to pass the tests (usually not too awful) then they were done, whereas the lower grade level programmers wrote the code first, then tried to debug (and debugging is always harder than test writing or coding)

random_coder• 11 years ago

It's also likely that these guys had already programmed before coming to the class. We all tend to get stuck when programming something for the first time. Second and third times are much smoother experiences.

evincarofautumn• 11 years ago

You’ve reminded me of a quote from Chuck Moore:

Before you can write your own subroutine, you have to know how. This means, to be practical, that you have written it before; which makes it difficult to get started. But give it a try. After writing the same subroutine a dozen times on as many computers and languages, you'll be pretty good at it.

Or, as Fred Brooks put it, “plan one to throw away”.

vanderZwan• 11 years ago

Hold it in your head! Or you won't have a clue what all your code, together, is doing.

Do you mean to say: if you can't hold it all in your head, you're overcomplicating things? Because if so, I agree.

I just finished giving an introduction to programming course, using Processing. I tried teaching my students to write structured code from the start, splitting functionality into smaller methods as much as sensible. That way they never have to juggle too much functionality in their head at once.

On the first day I had them call built-in methods. Then I introduced the concept of variables and types, then I introduced defining your own methods, and only after that did I move on to booleans, if/else, for-loops and arrays.

It seems to have worked well for getting them to structure their code from the start, using methods more or less the same way you would use chapters, headers, sub-headers and paragraphs to structure a paper.

I know this isn't the best way to program, obviously, but this analogy is (relatively) easy for them to understand and loads better than the ball of muddy spaghetti you often see with people who just start out. It's also probably fairly easy to make the jump to other more advanced forms of code organisation.

I also completely agree with your other two points by the way, I kept repeating similar statements throughout the course:

- By splitting functionality into small methods, it's easy to test if something does what it is supposed to do (and I could easily correct them: "Hey, this method drawBall() is also doing hit detection. That's not what the method is supposed to do, restructure the code!").

- "The computer only does what it is literally instructed to do. What do you literally want to do? What are you literally instructing the computer right now?"

chiph• 11 years ago

When I see students posting on /r/programming, what they usually seem to have a problem with is problem decomposition. i.e.: What are the steps needed to solve this part of the problem? Or algorithm design, critical thinking, or whatever you want to call it. How do I get from A to B, basically.

Remembering back to my first year, it seems that some people "get this" and some do not. Which is nothing against them -- everyone is different.

partition3• 11 years ago

- "The computer only does what it is literally instructed to do. What do you literally want to do? What are you literally instructing the computer right now?"

I feel like after you really know a system you almost start thinking only in terms of its components. "Hmm I want to do X...somehow I just automatically thought to modify function Y and call it in module Z."

humanarity• 11 years ago

Yeah I really reckon that "what is the simplest thing that could possibly work" is the vital question. If you stray from that, getting stuck wastes what could otherwise be productive effort. I do have to wonder tho -- why is programming so complicated? Silicon and neurons so fundamentally different that we're always going to be fighting ourselves -- or one day we're going to have interfaces make programming way more natural?

nerdy• 11 years ago

Begs the question: "Is there a 'right' way to think about programming problems?"

My knee-jerk reaction would be "No! Everyone learns and thinks differently"-- but being able to put programming styles into buckets and correlate final grades from it suggests otherwise, doesn't it?

brudgers• 11 years ago

I think the emphasis is more about the way people think about solving programming problems than the way they think about the problem itself. An analogy is that two people may differ in their mental models at the level of syntactic sugar rather than Turing completeness.

atmosx• 11 years ago

It might suggest something, but it's just a hint. In order to take that approach as conclusive much more comprehensive studies in to highly diversified environments should take place.

All that leaving aside that it's a social science we're talking so... It's not exactly science as in scientific method. It's just models that (under X circumstances) might or might not apply... Go figure ;-)

inglor• 11 years ago

From the actual article (and not the news article): "The results show that students’ change in programming patterns is only weakly predictive of course performance"

Not as "juicy" as the website - but then again, it's the actual results :)

ynniv• 11 years ago

It sounds like they discovered that some people already knew how to program.

FinnDS• 11 years ago

It seems to me that there are way too many variables that aren't really measurable for this to be completely accurate -- such as this example, some people probably had programmed before.

kifler• 11 years ago

Having recently (last year) enrolled in a Computer Programming program, I can say this is likely the case. The class has a number of people who are simply enrolled to get a piece of paper that states they know how to code and then there are those that are breaking new ground in learning.

I have the logic down quite well but my biggest issue is the syntax - it would appear that Stanford thinks I'm doomed to fail.

spion2• 11 years ago

One of the most touted advantages of Scheme (and other Lisps) as a teaching language is precisely the (lack of) syntax which allows the student to focus on logic.

willvarfar• 11 years ago

70 students enrolled in an introductory undergraduate course in programming methodology

I'm curious about that too; is this course optional? Would people who can already program take this course?

khalilravanna• 11 years ago

Anecdotally, at my university (graduated 2013 so still a recent experience), a lot of CS students took the intro to programming course. In fact, it was (and still is) a requirement for a BS in CS at the school. The only way you could get out of it was if you had taken AP CS in high school and passed the test for the credits.

A large majority of my CS friends in that class had all been programming for either a little bit or quite some time, and the class was mostly a joke for them. Lectures were optional, and a bunch of friends (including myself) only attended three classes the entire semester: the first class (where we found out we didn't need to go to class except for tests) and the mid term and final.

Of course on the other side of things, I had friends who had never programmed before and found the course grueling and struggled to pass.

Derander• 11 years ago

I TA for this class. The class is optional for non-engineering majors but has enrollment >= 700 people per quarter. So many people at Stanford take the class that it's a bit of a cultural phenomenon.

There are three "intro" programming classes at Stanford. CS106A is "intro to java", CS106B is "intro to C++", CS106X is CS106A + CS106B + a bit more in one quarter. People who already know how to program are encouraged to take one of the latter two, although there is always a contingent of people with /some/ background in 106A.

kriro• 11 years ago

I think data mining editor/IDE use is a pretty useful idea overall and the "paradigm shift" of constant monitoring which allows immediate teacher feedback is much needed (not just for programming).

The other interesting approach I read about a while back was using tracing quiz data to identify "blind spots". A typical example was identifying students that had problems understanding loop constructs (they would always assume one iteration for example).

I talked to one of the authors of this paper and he had some interesting ideas fro CS education in general: http://dl.acm.org/citation.cfm?id=2526978

My gut instinct and thinking back to my university days make me think that a blind spot for recursion may very well be a thing. I'd be pretty interested in identifying students that struggle with loops and/or recursion and investigating that further. For example...is there a correlation? If not what happens if you give a loop-struggler group only recursive tasks and a recursion-struggler group only loop tasks? Etc.

skywhopper• 11 years ago

It's not necessarily surprising that they might be able to detect students who will likely perform poorly in the course overall by monitoring their behavior early on. But the real challenge is not in determining which students are struggling, but rather in when and how you should intervene to improve these students' chances of success.

markbnj• 11 years ago

> Alpha students, explained co-author Piech, moved efficiently from one point to another as they wrote code, creating a streamlined program. Beta and gamma students, on the other hand, wrote themselves into so-called “sink states” in which their programs slammed into dead ends from which they had to back out before they could resume progress.

Very interesting, and intuitively right to me. The difference is likely in the ability to create and maintain a consistent mental model of the program's structure and behavior during the implementation.

AustinBGibbons• 11 years ago

Hey! I worked on the preliminary efforts of this project in 2012, and was also a grader for the course. (I'm "Gibbons, A." in the citation list)

Many people are making great points - there's a wealth of uncontrolled variables (ie some students already know how to program), but it was a fascinating dataset and I could even see the paradigms play out over the quarter during discussions with my students, both experienced programmers and students getting their first exposure to programming.

I would offer as a larger takeaway that work like this has the potential to vastly increase our understanding of learning and development with the rise of MOOCs, ipads-in-every-classroom, and of course IDEs that save every incremental compilation :-)

Oh and as another mention - working with Marcelo and Paulo was awesome, both are super passionate, intelligent dudes.

the_watcher• 11 years ago

> the authors did not find any correlation between the amount of “tinkering” — which is often thought to signify lack of programming expertise — and course performance. However, the authors found that students who changed their programming style — going from being a “planner” to a “tinkerer,” or vice versa — were the ones who performed best, suggesting that behavior change (rather than learning one single behavior) was determinant for course performance.

Not being flippant: The conclusion here seems to be evidence of an ability to learn something new, regardless of what, is strongly indicative of performance.

GrinningFool• 11 years ago

    > "The discovery of these “sink states,” and how students 
    > got into them, offers opportunities beyond predicting 
    > grades: They open the door for developing systems to 
    > encourage students to go down more fruitful paths before 
    > they become lost in the programming weeds."

In the real world, my most valuable learning experiences have been those dead ends. Because I don't repeat those mistakes - better to make them and learn from them than to be steered away from them before they happen.

taeric3• 11 years ago

Relatedly, I have seen plenty of very smart people that were able to bowl over these "sink states" completely unaware of just how unmaintainable their ultimate solution was.

convivialdingo• 11 years ago

Completely anecdotal, but I vividly remember my own experience learning to program.

I learned in a vacuum, having nobody around with any software experience. I was given a PC and then saved up for an Amiga 500 (and later a $500 Lattice C compiler). My only link to software was through the bookstore and magazines.

Being a nerd, my main motivation at the time (of course) was to make a D&D game. I remember my first code was in Basic (no subroutines!) so I eventually learned to create subroutines based on line numbers (100-900 was main, 1000 was map, 1100 was character A, etc)

I managed to make about 50% of a single game before I ran out of memory. So I then set out to shrink (optimize) what I had made. After reading an article on dungeon map creation, I figured out that large arrays could replace the hand-coded drawing I had written. Check.

Ok, so now I start doing everything in arrays and pretty soon my code is smaller but "fixing stuff" (maintaining code) was stupidly hard. Oh okay, so I learned then some that data and code should be together rather than global (encapsulation).

I then needed to create random maps. After struggling with that, I read a paper on fractals, and implemented my first algorithm.

You can probably guess by now where I'm going with all this... I had crudely recreated many things by knocking up against hardware and brain limitations and figuring out how to work around them.

I progressed into Pascal, C, C++, assembly and eventually created my own assembler and graphical operating system at 16 years old. It was not a thing of beauty, but it did work. (Time was free, and compilers weren't cheap.)

I think that most people learn programming by failing, hitting traps, finding limits, crashing, burning and trying again. The only real identifier of outcomes, in the long term, is the willingness to continue trying and learning from mistakes.

So from my point of view, the outcomes of the paper are arbitrary in the long term. I would definitely had been a "Gamma" programmer. I hit every problem, and got terribly stuck. It took years for me to progress, learn good practices and become a proficient developer.

Stanford study shows success of different programming styles in CS class

Related Stories