Italy did an accidental experiment on the effectiveness of GitHub Copilot

by dadadad100 • 3 years ago

23 16 3 years ago

dadadad1003• 3 years ago

By turning off access to GPTs the Italian government showed how their country’s developers felt about copilot

jruohonen• 3 years ago

Like with most event studies, they don't control any other events that might possibly affect the likelihoods. It seems that the ban also occurred in a weekend. They should also show the raw time series from a longer period in order to see how stable the series are and thus how realistic the effects are.

bonzini• 3 years ago

Copilot was not included in the ban, only ChatGPT. Flagging.

dadadad1003• 3 years ago

Yep. Sloppy on my part. Suggested better title above but I can’t change it myself.

koliber• 3 years ago

I wonder if the 50% drop was because developers were half as effective without copilot. Or was it because they have spent half of the day looking for ways to bypass the block while having angry conversations with colleagues, and therefore, were not coding.

boringuser2• 3 years ago

The idea that aggregate developer output is 50% improved by the simple existence of transformers seems to not pass my sniff test.

I haven't seen any of my coworkers improve their output in the past x years.

justrealist2• 3 years ago

It's easily true for me. Far more than 50% for someone doing simple React development or DS scripting.

thefz• 3 years ago

I still think that if your productivity is seriously impacted by chatGPT, you are easily replaceable as a programmer.

boxed• 3 years ago

Release event

I don't understand this. Did they measure the number of releases in the GitHub "releases" feature?

leadingthenet• 3 years ago

I was interested to see how it affected them in the medium-term, including the VPN/Tor usage, but the data range seems extremely short, a few days before the ban and a few after. Also doesn’t seem to account for any other events taking place in and around that time, or that it’s a weekend and so on.

Doesn’t really say much, imo.

Filligree• 3 years ago

The link says nothing about Copilot, this title is not just editorialised but wrong.

phreeza• 3 years ago

@dang: this post is marked as flagged at the moment, but I don't think it should be? Except maybe for the editorialized title, but that doesn't seem super bad.

dadadad1003• 3 years ago

OP here. Happy to change the title to “on the effectiveness of ChatGPT for GitHub users”. Sloppy reading on my part. Sorry

Italy did an accidental experiment on the effectiveness of GitHub Copilot

Related Stories