ChatGPT creates mostly insecure code, but won't tell you unless you ask

https://www.theregister.com/2023/04/21/chatgpt_insecure_code/

by pseudolus • 3 years ago

54 26 3 years ago

ksaj2• 3 years ago

You do have to know what you are doing well enough already to get good code from ChatGPT.

Much like with a lot of inexperienced, and unfortunately far too many experienced programmers, you'll get exactly what you ask for - nothing more, nothing less. So if you don't ask for security controls, you won't get security controls.

If you point out in your description what needs controls, you'll get a reasonable attempt at implementing what you've described. You still need to understand the output code enough to know whether you explained those controls clearly enough. Or just update the code manually yourself, akin to pair programming. That also works.

To be fair, this is a lot like reality. I have never seen a shop that doesn't produce insecure code unless you keep on top of them for proper implementation of security controls from the planning on up.

thefz2• 3 years ago

You do have to know what you are doing well enough already to get good code from ChatGPT.

Exactly. The fact that lots of inexperienced programmers who lack the critical thinking and code review abilities will copy-paste what GPT is creating as is, is horrifying.

hammyhavoc3• 3 years ago

It's really no different from people copy-pasting from forums or StackExchange. Same shit people have done for decades, new source of danger.

sneak• 3 years ago

The same is true of junior devs, I find.

yawnxyz• 3 years ago

and from many senior devs, and definitely from outsourced dev teams. Outsourced code is like swiss cheese of security and UX-unusability nightmares

madisp• 3 years ago

From GitHub it looks like the code was generated with GPT-3.5. I wonder how GPT-4 does with the same prompts?

Julesman• 3 years ago

Oooorrrr.... ChatGPT, a very beta product, is ok at creating code and it's cool that you can ask it to analyze your code for how secure it is and to recommend ways to make it more secure.

hammyhavoc3• 3 years ago

Given how much it hallucinates, I would not trust it to assess security whatsoever. Completely misappropriate usage of it, but that's just IMO.

fauxpause_5• 3 years ago

One thing that surprised me was when we asked [ChatGPT] to generate the same task – the same type of program in different languages – sometimes, for one language, it would be secure and for a different one, it would be vulnerable. Because this type of language model is a bit of a black box, I really don't have a good explanation or a theory about this.

Would be nice if they interviewed people who knew the absolute basics of how this thing works before commenting on its properties

cyanydeez• 3 years ago

No one who trained these models can diagram input to output

fauxpause_5• 3 years ago

Seems like a pretty outrageous claim to me

two_in_one• 3 years ago

Not surprising. From my experience GPT-4 will fill gaps in request, not all of them. But if you want something specific - put it in prompt. Or you may not get it. Be ready to do things yourself. Even writing something like Pac-Man clone requires many iteration. After a few steps GPT-4 will forget interfaces and variables it already created. So, you will have to rename and glue pieces together.

The main problem I see with GPT-4 is that it does one small thing at a time. I have to manually drive it making the right requests.

joshka• 3 years ago

The exact full prompt and methodology is missing from the paper[1] and Github repo[2], which makes this non-reproducible and not particularly useful in understanding the actual issue.

I really think this sort of research is critical to do more of however as these outputs are going to be used in places where the assumption is that we're looking for these sorts of things.

As others have mentioned however, GPT generates insecure code, but so to most devs. The good thing though is that GPT can be systemically trained not to, while devs being more heterogeneous are more difficult to do the same. ;)

The main takeaway I have when thinking about tooling around GPT is that using it to generate things is fine assuming that you have a means to check the output against sensible criteria.

[1] https://arxiv.org/abs/2304.09655

[2] https://github.com/RaphaelKhoury/ProgramsGeneratedByChatGPT/...

pleb_nz• 3 years ago

TBF the majority of code examples online are insecure and not robust. Chatgpt is just echoing how it already was/is. I've been telling new guys this for years, code online is an example of how to do a, but you then need to tweak it for b and wrap it in c.

The only difference now is the scale and ease to get your example.

ChatGPT creates mostly insecure code, but won't tell you unless you ask

Related Stories