Ask HN: Can ChatGPT generate fully functional code?

https://news.ycombinator.com/item?id=33919451

by midspectrum • 4 years ago

23 66 4 years ago

news.ycombinator.com

Fellow members,

I'm curious to know what are some of the best prompts to use on ChatGPT to understand its full-fledged capability as a newbie developer.

One of the things I noticed, is that it produces the requested code in the cleanest manner but it's not complete. For instance, I tried the prompt - "give me the python code for TicTacToe Game" and the code produced was incomplete.

Have you all encountered the same or my prompt is inappropriate?

anigbrowl• 4 years ago

YEs, but in small blocks and with specific asks. I asked it to generate code based on an existing project of mine, which generates analytics from social media activity. It does a good job of producing function bodies from prompts equivalent to a short comment you might use to document your function in the IDE.

For your example, I'd ask it to 'generate code, using Python, that accepts input and updates a display to allow a user to play an interactive game of TicTacToe with the computer.' I'm sure I could do better with some experimentation, but that will have to wait for some other day.

deutz_allis• 4 years ago

This is what i asked it for. In separate sessions it built functions slightly differently and only one of its programs had a minor bug. It was interesting to watch it answer the same prompt differently more than once.

herpderperator• 4 years ago

Take a look at this: https://www.dropbox.com/s/gq2hkp7w37c5y6q/ai.png?dl=0

ptsneves• 4 years ago

Oh my! It does not really matter if this is actually self-aware, but in terms of productivity: well it has the potential to be revolutionary. No longer does a person need to know how to program. Just talk to a computer and the computer will generate the code to execute such task.

Tozen2• 4 years ago

No longer does a person need to know how to program.

I think this is getting caught up in media sensationalism. ChatGPT is regurgitating copied code from various sources. A way to look at this is if you could type out questions to Stack Overflow, then it responded by giving you code that most closely matches to what you asked. The thing about that is it is taking and searching code that thousands of humans have created, "filtering" based on question asked, and giving it back to you. Instead of a person doing the hard work of searching Stack Overflow and other sources, ChatGPT can be the new centralized source to get copied segments of code.

Copying and pasting code from various sources is not anything new. Because a person has copied a piece of code from somewhere doesn't mean they don't need to know programming, what is going on, or how to troubleshoot it. Some people, who are being lazy, might think that ChatGPT is a way out of studying and understanding what is being done, and they will sooner or later found out that it isn't.

franky47• 4 years ago

It did write incorrect tests for non-Latin characters though.

scientism• 4 years ago

The future looks like debugging hell.

Calzifer• 4 years ago

It wrote a lot of incorrect tests at this point not only when non-Latin characters where involved.

Look at the mixed case tests below. Always the same string but sometimes a palindrome and sometimes not.

Same for the trailing whitespace test.

Maybe it got tired at this point. Could also explain why the performance test is missing warmup :D

sykofant• 4 years ago

holy cow

MrScruff• 4 years ago

There is a character limit on a single response. You can ask things like 'Continue the previous answer' to get it to generate more.

deadcoder09042• 4 years ago

just "continue" works :)

tux3• 4 years ago

Someone on /r/rust is currently marveling that it spit out a working bitmap allocator: https://reddit.com/r/rust/comments/zgkuq6/

ChatGPT is sensitive to how you prompt it, and it's designed to work as a dialogue where you ask further questions and it can refine its answer.

It can already write reasonable working code, although you shouldn't expect it to spit out entire working programs when the prompt is Gimme Teh Codez without further details. That being said, AI capabilities only get better with time, so I think it'd be shortsighted to dismiss its potential because of its current limitations.

If the pace keeps up, companies with access to large language models in a few years could have a large advantage on legacy programming, the same way programmers in high-level languages have a massive productivity advantage on 1970's low-level assembly programmers. You give up a little more control, but you save time.

hdjjhhvvhga• 4 years ago

You give up a little more control, but you save time.

Not just that: for an average programmer the compiler will optimize the code much better than if it was done by hand. Something similar can happen down the lane.

deadcoder09042• 4 years ago

type "continue" & it will continue where you left off.

it has a token cap so it stopped once it filled them.

midspectrum2• 4 years ago

Here is the response that I got after the "continue" command and its still incomplete.

https://pythonsandbox.dev/e6uevcuueka1?file=main.py

marceloabsousa• 4 years ago

It all depends on what you mean by generate fully functional code. In my experience, it is able to generate code snippets that are useful but very rarely can be simply copy & pasted into your work. The experience is actually quite similar to Stack Overflow.

gremlinsinc2• 4 years ago

yeah but the difference is I can immediately ask, hey why isn't this working, or why am I getting this linting error, and it'll maybe point me in the right direction (so far usually it works really well, sometimes I need to refresh the session, and start from scratch with my code base, but it's still instant hand-holding).

I was wanting to try to do something I wasn't familiar w/ at all: Building a mobile app with flutter, and so far I've got the gPT chat completely cloned in flutter, I'm just missing the additional features I wanted to add like saving chats, sharing screenshots and text chats, entering a url, and scraping the page text to ask ChatGPT to summarize, or maybe write a blog post similar to this but don't copy anything, etc....

nathias• 4 years ago

no it's mostly object oriented

ineedausername• 4 years ago

It does many mistakes that are only obvious if you know what you're looking at.

prohobo4• 4 years ago

First of all, don't use ChatGPT for that. Use full GPT-3, specifically the code models (ie. code-davinci-002).

ChatGPT is a subset & tweak of the full GPT-3 model to make a good chatbot, and it's actually less capable than the full model.

It's strange to me that people are going crazy over ChatGPT when GPT-3 has been out for months now.

More info on the difference between ChatGPT and GPT-3: https://dev.to/ben/the-difference-between-chatgpt-and-gpt-3-...

femboy• 4 years ago

Well, ChatGPT is free for now.

WheelsAtLarge3• 4 years ago

Yup, free will always help your popularity.

stevenhuang3• 4 years ago

That's flat wrong. That is ChatGPT's answer to the difference between ChatGPT and gpt3. ChatGPT does not really know about itself, its data cutoff is from 2021.

ChatGPT is trained on a variant of gpt-3.5 (instructgpt), read more here https://openai.com/blog/chatgpt/

prohobo4• 4 years ago

Okay, maybe I'm wrong, but the article doesn't explain how ChatGPT compares to GPT-3.

fvdessen• 4 years ago

For me the conversational interface is what makes the AI useful as a coding companion, how do you integrate GPT-3 in your workflow ?

prohobo4• 4 years ago

I've just been using the Playground. Also, working on creating a voice activated personal assistant.

JohnAaronNelson• 4 years ago

GitHub Copilot

arcturus17• 4 years ago

First of all, don't use ChatGPT for that
It's strange to me that people are going crazy over ChatGPT when GPT-3

People are going crazy because of the combination of the capabilities of the model and the UX.

Surely you can appreciate how deploying your own full GPT-3 based solution to assist in code generation is much more complex than going to a web app and typing "code me an Express server".

v4dok• 4 years ago

I gave it a .txt file that i wanted to parse its data into .csv columns. It actually got most of it right but after spending hours prompting it, it never really got everything right.

Its worst nightmare was that there were some nested values in the .txt. What amazed me the most is that pasting the error code from colab into the chat, it recognized the mistake and generated a fix.

That being said, most of the times it would try to fix a logical mistake by overengineering a solution to an error, or it would identify the mistake correctly, explain in natural language what needs to be done (impressive), but then give me the code unchanged.

WheelsAtLarge3• 4 years ago

Try a simple calculator. It will give you the complete code for that. I think there's a set limit on output text so that's why your tic-tac-toe was incomplete. I've asked for the calc in Rust and Python and I got a working app for both. It was simple but the fact that it gave me what I asked for is amazing.

I bet that you could break-up your app into functions and have ChatGPT write each function to get to a bigger program.

It's limited now but I can definitely see the possibilities.

It won't be free forever so we'll have to weight the usefulness vs cost eventually.

midspectrum2• 4 years ago

Yes! The calculator worked and I agree with you that in most of my prompts when I have requested to give elaborated answers, it doesn't give appropriate results.

Tozen2• 4 years ago

It's not that amazing. You can find complete examples of calc in Rust and Python all over the Internet. It just spit out a copy of someone else's code, and gave it to you.

Instead of you having to Google search everywhere or look in multiple sources, you were able to just ask ChatGPT.

WheelsAtLarge3• 4 years ago

If it stays at its current programming level then you're right but I'm more impressed with the ask-and-you-get capabilities and the future possibilities of the AI. My assumption is that it will get better as time passes, from past experience.

Overall, it's impressive tech.

spaceman_20203• 4 years ago

I've been using it for small blocks of code or specific functions - stuff I'd have to look up documentation or Stackoverflow for. I see it as a productivity booster, not a replacement.

You can't tell it to "create a CRUD app", but you can ask it to create a nodejs function that gets all the records from a mongodb database.

davidkunz• 4 years ago

You can ask for a simple example of a CRUD app, here's my code output:

const express = require('express')

const app = express()

// define the model for a user

const User = {

  name: '',

  email: ''

}

// create a new user

app.post('/users', (req, res) => {

  const user = Object.assign({}, User, req.body)


  // save the user to a database, etc.


  res.send(user)

})

// read a list of users

app.get('/users', (req, res) => {

  // retrieve a list of users from the database, etc.

  res.send([

    { name: 'Alice', email: 'alice@example.com' },

    { name: 'Bob', email: 'bob@example.com' }

  ])

})

// read a single user

app.get('/users/:id', (req, res) => {

  // retrieve a user with the specified id from the database, etc.

  res.send({ name: 'Alice', email: 'alice@example.com' })

})

// update a user

app.put('/users/:id', (req, res) => {

  // retrieve the user with the specified id from the

database, etc. const user = Object.assign({}, User, req.body)

  // save the updated user to the database, etc.

  res.send(user)

})

// delete a user

app.delete('/users/:id', (req, res) => {

  // delete the user with the specified id from the

database, etc.

  res.send({ message: 'User deleted' })

})

app.listen(3000, () => {

  console.log('Listening on port 3000')

})

spaceman_20203• 4 years ago

Well that's impressive :)

But of course unless you had programming experience, you would have no way to structure this output correctly and actually take it live

d4rkp4ttern3• 4 years ago

Is it better than Copilot with comment-based incremental code writing?

joshka4• 4 years ago

Better really is too subjective and context dependent. Play with both and see what works for you. Copilot is GPT3 specifically trained on code, so my understanding is that it should be better... It's also fairly convenient due to the IDE plugins.

warrenmiller• 4 years ago

Tim Corey has some thoughts on that https://www.youtube.com/watch?v=FEsahkX7ZF8

bspammer• 4 years ago

If you know roughly what you want and just don't want to dig through documentation to find the exact correct incantation, it's very good at that.

It's kinda crazy how fast my opinion went from "haha neat, the robot can write poems" to "this is an invaluable tool and I am begging you to let me pay for stable access". I use it multiple times per day, it's easily saved me hours already.

raesene9• 4 years ago

I've seen some experiments done with using ChatGPT to produce entire websites or programs

https://github.com/pdparchitect/ChatGPT-JobsForBots and https://github.com/pdparchitect/ChatGPT-Pong were two of them.

The author shows all the steps used to generate the programs.

nounaut• 4 years ago

From what I've seen so far you can tell it it's incomplete, tell it what errors you're getting etc, and it will agree and correct itself until you have a working application.

binarymax• 4 years ago

When it’s incomplete I ask it to refactor to smaller methods, and if it crops the output again I ask it to show the full text of a method one by one to get the full output. Tedious but it works.

Hakkin2• 4 years ago

I believe ChatGPT has some kind of response token limit, which is why longer code snippets commonly get cut off. I've seen some people recommend to enter something like "Continue your previous message starting from the last token", which sometimes works, but even when it does, it often breaks formatting, making it difficult to read the resulting code without copying it somewhere else. What I found to work better is instructing it to output a single function at a time, this usually results in short enough code that it can output all of it in a single message. Something like:

  I want you to send a single function in each response, I will say "Next" when I want you to send the next function.

I was able to get it to generate decently lengthy programs using this (on the order of a few hundred lines) that were syntactically and logically correct without any modification. This may work for some languages better than others, though I imagine you can find some way to logically split up the program for most.

stevenhuang3• 4 years ago

All you need is to prompt it with "Continue" and it will keep going exactly where it left off.

Hakkin2• 4 years ago

I've heard this quite a lot too, but it commonly doesn't work. I've had it respond thinking I was talking about using a `continue` statement somewhere in the code, and other times it simply just reprinted the whole code starting from the beginning. If you hit the "Try again" button enough it will usually eventually work, but it again has the problem where a lot of the time the formatting of the text gets messed up.

d4rkp4ttern3• 4 years ago

Somewhat related — I’m curious has someone built a ChatGPT based Linux command line helper? Ideally should be runnable via CLI

joshka4• 4 years ago

Take a look at Warp.dev

d4rkp4ttern3• 4 years ago

It does look nice, thanks! I recall it was discussed here before

duckydude20• 4 years ago

i really hate all this. codepilot and stuff. it takes away the joy of writing code. esp clean code.

geoah• 4 years ago

Golang results are amazing, they compile most of the time with some lib-specific issues from time to time.

You can ask to explain code, write tests, refactor for perfomance or readability etc.

gaurangt• 4 years ago

It's ok for small blocks or code, maybe a function or two, even a whole class, but beyond that, it's pretty pathetic. Even the small blocks of code it outputs only compile a few times and have a few errors. At this point, I would be wary of using it in Production systems.

Having said that, I usually compare it with Junior programmers, more like someone just learning a new programming language. You can expect that level of performance from ChatGPT!

sublinear• 4 years ago

I feel like the more interesting question is whether ChatGPT is better at generating political speeches instead?

Seems like there would be far more content to train on and less restrictive criteria.

gremlinsinc2• 4 years ago

It sure can, esp good at simpler things like ui, and popular frameworks etc. The only issue is when there's breaking changes between upgrades, like maybe the framework is at 9.0 but the model was trained on 7.0, so there's some things that don't work just right.

So far it's created some pretty amazing tailwindcss layouts for me, and a flutter app to interface with chatGPT that I want to add more features like saving, sharing, etc.. In 3 hours with zero flutter experience I can now at least do as much as the website does. That's pretty amazing.

Edit: YMMV -- imho you do need to be mid-senior level in tech/programming skills to know and find the best prompts to get it to go your way, there's some trial and error and your ide will usually let you know about errors/etc and you can ask them to fix it and gpt does a good job of correcting things.

joshka4• 4 years ago

Prompt:

  Write a small application in javascript that logs in to news.ycombinator.com with credentials passed on the command line and then posts the following message:

  >Fellow members,
  >
  >I'm curious to know what are some of the best prompts to use on ChatGPT to understand its full-fledged capability as a newbie developer.
  >
  >One of the things I noticed, is that it produces the requested code in the cleanest manner but it's not complete. For instance, I tried the prompt - "give me the python code for TicTacToe Game" and the code produced was incomplete.
  >
  >Have you all encountered the same or my prompt is inappropriate?

Output:

Here is an example of a small JavaScript application that logs in to news.ycombinator.com using credentials passed on the command line and then posts the message you provided:

  // First, we need to import the `request` module, which will allow us to make HTTP requests to the Hacker News API
  const request = require('request');

  // Next, we need to parse the username and password from the command line arguments
  const username = process.argv[2];
  const password = process.argv[3];

  if (!username || !password) {
    // If the username or password is not provided, print an error message and exit the program
    console.log("Error: Please provide a username and password as command line arguments");
    return;
  }

  // Now we can use the `request` module to make a POST request to the Hacker News login API endpoint
  // In the request, we need to provide the username and password as part of the request body
  request.post({
    url: 'https://hacker-news.firebaseio.com/v0/login.json',
    body: JSON.stringify({
      "username": username,
      "password": password
    })
  }, (error, response, body) => {
    if (error) {
      // If there is an error, print it to the console
      console.log(error);
      return;
    }

    // If the login was successful, the API will return the user's authentication token
    // We need to save this token so that we can use it in subsequent requests
    const authToken = JSON.parse(body).token;

    // Now that we have the authentication token, we can use it to post a new message on Hacker News
    // To do this, we need to make a POST request to the `item` endpoint, providing the authentication token as part of the request headers
    request.post({
      url: 'https://hacker-news.firebaseio.com/v0/item.json',
      headers: {
        "Authorization": `Bearer ${authToken}`
      },
      body: JSON.stringify({
        "title": "Best Prompts for ChatGPT",
        "text": "Fellow members, I'm curious to know what are some of the best prompts to use on ChatGPT to understand its full-fledged capability as a newbie developer. One of the things I noticed, is that it produces the requested code in the cleanest manner but it's not complete. For instance, I tried the prompt - \"give me the python code for TicTacToe Game\" and the code produced was incomplete. Have you all encountered the same or my prompt is inappropriate?"
      })
    }, (error, response, body) => {
      if (error) {
        // If there is an error, print it to the console
        console.log(error);
        return;
      }

      // If the request was successful, the API will return the ID of the newly-created post
      const postId = JSON.parse(body).id;
      console.log(`Successfully posted message with ID ${postId}`);
    });
  });

To use this code, you would need to save it to a file (e.g. hacker-news-post.js) and then run it from the command line, providing your username and password as arguments:

joshka4• 4 years ago

The code I got for "write a small python program to play tic tac toe" was complete (though I had to write "continue" to get all the output.

It made a playable game, that was close to the output that it suggested would occur, but it did have a few mistakes - it missed 2 win conditions (rows and one of the diagonals), and the output didn't quite align fully (i.e. "" instead of " " being printed and no lines for board edges, it also doubled up the diagonal checks oddly.

  # Check if there is a winner, and if so, print a message indicating who won
  if board[0][0] == board[1][1] == board[2][2] and board[0][0] != "":
    print(f"Player {board[0][0]} wins")
  elif board[0][2] == board[1][1] == board[2][0] and board[0][2] != "":
    print(f"Player {board[0][2]} wins")
  else:
    # If there is no winner, the game is a draw
    print("It's a draw!")

It took me a while to find the right prompt to fix this as it seems to confuse the code checking for game over, which checks for a winner as one of the conditions, for the code that checks for the winner at the end of the game.

One of my prompts led to the output being made the program be refactored into a function run in a loop rather than a one shot game. and then I tried:

the play_game function prints the wrong output when there is a winner in the columns or rows

and got the desired fix.

I like the stateful nature of this.

wolletd• 4 years ago

I let ChatGPT build a simple glue class in C++ to set the backlight of a display, just because that was what I was doing at the moment.

I noticed a bug, informed ChatGPT and it fixed it correctly, which is cool and all.

But I'm really interested in the next thing: I didn't save the code and so reran the same prompt a few days later. Now it was correct the first time. Did it learn the bug? It also lost the Exception handling it had in the first version, though. And it changed it's naming scheme for member variables from `m_foo` to `foo_`.

nicks23• 4 years ago

Yes constantly get this. It’s quite annoying and I’d love to know why. Doesn’t seem consistent or based on char limits.

Anyway, when this happens just use the prompt : “continue”. From this it’ll generally do a good job of continuing where it left off, but some refactoring is often required. As it doesn’t always exactly begin where it ended and leaves out vital code or in complete event handlers and functions for example.

From an initial prompt, you may need to continue several times Ron get to complete code you’re after.

Of course, you can throw the resultant incomplete buggy code back in as a new prompt and request analysis and bug fixes. Which it does a good job of resolving.

Of course, all of this subject to the complexity of your perk or/request

Ask HN: Can ChatGPT generate fully functional code?

Related Stories