byby

No. This is false. Even senior engineers can have this problem. It's not a differentiator for success.

This is counterintuitive but actually there is raw science to back the opposite fact up. I'm not joking.

The more delusional you are as in the less ability you have to take and accept criticism the more likely you are to be successful. There is a positive correlation for this verified by science.

It gels with my anecdotal experience too. At the top end of success those guys are the most delusional and least accepting of criticism, for example Elon musk or Steve jobs, etc. Etc.

But even on the lower end of the spectrum, senior engineers it's there too. For example people who claim they are able to unbiasedly accept criticism are often the people who are worst at it.

The reason for this is not any sort of deliberate emotional reaction or anything like that. But rather because the brain actively deludes you from seeing the complete truth. It pushes people to construct artificial scaffolds of logic to uphold their existing beliefs.

Senior engineers are not immune to it nor are they better. What determines the senior role has a lot of factors and definitely not all the factors are related to the actual formal senior rank. A lot of it is luck, time, politics and other bullshit that gets people that rank.

Everyone builds these ideals about their identity, their rank, and their engineering ability. The reality is that a lot of it is delusional. By deluding yourself it becomes easy to include other people in your delusion as well and that's how a lot of hierarchies work.

Not saying their aren't competent people out there, but don't fall for the false belief that a rank like "senior engineer" conveys a certain level of emotional maturity to accept criticism.

Yeah but to keep on topic you suggested that this sort of thing was happening in the thread.

I disagree, it's not.

You've written 3 functions instead off one.

3 functions is better. Think about it. Do people write all their stuff in one big function? No. Better to compose higher level functions with smaller ones rather then write one big monolith like you did. The more modular something is the better.

Also IO_function is there for illustration purposes. Technically it's just wrapping print with a name so you can understand the intent. In reality you just use the regular print here without a wrapper, so in actuality only two functions are defined.

The job of ChatGPT was to make cool_function unit testable. You haven't done it.

It did. By giving it a return value. Just like you did by giving it a new input value.

You still have cool_function using side effect generating code hitting the actual IO system.

Yeah but one component of cool_function is pure and you can unit test that. Cool function itself can never be tested because it generates no output, you test the unit components of cool function. That's the point of unit tests.

Genuinely the worst unit test I have ever seen written, on a poor form per line basis, absolute bananas. If you don't understand why [i for i in range(4)] is bad in a unit test and [0,1,2,3] is correct then I need you to walk away from the computer.

Let's just talk about it like adults. Just tell me what exactly about it makes you think it's bad?

Most likely it's some pedantic stylistic philosophy you have? I'm thinking you only want to test literals? Perhaps you prefer [0,1,2,3]? Am I right on the money?

Logic potentially has errors so you don't put logic in your test code. Makes sense, but who cares. For trivial shit it's fine. While in this case the logic in the test is identical to the function, typically 'logic_function' represents something significantly more complex and the list comprehension so I could care less if I'm not following the strictest form of testing. The comprehension is just something akin to an alias shortcut I prefer to use over writing out a massive literal. For the toy example the test is pointless because the logic is identical but typically it's fine to use range as an alias to represent a sequence of numbers.

Someone who strictly follows these stylistic rules without seeing intent or having the ability to bend the rules is just an inflexible pedantic programmer. It's not good to boast about it either by telling other people to walk away from a computer. That's just rude.

This is the function I gave chatGPT:

   ME:
   def (x: int):
      for i in range(x):
         print(i)

Alright fine, I can concede to this. ChatGPT should not have given me the best alternative but it should have given me the exact technically correct answer. You're right.

I've been writing them for over 20. Doesn't matter. What matters is factual correctness.

See here what a unit test is versus integration test: https://www.testim.io/blog/unit-test-vs-integration-test/

And also see my reply to the other guy where I definitively show that not only is what chatgpt produced correct, but better:

https://news.ycombinator.com/item?id=36064097

I'm sorry, I clearly haven't explained myself well as otherwise you would not have wasted a huge amount of text tying yourself in knots based clearly on a mistaken apprehension of what I was saying.

No need to apologize. This is a discussion. No one did anything wrong.

For clarity I reproduce the original function you gave and then I present what the change I am suggesting is

This is called dependency injection and it's a valid way of segregating IO away from pure logic. Although this pattern is popular among old school OOP programmers it's getting out of vogue due to the complexity of it all. You used a python trick here of default values, but typically dependency injection changes the function signature and ups the complexity of the code by a lot. Let me show you the full output of the code that chatgpt was implying:

   #unit testable code (without using dependency injection tricks)

   def cool_function(x: int) -> None:
       IO_function(logic_function(x))

   def logic_function(x: int) -> List[int]:  
       return [i for i in range(x)]

   def IO_function(x: Any) -> None:
       print(x)
       
   def test_output():
       assert logic_function(4) == [i for i in range(4)]

As a side note you will see it's actually an improvement to the code. It's simpler, no dependency injection, no confusing function type signature and a much simpler test case. The other thing that must be noted is the modularity.

Making tests unit testable in this way allows for your logic to be portable. What if I want to repurpose cool_function to output it's logic to another function? In your example you don't have the components to do that, it's harder for your case as you'd have to create another component for injection.

In short not only did chatGPT produce A correct answer. But it produced the better answer compared with your dependency injection. That being said your dependency injection is valid BUT you were not correct in saying that chatGPT's answer was worse or incorrect.

Output formatting is still a type of transformation!

I'll quote part of my reply (which you missed):

   (Btw making print formatting unit testable means segregating the formatting from the print. Produce the string first, test that, then print, because print can never be unit tested by definition)

BTW I gave ChatGPT the prompt I would give, and I have to say the answer looks pretty good, even if I'm not a Python programmer and it's not the way I'd do it (which would be to change the function to allow passing in an output stream):

It's wrong in this case. Unless you specifically asked it to write unit testable code, what it did here is write a hack that monkey patches the program. It's a huge hack. It didn't write unit testable code, but rather it wrote a integration test that monkey patches stdout, negating any need to make your code "unit testable" no refactoring needed using this method. The entire concept of refactoring code to be unit testable flies out the door in this case as you're just using integration tests to get around everything.

I mean yeah you use the unit test library but is not technically a unit test. It's fine I'm not a stichler for what style of testing is used in practice but what I am saying is that what chatgpt did previously was literally follow my instructions to the letter. It did it exactly 100% correctly. Think about it. I asked chatgpt to make the Code more unit testable. You didn't have chatgpt do anything to the code. You simply changed the test from a unit test to integration test. Huge difference. I mean if your case was the "proper" way then what does it even mean to make code "unit testable" if you're not even touching the code? Like why does the concept of "making code more unit testable" even exist if we're just changing tests to make everything unit testable? Ask yourself this and you'll realize that the only answer is basically what I just told you previously.

It is. There is literally zero other way to make that function unit testable. What are you gonna compare that data with in a test if it's thrown into IO?

By definition all unit testable functions have to return data that can be asserted. You throw that data to IO it's not unit testable.

IO is testable via an integration tests. But not unit tests. Which is what my query exactly specified. I specified unit tests.

You have to change the semantics of the function to make it unit testable. Literally tell me how else can you test that function with a unit test?

By definition a unit test can only test functions that return data. So there's no other option here.

That would be fine if the core thing needing unit testing was the data generation/ transformation logic, but just as often as not it's the output formatting too.

Output formatting touches io. In this case it is no longer a unit test that touches these things. Unit tests by definition test ONLY internal logic and transformations.

It is literally the definition of unit tests.

When you test things like stdout that becomes an integration test and Not a unit test. It requires some external thing or some global black magic monkey patch that changes what print does to do integration testing.

(Btw making print formatting unit testable means segregating the formatting from the print. Produce the string first, test that, then print, because print can never be unit tested by definition)

Typically programmers segregate these levels of testing because unit tests are easier to write. But to write unit tests your code has to be written in a way to cater to it. Often this style of coding actually improves your code it makes it much more modular. The reason is because pure functions that output data can be composed with all kinds of io functions. You can move it all over the place and to different platforms with different forms of IO. Print has no meaning in certain embedded systems so it can't be moved... By segregating the logic out it makes it so I can move the logic without the io baggage.

Chatgpt 100 percent gets the difference that's why it did what it did. I think you and the OP don't fully understand the meaning of unit testing.

Don't take this the wrong way, but just because you don't know this doesn't say anything about your skills as a programmer. But just recognize that this concept is basic and is pretty much something universal among testing.

Bro, dependency injection and mocking is the same thing as segregating your function from IO. Your replacing io calls to stdout with io calls to something else. But that doesn't make your code unit testable.

The function is still touching io. You gonna test it with another function that touches io? That defeats the point of the definition of unit testability.

and doesn't do what the first function does.

Are you serious? You mock your output streams with hacky monkey patching your function ALSO stops doing what it originally does. It's essentially black magic globals that mutate your program... very bad practice.

Chatgpt here just didn't write the obvious io component of the code because it would be freaking pedantic. The full code would include a function that prints lists composed with a function that produces lists. The composition allows part of the program to be testable while leaving the io part of it not testable. For the original program NONE of it was testable.

Your Monkey patching here would be replaced by different io functions. You want to change the output stream? then you change the IO function. Compose the list producer with another IO function. Play type Tetris and you can recompose your list producing function with all kinds of modular io. The point it you separated the core logic away from IO thereby making it more modular and more testable.

None of the io functions are testable via unit tests, that is the point. That is the definition of the most basic form of testing... Unit tests.

You literally HAVE to change your code in order to make it unit testable. If your code is throwing shit to io and retrieving values from io then none of your code is unit testable. You're at the integration test level and at this level things become hacky and more complicated. Your tests not have external dependencies like state, the operating system and you have to run hacks like your monkey patch.

Where ever you work or whatever you've been doing if you haven't been doing what I described then you (and your work buddies) haven't been testing your code via unit tests.

That's fine, whatever works bro. But chatGPT knows the common parlance for testing and unit testing, and it did exactly the correct thing.

Your interpretation of what testing is the thing that is strange and off here.

Yeah those crazies are far and few in between and none of them are on this thread. Throwing out religious accusations is going too far.

I think you don't understand what unit testability means. It means removing IO and side effects from your code.

How the hell do I test a print function? I take the print function and match it with what? It has no output so how can I test it printed the correct thing? I can't.

I can test a list. I just match it with another list. Making your code unit testable is about segregating IO from logic. Write pure logic where all functions have inputs and outputs and those things can be tested. Your io prints should be small because all functions that do io cannot be fully tested.

IO is pollution. Any output to IO is the program exiting the logical mathematical universe of the program and that output can be verified only by an external entity. Either your eyes for stdout or another process or files or a bunch of other ways.

Unit tests are about internal local tests that touch local functionality and logic. If you want something unit testable it needs a local output and an input and it shouldn't rely on io in it's data path.

I think your complaint here is an example of chatGPT superiority. It understood something you didn't. Well now you know.

Removing the print function from the logic and returning the data is 100 percent the correct move. Do you understand?

Yeah. Why did you get pedantic, rude and heated about the code quality for my toy example?

It's my example sure, but it's your reaction to it that makes no sense.

plus: that's not true.

Sort of. There's an input variable that adjusts the "creativity" of the LLM. If you adjust the variable the answers become more and more "creative" approaching the point where it can challenge you. But of course this comes at a cost.

As it stands right now, chatGPT can actually challenge you.

This is a good answer. You don't have the bias of years of programming experience or training. You don't have your identity tied to the job.

If AI helps you, you'll emphasize on the overall benefit rather then nitpick at the details because of the clear conflict of interest that LLMs present to programmers.

It's not a stance. I'm stating a fact of reality. Huge difference.

I didn't say don't bother extending it's capabilities either. You're just projecting that from your imagination. An hallucination so to speak not so far off from what LLMs do. I find your similarity to LLMs quite fascinating.

What I said is, the capability of doing the "extension" you want is already in the LLM. Sure go extend it but what you're not getting is that we've already arrived at the destination.

This is off topic. Clearly we all know the LLM is flawed. We are just talking about it's capabilities in debugging.

Why does it always get side tracked into a comparison on how useful it is compared to human capability? Everyone already knows it has issues.

It always descends into a "it won't replace me it's not smart enough" or a "AI will only help me do my job better" direction. Guys, keep your emotions out of discussions. The only way of dealing with AI is to discuss the ramifications and future projections impartially.

So I said it's like 50 percent of the way there implying that it gets things right at a rate of 50 percent. That's a fuzzy estimation as well, obviously so don't get pedantic on me with that number.

When you ask for large output or give it large input you are increasing the sample size. Which means more likely that part of the answer are wrong. That's it. Simple statistic that are inline with my initial point. With AI we are roughly half way there at producing answers.

If you keep the answers and questions short you will have a much higher probability of being correct.

So that 50k line program? My claim is roughly 25k of those lines are usable. But that's a fuzzy claim because I LLMs can do much better than 25k. Maybe 75% is more realistic but I'll leave it at 50% so there's a lower bar for the nay sayers to attack.

I mean. I said 50 percent of the way there right? So your example here is inline with what I said. It's not perfect, but it's halfway there.

Maybe not word for word, but similar enough

Find one. Dated before 2021. In fact, according to the theory that it's statistical parroting there should be multiple examples of for loops printing out numbers being converted to unit testable functions because AI needs multiple examples of it to form the correct model.

Find one. And it doesn't have to be from stack overflow either. Just a question and answer data point.

Probably not then. But I just post whatever I want and I'm already in the top 10 percent. And I'm not an avid user either. I just ask a bunch of questions.

I've had a few flamed and closed but that's just 1 or 2 out of I'd say around 13 or 14 questions. It's a low percentage for me.

And I absolutely assure you much of my questions are stupid af.

He's just talking _. Clearly nobody here on both sides are having religious fervor around ai. One side is saying we don't understand LLMs completely and the other side is saying we absolutely do understand it's all statistical parroting.

But to keep it with the religious theme... which side sounds more similar to religion? The side that claims it's absolutely impossible for LLMs to be anything more then a statistical operation or the side that claims they don't know? One side seems to be making a claim based on faith while another side is saying we don't know enough to make a claim... So which side sounds more religious?

I simply asked it to make it unit testable and it did the task 100 percent.

I'm not sure where your side track is coming from. Who in their right mind would ever check in code that prints a range of numbers from 0 to x?

The example wasn't about writing good code or realistic code. It's about an LLM knowing and understanding what I asked it to do. It did this by literally creating a correct answer that doesn't exist. Sorry it doesn't satisfy your code quality standards but that's not part of the task is it? Why don't you ask it to make the code quality better? It can likely do it Maybe that will stop the subtle insults (please don't subtly imply I'm incompetent that's fucking rude)

Like why even get into code quality about some toy example? What's the objective? To fulfill some agenda against AI? I think that's literally a lot of what's going on in this thread.

The code is not garbage, it's just your highfalutin python opinion makes it so you only ever use list comprehensions or return generators.

For loops in python that return non lazy evaluated lists are fine. Python was never suppose to be an efficient language anyways, grading python based off of this criteria is pointless.

It doesn't matter how snobbish you are on language syntax though. I fed it code and regardless of whether you think it's garbage it did what I asked it to do and nothing else.

Would you prefer the AI say, "this code is garbage, here's not only how to make it unit testable but how to improve your garbage code." Actually we can make the output more unpredictable as LLMs do have a non deterministic seed that can increase the creativity of the answer.

If you read my post I said 50 percent of the way there.

So that means a 10 line bash script at best would have 5 lines of bugs.

But the AI is actually better then that. You've just increased the bar.

Well I take myself to be an expert in this area, and I think it's fairly obvious how they work. Many of these so-called "Experts" are sitting on the boards of commercial companies with vested interests in presenting this technology as revolutionary. Indeed, much of what has been said recently in the media is little more than political and economic power plays disguised as philosophical musings.

Bro if you are an expert you'd already know that most of the exclamations that they don't fully understand LLMs is coming from researchers at universities. Hinton was my example on an "expert" as well and he literally quit google just so he can say his piece. You know who Hinton is right? The person who repopularized backprop.

A statistical AI system is a function `answer = f(question; weights)`. The `answer` obtains apparent "emergent" properties such as "suitability for basic reasoning tasks" when used by human operators.

Every layman gets its a multidimensional curve fitting process. The analogy your using here to apply properties of lower dimensional and lower degree equations to things that are millions of dimensions in size on a complex curve simply doesn't apply because nobody fully understands the macro details of the curve and how that maps to the output it's producing.

The properties of a 2d circle don't map one to one to 3d let alone 500000000d.

Much of its apparent properties are illusary, and much of the press around it puts in cases where it appears to work and claims "look it works!". This is pseudoscience -- if you want to test a hypothesis of ChatGPT, find all the cases where it doesnt work -- and you will find that in the cases where it does there was some "statistical shortcut" taken

You don't even know what science is. Most of software engineering from design patterns to language choice to architecture is not science at all. There's no hypothesis testing or any of that. An expert (aka scientist) would be clear that ML is mostly mathematical theory with a huge dose of art layered on top.

The hypothesis for the AI in this case is, and I'm parroting the real experts here,: "we don't understand what's going on." That's the hypothesis. How is that even testable? It's not so none of this is "science". ML never was a science, it's an art with some theoretical origins.

But your "hypothesis" is it's just "statistical parroting" which is also untestable. But your claim is way more ludicrous because you made a claim and you can't prove it while I made a claim that basically says "we can't make any claims because we don't understand". See the difference?

fear of being reprimanded.

Don't worry about this. You can get over the fear. I'm in the top 10% of stackoverflow users in terms of points and it's all because my stupidest questions from decades back gathered thousands of points from other stupid idiots like me. Who cares. Literally the line graph keeps climbing with no effort from me all from my dumbest questions. Just ask and don't worry about the criticism, you'll get a bit, but not too much.

However, in its current state users still have to know how to program in order to make good use of it. It will still give you lots of errors, but being able to get something close to your goal can save you a lot of time. Someone who does not know how to program will not be able to use these to put together a complex, useful and reliable system. It might change in the future, but these things are hard to predict.

Of course. I think the thing I was trying to point out is the breadth of what chatgpt can do. So if you ask it to do a really in depth and detailed task it's likely to do it with flaws. That's not the point I was trying to emphasize, not the fact that it can't do any task with great depth but the fact that it can do ANY task. It has huge breadth.

So to bring it line with the direction of this thread. People were thinking about making special LLMs that refactor code to be unit testable. I mean we don't have to make special LLMs to do that because you can already ask chatgpt to do it already. That's the point.

I agree. LLMs are very impressive, but it isn't helpful to think of them of magic. LLMs are a great tool to explore and remix the body of human knowledge on the internet (limited to what it has been trained on).

Of course you shouldn't think of it as magic. But, the experts self admit they don't fully understand how LLMs can produce such output. It's definitely emergent behavior. We've built something we don't understand, and although it's not magic, it's one of the closest things to it that can exist. Think about it. What is the closest thing in reality to magic? Literally, building something we can't understand is it.

It's one thing to think of something as magic, it's another thing to try to simplify a highly complex concept into a box. When elon musk got his rockets to space why were people so floored by decades old technology that he simply made cheaper?

But when someone makes AI that can literally do almost anything you ask it to everyone just suddenly says it's a simple stochastic parrot that can't do much?

I think it's obvious. It's because a rocket can't replace your job or your identity. If part of your skillset and identity is "master programmer" and suddenly there's a machine that can do better than you, the easiest thing to stop that machine is to first deny reality.