nadam

Does anyone know why these Claudisms exist? Are these kind of expressions a result of some kind of 'evolution', so they are more effective in chain-of-thought reasoning that other expressions? Or the preference of people doing some kind of human-feedback in the post training? Clearly probably not the 'average' expressions of their training data, and clearly if there would be no upside and it is easy to remove them they should be removed, as they are annoying, so I guess there is an upside or hard to remove them...

You put the "agent" word into apostrophes as if I use it as a marketing buzzword. No. An agent is an LLM in a loop with memory usage, with file system access, etc, which is usually more effective than just using an LLM as is, especially if you orchestrate these agents and subagents in a good way. In my opinion using the word 'ChatGPT' (a specific user-facing LLM brand) in these discussions is much more buzzwordy than using the word agent.

Maybe. But I am working with these agents and I see the sophisticated patterns using LLMs, using ground truth data and human experts working together. It could be much more effective than 'patient asking chatgpt' vs. 'general doctor one-shotting a problem without AI assistance and enough context'.

I am amazed at the false dichotomy (ChatGPT vs. Doctors) discussed in the comments. Let's look at it from the perspective of how a complex software is developed with a team of software engineers and with AI assistance. There is GitHub, Slack discussions, Pull Request, reviews, agents, etc...

Why a patient's health cannot be seen this way?

Version controlled source code -> medical records, health data

Slack channels -> discussion forum dedicated to one patient's health: among human doctors (specialists), AI agents and the patient.

In my opinion we are in the stone age compared to the above.

I find it amusing how people take this 'leveling game' at big companies so seriously. The title they get at these companies become their identity. They live by the rules the company imposes on them. Amusingly almost the opposite of independent thinking, which they are so proud of. What I found during my career is that some people blossom at these big companies, and some, with equal talent cannot really realize themselves, while they blossom at other organizations, perhaps in startups, or just at more interesting problems. What some people do not realize is that this aspect that as you level at these companies 'you get the big picture more and more' is not really true in a lot of cases. Efficiency and big picture thinking is almost orthogonal. Efficiency is context dependent, high level thinking is less so. I have seen people getting the big picture on things even at low levels or even considered a junior, while people on high levels suprisingly small minded. Higher level at these organizations means that you are more productive in the given field, given organization, for some reason, this can be because of good political skills, because you work your ass off, or because you have a very efficient brain, but it does not necessarily mean that you have better high level vision, or taste or maturity even. Sometimes it is almost laughable how these people who treat these leveling system seriously and get to a relatively high level treat young colleagues. They are almost naive about how young people think. They think that L3 juniors cannot do anything alone. While I see even young kids playing with suprising autonomy if they are thinking in the right problem space.

True role models of mine never were/are L3, L4, L5, etc... They were/are awesome engineers or scientist from the get go. (Like the Johns: The von NEumann and the Carmack.) Experience lets you get better and better, but if you are stuck at a role or organization, the problem might be that you need to find something that you are more passionate about and not necessarily that you are low level because your thinking is not 'independent enough' or you are not 'high-level enough'. At least that is my experience.

I was just thinking about what AI assisted coding brings to the design discussion, especially as AI become more and more powerful and we rely on it more and more. You still want to make things modular and easy to understand so that AI understands it easily and the needed info to modify a module can fit in a relatively small context window, but the difference is that it is very easy to make large scale measurement about which code style is understood better by an LLM, so maybe some of these debates will be decided relatively objectively!

For the topic: the discussed topics are relatively trivial surface level stuff, mostly I agree with POSD, but these will be handled by AI anyway. I guess humans will use the spare brain capacity to deal with the real deep design questions (for a while).

What I am starting to internalize nowadays is that working faster is useful, but what can really change one's life is learning faster.

Location: Budapest, Hungary

Remote: Yes

Willing to relocate: No

Technologies: I have the most experience in C++. But there are several others, like, Java, Typescript, Angular, etc...

Résumé/CV: https://docs.google.com/document/d/1UwV5np7atKgOdUsjcPETG-AX...

Email: nadam60@gmail.com

Linkedin: https://www.linkedin.com/in/adam-nagy-03575834

I am a senior software engineer. I am a generalist, but I have the most experience with complex performance-intensive C++ code. I am looking for remote work.

Hmm, I think 'fail fast' and 'embrace the grind' are popular and somewhat contradictory advices. Which is better? I think 'fail fast' is (or was?) a bit overhyped so I tend to err on the side of 'embrace the grind'. But obviously the art is in deciding which one to follow in a case by case basis. Working on your dream game for years only to find absolutely no traction is not a good place to be in, but constantly chasing low-effort ideas without any 'moat' can be also fruitless. Moat usually comes with time, effort, and resources invested.

"But when you have to scale the work, then the code has to be easy to read, easy to extend, easy to test, have extensive test coverage, have proper dependency injection / be easy to mock the 3rd party dependencies, be easy to configure so it can be deployed in every cloud provider (i.e. by using env vars and config files for modifying its behavior).."

Interestingly I do not find that the stuff you mentioned are the things that LLLMs are bad at. It can generate easy to read code. It can document its code extensively. It can write tests. It can use dependency injection especially if you ask it to. What I noticed where I am currently am much better than an LLM is that I can still have a very nuanced very complicated problem space in my head and create a solution based on that. The LLM currently cannot solve a problem which is so nuanced and complicated that even I cannot fully describe and work partially from insincts. It is the 'engineering taste' or 'engineering instincts' and our ability to absorb complicated nuanced design problems in our head that separates experienced developers from LLMs.

Unless LLMs get significantly better and just replace humans in every task, I predict that LLMs effect on industry will be that much less developers will be needed but those who will be needed will be relatively 'well paid' as it will be harder to become a professional software developer. (more experience and engineering talent will be needed to work professionally).

"When every supermarket aisle looks like a sea of sameness, when every category abides by the same conventions, when every industry has converged on its own singular style, bold brands and courageous companies have the chance to chart a different course. To be different, distinctive and disruptive."

I don't think it is that simple. There is stuff with different asthetics today, they are just not always successful among the mainstream audience. It is always a choice to substantially differ from the mainstream look, and in industries where the cost of entry is not too high plenty of people do it, they are just not successful or too niche most of the time. Where there is a high cost of entry obviously less players try to be massively different, because being too niche is not profitable in that case.

An example is the material editor. When dragging a value of a property of a node (like color) you see it immediately in the editor. You apply a material on an object, (regular object, water, Landscape), and while changing stuff in the node graph, you can see it applied to the object on the scene almost immediately (depending on some factors of course). A Material can be as complex as a Landscape auto-material which magically auto-generates the surface appearance of a complex landscape (with even foliage). But this is just an example, there are other node graphs in Unreal, like PCG. Hot-reloading my C++ code is not strictly necessary as I write C++ code mostly for really messy complex stuff, where I need to think a lot between tests. So I think Unreal mostly has immediate feedback where it is most necessary.

Re-watching Bret Victor's "Inventing on Principle" was an interesting experience today. At the time of recording (2012) his talk was magical and ahead of its time. Today his principle (immediate feedback to the creator) is fully applied in tools for technical artists. (Unreal Engine, Houdini, etc...) It is funny to see what happened with his ideas: basically what happened is that the software engineer and technical artist roles has got really separated. For technical artists his ideas are everyday life today, almost trivial seen from 2024 (although they are not), although they work mostly not in text-based, but node based programming languages. For software engineers his ideas are not so relevant as software engineering is specializing on the more messy, more abstract or more complicated problems. Frontend development is somewhere in between. I think Tim Sweeney (and his team) is missing from the list, who I believe has seen this coming already in 2012 and built Unreal Engine to its current form.

"compare this with enterprise software, which is orders of magnitude more complex than video games in terms of business logic" Maybe this was true 20 years ago, but I do not think this is true today. Game code of some games is almost as complex as enterprise software or even more complex in some cases (think of grand strategy games like Civilization or Paradox games). The difference is that it still needs to be performant, so the evolutionary force just kills programmers and companies creating unperformant abstractions. In my opinion game programming is just harder than enterprise programming if we speak about complex games. (I have done both). The only thing which is easier in game programming is that it is a bit easier to see clearly in terms of 'business requirements', and also it is more meritocratic (you can start a game company anywhere on the globe, no need to be at business centers.) And of course game programming is more fun, so programmers do the harder job even for less money.

For people who think game programming is less complex than enterprise software, I suggest the CharacterMovementComponent class in unreal engine which is the logic of movement of characters (people) in a networked game environment... With multiple thousand lines of code in just the header is not uncommon in unreal. And this is not complex because of optimization mostly. This is very complex and messy logic. Of course we can argue that networking and physics could be done in a simple naive way, which would be unacceptable in terms of latency and throughput, so all in all complexity is because of optimization after all. But it is not the 'fun' elegant kind of optimization, it is close to messy enterprise software in some sense in my opinion.

His system prompt is too specific. It worked for me with GPT 4o using this prompt:

----

System prompt: Please note that words coming to you are tokenized, so when you get a task that has anything to do with the exact letters in words, you should solve those using the python interpreter.

Prompt: How many r's are in the word strawberry?

----

This whole thing is a non-problem. Adding in this hint into the system prompt the whole topic is solved once and for all. I would argue that if the training data would include this wisdom, it would be even a non-problem without the system prompt.

Or alternatively:

1. Programming in very concrete/practical terms because you do not know how to think in precise and abstract terms (do not know math)

2. Thinking more precisely and abstractly (more mathematical way)

3. Only do some key important abstractions, and being a bit hand-wawy again in terms of precision. The reason: important real-world problems are usually very complex, and complex problems resist most abstractions, and also being totally precise in all cases is impossible due to the complexity.

All-in-all it is due to increased complexity in my opinion.

Example: 1. Writing some fun geometry related programs 2. learn about geometry more seriously 3. write software based on a multiple hundred thousand line CAD kernel.

Other example: 1. Write fun games on C64 2. Learn about computer graphics in University 3. Contribute to the source code of Unreal Engine with multiple million lines of code with multiple thousand line class declaration header files.

Amazing work, prompt engineering at its finest. One future direction for Arc AGI could be to use not Python, but a much more concise programming language that is more suited for brute-force methods like genetic mutations. The problem would be of course to train an LLM that is proficient enough in such a language. I am thinking about stack based languages. For this competition I would develop a careful bit-level encoding of a variant of the 'Joy' programming language. (https://en.wikipedia.org/wiki/Joy_(programming_language)) It would be a considerable effort though which I don't have time for, hence I post this idea publicly. A promising direction is a mix of things in my opinion: Special stack-based concise language, consulting LLMs like the OP did, and genetic algorithms combined.

I love this, this is super interesting, but my intuition based on looking at a dozen examples is that the problem is hard, but easy enough that if this problem becomes popular, near-human level results will appear in a year or less, and AGI will not be reached. The problem seems to be finding a generic enough transformation description language with the appropriate operators. And then heuristics to find a very short program (in the information theoretical sense) in this language that produces all the examples for a problem. I would be very surprised if we would not increase the 34% result soon significantly, and I would be surprised if this could be transferred to general intelligence, at least when I think of the topics where I use AI today and where it falls short yet. Basically my intuition is that this will be yet another 'Chess' or 'Go'-like problem in AI. But still a worthwhile research topic, absolutely: the value that could come out of this is well worth the 1M dollars.

'The breaking tech news this year has been the pervasive spread of "AI" (or rather, statistical modeling based on hidden layer neural networks) into everything'

Calling current generative AI tech as 'statistical modeling based on hidden layer neural networks' is as insightful as calling any new development in the tech industry (like personal computers, software as a service, or game engines) as 'usage of stored program computers' or even 'usage of transistors'. Although I understand that the author's aim is to somehow discredit generative AI this way, but referring to something as one of its very basic, old and low-level building blocks feels just dumb.

I think it is important to discuss the notion of accidental complexity and essential complexity here. If your organization's strength is that you have world class engineering essential complexity is your friend: a problem domain with big essential complexity is really a moat: it keeps the barrier to entry into the market high. If there were less essential complexity in the world there would be much less money in software engineering and much less software engineer jobs would exist. Case in point: markets where barrier to entry regarding technical complexity become too low degrade into a race for the bottom. (like the flood of indie games that do not make money.) On the other hand accidental complexity is not our friend: if you maintain a system with too much accidental complexity there is a great risk that a smarter competitor could create something at least as good with less resources.

It is too early to tell for my current project. I know that it is very hard to create a successful game. Certain genres are more crowded than others though. Creating a successful 2D platformer is almost impossible nowadays for example. I am creating a grand strategy game. It is a niche where the supply and demand dynamics is reasonable, but it is still hard to create something successful.

It is not just about the financials. It can be a very lonely activity to create something that other people do not care about, or just very very few people care about. (As it is described in the article.) Besides financial reward, social reward is important too.

You are right: it could still work as something I enjoy. It is just that I am in a state of my life where I do not want to put too much time into such an activity: I have a greater desire to create things that people want. But I still watch interesting movies and occasionally it could be joyful to solve a math problem. But as I do not watch movies dozens of hours weekly, I also do not want to solve math problems dozens of hours weekly. This has probably to do where I am in life currently. This may change later in life (I am middle aged.)

The problem with nerd sniping is that it is instant gratification in a way. Always focusing on tech and math problems because you like solving them can lead to neglecting other areas. But maybe learning more about those other areas could lead to better success in life. At least this is my experience. It solved itself in my case. My interest just shifted to be more end-product focused gradually.

Not very theoretical. But with 'nerd sniping' focus I tended to create my own tech unnecessarily, and wasting too much effort to optimize things that do not necessarily need to be that much optimized. Nowadays I try to reuse other people's code as what I really enjoy is to create the end-product.

I try to find joy in creating stuff that some other people find useful or enjoy. Creating something just so that I enjoy the process but society do not value at all is something I have a very bad feeling about nowadays. Like I am wasting my time/life. As an engineer it is more rewarding for me to create a videogame that people enjoy than creating a nice optimal algorithm or solving a hard math problem.

Joy with creating something other people enjoy or find useful. For example creating a videogame. But formerly I enjoyed creating the underlying technology (engine programming, which is a nerd sniping too), nowadays I am more interested in the game as a whole (more game-design focus and less game-tech focus.)

"If you want to go into mathematics, doing the mathematics itself has to be the thing that’s the reward" True. In other words mathematics is the biggest 'Nerd Sniping' source ever existed on Earth. Math was my hobby when I was younger, and every minute of learning and thinking about theoretical math had a huge opportunity cost of not learning something more useful. Fortunately math is no longer a hobby of mine, so my knowledge is becoming more rounded/practical nowadays. I have became an expert of detecting any nerd-sniping, which at a younger age I was prone to. Surely I still teach math to my children when they need help, but otherwise I use math only when needed, not l'art pour l'art. Coincidentally I comment much less on public forums too. (Another non-useful activity.)

In my perspective it is more believable to get into AI as a young guy without higher education than someone like me: a middle-aged person with higher education with a family and a reasonable corporate software engineering job. For me to get into AI research, I guess I would have to work for less money for a while than my current boring Software Engineering job, so the golden handcuff prevents me from doing it. While for a young guy there is nothing to lose, they can shoot for the most interesting thing right away. When I was young, I graduated from university with an AI specialization, but no one cared, especially not in Eastern Europe where I live. So I got into reasonably-paying slightly boring software engineering.

The main problem is that as a software developer most of us worked at software product companies all our life and I noticed that often we know more about software product design and product strategy than some of the 'idea people'. People who never worked as developer OR product manager OR designer and have no knowledge in any of these fields and have no money to afford a payed developer are rarely the good co-founders. I absolutely agree with the OP that most of these people need simple CRUD apps, and the reason they need a 'cofounder' is that they need a mediocre programmer but have no money to pay them. I don't even like the term 'technical' vs. 'non-technical'. Ideally a smart person in the tech business should be somewhat 'technical'. If absolutely and categorically non-technical then they should be really-really exceptionally good in something that is absolutely needed for the business. (For example having a large following.)