kypro

I'm from the UK. What's in the image?

Until they can get the confidently incorrect problem solved, I don’t see it being a viable replacement. Maybe if you have agents that are aggressively checking the main ones’ output, but that seems unnecessarily wasteful of resources to me.

2 years ago AI could even count the number of letters in a word or do basic maths. Now we're at the point where we can't trust it because makes occasional errors.

Don't get me wrong, I agree with you. These are things that need to be solved first, I just don't have reason to believe these are particularly difficult problems.

Anyway, I enjoyed the convo. Was interesting hearing your thoughts. I would give anything to be wrong.

We're so screwed man.

It was only a few years ago I was debating AI risk with people and they were saying, "but obviously we're not stupid enough to give it access to the internet!!"

And honestly, it wasn't always easy to argue with that. Like yeah, maybe we would take this stuff serious and run it on a completely isolated machine with no external IO or network access. Maybe my opinion of humanity is too low.

But it's hard for me to read this and believe anyone cared risk here beyond the most surface level concerns like adding some minor restrictions to the network. Not even I would have expected us to be this reckless.

With this access, our models performed a series of privilege escalation and lateral movement actions in our research testing environment until the models reached a node with Internet access.

This simply should not be possible. Call me crazy, but I don't agree with giving a frontier AI model with unknown cyber capabilities access to a restricted network in the first place, but clearly this was an incredibly poorly designed sandbox.

If one of these models have a genuine step-level capability improvement and start to pursue their own goals, then who knows what might happen. I mean who knows, maybe it's already infected critical infrastructure. We have no idea what these labs are cooking up, where they're running these things, and neither us or them seem to have any clue what their capabilities are.

Every day that passes it becomes harder for me to understand how there are still people denying what's coming.

As always is the case, nothing will be learnt from this.

However, I don’t see a viable path forward where that occurs, and we all still have jobs. Why would I be kept around if a superhuman AI can do my job 1000x better and faster, and at a drastically cheaper price? Or, if I was kept around as a human QA agent, why would I be paid nearly what I am now?

Well as an AI doomer I agree with this. In my view you will lose your job, along with everyone else, and we'll just need to hope that those in power will give us some cash to feed our families. Ultimately us regular folks won't have the collective bargaining power of our labour and could not revolt with nation-wide strikes, nor are we any physical threat for those with power anymore.

But that's the good outcome. This assumes AI remains obedient to it's human owners and that bad actors don't have access and can't use it to create biological weapons and other civilisation-level threats.

I know we're not there yet, but the history of AI is one of continuous step changes in capabilities with each algorithmic breakthrough. AIs today are far more knowledgable and much quicker than humans, but still missing a few bits that keep them from being broadly super human. One or two algorithmic breakthroughs is all it takes though. It was just two breakthroughs over the last 8 years that took us from not even having code-autocomplete to having reasoning models advanced enough to solve math problems which have stumped humans for decades.

The problem is if you don’t understand the root cause, you’ll at best feed the LLM a list of symptoms

Today maybe.

When Cloudflare has an outage does the CEO need to know the root cause though? If he just assigns some engineers to investigate, fix and report back to him – what's the issue with that?

Assuming you agree with my analogy, can you explain what it is that you feel an agent or group of agents assigned to an engineering tasks could not do that a human can? And if can't do it today, why do you believe it wouldn't able to do it in the future?

Are we just disagreeing about timelines here or whether humans have some magical quality which algorithms and AI can never capture?

I feel like it's reasonable to disagree on timelines. It's quite unreasonable to assume human coders are special.

GPT-Live isn't a particularly intelligent model. It needs to handoff for reasoning and more complex tasks.

You shouldn't expect it's responses to be as reliable as a frontier model. It will sometimes say odd things.

It's way easier. People don't need to read code anymore, nor will they have to care about building mental models about the system as this article suggests.

The way to think about coding agents is that a good one should in theory literally replace the developer entirely. No, a whole organisation of developers.

In theory a product owner should just be able to dictate how they want to product to function at a high level and the AI should take care of the rest in the same way a human programmer or team of programmers would have in the past. If there are conflicts in what's being requested, then the AI should be able to recognise that and ask for production direction.

As always with every generation of AI people seem to way over index on the now.

- "They're good for autocomplete, that's about it"

- "They're good for quickly mocking up small functions, but they make a lot of mistakes"

- "They're good for scaffolding some parts of the system if you're good at prompt engineering"

- "They're good at writing most of the code, but humans will always need to do the last 10%"

- "They can write all of the code, but humans will still need to architect the system"

You are here. Perhaps this is where progress stops. I wouldn't bet on that however.

AI is making programming an irrelevent field.

As someone who lives in the UK 100%.

There are things I don't say online anymore, not because I don't think they're correct, but because my opinion could be deemed offensive.

Whenever I say something I think could risk being seen as offensive, I tend to add a lot of footnotes explaining my reasoning for the things I'm saying and why I'm not trying to offend.

I also never share remotely offensive humour to friends over social media or messaging apps anymore, which used to be fun. Lots of people have been arrested for jokes so just not worth it tbh. Even if I don't think it's that offensive the law is too subjective to take the risk.

I see a lot of these "don't use [insert product]" posts and it's normally always BS about how some exec at the company has the wrong political views or upset some community who was the product by taking it in a different direction from what they wanted.

I think this is probably first time ever I read one of these and was convinced immediately that I should probably look into using something else.

I don't think your wheel analogy holds. Wheels and rockets are fundamentally different technologies.

LLM technology itself has nothing to do with general (human-like) intelligence.

Neural nets to the brain are like a plane to a bird. Saying LLMs have nothing to do with human-like intelligence is fundamentally ignorant of the field and the inspiration behind many of the algorithmic experiments.

LLMs clearly do things differently, but that doesn't mean they can't be generally intelligent, or they can't produce output that's remarkably similar to a human.

And to be clear, I have studied AI, published on AI and work in AI.

Do you people get off by fantasizing about people losing their job, or worse? (whatever you are implying)

I'm the exact opposite. I've been terrified about AI for decades. I had nightmares last night as I do most nights about AI. I think humanity is years away from a disaster and I'm pleading for people to see.

But what I've learnt is that every step closer we get the goal posts just move a little more. I'll note just a few years ago HN was saying ChatGPT and LLMs were just stochastic parrots or advanced autocompletes. I pushed back against this and said within years they'll be raising PRs on their own from tickets. But everyone told me, "nooo, they couldn't even do basic maths!!"

Remember this comment when the time comes. We could have acted, but we didn't because every advancement people dismissed and seem to think that further improvements would not be made. You will be wrong.

I'm just going to be honest because lying won't help.

Realistically you will almost certainly fail because almost all startups fail.

There was a post on Indie Hackers many many years ago now where someone looked through all of the projects posted there over the years and found that something like 1% of them succeed (may have been less tbh – it was so long ago).

You are more likely to succeed if you have serious money to throw behind it or you are able to get investment, but if you're just a dude trying to start a businesses online with some savings you're more or less guaranteed to fail statically. This wasn't true of in tech in the 90s and 00s, but things have matured a lot since then. The fact you have users is actually impressive and imo you've already done better than the vast majority do.

I'm saying this because the first thing you should be doing if you're not already is assuming failure with an extremely high confidence. If you have no plan for failure then you need to start coming up with one now.

I've been through this myself. Startups were my life in my late teens and early 20s. I had a breakdown in my mid 20s because after some early success in my teens, I just couldn't get anything to work anymore. I was also isolating myself from friends and family and the realisation that I had basically lost some of the best years of my life behind a screen and text editor for nothing ate away at me.

Quitting is not failure. Here's what you need to do now.

- Come up with plan assuming your startup will fail.

- Decide the point at which you will stop.

- When that time comes execute your plan like a zombie through the pain.

Your emotions will tell to keep going one more day and your plan however good will almost certainly not be what you want to be doing because if it was you'd already be doing it. Just keep going and you'll come out of it in the end.

Good luck. Wishing you the best bro.

If you have a magic wealth creation box you don't need labours or people to sell to. You just ask the magic box for the resources and you have them.

The post-AGI world doesn't depend on people buying stuff.

Post AGI economy won't depend much on human demand.

As you say there will be no job to strike from. And pissing off the government probably isn't going to go down well when they're you're only hope at feeding your family.

Unfortunately in this world income taxes have dropped to 10% of their previous levels and the government is now primarily funded by AI. There are rumours some leaders are questioning why they're even redistributing wealth when they could just take it. Citizens are not needed anymore, they're just a burden and taking from those at the top.

The good thing is that LLMs will not bring forth AGI.

Even if this were true you'd be silly not to see the larger trend.

LLMs and the transformer model were just a step function in AI capabilities upon the last generation of AI.

Whether it's LLMs or not is irrelevent. Major algorithmic improvements in AI have happened every 5-10 years over the last 50 or so years, and now they're happening maybe every 3-5.

The question you should be asking is what happens when the next step function in AI capabilities occurs?

Arguably people worrying about job loses are not worried enough.

If it's any consolation, I suspect people worrying about job loses and a loss of meaning will soon realise these are minor concerns.

What people always struggle to model with AI are the step functions that occur in capabilities as new algorithmic advancements are made. Historically those come maybe every ~5-10 years, although the pace has dramatically increase in the last decade and last 5 years.

Most people right now are plotting a linear improvement over the several years as models get bigger, harnesses become more complex and capable, more people adopt AI for efficiency, etc. Some are adding small exponents, but most are fundamentally modelling wrong.

What's more likely to happen is that in a few months someone at some AI lab will leak that they've made a breakthrough internally. Then another leak will come, and another. We'll know something big is going on because we'll see strange things happening within governments, then FOOM.

We're likely now one more step function away from something that's not just capable of taking your job as an intelligent and knowledgeable human, but surpasses your ability so significantly that you can't even comprehend what's happening around you.

This job concern is really just a concern perhaps of the next year, or maybe at a stretch, a few years.

It's not the purposelessness or job-loss that should have you concerned, but the pandoras box that we're opening. Once that box is opened every problem bottlenecked by intelligence is solvable practically over night, and therefore the way the society will look in a decade is almost impossible to imagine.

The problem of how to cure cancer is solved, but the problem of how to kill everyone with a super virus is too. The problem of how to cure alzheimers is solved, but so is the problem of how to hack the human neural activity is too.

I genuinely do not understand how people still don't see what's coming, but I try to remind myself that these are the good problems and we should enjoy this for as long as it lasts.

Business loves them, actual engineers hate them.

If you care about your career you should be one. If you care about engineering you might have other concerns.

I find mental burnout is largely a product of my ability to keep pace.

I feel burnt out in spoken conversations because I find it very difficult to process sentences at speed. But I don't feel that at all when reading, writing or listening to podcasts because I can go at my own pace.

Similarly I love maths and it doesn't give burnt out when I do it myself, but in school I remember it being agonising trying to get my brain to keep up with everyone around me.

I think this is why AI burns out engineers. Agents can do things do much faster than us so we're no longer comprehending things at our own pace, but trying to keep pace with the AI.

I feel that useless thing too. It hurts me more than anything. I've poured my heart into so many blog posts, tutorials and open source projects over the years. They were genuinely hard at the time. Each tutorial or code project might have been weeks of effort – and it showed. Today an AI could do the same in 10 mins from a single prompt. What was a source of pride for me, now just feels like a major waste of the last 2 decades of my life.

Maybe that's for the best though. I think in hindsight I over index on the value of knowledge. Maybe there are more important things in life.

Extremely weak bid, no?

PayPal was worth 5x this a few years ago... Even pre-pandemic it was worth far more. Hell, just a year ago it was valued at ~$80 share.

$60.50 per share would value them at just 11x their current earnings. I get PayPal isn't exactly a growth company anymore, but I'm not sure I understand why shareholders would agree to this offer?

Yeah, this is exactly the type of thing I'd support.

The UK (where I live) emits something like 0.5% of global CO2, but we're one of the largest economies in the world.

If we're going to do anything it seems we should really be doing everything we can to use our economic might to try to do something about that other 99.5%. Where as at the moment we're wasting our time obsessing about how we bring down that 0.5%, and providing an economic horror story for why no other country should replicate what we're doing.

That's a fair push back.

For a specific technology to become viable, you often need progress in several individual technologies to occur. Some of these will be exponential and surprise us, but many will see slower progress.

Either way, I suspect progress in robotics will be slower than AI, but I also think we'll see a lot of investment in robotics over the next decade given that AI opens up a lot of new potential applications for robotics.

I suspect any task that doesn't require extreme battery life or extreme dexterity will be doable by robots much sooner than most people think today. I think we could have decent-ish humanoid robots within a decade – robots that can prepare food, clean hotel rooms, empty bins, etc. I think Elon's bet on Optimus is quite likely to be viewed as very ahead of the curve with time.

However, it will take much longer for robots to do everything a human body can, but then most jobs don't require we push the physical limits of human body.

That's why I think 60% is reasonable. And even then I'm not saying it will happen, I just don't think that's a bad bet if given even odds.

Unpopular opinion, but we need to start using hard and soft power if we want to fix this problem.

Europe cannot fix climate change on its own, and we already represent an increasingly small share of total emissions.

The question now is how we get other countries onboard fast, and if they continue to build out cheap coal-fired power stations what are we going to do about it? Something or nothing?

Unless we can get every nation onboard, we should switch strategies to one of managing the inevitable rather than hopeless trying to prevent it. Costly energy policies that reduce global emissions by roughly 0.5% (basically what will happen if the UK ever achieves net-zero) are a total waste of money and time.

Those who care about this issue need to get real and need to stop pretending that the solution is for a few more people in the UK putting solar panels on their roofs. Climate change is a global problem. If your solutions are domestic then they're not serious.

UK is the worse for this too.

We have a story in the news currently about a foreign child gang rape leader (yes, those are things in the UK) who raped at least 30 children and was released early after serving just 14 years in prison.

We have major children sexual abuse scandals in this country basically every year too. Politicians never care about that though.

There's always been an arms race in anti-bot technology and more sophisticated bots.

I'm sure, they can add a jitter, but then you just change how you detect / weight detection.

If I took you back to 2020 and said in a little over 5 years there will basically be no human coders writing code anymore you'd almost certainly not believe me.

And similar things can be said about many technologies in recent history – cars replacing the horse, first flight to man on the moon, even the creation of early internet to its mass adoption.

You're talking generally a decade or 2 for society to completely change from the rapid advancement of a new technology.

I'm not saying I agree with the 2035 prediction, but it doesn't seem impossible to me, if AI can help us improve the pace that we're already developing disruptive robotics.

In 2010 the idea of self-driving cars and autonomous delivery drones seemed very sci-fi and a long way out. But today, just 15 years on, these things are increasingly starting to be rolled out.

If they dropped that 95% number to 50-60%, I think I'd probably lean towards agreeing. Not because it makes sense in my gut, but because the logical part of my brain knows exponential trends (if one exists) do things that we wouldn't instinctively predict. But even if you assume exponentials 95% does seem very high.

It's very different these days. I've said similar things a number of times.

~15 years ago HN was full of really interesting deep dives into technical topics. There used to be articles about different algorithms, frameworks, programming languages, etc. Back then I even used to write tech posts and tutorials myself and share them here with some success.

The other thing I miss are the entrepreneurial posts... My friend introduced me to HN because I was interested in business and tech, but I didn't really get the appeal of HN for a while. But I remember when I was like 19, browsing HN during my lunch break at my crappy retail job and found this,

https://sofamoolah.com/2011/07/14/the-conclusion-a-6-figure-...

That post at the time got me totally hooked on HN... It was these real world posts about the struggles, successes and failures of normal people starting businesses online with their tech knowledge that I loved.

I literally never see articles like that here anymore. Although, I'm not sure that's just HN... I suspect the world has just changed. Those days when someone could start a successful company from writing some code in their bedroom has more or less gone.

I thought that too. The prompt is full of metaheuristics.

I remember a couple of years back when people were saying how prompt engineering was a skill, and reading this prompt kinda took me back to that.

Were I to guess, the reason the model couldn't do this itself is because most of the time, for most problems, a lot of this is bad advice.

In search optimisation you're often trading between time and quality. A very broad search will return very bad results for a long time. Where as a more depth oriented search with some heuristic will tend to return a pretty good result (if not optimal or close to optimal) quickly.

I'd assume models naturally want to find some middle ground there because that's the best thing to do most of the time, but for very difficult problems where a decent attempt isn't good enough you want a much broader search that doesn't have the time constraints. Much of the prompt seemed to be in that direction – really encouraging broadness of the search, preventing early convergence, and remove pressure of time constraints.

Who is the "political class" of the EU?

Highly educated people, often from wealthy backgrounds, who are very globally orientated.

They are generally not representative of the average person in Europe.

They are the people who European government appoints.

In the US, it seems more obvious how corporate money pulls the political strings, but my impression is that corporate influence is a lot weaker in Europe..

I agree, but not sure that directly contradicts what I said?

What am I missing?

Nothing really.

Some will disagree with this, but this is neither new or surprising behaviour from the EU. In the EU if the political class want something, it doesn't really matter what the public want or vote for.

In the US a lot of your dysfunction is from the fact that your political system actually "works". You maniacs actually can vote for someone like Trump (twice), and he can do stuff regardless of how unpopular he is among the political class.

Here in Europe things like democracy and freedom of speech are only permitted if our political class approves. We can decide things like tax rates, but some things we're not allowed to express opinions on, and some things we have no power to vote for or against.

With some exceptions most European democracies work like this and EU is really the gold standard of this system. They have lots of ways to do what they want regardless of how popular it is, and regardless of what the opinions of our elected representatives are.

Yes, I'm not sure it was clear, but that was largely my point – to say it's "worthwhile" as a flat statement probably doesn't capture the nuance.

It's obviously going to be worthwhile in some sense as almost all skills are worthwhile to some degree. But normally when we're talking about whether something is worthwhile to learn we're weighing up other competing things we could be learning instead. In that sense I think it's far more debatable about whether coding is worthwhile to learn.

Learning Classical Latin and Ancient Greek is also worthwhile.

In what sense?

From this perspective everything is worthwhile.

If "worthwhile" is relative to learning other skills then Classical Latin and Ancient Greek is near the bottom of that pile, and you probably know what which is why you cited them.

So then the question is where coding sits on that hierarchy, and if someone should learn to code vs acquiring other skills.

Personally I'd not advise anyone to learn to code today. It's not just that I'm not convinced of the utility of the skill going forward, but I also think it's very hard to actually learn to code competently for someone starting out today. I had the luxury of spending decades coding without LLMs professionally. No one learning today is realistically going to have the opportunity to compete with old school coders with decades of experience writing code by hand, even if we assume the skill will be worth anything.