rafiki6

No technical analysis, but all models experience drift eventually.

Pretty fun! A few questions:

1 - are you planning to let people write their own prompts?

2 - when will you share the model names?

Because OpenAI is a single team with a singular focus.

Google is a behemoth with multiple products and a lot of people with opinions who you have to get through to launch a product.

Also, OpenAI has not unseated Google's dominance in search nor do I see this happening.

More specifically validation loss is irrelevant when you can't even sample out of distribution anymore.

You seem to have a serious attitude problem in your responses so this is my last one.

It's propietary company evaluation data, and it's for a specific domain related to software development, a domain that OpenAI is actively attempting to improve performance for.

Anyways enjoy your evening. If you want to actually have a reasonable discussion without being unpleasant I'd be happy to discuss further.

Did I say that?

The OP's post was saying it's somehow able to solve something new. It's showing a severe misunderstanding how how language modelling works.

I never said: - "it's not me it's the test" - "These systems are not just next token predictors"

None of the papers or blogs you've shared offer any points that actually rebutt what I'm saying.

And yes, we will eventually have them work in real time. Can't wait.

I have no idea what is shifting in real time. I formed this opinion of GPT4 by running it through several benchmarks and making adjustments to them, so my view is empirical and it was formed 1 week after it came out.

Your post says nothing of substance because it offers no substantial rebuttal and seems to just attack a position by creating a hand-waved argument without any clear understanding of how parameters in-fact impact a model's outputs.

You also completely missed my point.

Just because it's newly created doesn't mean that the structure of the language and the concepts it represents are actually new.

It's clear that whatever tests he writes cover well established and understood concepts.

This is where I believe people are missing the point. GPT4 is not a general intelligence. It is a highly overfit model, but it's overfit to literally every piece of human knowledge.

Language is humanities way of modelling real world concepts. So GPT is able to leverage the relationships we create through our language to real world concepts. It's just learned all language up until today.

It's an incredible knowledge retrieval machine. It can even mimick how our language is used to conduct reasoning very well.

It can't do this efficiently, nor can it actually stumble upon a new insight because it's not being exposed in real time to the real world.

So, this professors 'new' test is not really new. It's just a test that fundamentally has already been modelled.

The problem is everyone is doing this.

There's no moat anymore. Building some dumb webapp to sell to people to make their lives marginally more convenient is not sustainable model.

So what do you do now? Seems like the only option is to move towards a life and death industry.

But when everyone does this, it's game over.

It's easy to say embrace it, but we aren't that far off from a fully capable developer. People are already stitching task based flows to make it basically build a whole thing and a big thing.

The majority of the software industry is in a lot of trouble and honestly given that it was one of the highest paying industries that's not great for the economy.

What are people supposed to even do? What's next for all these displaced workers?

That's pretty much what GPT is.

That's an interesting take on what to do personally. I've been thinking the best think to do is just to work hard, keep head down, and if the axe falls, expect that it will likely take at least 1 year to land the next role and it won't be an ideal setup and to just ride it out.

Generally speaking no company will willingly canabalize it's current product line for an unproven and premature technology in a new product line.

That's also usually how companies get displaced. I'm not sure we'll be driving any cars from established manufacturers if EVs gain wide adoption and reach a price level that's affordable to the average person.

I don't quite understand the strategy of sitting and waiting on a technology competitive advantage. Surely, to get the technology ramped up and integrated into their manufacturing process isn't an instantaneous thing?

I do agree that Tesla's entire value is in their battery tech. Their 'premium' cars are actually about on par with Korean and Japanese mid-tier vehicles in terms of quality at best, by every objective measure. But their pricing is luxury level simply due to the cost of batteries.

Voice assistants, chat bots etc. are all premature technologies that are dying slow deaths.

The primary reason is quality control. The way these devices are tested can never truly represent the massive variation which would impact their ability to process and parse sound. For example, the wide range of accents for a language like English. The variations in ambient noise in real world environments etc.

Beyond that, generative language models have only recently become powerful, but they need server side processing which is incredibly expensive for the majority of contexts where an AI is useful. Think of call centers. I HATE when companies try to use voice AI in call centers, thinking it's a good way to save money.

Bank Call Center Phone Cal example:

Voice AI: "tell me, how can I help?" Me: "I'd like to request my final statements for a recently closed account." Voice AI: "I'm not sure I heard that correctly" Me: "Statements for a closed account" Voice AI: "Do you want to close an account?" Me: "Statements" Voice AI: "I'm not sure I can help with that, let me get you to a customer care representative. Please enter or say your 16 digit account number"

What was the point of that? The vast majority of customers know how to use online banking to get information at this point. Why did you make me do this? And then, imagine I get disconnected and need to call back. Go through the same process again. The bank may have saved some money (questionable, as they have already outsourced the call center anyway to somewhere cheap), but they've irked me so much, I'm always ready to switch. To bad all banks are the same where I live.

Point being, the tech is too premature, unfinished and hard to build and it offers questionable value.

Voice AI is mostly useful in situations where I need to be handsfree. I think what SoundHound is doing makes the most sense. Sell your Voice AI as an API to manufacturers who build good quality speakers.

Everything else is pointless right now.

I just want to say I find it very enlightening to see my previous post have dual interpretations because I wasn't precise. It really explains why ancient literature can cause so much grief!

To clear it up, I did mean that despite the massive increase in workers, with today's layoffs most of them have not yet reverted back to their pre-pandemic 2019 headcounts.

Where did all of these workers come from? Overall as an industry we did not actually substantially increase our headcount. We're about at par. Why?

- People did retire/leave the industry due to 2020 layoffs/Covid - Big tech, who had the largest "influx" actually just vacuumed people from other non-big tech (e.g. banks, mid-size, startups, etc.) - Much of the headcount growth was actually in non-technology positions, especially HR, marketing, sales etc. because many companies wanted to capitalize on pandemic growth

It seems to me like we'd probably be better off partnering with domain experts in Genomics who want to build software that can be used across the board. Sounds like an interesting opportunity for a business. I'm open to the idea if anyone wants to chat, let me know. I'm SWE but would want to partner with a Genomics Expert.

Honestly, I think people like you are the right kind of people to start private enterprises and bring along professional software engineers to help you build something incredible in the space.

I wish more fields would just start adopting the product/engineer partnership that Software companies have perfected. Engineers are very good at what they do. Product people are very good at what they do. They need each other to build things. Sure, engineers might know enough about product to get by and product people might know enough about coding to get by, but the reason it works is because each one is an expert in what they do and are equal.

Its no different in finance, healthcare, genomics etc. I'd love to work in a setting where I'm paired with an SME product manager in a domain I have no clue about and they respect my work and I respect theirs and we are partners.

This is one of the biggest factors that made software/internet companies explode. They respected people who build software. They didn't need to. A bunch of MBAs could have easily just decided that the best way to run the company was to treat the people building the product as a cost center. Many did. I think that's probably one of the reason for the lack of innovation and down fall in many old tech companies like HP/IBM.

The ones that treated SWEs properly and valued them accordingly, did very well.

I'd flip the question around. Would engineers benefit from fewer big tech?

It's important to put this in perspective.

1- layoffs are happening primarily to non-technologist roles

2- even with the layoffs, we're still no where near head counts for 2019.

Companies are taking advantage of the macro situation to clean house. It serves a dual purpose. Get rid of low performing employees and clean up any non essential functions, while also pleasing investors.

If by Web3 you mean decentralized networks, then I'm not sure it has much of a future. I'm not sure why we thought there was. We've had decentralization since the dawn of the internet, and it never took off.

My view is that decentralization is not the panacea it's being claimed to be. Humanity is not inherently trustless and we naturally want to hold something or someone accountable when things go wrong, and if we think this is solvable by complex technical transparency, we've missed the whole field of UX and why Apple is so successful. Do you really think the average person cares about a decentralized version of Instagram? Even in networks that seem decentralized, a lot of times scalability issues can only be solved through centralization.

Beyond that decentralization is naturally against the establishment, and the establishment won't go down with a fight. The only future decentralization has is if the establishment keeps getting weaker.

If you mean anything else by Web3, I'm not 100% sure. I have no idea what the term means tbh. It's being used to peddle a whole host of things.

I also think Web3 assumes that we've finished being in "Web2" whatever that even means. I guess Web2 is really the emergence of cloud computing, SaaS and mobile computing networks. I'm not sure that's quite over. I'm not really sure we need to evolve this. Or does Web2 mean big web companies who sell ads? If that's what it means, we can deem these companies (e.g. Google) as necessary infrastructure and regulate the crap out of them. The problem with these terms is they're just meaningless mumbo jumbo mostly created by people who have no depth.

Absolutely nothing is wrong with it.

Canada's main issue is that we've allowed duopoly on ALL telecom (i.e. we essentially have 2 major national communications networks encompassing all forms of telecom, including phone, TV, cell phone, internet etc.)

Rogers is one of those networks, and it basically covers 30-40% of the country's communications.

There are a few smaller regional players, such as Shaw, Cogeco, Videotron etc. I think we might be the only developed country that's allowed this. I know in the US, there are 5 major mobile networks, some of who offer internet etc., but there are major internet providers separate from the mobile network providers.

The reason this has happened, is because building networks is really expensive making it so that being a regional provider is a tough business, and the 3 major companies have played the laws to their advantage. They've done acquisitions over time, and used the rules around foreign investing to block a lot of major development in 3rd and 4th networks from happening.

This is why the Rogers/Shaw deal should be blocked and the government needs to allow foreign investment to be able to come in and build out networks.

This problem can only be solved in 1 of 2 ways, and only through regulation unfortunately: - A complete overhaul of our rules which will allow foreign investment in our communications networks in Canada - A break up of the 2 networks

The second is not likely to happen. Rogers/Bell/Telus are some of the wealthiest companies in the world, and are at the top in terms of deep pockets in Canada. They will essentially fight the government on any sort of attempt to break them up and handily win.

The government has a much better shot at changing the rules to allow foreign investment, but they don't want to do it. The reason that's the case is because the leadership of the telecom companies has many shared members with prominent politicians, and also outright owns all forms of media in the country except for CBC, Post Media, and Corus.

Post Media is a known right-leaning org and Corus is effectively owned by Shaw, so their sway in the country's politics is not as big as Bell Media and Rogers Media who both own much bigger networks.

That means that they can run a very effective smear campaign against any politician or part who dares to allow foreign competition to enter Canada.

Yes, but even if they take him to court, it's highly likely that Musk and his legal team drag this out. It's not going to be a simple case. No contract law case ever is. And at that point, Twitter's business might keep suffering, to the point that their fair market value plummets, and they enter a very difficult financial situation. Then Musk can swoop in with a much lower price offer and they will accept simply due to their financial position and not being able or willing to continue the case in the courts. I think Musk will end up buying Twitter but at a much lower price.

This is mostly from the perspective of software companies, but I've noticed it applying more and more to non-software companies. (Notice I didn't say "tech" companies, because that's a meaningless term).

The only limit you'll experience from job hopping is that you'll essentially be stuck in an individual contributor (IC) role if you don't stay in anyone place past 2 years, although even that isn't always true. BTW, IC here means any type of role where you're mostly limited to project work and not strategy work (team leadership, technical leadership etc.) Generally the split is senior/staff vs principal type levels, or manager type levels.

The reason this is the case is because to have org-wide impact and get promoted you usually need a few years, have seen through a few big projects, and have built a reputation at one place. This is especially true of large organizations, maybe less so of smaller sub 500 person companies.

But if you want to hop every 2-3 years, especially in software, go for it, especially if you have no ambition for leadership or feel that you're stagnating where you are. You'll keep your IC level salary at market rate which is always going to be much higher than whatever your company is paying you.

Granted, there will be some managers who don't want to hire you because you're a "job hopper" but in my experience, these managers are not good managers and not people you want to work with.

I'd say the minimum at one place is essentially 2 years, although the conceptual minimum is seeing through work at your level end-to-end (e.g. finishing a big project). The reason I'd stick to 2 years though is because most people don't believe that anything can be achieved in less than 2 years.

Note, this advice likely doesn't apply in a bad job market. The reason that's the case is because bad job markets actually tend to allow people who are entirely unqualified to become hiring managers at their companies at a much higher rate than good job markets. That's usually the case because when companies start cutting costs they might fire a bunch of people, including middle management, and assign the responsibilities to cheaper people. There's a reason those people are cheaper.

Anyway, these people then become hiring managers and many of them hold the "job hopper" stigma. Again, bad managers.

This is why, as much as people love to complain about it, the big tech companies might actually be doing hiring right. Hire for the core competence and skills and not for the frameworks/languages. The thing that is missing in the process is the core skills should be tested only once, not every time you interview.

Yes, but then this takes away the idea of one private key to rule them all for identifying someone. The moment you've lost that key, you've lost access to all the services relying on it. Each might have their own backup mechanism to fall back on, but suddenly the UX really sucks.

You're confusing a highly optimized loss function with AGI. Robots will continue being better at doing rote things humans do, and we will continue building them predictable environments to do them in.

But AGI is another beast. We don't understand causality in ML. We are no where near getting there.

I could foresee building some incredible human-computer interfaces, but I don't think we will get to a self-motivated thinking machine any time soon. We don't understand how our own consciences work. If we did, we would be able to better model human psychology and behavior, but we can't yet.