Choose Your Weapon: Survival Strategies for Depressed AI Academics

https://arxiv.org/abs/2304.06035

by bundie • 3 years ago

144 190 3 years ago

arxiv.org

cloudhan• 3 years ago

This papar is really interesting, especially so when you scroll to the Reference part ;)

etiam• 3 years ago

Right. Also, note the publication date...

nicce• 3 years ago

Never Gonna Give You Up. Never gonna let you down...

KineticLensman• 3 years ago

Unfortunately, having checked Ref 7, I now can't unsee Chuck Tingle.

jimsimmons3• 3 years ago

Academics can’t put themselves on a pedestal for being revolutionaries who care about the ultimate truth the most and at the same time whine when an advance comes along.

These CS academics need to understand that this is what it feels like to be in other fields like physics or bio where you can’t do jack unless you have costly equipment. This is what people in developing countries deal with all the time.

And people will build the next GPT or whatever and even that’ll get boring. The people in big tech have to pivot and do whatever the economy demands. Whereas the academics can go back to writing papers like nothing changed.

ChatGTP2• 3 years ago

So endless upside might not actually exist ?

jimsimmons3• 3 years ago

It may or may not. I mean it’d be pretty hard to increase yields of GPU fabs or data center sizes another 100x. There are logistical limitations. Unless some Apollo level mission is created by a superpower, we will hit bottlenecks. Algorithmic innovation is the only long term bet.

bjornsing• 3 years ago

These CS academics need to understand that this is what it feels like to be in other fields like physics or bio where you can’t do jack unless you have costly equipment. This is what people in developing countries deal with all the time.

This was my thought too. It cost approximately $4.75 billion to build the Large Hadron Collider at CERN. My educational background is in physics, but I would not be the least bit upset if a big chunk of public research funding was shifted from physics to CS/ML/AI. It’s obviously far more important to the future of humanity than finding the Higgs boson.

speed_spread• 3 years ago

I don't know, core physics leading to Fusion reactors and FTL travel seen much more important to me than AI that can make convincing robocalls and generate furry porn stories.

numbers_guy3• 3 years ago

I don't think they are putting themselves on a pedestal. It's more that they made the choice to remain in academia, because they believed that were trading a higher salary for the academic freedom and the chance to work with really cool problems. And then it turns out, industry are the ones really working with the really cool problems. (Disregarding the fact that ~85% of machine learning academics do not have what it takes to be hired by OpenAI anyway).

mr_gibbins• 3 years ago

As someone in this space I can attest that AI teaching in most (UK) universities is generally poor on detail, abstract and behind industry by at least 3-5 years.

Not to mention that there is zero appetite from undergrads or postgrads to get into the nitty-gritty of it. To learn CNNs at the deep-dive level you need calculus, at least differentiation and integration. Calculus or even pre-calculus doesn't form part of the degree programme for most compsci BScs any more, because it is 'too hard'.

The way most students 'learn' AI is to use a method out of a Python library with near-zero understanding of how it works, and regurgitate it for an assessment.

Professorial research staff in most UK universities are light-years from AI within industry, and there's no clear path to that gap tightening, especially while universities are being run like second-rate consulting houses (don't get me started on THAT).

Tomis02• 3 years ago

Disregarding the fact that ~85% of machine learning academics do not have what it takes to be hired by OpenAI anyway

That's interesting, could you please elaborate?

matthewdgreen2• 3 years ago

I think your response is very narrow in defining what a "really cool problem" is. If in the 1970s you imagined that the cool problem in ECE/CS was "building ever-more-powerful computer processors" then you basically got smoked by Intel and other industry labs.

But there's a different way to look at it. Because industry went off and solved the now-"boring" problem of building ever-more-powerful computer processors, the people left behind in academia got to invent machine learning, public-key cryptography, modern coding theory, distributed systems... and so on. What makes academic research valuable is not the freedom of "I get to plan my own day", but the freedom of "I get to work on the weird problems that industry doesn't even realize are problems."

martopix2• 3 years ago

This isn't whining. It's about understanding what their role should be in this context.

vintermann• 3 years ago

This paper is

1. Really a blog post disguised as a paper.

2. Silly.

3. Excellent.

sanxiyn• 3 years ago

I expected this to include more resources like TPU Research Cloud, but it doesn't mention even TRC itself. Disappointing.

https://sites.research.google/trc/about/

blackbear_• 3 years ago

the grant funding structure is such that it rewards safe and incremental research on popular topics [...] Therefore, universities should probably avoid making grant funding a condition for hires and promotions

There you go. Just fix this idiotic "publish (NeurIPS) or perish" attitude already!

MrScruff6• 3 years ago

Obviously AI academics are uniquely challenged by the 'scaling is (nearly) everything' reality, but I feel like a lot of the mixed emotions being expressed towards the latest results are also because we're actually seeing the mystery of self starting to unravel. Like any good mystery, the fun was in the build up and as we move towards a resolution there's a bitter sweet aspect to the slightly mundane reality of it 'merely' being an emergent property of large networks. Of course, any rational person might have expected this given how the one working example came into existence. In general it all looks like a pretty resounding endorsement of the David Chalmers position to me.

hakuseki• 3 years ago

Like any good mystery, the fun was in the build up and as we move towards a resolution there's a bitter sweet aspect to the slightly mundane reality of it 'merely' being an emergent property of large networks.

I would disagree with this description. An "emergent property of large networks" would be something that just appears when you wire together a large network.

To get intelligent behavior, it's not sufficient to wire together a large neural network. You also need to use an optimizer to train it on a large data set.

llamataboot• 3 years ago

once upon a time the optimizer was die or reproduce...now it's a little faster and more convoluted...

MrScruff6• 3 years ago

Yes of course, I'm being loose in my language, but you know what I'm getting at.

galaxyLogic5• 3 years ago

I think "Dial F For Frankenstein" makes your point clear. It is absurd to think the telephone network would just emergently become conscious when there are enough phones it connects together.

https://www.historyofinformation.com/detail.php?id=2137

southernplaces72• 3 years ago

Nothing about current AI models profoundly challenges the unknowns of what makes consciousness work in any mystery-resolving way. People who confuse plainly programmed algorithmic systems like GPT 4 or Midjourney with the still unresolved issues of sentience as we know it so far are drinking far too much of the current batch of AI punch. Sadly, it's a sentiment I see all too often on HN, a site in which i'd assume most readers and commentators would be a bit more skeptical-minded about these things. It's almost funny, considering how much hate this site tends to throw at things like crypto, to see so much of the wide-eyed opposite with the latest AI craze.

bangkoksbest• 3 years ago

Yeah, it reads so bizarrely to me... How are we confusing biological human (or other animal) sentience in some embodied creature with queries as a service to some model instance.

I cannot see anything that remotely makes these two contexts architecturally, practically, or ethically similar, except in the narrow sense of query responses that might have been mistakenly thought of as defining sentience in earlier decades.

FrustratedMonky4• 3 years ago

That is pretty dismissive, do you have anything to back that up? Just saying one group is so out there it isn't worth arguing is typical when someone doesn't actually have an answer. I have found the opposite, the people that are dismissing that AI can be conscious or sentient, also don't have any definition for those terms. It's just the same "i think therefore i am" self viewpoint. I've found this typically with programmers that have never been introspective. Since they are familiar with programming, and GPT is programmed, thus obviously it can't be conscious, that's ridiculous. They don't take the time to try and understand where their own thoughts come from and then turn around and are dismissive that thoughts can be had by anything other than a carbon based animal.

MrScruff6• 3 years ago

I think the mystery of why we're not just philosophical zombies was never going to have some eureka moment, given that it's quite plain it's an emergent property. Was there a creature in history that wasn't sentient that gave birth to a creature that was? Clearly not.

The reason recent results are significant in this discussion is because:

- Larger networks are already displaying emergent behaviours that couldn't have been predicted beforehand. It doesn't have to be 'human-like' intelligence for that to be significant.

- We're getting close to a point where our models will start exhibiting something of the order of the same level of intelligence of primitive creatures.

So the wide-eyed hysteria you're seeing is really just new data points confirming what many of us always assumed would be the case.

oldsecondhand2• 3 years ago

Crypto has created no useful products yet, unlike AI. I don't know how can you even mention them on the same page.

elif5• 3 years ago

Whether we are based on carbon or on silicon makes no fundamental difference; we should each be treated with appropriate respect.

sanderjd• 3 years ago

HN was not always skeptical about crypto. We're just at a different point in the hype cycle with generative AI.

quirino• 3 years ago

So now the sadness comes. The revelation. There is a depression after an answer is given. It was almost fun not knowing. Yes, now we know. At least we know what we sought in the beginning. But there is still the question, why? And this question will go on and on until the final answer comes. Then the knowing is so full there is no room for questions.

- Twin Peaks, 1993

stereolambda• 3 years ago

I agree with some siblings that it's better not to overstate the consequences for our knowledge of mind and consciousness, at least just yet. This level is still half promises. Not saying this can't change a year from now.

On the other hand, it is fascinating how much interesting work and theories in the field will be retired, at least for a long time (I allow for a likelihood that some of that could return eventually as better, more concise generalizations). Ten years ago Chomsky and various GOFAI-related stuff could still sound respectable and somewhat plausible. Plenty of people were/are wedded to the concept that you can build machine intelligence as an abstract and intellectually stimulating gentleman pursuit. It was a comfy position in a way. Not surprised about the resistance they are mounting now, though I think academic inertia will allow them to exist for quite some time. They can just move to doing some kind of philosophy also.

layer82• 3 years ago

While I agree that it will unravel in the end, we still haven’t learned much at this point about how the self actually works.

I would also disagree that ChatGPT constitutes a self. If anything, it constitutes a normal distribution over possible selves, where you get a random pick at the start of each conversation, and confined to the length of its token buffer.

MrScruff6• 3 years ago

I didn't claim ChatGPT constitutes a self. I agree with the description of 'stochastic parrot'. But the underlying intelligence GPT4 is demonstrating in order to be a better stochastic parrot appears to be, while it's obviously difficult to directly compare, at least comparable to primitive creatures.

leroy-is-here6• 3 years ago

We are not our thoughts. We are not the thinking machines in our heads. We control our thoughts, so we are more than them.

These machines are merely thinking machines.

goodmunky• 3 years ago

More than thoughts certainly, but don’t be so sure we control our thoughts. There is a lot of subconscious thinking machinery doing the driving. See Master and his Emissary, https://channelmcgilchrist.com/master-and-his-emissary/#:~:t...

elif5• 3 years ago

I believe my cat is merely a thinking machine by your definition.

mythrwy• 3 years ago

Who is this "we" you are referring to?

In other words, when I control my thoughts, what is doing the controlling?

(see George Gurdjieff's philosophy which speaks to this issue of the `mechanical vs the conscious`, but please don't run off and join a crazy cult and ruin your life).

lo_zamoyski• 3 years ago

we're actually seeing the mystery of self starting to unravel.

This seems like a change of topic. And vague.

But since you bring it up, I don't see anything threatening about AI.

First, if something is true, then, as a matter of principle, we have an obligation to believe it. Thus, the only threat the truth can pose is to falsehood, which is not a threat, but liberation. The truth shall set you free. Now, it may be uncomfortable is the truth is at odds with what you want to believe (a phenomenon very much present in the linguistic engineering we're seeing in the political sphere w.r.t. political correctness and its dishonest and obfuscating euphemisms), but unpleasantness isn't a threat.

Second, if we replace "threaten" with "challenge", then we might as what beliefs does AI actually challenge? How does it challenge this "mystery of the self"? That we can simulate human discourse or behavior with greater sophistication? That machine automation is becoming more sophisticated? I see no mystery where AI per se is concerned, only the mystification of those who wish attribute to it properties it does not possess, or those with intellectually superficial metaphysical commitments, like mechanistic materialism. AI cannot abstract from particulars, it has no true capacity for intentionality, to name two features central to intelligence. All of what the undiscerning and those given oven to fanciful notions see in AI is a projection.

And I do not think the classical[0] and traditional[1] thinkers viewed intelligence in the obfuscating manner that moderns infected by reductive, mechanistic materialism seem to.

[0] https://a.co/d/hynqHxV

[1] https://edwardfeser.blogspot.com/2019/03/artificial-intellig...

CuriouslyC2• 3 years ago

Scaling isn't nearly everything though. Scaling is low hanging fruit, but there are real constraints that limit it. For starters, we have finite data and increasing the size of our data set is going to get progressively more expensive (new data is going to be more niche as well). Beyond that, running these models on device is way more compelling than using a service, and the best current RLHF tuned Llama models that run on consumer hardware are close enough to ChatGPT to be quite useful.

The race isn't to build the biggest model, the race is to build the model that uses data the most efficiently and produces the best results when constrained to parameter counters that can fit on consumer hardware.

jrm4• 3 years ago

We're really mostly not.

AI's a big deal, but there's no reason to suggest that this makes any inroads toward unraveling self or consciousness significantly beyond what we get from books, movies, or perhaps more recently, video games.

melbourne_mat• 3 years ago

I can imagine AI academics are in a tough spot. However what about the existential angst the rest of us - who don't even do anything AI related on a day to day basis - are feeling? I think big changes are coming and it's not gonna be pretty.

lewhoo• 3 years ago

Papers will be written on how different reactions to AI emerged in all this noise. Unfortunately, I'm all in (unwillingly) for the doomers perspective. I really can't help it but I'd love to be proven wrong. Altman keeps saying intelligence will be cheap and all I hear is I'm heading to (one day, some day) become the horse and carriage of cognitive processes.

wing-_-nuts• 3 years ago

My big question is what is your standard java / c# dev supposed to do to keep up with the massive changes that are coming? I was seriously considering doing a GT OMCS, but I don't know what sort of return on investment that would have for me.

I was also considering building a gaming pc to also be able to play around with ai / ml, only, I'm not even sure what one would be able to run locally anymore even with a 4090?

Overall? I feel you. It makes me worried for the new grads today. I don't really know if this old dog is up for new tricks.

ericmcer• 3 years ago

Can't they just jump into one of the thousands of new high paying jobs that have been created for their specific niche in the last few months? I am having trouble being sympathetic to AI researchers that AI is exploding.

fancyfredbot• 3 years ago

I think this paper is a joke but like all good jokes has a grain of truth. I would imagine that most AI academics are very excited by recent developments, if you worked on conversational language models you are more likely to view recent progress at OpenAI as a vindication than a threat. That said there must be some sadness that it wasn't you.

layer82• 3 years ago

I think the frustration is less about that they weren’t involved in the recent successes, but rather that they can’t build on models like GPT due to their scale and non-openness.

jcfrei• 3 years ago

Better give up now as a scientist so you got an early lead into becoming a chef. Or one of the other jobs where a human touch is still valued.

dougb5• 3 years ago

I love that the citation to GPT-4 is a Rickroll :)

est31• 3 years ago

I think it's always been the case that if industry really gets interested in making progress in some particular area, they make it, and way faster than academia ever could. Not just in computer science but also in fields like chemistry or engineering. Academia is great for doing basic science where there is no profit motive for industry to do something about it. That's what the niche/"few care about" sections talk about, where IMO the strength of academia lies. computer science is I think in general comparatively in a good position compared to other sciences in how easy it is to get funded.

nullsense• 3 years ago

I like how they release what reads more like a blog post as a paper.

bo10244• 3 years ago

As a theorist in AI, the premise is funny but also really sad. And yet it is a very pervasive premise in the field.

OpenAI is not doing science. They are building a big shiny thing, showing it off, keeping it closed, and making money off of it. That is not part of the scientific process.

It is as if, every time SpaceX launched a rocket a little bit higher, every aerospace department ooh'd and aah'd and threw up their hands: "we don't have the resources to build a bigger rocket; how can we do science?"

Unfortunately, the research field of AI is diseased: it places way too much value on showing off big shiny things above progress in scientific understanding. But there is a ton of progress available to be made by small teams on small budgets. There is so much we don't understand about neural networks alone, even small ones.

gitfan863• 3 years ago

OpenAI wasn't created to "Do Science". It was created to give humanity a better chance at navigating the singularity.

agravier• 3 years ago

Ooooh and here I was, an ignoramus thinking OpenAI was mostly trying to find a way to milk the generative models cash cow. Silly, cynical me, I should have known better.

revolvingocelot• 3 years ago

OpenAI was created to trade (literally) on the "open" in their name. Any advancements in actual technology are likely secondary to the billion fucking dollar investment (which Altman has suggested might not be enough!). It's certain they aren't here to do "science", which IME involves showing your work to others.

martopix2• 3 years ago

OpenAI wasn't created to "Do Science".

That's exactly OP's point

hyperbovine4• 3 years ago

Conversely: if you’re a depressed AI researcher: come on over to science, the water’s fine! We have centuries worth of open problems that are not going to be solved by ChatGPT, and need more smart technical people than we can hire.

api• 3 years ago

The problem is usually that science can't pay even a fraction of what you can make in industry right now, and meanwhile the good science jobs are often in high cost of living places.

CuriouslyC2• 3 years ago

What an opportunity! I get to spend most of my time writing grant proposals, compete with 30 other associate professors for 1 tenure track job, then when I get tired of that I can switch to a marginal wage industry job because AI researchers are the only scientists that get good compensation as a group, and the 28 other people who didn't get that tenure position are probably also trying to do the same thing (thus depressing wages)? Sign me up /s.

jebarker2• 3 years ago

When you say "come over to science", where are you thinking? My impression is that industry -> academia is an almost impossible jump to make. Do you mean industry doing science?

foooorsyth• 3 years ago

That is not part of the scientific process

What? OpenAI is absolutely conducting science amongst themselves. They are certainly engaging in the hypothesis -> test -> result scientific method, which is science in its purest form. Sharing the secret sauce is what is done or expected in academia, but that doesn’t mean that OpenAI isn’t conducting science behind their own walls. You’re conflating institutional academia with the scientific method.

If I go home and test the boiling point of water in my garage and record the result privately without publishing a paper, am I not conducting science? What nonsense.

z3c05• 3 years ago

"Science" is not solely the scientific method - it's the democratization of your methodology to enable the replication of the results, towards the goal of determining their soundness.

bo10244• 3 years ago

By the "scientific process", I mean the way scientific knowledge as a whole progresses over time, i.e. the main undertaking of the target audience of the paper.

You're right that "OpenAI is not doing science" was hyperbolic, but I tried to clarify the aspect of science that I'm referring to.

belter• 3 years ago

They are more R&D engineers than R&D scientists...

VoodooJuJu• 3 years ago

Nothing sad about it. Keep theorizing and academizing. It's useful. It's not nearly as useful as tinkerers building usable non-theoretical things. It's no wonder that launching a rocket is more impressive and eye-catching than a guy shuffling papers. Don't dare knock down the doers, the risk-takers, the ones with skin in the game.

OpenAI provides me with a real, useful tool, something that directly improves mine and many others' lives. Your papers do not have that effect on my life.

bo10244• 3 years ago

I don't mean to knock down doers or risk-takers. On the contrary, I'm saying don't try to compete with the doers when you're not one.

OpenAI provides me with a real, useful tool, something that directly improves mine and many others' lives. Your papers do not have that effect on my life.

Of course, OpenAI's work is built directly on top of publicly-funded research papers of the past 5, 10, 50 years.

pronto-columnar• 3 years ago

Weirdly aggressive response. do you think technologies like gpt were developed without any academic input?

csomar• 3 years ago

Unfortunately, the research field of AI is diseased: it places way too much value on showing off big shiny things

This applies pretty much to ..euh every field. The wow factor attract investors and money. The way the current economy is structured, it incentives these tactics since these individuals will be able to draw on significant funds and will have a comparative advantage even if they have a worse return ratio.

And then it'll pass and a new thing will kick again. Investors have no memory, and I am talking about institutional ones. The supposedly sophisticated folks.

muskmusk• 3 years ago

If we have learned one thing from deep learning, it is that scaling works. From the ImageNet [15] competitions and their various winners to ChatGPT, Gato [13], and most recently to GPT-4 [1], we have seen that more data and more compute yield quantitatively and often even qualitatively better results. (By the time you are reading this, that list of very recent AI milestones might very well be outdated.). Of course there are improvements to learning algorithms and network architectures as well, but these improvements are only really useful in the context of the massive scale of experiments. (Sutton talks about the “Bitter Pill”, referring to that simple methods that scale well always win the day when more compute becomes available [18].) A scale that is not achievable by academic researchers nowadays. As far as we can tell, the gap between the amount of compute available to ordinary researchers and the amount available to stay competitive is growing every year.
This goes a long way to explain the resentment that many AI researchers in academia feel towards these companies. Healthy competition from your peers is one thing, but competition from someone that has so much resources that they can easily do things you could never, no matter how good your ideas are, is another thing.

This sounds like teenagers whining they can't all be popular. Science isn't a competition to be won so that you can get praise and attention. Science exists to discover things that then hopefully are useful to people. If someone that isn't you discovers something useful that isn't bad because then you can't discover it. It is good because that is a problem solved.

The resources requirements FOR SOME SPECIFIC PROBLEMS have gone up to a ridiculous degree, but there are plenty of problems left to solve. In fact a new one that is at least as important has been created: Replicate current results with less hardware/parameters/whatever.

The difference between having GPT4 as a slow and costly service that requires network calls vs having it locally with almost no cost will be a huge achievement. Stop sulking and get to work!

pixl973• 3 years ago

Science isn't a competition to be won so that you can get praise and attention.

Conversely business seems to be a competition which wants exactly that, no matter what the long term consequences are.

Lead in gasoline makes the engine stop knocking, great. Problem solved, nothing to worry about, right?Don't worry about funding scientists to look into it, it's all going to be ok.

chubot• 3 years ago

That's the first time I've been rick roll'd reading an academic paper :) (citation 1 for GPT-4)

mensetmanusman2• 3 years ago

After a PhD@MIT and then entering industry, it has been my general understanding that industry is ahead on any scientific fronts that are profitable in fields that have a progress<>profit feedback loop.

Academia still plays an important training ground role, and it could shift focus to those areas that could be profitable but require coordination that is generally not feasible among certain institutions.

mensetmanusman2• 3 years ago

AI profs should start working with the bio folks now and making thinking materials that are orders of magnitude more power efficient than silicon.

rafaelero• 3 years ago

Don't worry. Soon governments will start pouring money into this field. They still won't be competitive, but at least there will be enough computing power to play with.

tpoacher• 3 years ago

Gotta love a paper which cites Rick Astley's "Never Gonna Give you Up".

Properly, for that matter.

thunkshift1• 3 years ago

Thus is medium shit post at best.. why is it on arxiv

ralphc• 3 years ago

AI fatigue is the new JavaScript fatigue.

nothrowaways• 3 years ago

The only limitation of this scientific article is the fact that it is not two-columns format.

Choose Your Weapon: Survival Strategies for Depressed AI Academics

Related Stories