reqo

Isn't there some value in handling the infra for the largest genai products currently?

Yes, the correct link is this: https://corporate.mcdonalds.com/corpmcd/our-stories/article/...

I am curious, what do you mean by "Adapter based weight selection"?

While fundamentals are important to learn, there is also a huge benefit in learning specific tools and frameworks. There is no ”one size fits all” when it comes to software and more often than not, you need very customized solution for optimal performance. Moreover, learning to master a tool often means you are also able to improve it, which is part of the reason open-source tools usually improve with the loyal contributers!

It would make languages like Python quite unreadable as it would be hard to distinguish between variables and keywords!

This ‘goal drift’ means that agents, or tasks done in a sequence with iteration, get less reliable. It ‘forgets’ where to focus, because its attention is not selective nor dynamic.

I don't know if I agree with this. The attention module is specifically designed to be selective and dynamic, otherwise it would not be much different than a word embedding (look up "soft" weights vs "hard" weights [1]). I think deep learning should not be confused with deep RL. LLMs are autoregressive models which means that they are trained to predict the next token and that is all they do. The next token is not necessarily the most reasonable (this is why datasets are super important for better performance). Deep RL models on the other hand, seem to be excellent at agency and decision making (although in restricted environment), because they are trained to do so.

[1] https://en.wikipedia.org/wiki/Attention_(machine_learning)

The facilities will reportedly consume as much as 13 percent of the plant's output.

Why are AI products being shipped so aggressively despite being so inefficient? Is code autocompletion and generating random images really worth so much electricity? Shouldn’t we wait until the research has created an efficient architecture that is easily scalable first?

Fine-tuning on a niche task I guess! E.g. a bunch of LoRAs that can be easily swapped depending on the task!

Isn’t that what the softmax layer is doing? The token with highest probability among all the available tokens in the model dictionary is chosen as the next token!

Yes but it does not invalidate the fact!

It does not have to have a use or a point, it is up to you to decided what it represents. I see at as an interactive yin and yang!

Is it common to publish books directly to ArXiv, especially books that have just been released?

Many small owners together can bring a class action though

I am sorry but is that a password on the pink post-it note?

I think in the long run, it is in the interest of AI companies to incentivize creators to create high quality data! Not paying them their fair share will likely decrease the volume of high quality data available (or make it much less accessible). Unless these companies already have developed another architecture that can learn much more from the same dataset, the lack of new high quality data will be a problem for future larger models!

Merry Christmas everyone! Thanks everyone for you great contributions to the community and thanks to the dear moderators for the great job they are doing!

It is the first time, to my knowledge, that there is strong evidence that indicates it could actually happen! (The evidence being the rapidly changing climate and our inability to adapt quickly enough)

Please provide a source, sounds interesting!

You can see the results if you download the pdf from [0].

[0] https://jamanetwork.com/journals/jamanetworkopen/fullarticle...

A lot of AI/ML papers these days have an accompanying interactive page like [0], will we see anything like these now directly in arXive?

[0] https://voyager.minedojo.org/

Any evidence to back that up?

I believe RAG is more appropriate for this! While you can certainly fine tune on a pdf, you are essentially fine turning with batch size == 1, so you should not expect good results! Also you need a label (for example summary) in order to fine-tune!

Unrelated. How hard is it to create another “internet” which is completely detached from the current internet?

I honestly don't think there is any incentive for large corporation to care too much about things like this and, on the contrary, they are incentivized to do more! The fine they get is basically cost of doing business and since our attention is engineered to be short, the negative PR won't last long enough to have a huge impact. Unless we see a "too big to fail" company actually fail because something like this, we won't see any change in their behvaiour!

I think this is really interesting from a meta (in-context) learning pov, but I think at some point prompt engineering stops being prompt engineering and instead becomes in-context training. This is problematic for applications such as search or completion, unless some other software/model does this automatically for you!

I bet this will make Neuralink useless! It would be great for the poor animals getting operated!

I don’t see the contradiction. Humans can be emotional and at the same use science to make all humans life better! In fact why would we ever develop any technology that makes life better for others if we don’t have any feelings for them?

Finding that 70% of attention heads and 20% of feed-forward networks can be excised with minimal effect on in-context learning suggests that large language models are undertrained.

I thought that it was common knowledge that LLMs are undertrained, none of the publicly available loss graphs show any sign of convergence!

it makes me think of the flashes in the game of life! who knows how this released energy is going to affect the celestial objects nearby and these in turn affect other bodies and so on...

Another probable bad side effect of AI will be that a generation of developers will become dependent on copilot/chatgpt to be able to write code! They will definitely perform worse than before llm era without the help of a llm!