ml_basics

Read about half of the article wondering when it was going to explain why carplay is addictive

January 2026 when i started using opus 4.5 and understood that it could do actual useful work beyond coding small snippets

maybe we will look back and also think that the current slate of IPOs were reasonable valuations?

this will change as inference demand increases (which is happening right now faster than many people expected)

During my time at university studying pure mathematics I had an interesting experience of doing a challenging sheet of combinatorics problems during a vacation. Every day I attempted one question and got stuck on it. Then the next morning I woke up knowing the solution. It was a recurring thing: this happened every day for about 2 weeks until I had solved all the problems.

For me this a big eye opener about the importance of sleep and relaxed thinking to solve challenging problems.

how much of the cloud revenue is from Anthropic revenue sharing?

They are referring to a thing called "speculative decoding" I think.

This is from 11th September

The graph at the beginning showing the cost of sequencing over time falling faster than Moore's law stops in 2015. Would love to see how things have progressed since then. Casually googling i only saw plots up to 2021 but looks to me like progress is now slower than Moore's law since ~2015. Maybe things will change when Nanopore gets more reliable

what's the relationship between this work and the recently announced voice models from Microsoft AI? https://microsoft.ai/news/two-new-in-house-models/

It's a europeanism. In both French and German (and probably other EU languages) the word for "subsidy" is something like "subvention" so native speakers of these language often reach for an unnatural word in English.

Btw other examples include "actually" which is used to mean "currently", and "eventually" which is used to mean "maybe".

Personally I'm torn whether to consider this incorrect use of the language as it is quite widespread. Maybe it would be better to consider this as the emergence of a new dialect.

And American tech companies have been substantially more successful that European tech companies so maybe it does actually make sense to have stock options as part of compensation rather than employer bonuses?

It's quite remarkable how much the goal posts have shifted when it comes to what is impressive with AI/ML. Things like this are a good reminder.

10 years ago the GAN paper came out and everyone was excited how amazing the generated image quality was (https://arxiv.org/abs/1406.2661)

The amount of progress we've made is mind boggling.

To be fair to the authors they are affiliated with a university and not a big industrial lab, so they may be working with significantly constrained resources. Not sure exactly what the best solution is for this case given that it affects most people outside of a very select few.

Cool paper. Really interesting to see how even quite straightforward architectural modifications haven't yet all been exhausted yet, despite all the resources being poured into LLMs

They make the yoghurt, then pasteurise it (I guess so it has a longer shelf life). So it tastes roughly like yoghurt but doesn't have any of the good bacteria. I've also seen that sometimes lactobacteria are then artificially added back in so that they are present but in a controlled way.

As of July 3, 2023, we’ve disabled the Browse with Bing beta feature out of an abundance of caution while we fix this in order to do right by content owners. We are working to bring the beta back as quickly as possible, and appreciate your understanding!

Demo website with speech-speech translation examples https://google-research.github.io/seanet/audiopalm/examples/

We introduce AudioPaLM, a large language model for speech understanding and generation. AudioPaLM fuses text-based and speech-based language models, PaLM-2 [Anil et al., 2023] and AudioLM [Borsos et al., 2022], into a unified multimodal architecture that can process and generate text and speech with applications including speech recognition and speech-to-speech translation. AudioPaLM inherits the capability to preserve paralinguistic information such as speaker identity and intonation from AudioLM and the linguistic knowledge present only in text large language models such as PaLM-2.

Direct link to demo video showing speech-to-speech translation: https://google-research.github.io/seanet/audiopalm/examples/... (see website for more example)

I was thinking atomic as in "atomic operations"

bfloat16 is probably familiar only to ML practitioners; it's a reduced precision floating point format designed for ML models. I was surprised to learn that the "b" stands for "brain", as in the team at google that developed it along with many other advances in machine learning.

Pretty cool landing page, listen to the audio at the top of the page which gives a high level explanation and demonstrates the method

Check out the scale, those are very tall peaks in the centre at 21km!

Congrats to all the people behind this!

To the naysayers that point out that OpenAssistant is not as good as ChatGPT: all of LLM development is still in the early days; the work being done by teams like OA is helping to shape the future development of the entire field by creating an actually-open-source foundation. The results of this work will only get better with time.

Yep totally agreed. One of the things I'm excited to see going forward is multimodal models (trained e.g. on text + video + audio + images).

I'm sure there's a lot more to it than this, but maybe one factor that makes humans a lot more data efficient is the multimodal input we receive.

If that's the case, imagine how much better things could get when we train with all the videos, podcasts, radio etc in the world, in addition to all the text out there!

Where's the predicted society where automation allows us all to work for two hours a day, and spend the rest at leisure?

Others have already commented on the fact that quality of life has generally increased.

I think it's also worth realizing that over the course of a lifetime people today spend a much smaller fraction of their waking hours engaged in work than before.

100 years ago a typical man in a developed country might have started work aged 14 and then stopped working a few years before death if they were lucky to not die before.

Nowadays even someone "uneducated" would start working at 18, while others might start aged 22 or older. And a common expectation is to have a retirement of 15+ years.

So averaging over a lifetime we do work a lot less as a fraction of our lives than in earlier times.

Edit: numbers were plucked from thin air based on my experiences of my own country, but i guess the general point applies to most developed countries

Which ones do you mean? From my coarse understanding, seems like most places that didn't industrialise got colonised by ones that did. Maybe that's not the case with the development of the internet, but still it's the case that the main center of innovation during the early phases (Silicon Valley) is still way ahead of everywhere else (example: OpenAI is based in SF)

Especially in a chatbot use case with cumulative prompting, which is the best use case for such a large context vs. the default cheaper 8k window.

Depends on what is up with the images and how they translate into tokens. I really have no idea, but could be that 32k tokens (lots of text) translates to only a few images for few-shot prompting.

The paper seems not to mention image tokenization, but I guess it should be possible to infer something about token rate when actually using the API and looking at how one is charged.

What's the lifespan of an LLM going to be in the next few years? Seems like at the current pace, cutting edge models will become obsolete pretty quickly. Since model training is very expensive, this means the LLM space has some parallels with the pharmaceutical industry (massive upfront capital costs, cheap marginal costs relative to value produced).

I find it quite fascinating how quickly machine learning has changed in this regard.

From the paper:

Given both the competitive landscape and the safety implications of large-scale models like GPT-4, this report contains no further details about the architecture (including model size), hardware, training compute, dataset construction, training method, or similar.

I'm curious whether they have continued to scale up model size/compute significantly or if they have managed to make significant innovations there.

I just skimmed the paper but seems they are also omitting details about how they actually feed the images in too, which is a shame as a curious outside observer.