rryan

ehh, no? definitely not.

https://kyutai.org/blog/2024-07-03-meet-moshi/

As a human emdash enjoyer, I'm pretty damn grumpy that everyone assumes I'm a chatbot now. :P

I think the super long timelines in automotive make it really hard to succeed. Really impressed with what ClearMotion accomplished given that

Don't make me tap the sign: There is no such thing as "bytes". There are only encodings. UTF-8 is the encoding most people are using when they talk about modeling "raw bytes" of text. UTF-8 is just a shitty (biased) human-designed tokenizer of the unicode codepoints.

RMSNorm is pretty insigificant in terms of the overall compute in a transformer though -- usually the reduction work can be fused with earlier or later operations.

ML 101: Do not evaluate on the training data.

Yes of course it can, because they fit in the context window. But this is an awful test of the model's capabilities because it was certainly trained on these books and websites talking about the books and the HP universe.

Thanks for everything, Rich. You inspired me repeatedly.

https://elevenlabs.io/

https://twelvelabs.io/

.... wat

Yea uh, don't know what was going on there but there are roles with direct reports that aren't in mountain view (unless you mean, like in 2004). NYC has 1000s of googlers.

This is ... not what I expected. It's basically wiring up pre-trained models to ChatGPT via a router and "modality transformations" (a.k.a speech-to-text and text-to-speech).

I expected it to be a GPT-style model that processes audio directly to perform a ton of speech and maybe speech-text tasks in a zero-shot manner.

Except when your doctor won't prescribe it you offlabel because you don't have a condition they were taught about in med school

In the past few months I've been getting much more fake order / payment spam.

I agree that a Transformer is an example of a "reflexive" behavior because it learns to react in a context (via gradient descent rather than evolution as the learning algorithm). It's a conditional categorical distribution on steroids.

I also agree it's not much different than what's going on in this petri dish with pong.

But I don't think that's a profound statement.

What I'm saying is that calling what a Transformer does "language development" isn't accurate. A Transformer can't "develop" language in that sense, it can only learn "reflexive" behavior from the data distribution it's trained on (it could never have produced that data distribution itself without the data existing in the first place).

There's a huge difference between fitting a probabilistic model to a data distribution then sampling from it (what GPT-3 is) and agents that invent language and use it to communicate.

You're using a pointwise loss, which means you are treating each timepoint as conditionally independent. That's a deeply flawed assumption, probabilistically.

Try an autoregressive model of the joint probability distribution of the sine wave timepoints -- like WaveNet [1]. It will nail your sine wave -- just as it nails mixtures of sine waves (speech and music). :)

[1] https://arxiv.org/abs/1609.03499

"I could go pro, but nah" -- I read this and closed the tab. What a whingefest from a free tier user.

The Reddit crawling problem is because Reddit rate limits their crawling so they have to prioritize the most popular feeds. What's the problem with linking your account, or making a dedicated feedly throwaway for crawling?

Been a pro user since the beginning because I want the service to stick around. It works just as well as it always has and I don't mind that they're adding new features even if they aren't for me.

Sheesh.

That makes no sense. Why is "YouTube Premium" capitalized?

The main source of income for youtube isn't ads. YouTube revolves around the merchandise and YouTube Premium subscriptions.

If you are talking about creators who are not earning money for using vanced, you should know they won't make millions out of those ads.

... right. Feel free to keep telling yourselves that.

N=20

* YouTube Kids with allowlist.

* App Timer of zero minutes on YouTube app.

Straw man argument. OP was suggesting you pay for YT premium, which does away with the ads completely. Removing your web adblocker is a non sequitur, and exposes your computer to malware / spyware in addition to ads.

Parents are treated equally. If you give birth you get an additional 6 weeks of medical recovery leave. If two parents (any gender) adopt a child then both receive an equal length baby bonding leave.

I've taken parental leave twice at Google and used the full amount. No ill effects, as far as I can tell. Everyone I work with and my management chain was fully supportive.

nit: Kubernetes is the container orchestration, not the container technology itself. So the comparison is apples vs. oranges.

You could replace the containers that are being scheduled by Kubernetes with WebAssembly. Others already linked to Krustlet which is effectively this.

No one at Google is responsible for these half baked and largely irrelevant widgets or wants to stake their career fixing them.

You're just ... wrong about this. There's an entire team of dozens of people (maybe hundreds now) focusing on this specific web answer feature. I personally worked on the team (not this feature, though).

I don't understand why people say things they know nothing about.

AIUI, AssemblyScript is a TypeScript-like language that is designed to compile to wasm.

I love that Firefox exists and Quantum is an amazing step forward, but Firefox still regularly runs away with gigabytes of RAM and hung worker processes. I have no problem with long-lived Chrome sessions but I need to restart Firefox ~daily. It's not bad memories of 10-years ago.

Uh, 2040? Seattle will be underwater by then.

The grandma stereotype is discriminatory towards both age and gender.

Cloud TPU pods are seriously amazing. I'm a researcher at Google working on speech synthesis, and they allow me to flexibly trade off resource usage vs. time to results with nearly linear scaling due to the insanely fast interconnect. TPUs are already fast (non-pods, i.e. 8 TPU cores are 10x faster for my task than 8 V100s) but having pods open up new possibilities I couldn't build easily with GPUs. As a silly example, I can easily train on a batch size of 16k (typical batch size on one GPU is 32) if I want to by using one of the larger pod sizes, and it's about as fast as my usual batch size as long as the batch size per TPU core stays constant. Getting TPU pod quota was easily the single biggest productivity speedup my team has ever had.

Travis has weird priorities. The Ubuntu environment has been stuck on Precise/Trusty for years now. And the only place you can upload artifacts is Amazon S3. I really hate to be that person who sees a new offering and is like "but when will you fix your busted junk?!" but.. when will you fix your busted junk?!