memothon

Was this announced beforehand? How do you double prices for customers so abruptly with no transition period?

I'm always skeptical because you can make it pass the benchmarks, then you use it and it is not practically useful unlike an extremely general model.

Cool work though, really excited for the potential of slimming down models.

Yeah I agree with this. I will try to use it in earnest on my next project.

That metric is the key piece. I don't know the right way to build an automated metric for a lot of the systems I want to build that will stand the test of time.

I think the real problem with using DSPy is that many of the problems people are trying to solve with LLMs (agents, chat) don't have an obvious path to evaluate. You have to really think carefully on how to build up a training and evaluation dataset that you can throw to DSPy to get it to optimize.

This takes a ton of upfront work and careful thinking. As soon as you move the goalposts of what you're trying to achieve you also have to update the training and evaluation dataset to cover that new use case.

This can actually get in the way of moving fast. Often teams are not trying to optimize their prompts but even trying to figure out what the set of questions and right answers should be!

I've used browser rendering at work and it's quite nice. Most solutions in the crawling space are kind of scummy and designed for side-stepping robots.txt and not being a good citizen. A crawl endpoint is a very necessary addition!

This model is probably really strange to people who haven't spent a ton of time working in something like Clojure.

Turso offers a cloud-hosted sqlite product. The idea is you can easily spin up per-tenant databases and have a "serverless" sqlite interface.

It feels like it has a lot of the same downsides of hosted databases so I'm not sure what the specific value is.

From their site they really emphasize local first syncing (so mobile apps and electron/tauri apps) and multi tenancy (hard database boundaries)

If you implement different local payment providers that you could see higher conversion rates.

Lots of companies have automations with Zapier etc. to upload things like invoices or other documents directly to notion. Or someone gets emailed a document with an exploit and they upload it.

Some people have startup credits

I made a quick site so you can see what tools are using the most context and help control it, totally free and in your browser.

https://claude-code-analysis.pages.dev/

Yeah this was a huge foot gun

Those WAL entries streamed to object storage I think are just for backups.

Each DO is globally unique (there's one DO with a given id running anywhere) and runs sqlite on its own local storage in that datacenter.

I'm imagining your poor 90 year old aunt playing this wild game of Simon says with you and having no idea what's going on.

Maybe just ask the cousin not to send any more money?

Yeah I get what you're saying. totally agree.

Replying to bookmark

Good question, remains to be seen. Will need to see how it plays out!

I don't disagree it is unsustainable, I'm really trying to be more precise about whether anyone is getting value out of the tools. I'm just really skeptical that nobody is getting value.

Sorry I should have been more specific.

The article does mention that OpenAI has huge revenue.

While The Information reported that OpenAI's revenue is $3.5 to $4.5 billion a year in July, The New York Times reported last week that OpenAI's annual revenues have "now topped $2 billion," which would mean that the end-of-year numbers will likely trend toward the lower end of the estimate.

But then the author claims that the business value is questionable.

And even if they did, it isn't clear whether generative AI actually provides much business value at all. The Information reported last week that customers of Microsoft's 365 suite [snip]

I would have appreciated a deeper discussion of why OpenAI's revenue isn't a data point toward generative AI having some business value. Presumably if nobody was using generative AI in a way that gives them value, OpenAI wouldn't be using all those GPU hours. That's what I was missing from the article personally.

I always find it really strange when articles like this claim nobody is paying for generative AI. I can't find reliable stats on this but there are at least a million ChatGPT Plus subscribers. Does that not count?

It takes time for new advancements to get proliferated through the economy.

The A-Z of Product Manipulation would be a more apt title.

UAE maybe?

I think the discussion around "exponentials" with top-end LLM (think 3.5 sonnet, gpt-4 not the smaller models) scaling is really pointless. The heuristic we have for what to expect from performance is just scaling, which has worked pretty well. These benchmarks are imperfect in lots of ways, aren't necessarily sensitive to showing exponential progress and it is difficult to predict step changes in capability in advance.

If you zoom out on the first graphic from December 2023 back to 2020, the capabilities of models released at that time on these benchmarks would be much much lower. The best lens for future performance of large models is uncertainty.

In the specific case of the Drake track, hateful is an appropriate word.

He was using Tupac's voice on the TaylorMade freestyle in a really disrespectful way that would borderline on hateful of his artistic legacy. Just read these lyrics...

Verse 1: 2Pac (AI)] Kendrick, we need ya, the West Coast savior Engraving your name in some hip-hop history If you deal with this viciously You seem a little nervous about all the publicity Fuck this Canadian lightskin, Dot We need a no-debated West Coast victory, man Call him a bitch for me Talk about him likin' young girls, that's a gift from me

I think your post is taking these AI voices out of their original context.

First let's consider CD -> streaming as a media change. Streaming didn't really exist when Tupac was around. But nobody would say putting Tupac's catalog on streaming is inherently disrespectful to his artistic legacy because it's preserving (more or less) the same artistic product.

Here are a couple other examples that I do think are more analogous than improvements in recording technologies:

Posthumous releases with material not created by the artist. Sometimes record labels will try to capitalize on the brand of an artist and release material that really only has snippets of random recordings that an artist made. In my view, this is disrespectful to the artist because it's not a piece of artistic material they wanted to release.

Another example is colorizing black and white movies. Similarly, this action changes the actual artistic product in a way that's disrespectful to the creators of those films.

Creating AI voices of artists is similar to these examples because it's changing the artistic output of an artist and disrespecting their artistic legacy. It's creating content under their name without the ability for them to say no or have any input into the output.

Thanks all!

Did you happen to remember the domain-specific LoRA startup?

I really like the point about getting AI to ask you questions.

The focus in the AI tutor world is basically a chatbot to ask questions of. But if you're trying to learn something, it's really helpful to have targeted questions asked of you!

Zuck has mentioned recently

That's a really surprising thing to hear, where did you see that? The only quote I've seen is this one:

“One hypothesis was that coding isn’t that important because it’s not like a lot of people are going to ask coding questions in WhatsApp,” he says. “It turns out that coding is actually really important structurally for having the LLMs be able to understand the rigor and hierarchical structure of knowledge, and just generally have more of an intuitive sense of logic.”

https://www.theverge.com/2024/1/18/24042354/mark-zuckerberg-...

I actually really like Anki (and think it's a great tool!) but this is one of the biggest problems I see for spaced repetition to get in the hands of more people.

You can change the number of review items but it doesn't change the fact that you have an impossible backlog to get through. Then people just get bored and churn.