exhaze

Wasn’t that almost the norm for both new OS drivers and subsequently GPU SoC firmware of ‘gamer cards’ to literally optimize for the latest AAA titles? I feel in that case, the interests aligned - in a world where I had a chance to actually play Crysis, I wouldn’t care why the frame rate was decent, no?

“Today, the NASDAQ fell by 18% moments after Simon Willison published his latest blog evaluating the Legend 7.4 model that rendered an animated SVG of a pelican on a bicycle and the pelican fell off”

Agree directionally - even back in sonnet 3.5 days, I was helping some friends by showing them how to create intermediate representations for SVG building blocks mapping to parametrizable functions that can be used to make interactive SVG-rendered visualizations for various medical needs.

Not an expert on this, so won’t speculate regarding what traits would create robustness specifically attributed to SVG visual representation capability, but felt you might find this paper on reasoning models trained on physical world video data becoming better at general reasoning interesting:

https://arxiv.org/abs/2210.05359

What if they’re just unsuccessful at it and something about a model grokking how to create a realistic visual representation of a pelican on a bicycle ends being key to the next OOM of capability unhobbling.

And since we have established this silly routine once, I must keep going and ask - yes, you have quite a collection of bona fide pelicans you’ve seen and photographed.

Have you physically seen all the other animals you’ve evaluated as well?

I was born in Soviet Russia, so trust but verify and if still around, maybe there pelican make LLM draw Simon ride bicycle and notice Python code improve.

Simon - thank you for not dismissing it (and surviving the text that came before the question).

I asked because I genuinely feel that the % of people working on some of the most important technology these days - things such as these 'strangely shaped tools' (to borrow from nearcyan) - large language models - the younger generation (folks in their early/mid 20s) - it is not unlikely that they have not physically seen the meatspace version of whatever digital correspondence of it that is being packed into latent space.

After all, why waste time going to the SF or Oakland zoo? One can just check Simon's latest pelican blog post and skip the zoo trip - the harnesses are waiting.

I thought my joke post was silly and then I read new comments and I'm like, "I didn't try hard enough" lol

Simon - has no one told you about the Willison-Pelican Scaling Law?

```

if is_willison_pelican_blog_post:

[redacted]

```

You haven't seen their final form [1]

[1] final form is a frontend/react/let's not talk about it, library - it caused a great deal of PTSD to me and my previous company's team due to its dogmatic preference for "we use these axioms, end of story", over practical utility - so it was quite challenging to do state of the art tasks such as nested form fields (e.g. 'user.address.personal.line-1'). The PTSD it caused made us all block out the memories, I suppose. But - it had zero dependencies. That is what mattered. It kept us going. We weren't reaching for more. We had plenty of time.

And thank god for that. Because I'd forgotten my watch in California - and this was in Tokyo [2]

[2] a joke within a joke about Jensen's Kyoto gardener story. Beautiful story, drowned out by WatchGate memes. Why can't jokes have layers? Models have trillions. If you miss 100% of the jokes you don't make, make all the jokes. Someone will laugh (eventually, maybe?) Even if it's: "this person + comedy club = full secret service detail". If someone laughs at that - at my own expense? I don't mind. They laughed. I know this is a gibberish, off-topic message - it's also a human message. I just felt we need more such things in our lives these days.

PS: have you physically seen a pelican in real life? (not a joke)

My best learnings came from mistakes - mine, my team's, my company's, other companies - but almost always, it was when those mistakes were promptly followed up with a thoughtful incident report.

For example, when I launched what is now Uber One, two days later, I got a call that I broke something. It turned out I had forgotten an edge case in a completely different service. Yes, this was 2016 Uber, 3000 microservices, but I don't think the users who didn't get their discount really cared.

The takeaway I still remember from that today is about always looking for "works by coincidence"; I often ask myself about the preconditions that must be in place for some class of mistake to be impossible.

Nice! Just added your comment to my list of Perl script-inspired lists

Claude Code’s feature cardinality is breathtaking. At this rate, the next pope will be from Anthropic

I highly recommend talking to strangers! People are lovely. Go out and try it.

I’ve been here since 2009 and this is one of the loveliest comments I’ve read.

At face value, it may seem ‘duh!’.. but there’s a distinct aesthetic to it that resonated with me.

Perhaps the best analogy I can think of is Asimov’s philosophy about writing.

‘I want the reader to forget they are reading as if my thoughts are being transmitted directly from my brain into theirs’ [sic]

Recently, a research publication demonstrated that an LLM.. nah, not today. Sometimes knowing the underlying theory and deciding to disengage from it and just appreciate the moment is fine.

Because I can go outside my apartment here in Tokyo right now and try it. I already do, but each of us has our own unique loveliness. So I’ll keep trying. Just because.

Push notifications and mental real estate by being “an app” are the primary business reason (based on both statsig experiments I’ve seen across my career as well as some intuition about behavioral psychology regarding the app mental real estate bit).

There are a few ways, but I believe SSOT (single source of truth) is key, as others basically said. Some ways:

1. Shared TypeScript types

2. tRPC/ts-rest style: Automagic client w/ compile+runtime type safety

3. RTK (redux toolkit) query style: codegen'd frontend client

I personally I prefer #3 for its explicitness - you can actually review the code it generates for a new/changed endpoint. It does come w/ downside of more code + as codebase gets larger you start to need a cache to not regenerate the entire API every little change.

Overall, I find the explicit approach to be worth it, because, in my experience, it saves days/weeks of eng hours later on in large production codebases in terms of not chasing down server/client validation quirks.

Tangential, but thought I'd share since validation and API calls go hand-in-hand: I'm personally a fan of using `ts-rest` for the entire stack since it's the leanest of all the compile + runtime zod/json schema-based validation sets of libraries out there. It lets you plug in whatever HTTP client you want (personally, I use bun, or fastify in a node env). The added overhead is totally worth it (for me, anyway) for shifting basically all type safety correctness to compile time.

Curious what other folks think and if there are any other options? I feel like I've searched pretty exhaustively, and it's the only one I found that was both lightweight and had robust enough type safety.

Back in 2015ish Uber we liked to emphasize everyone building at the company to never forget about the long tail [distribution].

When your N crosses gets above N per day, even the 0.1% edge cases happen several days times a day. When this has real world implications, even a single instance can matter a lot.

Cool project! I built a similar tool [0] last year, but:

1. Targeting fbt (Meta's internal i18n tool)

2. Used CST (<3 ast-grep) instead of AST - really useful here IMO esp. for any heuristic-based checks.

3. Fun fact: this was made entirely on my phone (~2.5h) while I was walking around Tokyo. Voice prompting + o1-pro. Why? My friend was working on porting fbt to TS and said he was planning to build this. I wanted to one-up him + convince him to start using LLMs =)

One thing you should be aware of is that for at least Japanese, localization is far from just translating the text. There are lots and lots of Japan-specific cultural nuances you have to take into account for web users and even down to actually just having an entirely different design for your landing page often because those you'll find those just convert better when you know certain things are done that are typically not done for you know non-Japan websites.

Notta (multi-lingual meeting transcriptions + reports) is a great example if you compare their Japanese [1] and English [2] landing pages.

Note how drastically different the landing pages are. Furthermore, even linguistically, Japanese remains a challenge for proper context-dependent interpretation. Gemini 2.5 actually likely performs best for this thanks to Shane Gu [3], who's put in tons of work into having it perform well for Japanese (as well as other "tough" languages)

[0] https://github.com/f8n-ai/fbtee-migrate

[1] https://www.notta.ai (Japanese version)

[2] https://www.notta.ai/en (English version)

[3] https://x.com/shaneguML

Food for thought, a snippet from a highly specialized project I created two months ago:

https://gist.github.com/eugene-yaroslavtsev/c9ce9ba66a7141c5...

I spent several hours searching online for existing solutions - couldn't find anything (even when exploring the idea of stitching together multiple different tools, each in a different programming language).

This took me ~3-4 hours end-to-end. I haven't seen any other OSS code that is able to handle converting unstructured JSON into normalized, structured JSON with a schema, while also using a statistical sampling sliding window method for handling for all these:

- speculative SIMD prediction of end of current JSON entry - distinguishing whether two "similar" looking objects represent the same model or not - normalizing entities based on how often they're referenced - ~5-6 GB/s throughput on a Macbook M4 Max 24GB - arbitrary horizontal scaling (though shared entity/normalization resource contention may eventually become an issue)

I didn't write this code. I didn't even come up with all of these ideas in this implementation. I initially just thought "2NF"/"BNF" probably good, right? Not for multi-TB files.

This was spec'd out by chatting with Sonnet for ~1.5 hours. It was the one that suggested statistical normalization. It suggested using several approaches for determining whether two objects are the same schema (that + normalization were where most of the complexity decided to live).

I did this all on my phone. With my voice.

I hope more folks realize this is possible. I strongly encourage you and others reconsider this assumption!

Cite things from ID based specs. You’re facing a skill issue. The reason most people don’t see it as such is because an LLM doesn’t just “fail to run” here. If this was code you wrote in a compiled language, would you post and say the language infuriates you because it won’t compile your syntax errors? As this kind of dev style becomes prevalent and output expectation adjust, work performance review won’t care that you’re mad. So my advice is:

1. Treat it like regular software dev where you define tasks with ID prefixes for everything, acceptance criteria, exceptions. Ask LLM to reference them in code right before impl code

2. “Debug” by asking the LLM to self reflect on its decision making process that caused the issue - this can give you useful heuristics o use later to further reduce the issues you mentioned.

“It” happening is a result of your lack of time investment into systematically addressing this.

_You_ should have learned this by now. Complain less, learn more.

Install MCP plugin and call a search engine of your choice.

If you’re unhappy about something, try to first think of a solution before expressing your discontent.

It would be great to hear from actual folks working for one of his companies rather than folks that presume that they can speak on behalf of those people, and, even worse, go so far as to liken them to "people in abusive relationships".

Can you show me any longitudinal studies that show examples of a causal connection between incrementality of latency and churn? It’s easy to make such a claim and follow up with “go measure it”. That takes work. There are numerous other things a company may choose to measure instead that are stronger predictors of business impact.

There is probably some connection. Anchoring to 10ms is a bit extreme IMO because it’s indirectly implying that latency is incredibly important which isn’t universally true - each product’s metrics that are predictive of success are much more nuanced and may even have something akin to the set of LLM neurons called “polysemantic” - it may be a combination of several metrics expressed via some nontrivial function that are the best predictor.

For SaaS, if we did want to simplify things and pick just one - usage. That’s the strongest churn signal.

Takeaway: don’t just measure. Be deliberate about what you choose to measure. Measuring everything creates noise and can actually be detrimental.

Curious - how would one frame the cat before/after box reveal within the conceptual framework of causal closure?

That’s a nice generalization from a single data point but the problem here is that I have experience with… 10-15? across 7 companies in my career and it’s worrying to me to hear you say it takes you a few months - especially in today’s world where you can rapidly accelerate your rate of learning and understanding of existing idioms within a framework/specific codebase with LLMs.

As LLM capabilities and context windows advance, if you’re blaming the LLM for this it means you don’t know how to use it.

Sorry, but there is too much empirical evidence of this both from first-hand experience as well as what I’ve seen from many other practitioners who started with the question of “how can I improve this?” instead of “this sucks”.

LLMs output can often be like a prism in front of a mirror - a fusion of humanity with the specific human, reflected back at the human.

Simon, perhaps you're just not funny.

Using LLMs to bootstrap development in unfamiliar frameworks is a valuable approach when balanced with pragmatic engineering principles. While framework idioms are important, they're secondary to delivering working, maintainable software that solves real problems.

The focus on "terrible structure" misses the point - what matters is whether the system meets its requirements efficiently and can be maintained effectively. Have you measured any actual negative impacts on system performance, maintainability, or customer value? My experience suggests that starting with working code and iteratively improving it as patterns emerge often leads to better outcomes than delaying development for complete framework mastery.

The interview analogy is particularly misleading - success in a software engineering role isn't measured by framework knowledge, but by the ability to deliver value to customers effectively. Learning framework idioms can happen in parallel with productive development, especially when using LLMs to accelerate the initial learning curve.

Sounds plausible given I’ve recently observed a ton of research papers in the space that in some way or another incorporate MCTS

I wasn’t aware of others attempting manifolds for this before - just something I stumbled upon independently. To me the “eerie” part is the thought of an LLM no longer using human language to reason - it’s like something out of a sci fi movie where humans encounter an alien species that thinks in a way that humans cannot even comprehend due to biological limitations.

I am hopeful that progress in mechanistic interpretability will serve as a healthy counterbalance to this approach when it comes to explainability.. though I kinda worry that at a certain point it may be that something resembling a scaling law puts an upper bound on even that.

I think the wildest thing is actually Meta’s latest paper where they show a method for LLMs reasoning not in English, but in latent space

https://arxiv.org/pdf/2412.06769

I’ve done research myself adjacent to this (mapping parts of a latent space onto a manifold), but this is a bit eerie, even to me.

Hey, that's a great question. I should have been more clear: for deterministic generation that's not done using an LLM. It's done using just regular execution of TypeScript. The code generators that were created using an LLM and that I manually checked for correctness, they're the ones that are generating the other code - most of the code. So that's where the determinism comes in.