LASR

Oh you wait until LLMs come up with frameworks that allow multiple LLMs to collaborate effectively. Then you’ll have new frameworks every 3 days.

Ridiculous in the first place.

Forget all the AI stuff. Please make the subject editable by default. Come on Google.

Incredible. Since you’re the dev, please please please isometric camera.

It worked. Most people under 30 don't know Apple existed before the iPod / iPhone. ie: Before Jobs.

Just get rid of cheapstake garbage all around the OS and you’re golden.

Yes this is pretty common in large enterprise-ey tech companies that are successful. There are usually a small group of vocal members that have a strong conviction and drive to make a vision a reality. This is contrary to popular belief that large companies design by committee.

Of course it works exceptionally well when the instinct turns out to be right. But can end companies if it isn’t.

Yep. There is no being nice in the business of copyright. But to be fair, they’ve owned the original base game assets.

Not to mention it’s a reverse engineered version of the base game.

Just tried this. Holy fuck.

I'd take an army of high-school graduate LLMs to build my agentic applications over a couple of genius LLMs any day.

This is a whole new paradigm of AI.

With so many things being called Flow these days, this one is probably the most fitting.

I’ve been following share prices for Micron, Seagate, Western Digital and Sandisk.

They’ve all pretty much 5x’ed YTD. That’s completely wild.

As someone paying some vague attention to market movements, this was predictable.

News of a deal and hype was largely responsible for the rise. Now that the sentiment is cooling off, it’s dropping back to a more reasonable level.

There is no free lunch. The amount of prompt writing to give the LLM enough context about your codebase etc is comparable to writing the tests yourself.

Code assistance tools might speed up your workflow by maybe 50% or even 100%, but it's not the geometric scaling that is commonly touted as the benefits of autonomous agentic AI.

And this is not a model capability issue that goes away with newer generations. But it's a human input problem.

This is an interesting approach.

My team has been prototyping something very similar with encoding business operations policies with LEAN. We have some internal knowledge bases (google docs / wiki pages) that we first convert to LEAN using LLMs.

Then we run the solver to verify consistency.

When a wiki page is changed, the process is run again and it's essentially a linter for process.

Can't say it moved beyond the prototyping stage though, since the LEAN conversion does require some engineers to look through it at least.

But a promising approach indeed, especially when you have a domain that requires tight legal / financial compliance.

Zen5?

Ability for normal people to set up reasoning chains.

Honestly, GPT-4o is all we ever needed to build a complete human-like reasoning system.

I am leading a small team working on a couple of “hard” problems to put the limits of LLMs to the test.

One is an options trader. Not algo / HFT, but simply doing due diligence, monitoring the news and making safe long-term bets.

Another is an online research and purchasing experience for residential real-estate.

Both these tasks, we’ve realized, you don’t even need a reasoning model. In fact, reasoning models are harder to get consistent results from.

What you need is a knowledge base infrastructure and pub-sub for updates. Amortize the learned knowledge across users and you have collaborative self-learning system that exhibits intelligence beyond any one particular user and is agnostic to the level of prompting skills they have.

Stay tuned for a limited alpha in this space. And DM if you’re interested.

I am genuinely curious how demand for such things spike up. Is it social media? Influenced by someone etc?

I am not on SM, and I really cannot even imagine the peer?-pressure that drives normal, sane and reasonable people living in modern society to go after something so silly and be willing to part so much money for it.

I understand how it works with designer bags and watches etc where it's a signal for social status. But this bag? Come on.

Anyone know why? Or is it just that there is a positive feedback loop here of people wanting to buy this bag only to resell it for a huge return, and that causes the prices to shoot up. When really nobody actually wants to buy this bag to use? Sounds like a group-think ponzi scheme.

TLDR: I’ve got stuff to do.

I feel exactly the same way. I recently bought my father-in-law an M4 iMac which he thought was a disproportionately nice gift for no apparent reason.

Oh there are some very compelling reasons for it. My tech support load went way down and he’s super happy. Win-win.

They keep pushing. We keep tolerating.

Every Tv in my home is paired with an Apple TV. I leave the display unconnected to the internet.

The day when TVs require an internet connection to show external inputs - that will be a sad day.

Or when they come with cellular modems.

Oh how far we've come.

My home internet is a fiber gigabit 3g/3g up/down. Tucked away under the staircase is where my fiber ONT terminates and it is my server room. I have half a dozen boxes running various things. 4 symmetric 2012 i7 mac minis running linux KVM, and hosting various critical services - pihole, home automation, Homekit Secure Video etc.

Then there a giant former gaming PC with 7 HDD bays running the entire storage backend for a whole load of GoPro/Osmo/Insta360 videos I capture. Rclone to Google Photos for back-up. I don't edit any videos. Just there to capture memories so I can at some point when AI tools get good enough just have it generate clips. Same box runs my plex server with HW transcoding.

Then there is the actual gaming PC, a mini-ITX running steam remote play. Has power, a network cable and a fake HDMI dongle that emulates a monitor to trick the GPU into thinking something is actually plugged in.

Basically everything I do with desktop PCs at home is via some sort of remote interface.

Remote gaming is probably the most demanding of all of these. Low-latency HW-accelerated solutions eg: Parsec / steam-link are incredible technologies.

I carry an AppleTV + PS5 controllers to friends' houses and play the latest games across the internet.

We use mermaidjs as a supercharged version of chain-of-thought for generating some sophisticated decompositions of the intent.

Then we injected the generated mermaid diagrams back into subsequent requests. Reasoning performance improves for a whole variety of applications.

When will this tool have SOC compliance and SSO support?

The devs on this must be sleeping. F for not paying attention to your users’ needs.

This concept blew my mind when I internalized it.

Same reason why honest signals exist. A peacock with very rich feathers is a fitness disadvantage. But they find mates more successfully. These traits persist in the gene pool.

It’s so much easier to just evolve a cheating trait that does the job of finding a mate even without the required fitness.

But the signals stay honest for the most part.

Why?

It’s because ultimately the species survives, not the individuals.

In a lot of cases, something that makes the individual more fit also makes the species more fit. But in some cases, they are inversely proportional.

Hence you end up with suicidal genes that favor the death of the individuals for the greater good of the species.

Now extrapolating to human society, most nations have landed on a system where taxes are paid to the government. Every individual might complain and try to get out of paying. But we do. Why? Maybe because societies where that wasn’t a thing were less fit and didn’t last long enough to still be around.

So the team I lead does a lot of research around all the “plumbing” around LLMs. Both technical and from a product-market perspectives.

What I’ve learned is that for the most part that AI revolution is not going to be because of PHD-level LLMs. It will be because people are better equipped to use the high-schooler level LLMs to do their work more efficiently.

We have some knowledge graph experiments where LLMs continuously monitor user actions on Slack, GitHub etc and build up an expertise store. It learns about your work, your workflows and then you can RAG them.

In user testing, people most closely associated this experience to having someone just being able to read their minds and essentially auto-suggest their work outputs. Basically it’s like another team member.

Since these are just nodes in a knowledge graph, you can mix and match expertise bases that span several skills too. Eg: A Pm who understands the nuances of technical feasibility.

And it didn’t require user training or prompting LLMs.

So while GPT-5 may be delayed, I don’t think that’s stopping or slowing down a revolution in knowledge-worker productivity.

What you are saying is totally correct.

And this applies to language / code outputs as well.

The number of times I’ve had engineers at my company type out 5 sentences and then expect a complete react webapp.

But what I’ve found in practice is using LLMs to generate the prompt with low-effort human input (eg: thumbs up/down, multiple-choice etc) is quite useful. It generates walls of text, but with metaprompting, that’s kind of the point. With this, I’ve definitely been able to get high ROI out of LLMs. I suspect the same would work for vision output.

It's slow. So we use hypothetical mostly for async experiences.

For live experiences like chat, we solved it with UX. As soon as you start typing the words of a question into the chat box, it does the FTS search and retrieves a set of documents that have word-matches, scored just using ES heuristics (eg: counting matching words etc)

These are presented as cards that expand when clicked. The user can see it's doing something.

While that's happening, also issue a full hyde flow in the background with a placeholder loading shimmer that loads in the full answer.

So there is some dead-time of about 10 seconds or so while it generates the hypothetical answers. After that, a short ~1 sec interval to load up the knowledge nodes, and then it starts streaming the answer.

This approach tested well with UXR participants and maintains acceptable accuracy.

A lot of the times, when looking for specific facts from a knowledge base, just the card UX gets an answer immediately. Eg: "What's the email for product support?"

What you can do with current-gen models, along with RAG, multi-agent & code interpreters, the wall is very much model latency, and not accuracy any more.

There are so many interactive experiences that could be made possible at this level of token throughput from 405B class models.

So I've done a ton of work in this area.

Few learnings I've collected:

1. Lexical search with BM25 alone gives you very relevant results if you can do some work during ingestion time with an LLM.

2. Embeddings work well only when the size of the query is roughly on the same order of what you're actually storing in the embedding store.

3. Hypothetical answer generation from a query using an LLM, and then using that hypothetical answer to query for embeddings works really well.

So combining all 3 learnings, we landed on a knowledge decomposition and extraction step very similar to yours. But we stick a metaprompter to essentially auto-generate the domain / entity types.

LLMs are naively bad at identifying the correct level of granularity for the decomposed knowledge. One trick we found is to ask the LLM to output a mermaid.js mindmap to hierarchically break down the input into a tree. At the end of that output, ask the LLM to state which level is the appropriate root for a knowledge node.

Then the node is used to generate questions that could be answered from the knowledge contained in this node. We then index the text of these questions and also embed them.

You can directly match the user's query from these questions using purely BM25 and get good outputs. But a hybrid approach works even better, though not by that much.

Not using LLMs are query time also means we can hierarchically walk down the root into deeper and deeper nodes, using the embedding similiarity as a cost function for the traversal.

Question for the group here: do we honestly feel like we've exhausted the options for delivering value on top of the current generation of LLMs?

I lead a team exploring cutting edge LLM applications and end-user features. It's my intuition from experience that we have a LONG way to go.

GPT-4o / Claude 3.5 are the go-to models for my team. Every combination of technical investment + LLMs yields a new list of potential applications.

For example, combining a human-moderated knowledge graph with an LLM with RAG allows you to build "expert bots" that understand your business context / your codebase / your specific processes and act almost human-like similar to a coworker in your team.

If you now give it some predictive / simulation capability - eg: simulate the execution of a task or project like creating a github PR code change, and test against an expert bot above for code review, you can have LLMs create reasonable code changes, with automatic review / iteration etc.

Similarly there are many more capabilities that you can ladder on and expose into LLMs to give you increasingly productive outputs from them.

Chasing after model improvements and "GPT-5 will be PHD-level" is moot imo. When did you hire a PHD coworker and they were productive on day-0 ? You need to onboard them with human expertise, and then give them execution space / long-term memories etc to be productive.

Model vendors might struggle to build something more intelligent. But my point is that we already have so much intelligence and we don't know what to do with that. There is a LOT you can do with high-schooler level intelligence at super-human scale.

Take a naive example. 200k context windows are now available. Most people, through ChatGPT, type out maybe 1500 tokens. That's a huge amount of untapped capacity. No human is going to type out 200k of context. Hence why we need RAG, and additional forms of input (eg: simulation outcomes) to fully leverage that.