fxnn

We can expect the model landscape to consolidate some day. Progress will become slower, innovations will become smaller. Not tomorrow, not next year, but the time will come.

And then it'll increasingly make sense to build such a chip into laptops, smartphones, wearables. Not for high-end tasks, but to drive the everyday bread-and-butter tasks.

Read the blog post. It mentions that their chip has a small SRAM which can store LoRA.

No, it did not mostly refer to to-do lists. At least that's not what I understood.

It was an example, yes. But he also referred to Luhmann’s Zettelkasten, a huge collection of notes, systematically interconnected, see https://en.wikipedia.org/wiki/Niklas_Luhmann#Note-taking_sys.... Looks like the author followed a similar approach in Obsidian, a tool to store markdown notes in an interconnected manner.

He wrote: "Roam Research turned bidirectional links into a cult. Obsidian let the cult go off-grid. The lore deepened. You weren’t taking notes. You were building a lattice of meaning. ... A quote would spark an insight, I’d clip it, tag it, link it - and move on."

But it didn't work out: "the insight was never lived. It was stored. ... In trying to remember everything, I outsourced the act of reflection. I didn’t revisit ideas. I didn’t interrogate them. I filed them away and trusted the structure. ... The more my system grew, the more I deferred the work of thought to some future self who would sort, tag, distill, and extract the gold. That self never arrived."

Hence, what he mostly wrote down were thoughts, ideas, quotes, hoping that insights and value will come from this huge collection. Turns out, it wasn't so easy for him. I believe he'll need more focus, more curation, a more targeted and heartfelt approach.

Well, it sounds like it did work out back then for Niklas Luhmann. He said that all his famous publications wouldn't have been possible without his structured note taking approach. His 90,000 cards were digitized and are now available online, see https://niklas-luhmann-archiv.de/bestand/zettelkasten/zettel...

That's one thing where I love Golang. I just tell Aider to `/run go doc github.com/some/package`, and it includes the full signatures in the chat history.

It's true: often enough AI struggles to use libraries, and doesn't remember the usage correctly. Simply adding the go doc fixed that often.

The author would be excited to learn that CouchDB solves this problem since 20 years.

The use case the article describes is exactly the idea behind CouchDB: a database that is at the same time the server, and that's made to be synced with the client.

You can even put your frontend code into it and it will happily serve it (aka CouchApp).

https://couchdb.apache.org

Even if I don’t care much about the „digital garden“ concept, I was much surprised to visit a page called „blogscroll“, click on a link, and then not find a blog.

When we talk about understanding a simple axiomatic system, understanding means exactly that the entirety of the axioms are modeled and applied correctly 100% of the time.

Yes, but then, when we talk about understanding in LLMs, we talk about existence, but not necessarily about determinism.

Remember that, while chess engines are (I guess?) deterministic systems, LLMs are randomized systems. You give the same context and the same prompt multiple times, and each and every time you get a different response.

To me, this, together with the fact that you have an at least 1-in-10 chance of getting a good move (even for strange scenarios), means that understanding _does exist_ inside the LLM. And the problem following from this is, how to force the LLM to reliably choose the right „paths of thought“ (sorry for that metaphor).

Maybe it needs even smaller steps, and a programmatic (i.e. multi prompt) habit to always double-check / validate the assumptions and outcomes.

But that also exists in the AI world. It’s called „fine tuning“: a LLM trained on a big general dataset can learn special knowledge with little effort.

I’d guess it’s exactly the same with humans: a human that received good general education can quickly learn specific things like C.

Hmm. You sure this is the same thing? I would say it’s more about confidentiality than about value.

Because what companies want to hide are usually secrets, that are available to (nearly) no one outside of the company. It’s about preventing accidental disclosure.

What AIs are trained on, on the other hand, is publicly available data.

To be clear: what could leak accidentally would have value of course. But here it’s about the single important fact that gets public although it shouldn’t, vs. the billions of pieces from which the trained AI emerges.

Okay if it’s about looking at one painting and fake that. However, if you train your model on billions of paintings and create arbitrary new ones from that, it’s just a statistical analysis on what paintings in general are made of.

The importance of the individual painting diminishes at this scale.

Okay, so the scale at which they sale their service is a good argument that this is different from a human learning.

However, on the other hand we also have the scale at which they learn, which kind of makes every individual source line of code they learn from pretty unimportant. Learning at this scale is statistical process, and in most cases individual source snippets diminish in the aggregation of millions of others.

Or to put it the other way round, the actual value lies in the effort of collecting the samples, training the models, creating the software required for the whole process, putting everything into a good product and selling it. Again, in my mind, the importance of every individual source repo is too small at this scale to care about their license.

For me that discussion is always hard to grasp. When a human would learn coding autodidacticly by reading source code, and later they would write new code — then they could only do so because they read licensed code. No one would ask for the license, right?

So why do we care from where LLMs learn?

Sounds like the perspective of someone who never gave that tool a real chance. To me it’s primarily just another aid like IDE inspections, IDE autocompletions etc.

I use GitHub Autopilot mainly for smaller chunks of code, another kind of autocompletion, which basically safes me from typing, Googling or looking into the docs, and therefore keeps me in the flow.

Highly recommending Martin Kleppmanns „Designing Data-Intensive Applications“ book (O‘Reilly). Covers virtually any topic around database implementation. Well written, big source of knowledge. Introduces concepts gradually and with many examples. Lots of references to actual software out there, and how they are done.

KV persistence is covered in Chapter 3, right from the start. Redis is also mentioned as an example for an in-memory store with „weak durability by writing to disk asynchronously“.