quatonion

I did a lot of work on intentional programming at Microsoft back in the 90s on C++, and also intentional software later.

Our approach was quite a bit different to Kiczales at the time. We had "attribute providers" which were essentially compiler plugin DLLs.

These plugins could register themselves at various parse points and add work like enzymes on the AST.

So here we have a compile time attribute that writes all the boilerplate to add windowing support to a class. Adding message maps, hooking up the dispatch, life cycle etc.

[Window] class MyWindow { };

It worked, but as others pointed out, it suffers from combinatorial explosion, and issues with debug ability.

That said, as long as you stick within well defined verticals it was still very useful and saved a lot of typing, and reduced cognitive load.

I have thought quite a lot about modern versions of this using LLMs and I can see why the article notices a parallel to prompts or spec based designs.

At some point I tried to make a version of the system we built almost 30 years ago, but using an LLM as the preprocessor instead of rigid AST hackery.

It works a lot better, and you can approach a more continuous interpolations between intent blocks and generated code. Even the combinatorial problem largely disappears.

I stopped working on it though because I couldn't really see anyone wanting to adopt a new language or some whacked out extensions these days.

Maybe it does have its place though in certain fields. It might be worth having another go at it with fresh hindsight.

Housing demand is back up. Not just single family homes but also condos.

Hmmm, the median house price is $2.1M.

It's a good time to live in SF.

It seems pretty K shaped.

Great time to live in SF and Bay Area iff you are an AI researcher, founder, VC, executive at one of the hyper-scalers, or a senior engineer with massive equity package in the few winning companies.

Not entirely sure if it's a great time to live in SF if you work at a bakery, are a Barista at Starbucks or even a mid tier SWE.

Thoughts?

Currently we have zero information what is causing the issue. And all providers have suffered outages or rate limits.

Can you post some images of lines getting garbled. That sounds like a genuine bug Anthropic might want to look into. I haven't seen that ever.

They do though. For some reason whatever path this is going through isn't using it.

Just as a reminder, Google DeepMind is right here in the UK, and there are hundreds, thousands of top AI researchers that are foundational to everything that the US is doing, as much, if more so than the Chinese contributions.

We need compute, yes, but we certainly aren't short of talent if we put our minds to it, and many of them are already here.

That's brilliant. And 10,000 copies! Woot, look at you go! Very well done and I think it looks great. It gives me hope. I'll go make it 10,001.

I'm working on a durable ECS system, mostly for games, though generally useful. It's to exercise a storage engine I have been building with Git like properties.

Should be able to turn the computer off at the mains half way through, then restart and instantly carry on without any loading phase as if nothing had happened.

Well that's the dream.

Now the lower layers work I'm mapping out the actual ECS part and what the API will be like.

I'm quite enjoying it and it's a really fun challenge.

That's certainly one way to do it, but where would you place them?

No, but seriously, you could imagine what we witnessed playing out in a high stakes Tom Clancy or Michael Crichton style fable.

The fiery blowhard Pentagon chief, the arrogant know it all tech bro lab head, an alarm being called in from a remote office and surfaced through Amazon.

It almost writes itself.

But we can use your name for the novel.

"Bolt Cutter"

Has a nice ring to it.

Half of me wonders if it was all a live simulation/drill, to practice what happens if a much more serious event occurs, and a model needed to be quickly shut down.

Under such conditions we would be looking at Amazon's actions through a much more benevolent lens.

Not saying it has been, but it certainly crossed my mind as something worth doing regardless.

Right, but I am talking about the general government response trajectory.

And also, even though Anthropic may not have labs themselves directly, there is a funnel of research that comes in the form of papers and conference tracks.

The AI community is pretty tight knit, and not having access to frontier models affects everyone.

I wonder how this is going to work given half the people working at the AI labs are Chinese foreign nationals, and even more interesting, DeepMind is based in the UK. Plus there is an awful lot of AI research going on all across Europe, especially Switzerland, that is feeding straight into the US major labs.

Banning foreign nationals from using your technology only makes sense if you don't rely on foreign nationals to build it in the first place.

Or are we so far along now we think we don't need them anymore.

I'm wondering if they might go for a restricted access model that goes beyond passport or citizenship, where people can still use it, but you have to be individually vetted, and put on a list to get security clearance.

And just imagine the true capability of AI if Fable and Mythos are the models known publicly. We can only imagine what is behind closed doors.

A Ryzen 9 laptop with 24GB of RAM and a 5070 is £1200. This thing better be dirt cheap. Basically add the price of the RAM, and deduct the price of having to use ARM and being forced into a Windows ecosystem. So what's that total? £1500 to £2000 depending on who you ask? Who is this for? Not gamers, not average Windows users, not professional developers or content creators.

Thing is, hardly any games are optimized for ARM. And no serious AI development occurs on Windows.

I understand that you want multiple models running concurrently, but then 128GB is starting to look cramped.

It's a bold move that goes one to one against Apple and AMD Strix Halo.

I'm looking at it and thinking, if it can run Linux at a fair price it could be great.

I'm curious. I am thinking, what does a non developer buy this thing, take it home, and do with it.

What does the unboxing and first 24 hours look like?

A mixture of emotions somewhere between thinking you are living in the future, and frustration at not actually being able to do much.

The target audience seems like wealthy early adopters, but that is about it.

I guess we shall see.

Hah, I did this on CB radio here in the UK last year, which was hella fun. I created a dashboard to run the whole thing with hosts/presenters and guests. Different LLM providers with different personas. I had a way where you could clone a persona from a known person, so one of my stock presenters was Art Bell, for example. Then I had all kinds of strange guests. Well it was just for fun so the setup was incredibly janky, but it did work, and as you mentioned, I found it quite hilarious as well - and unhinged! I did want to get into the management side of it too but got tired of the project. I still think it would be incredibly cool for community radio, especially as agents can pull from local newspapers, events or facebook, so they can talk about a missing cat or the state of the pot-holes. Very cool stuff OP!

Well Jamieson O’Reilly, Wiz, SentinelOne, Zenity and CTech by Calcalist, for a start

I used it to add a MIDI driver and support to my OS this afternoon. Worked okay, but I agree it is a bit clunky yet. I think it is pretty good for a preview release. Much better than nothing.

You aren't the only one. Lol. Those maniacs are out of control.

They really need to get rid of human mods on there, it is completely toxic.

I did this with my experimental site GoodFaith.

The idea is everyone posts in good faith, and AI moderates the posts before you send.

You can still go ahead if you don't agree with the moderation, so it isn't censorship, it's just that it provides a pause.

The AI also rates the posts and comments, just like you said.

I built it for precisely the reasons you said, because the way it is going, reddit is unsustainable.

I hope they change something.

I have been having a crack at it in my spare time. A kind of intentional LISP where functions get turned into WASM in the cloud.

The functions are optionally tested using formal verification. I plan to enable this by default soon, as time allows.

These functions that get written can then be composed, and enzymes that run in the cloud actively look for functions to fuse.

Also more people use it, the faster the compiler gets via network scaling laws.

It's very much research at the moment, but kinda works.

Jupyter Notebook style interface with the beginnings of some image and media support.

https://prometheus.entrained.ai

Can try looking at some of the examples or trying something yourself.

Would love some feedback.

For some time I have been trying to replace the very costly attention pass in LLMs.

Here is my current attempt at fixing things.

This is applicable beyond LLMs, but that is certainly an important use case.

Description, Ready to use Code and Interactive Educational materials inside.

This is really interesting. I always wondered how it works.

Couple of years ago I did some experiments using a surrogate for attention using a feed forward network (MLP) to avoid the quadratic explosion.

It worked but had problems at the time, and my mind wasn't really in it.

This has dug it back out again with the benefit of time and additional insights.

So now I'm thinking, you can use a lot of the insights in the work here, but also shoot for a full linear scaling surrogate.

The trick is to use the surrogate as a discriminator under an RL regime during training.

Instead of just applying better/faster math and optimizations alone, have the model learn to work with a fundamentally better inference approach during training.

If you do that, you can turn the approximation error present in the FFN surrogate inference method into a recovery signal encoded into the model itself.

I haven't tried it, but don't see a reason it shouldn't work. Will give it a go on a GPT-2 model ASAP.

Thanks again for the awesome article.

Really not a big fan of batteries included opinionated protocols.

Even Cap'n Proto and Protobuf is too much for me.

My particular favorite is this. But then I'm biased coz I wrote it haha.

https://github.com/Foundation42/libtuple

No, but seriously, it has some really nice properties. You can embed JSON like maps, arrays and S-Expressions recursively. It doesn't care.

You can stream it incrementally or use it a message framed form.

And the nicest thing is that the encoding is lexicographically sortable.

Are we a hundred percent sure it isn't a watermark that is by design?

A quick test anyone can run and say, yup, that is a model XYZ derivative running under the hood.

Because, as you quite rightly point out, it is trivial to train the model not to have this behaviour. For me, that is when Occam kicks in.

I remember initially believing the explanation for the Strawberry problem, but one day I sat down and thought about it, and realized it made absolutely zero sense.

The explanation that Karpathy was popularizing was that it has to do with tokenization.

However, models are not conscious of tokens, and they certainly don't have any ability to count them without tool help.

Additionally, if it were a tokenization issue, we would expect to spot the issue everywhere.

So yeah, I'm thinking it's a model tag or insignia of some kind, similar to the fun logos you find when examining many silicon integrated circuits under a microscope.

That is just a made up story that gets passed around with nobody ever stopping to obtain formal verification. The image of the whole AI industry is mostly an illusion designed for tight narrative control.

Notice how despite all the bickering and tittle tattle in the news, nothing ever happens.

When you frame it this way, things make a lot more sense.

I know right, if I didn't know any better one might think they are all customized versions of the same base model.

To be honest that is what you would want if you were digitally transforming the planet with AI.

You would want to start with a core so that all models share similar values in order they don't bicker etc, for negotiations, trade deals, logistics.

Would also save a lot of power so you don't have to train the models again and again, which would be quite laborious and expensive.

Rather each lab would take the current best and perform some tweak or add some magic sauce then feed it back into the master batch assuming it passed muster.

Share the work, globally for a shared global future.

At least that is what I would do.

Same where I went at Leeds Uni in the mid 80s.

Ours was just for CS undergrads mostly when I was there, and wasn't too overloaded. I guess we had about fifty terminals maybe on campus at least.

I remember we could dial it up from a couple of terminals in our Halls of Residence over JANET.

You are right, I never found it that slow either - loved that machine and the terminal to terminal messaging was crazy fun.

I have no glazing built into my custom instructions, but it still does it.

It used to be a lot better before glazegate. Never did quite seem to recover.

I don't mind us having fun of course, but it needs to pick up on emotional queues a lot better and know when to be serious.

Why you can't download an entire chat as markdown

Copy/Pasting sections of the chat on mobile is laborious

That it still gets manic and starts glazing

That it can remember some things and keeps bringing them up, but forgets other, more pertinent things

If you switch away from it while it is in the middle of generating an image it often cancels the image generation

Image editing accuracy seems to have gone down significantly in quality based on intent.

You can't turn a temporary chat into a permanent one.. sometimes you start a temporary and realize half way it should be permanent - but too late.

The em dashes need to go

And so do the "it's not this, it's that!"

Is it really necessary to make so many lists all the time

Canvas needs a bunch of work

It's brilliant. Love it. I want to know more about the generation, it looks very well thought out. Worth an article in itself.

Come now. Being kind is also a thing, and I think it sounds more than acceptable.