WhitneyLand

The silence is deafening.

Google watches over the last few months a flat out assault on the Pareto curve from American and Chinese companies. Release after release pushing the boundaries of frontier intelligence and price/performance.

And the response from arguably the biggest AI research labs in the world by headcount is Flash 3.6.

What do you do when you are given essentially unlimited resources and still find yourself falling behind?

The world doesn’t need advisors like Moh that are bitter enough to gossip about a family funeral.

Not convinced just using top n sigma is going to beat a SOTA detector. The fingerprints Pangram uses should in principle be able to detect style above the token selection level.

Other papers have tried to beat it with temperature and it didn’t work, although I haven’t seen anyone try insane levels.

Give it a shot and let me know if you have any success.

I’m personally interested in it as part of the research to improve LLM writing. Detecting “AI voice” is part of understanding what’s wrong with it in the first place and how to improve it.

But yeah, in general I think you’re right, the actual utility is pretty niche.

I’m not sure what you’re saying I’m wrong about.

The comment I was responding to, about changing LLM output, referred to prompting, not temp/sampling tricks. I’m not aware of Pangram being beat by clever prompting. There’s some interesting work on creative writing using contrastive prompt techniques, but I haven’t seen it tried as evasion.

Even if you control temp and sampling, they’re not magic. If you raise the temperature too much writing can go to hell, so you may beat the detector but end up with junk. There are some ways to mitigate such a quality drop like raising temperature in conjunction with min-p, but still, I haven’t read any research that shows it getting good results at anything close to 10.

Now you want to get more clever and manipulate logprobs…well ok, you could come up with elaborate strategies designed to evade specific detection methods. But I don’t see that getting done as a weekend project while maintaining writing quality. And if it does happen there’s no guarantee the detector can’t train on its characteristics and start an arms race.

That’s a different point.

I’d want detectors to be as accurate as possible, false positives of 1 in 10000 seems like a good starting point. I believe their results have been independently tested.

And as a separate matter, any tool for evaluating students should be applied fairly, safely, and with adequate human review and due process.

You need good tools and good oversight.

"It's really easy to have a false positive"

Not really. The false positives for the SOTA detector are very very low.

"It's also very easy to change the pattern of LLM output."

Not in a way that can reliably avoid detection. The problem is the patterns are baked into the distribution itself. It's smoothed over, so it becomes difficult to prompt your way out of that.

"Text is simply not information dense enough to be able to decode some arbitrary signal of provenance from it...it's a bad fiction to perpetuate that any of this is anything more than tarot card reading."

Not true at all. Pangram is highly effective and has a very low false positive rate.

The post here is impressive for a small project, it looks like they independently thought of one of the core ideas Pangram uses of creating twins to compare.

You can see how it works here: https://arxiv.org/pdf/2402.14873

I find it hard to see a language as beautiful that’s grown too complex for a single person to hold a complete mental model of.

I used to think that was a personal limitation, until I saw an interview with Bjarne explaining that he used to understand all of it but at this point it’s too big, no one can anymore.

Everyone will still need to use Xcode for at least some debugging, no way around that.

As for the builds, your agent probably already knows how to do a lot of this from the command line, although explicit suggestions can help it build faster for different situations.

As for XcodeGen, you may find it unnecessary overhead if you're already using Xcode file system synced groups.

For iOS my biggest suggestion would be to enable App Store Connect skills for your agent (https://github.com/rorkai/App-Store-Connect-CLI).

With this not only do you not need Xcode all the time, you also don't even need to be near a MacBook.

Just make changes via Codex on your iPhone, the tell asc to build and upload to TestFlight, download and run the new version, iterate.

Your value is intrinsic as a human being. We’re capable of love and shared experiences that a machine will never know.

The prompt does matter. They specifically told it to assume a proof exists so it would not too easily dismiss the possibility.

If all checks out this is a huge milestone. AI has now solved one of the most famous open problems in graph theory, using an off the shelf model, in one hour.

It might be a better mathematician than most humans at this point. Kind of like when chess software started beating everyone except grandmasters.

What’s left? Proposing and building out entirely new theories and frameworks? Then better than any human? Then alien math results we struggle to comprehend?

Codex is supported well on iPhone/iPad, it’s inside the ChatGPT app.

It’s amazing how much work you can get done on your phone now, especially if you already have a design mapped out in your head.

That would be cool, but is there a way to differentiate between voice mode and text only output in the custom instructions?

Otherwise, it seems like preferences could be competing.

Much better than it was before but it’s still significantly weaker than a direct chat.

For example I asked

“Why should LLM attention use dot product instead of cosine similarity, being that we often hear vector magnitude does not encode most of the useful information needed”?

The voice response was directionally right but lacked detail and was a little hand wavy.

The answer to the same question in a text chat was much higher quality.

The voice response replied “let me think about that…” so it appears to be invoking 5.5 as advertised, but it’s definitely weaker.

I had reasoning set the same for both.

Nice work.

It’s advertised 7MB, but also comes with a 5MB mini version.

Looks like mini saves space by using 256 element vectors internally instead of 384, but then projects it up to 384 at the end for compatibility.

It’s a third smaller, but the loss is not linear, looks like you give up less than 1/3 of information with the smaller data path.

“Z.ai provides a replacement MCP for web search, but it's pretty awful and slow”

I’ve had good results with Tavily so far, might be worth checking as an alternative for agent search.

This post seems like some decent advice mixed in with a lot of overconfidence and unverifiable claims.

“expert developers whose skills have reached the point where they outclass any and all “frontier AI models” in their area of expertise”

Are any developers saying they outclass any and all frontier models? I’d say at best it’s mixed at this point. The best developers still do certain things better, but not even close to all things.

“The problem is that even code written and/or reviewed by Fable 5, will stink”

I’m skeptical. Example prompt and output please.

The premise of a secure open codebase is fine.

The problem is being more auditable does not automatically make it more audited.

There have to be enough people with skill taking enough time to work on it.

What’s with the 3 subscription plans that are suggestive of being mapped to plans from Anthropic and Open AI?

Do they really correspond roughly? Seems like they’re trying to suggest a discount while still being worth a significant amount of monthly spend.

This is some high level hard earned wisdom.

How strict is this no AI policy?

Say AI is used to identify and rewrite a single function that improves performance or fixes a bug, then the developer carefully reviews and tests it and submits a nice tight PR with all human communication.

So they don’t want that? They would just reject it?

If I’m understanding correctly, under the policy the higher performance function / bug free submission would be rejected and they could ask for a rewrite.

Should it then be rewritten from scratch, and clean room engineered so it doesn’t resemble the AI too much?

Fredrik, while acknowledging everything you said, the fact remains some of my money can end up empowering politics I find repugnant.

If it were a small amount of money, it wouldn’t be an issue. If the politics at stake were less important, it wouldn’t be an issue.

They’re not going to stop at immigration, look at other places in the world to see the future risk.

Sorry, I was a paying and satisfied customer, and now I’m out.

I’m not sure how much we’re disagreeing, as I mentioned I don’t claim the gap can’t be closed. I’m saying as of now the gap is real and quantifiable.

That observation doesn’t really prove that the argument fails, right? Just because analogies were used powerfully beyond the experience of a writer doesn’t let us conclude that AI has transcended or will soon transcend the value of all qualia (Mary’s room thought experiment).

And on the other hand we do have evidence to the contrary. AI is not capable of writing a movie script that resonates with people like this. It’s not capable of doing what a stand-up comedian does.

People will argue if and when these gaps can be closed. I think they can be in principle, but it’s not going to happen anytime soon.

Well image generators work differently…

Do you mean that they may get away with less oscillators because of the decoder layer? Well there’s the rub isn’t it, the more work you have done by a software layer the less power you’ve proportionally saved by having it be done by physical computing.

But let’s spitball here what would you estimate would be needed in number of oscillators and interconnects for a 4K image?

Which idea is not outlandish? Physical computing? I agree physical computing is a fascinating topic.

But specifically what they’ve simulated here? I don’t see how that would ever work in real life scaled up to any kind of real size.

I’m not criticizing them for starting out small. Lots of things can be proven with small models. I’m saying in principle, I don’t see how this will work unless there’s some fundamentally new technique that is currently not known about. Maybe they have some secret idea but they haven’t shown it here.

It’s not clear to me how this would ever be practical since it seems dependent on n^2 scaling.

You’ve got to wonder when you have an image generation demo why would you possibly have 64 x 64 pixel output as your demo?

If I’m understanding this properly to generate a 4K image, you need like 5 trillion point to point connections on the chip. Even if power use from the oscillators is zero that’s going to be an issue.

First great work.

Reminds that I wish there was a modern way to do this for the words people speak and write online with. I want to literally know when people started putting literally twice in sentences.

Ngram seems is out of date a piece meal. Now Corpus seems like they try but UX terrible.