meander_water

I thought all model providers are doing this under the hood anyway in their UI?

They certainly seem to when A/B testing different models, and Fable routes to Opus 4.8 when guardrails fail.

Also, openrouter recently released a fusion router - https://openrouter.ai/blog/announcements/fusion-beats-fronti...

GPTZero is much better at handling humanized outputs. Also has a similar false positive rate to Pangram.

However, it’s your job to go down the rabbit hole, learn the 100%, and sprinkle in your 3%.

I would say that there is a big difference between stealing without acknowledgement, and stealing with acknowledgement and actively learning through reverse engineering.

I don't think you should waste time reviewing every single line of code in here and just use AI to review it!

What you bring is the knowledge that the author nor the LLM doesn't know.

How can you possibly know what relevant context to provide the LLM unless you read the 10k loc? Now you've wasted double the time.

Thanks, I didn't mean to be brusque, but I have seen a lot of these vibe tests lately that come to grand conclusions like "X model is better than Y" from the result of a single prompt.

Appreciate you sharing the results of your tests though!

So we ran it head-to-head against Claude Opus 4.8: same one-shot prompt, build a 3D platformer in raw WebGL from scratch

Running a single one-shot prompt is not a benchmark, not is it representative of any sort of real-world usage.

Most agent usage is collaborative so you need to test things like reliability (when I delegate a task, does it complete it without making up test results for e.g.) and steerability (does it obey my instructions or does it just do what it thinks is best).

Really curious to understand why I'm being downvoted. I don't think it's a particularly spicy take - Just choose the right tool for the job.

As someone who has built both react based frontends and html based ones (with htmx), there is a law of diminishing returns at play.

To start off, writing a basic crud website with forms is much easier with htmx.

But when you start building more complex components, and integrate with other systems (OAuth for e.g.) there are tons of libraries and SDKs for the react ecosystem, but not many for pure html components.

At this point, it's much easier to use off the shelf components than it is to manually write html to handle all the bizarre UI edge cases.

All the model releases we've seen this year have only made incremental improvements in benchmarks.

This feels like the first release that feels like a significant step up in terms of benchmark results.

Can anyone make an educated guess what the secret sauce in the model architecture is between 4.8 and Fable?

Not the first study, and they all largely report the same results:

https://www.nature.com/articles/s41562-025-02259-6

https://www.theguardian.com/money/2019/feb/19/four-day-week-...

https://www.4dayweek.com/research

Lovely sentiment in the article, which was unfortunately AI generated.

Can we start tagging titles in HN with [AI-generated] or something?

I know some people have no problem with it, but it might help others (like me) to steer clear

Other gems in a similar vein

https://github.com/narze/awesome-websites-as-answers

That part of the article almost read like clickbait, because at the end he admits there is an upper bound arg:

uv add pydantic --bounds major

So not really sure what he's complaining about

the model has its own emergent guardrails that sometimes cause it to push back on legitimate security research requests. But as we found, these organic refusals aren’t consistent - the same task, framed differently or presented in a different context, could produce completely different outcomes as illustrated in the examples below.

This was new. I'm surprised that a model specifically designed for security research and gated to professionals is refusing legitimate requests

I'm so excited for this, nice work!

Gemma4 edge models were promised to be great for agentic use, but have been really disappointing in all my tests. They fail at the most basic tool use scenarios.

Have you run any tool-use benchmarks for Needle, or do you plan to? Would be great if you could add results to the repo if so.

This appears to be part of the same Mini Shai-Hulud campaign affecting Tanstack Router https://www.securityweek.com/tanstack-mistral-ai-uipath-hit-...

Sure, but will they download the right version? And will they be inspecting the right files on disk? There's a whole lot more that can go wrong

One underrated advantage of using Python or Typescript is that AI agents can inspect the code of installed dependencies.

This means you don't have to muck around with supplying the right documentation for each version of each dependency, or worry about hallucinated interfaces (at least with the latest models).

In the past you'd have to dig through a foreign codebase manually to figure out why a documented interface for a dependency is not working as expected, but frontier models automate that quite well.

I don't understand why people were voting this comment down in the issue page

We find that models are not failing due to “death by a thousand cuts” (i.e., many small errors). Instead, they main- tain near-perfect reconstruction in some rounds, and experience critical failures in a few rounds, typically losing 10-30+ points in a single round trip

We find that weaker models’ degradation originates primarily from content deletion, while frontier models’ degradation is attributable to corruption of content.

I think we largely already knew this. This is why we fudge around with harnesses and temperature etc.

Agree. All of the major AI model labs have designed their user interfaces in entirely the wrong way.

Prompting via text alone is a really bad way to generate images. Ideally you want Canny Control to draw an outline of the image with elements in the exact locations where you want them. It's why comfyui is so great.

The ability to edit images and specify regions in the image for the prompt is a step in the right directions though. ChatGPT and Gemini have this.

I think one of the reasons for sloppy images is that non-artistic people don't have the vocabulary to describe images to be produced in interesting styles.

Yes, you can do image-> text on existing styles, but something always gets lost in translation.

Midjourney probably has the best baseline, and --sref is a really easy way to differentiate

Agree with JSON. But, surprisingly html and latex perform slightly better than markdown for more complex tables.

Check out this paper - https://arxiv.org/abs/2506.13405

Surprised they didn't mention docker compose secrets - https://docs.docker.com/reference/compose-file/secrets/

Isn't this just HATEOAS as espoused by libraries like htmx, datastar, hotwire etc.

https://htmx.org/essays/hateoas/

Thanks for the insider info! Do you know if there are any published benchmarks for Nano 3?

This looks like it uses Gemini Nano under the hood. But the latest Gemma4 E2B and E4B models appear to be much better, so you'd probably be better off deploying quantized versions through an extension for now.

- Gemini Nano-1: 46% MMLU, 1.8B

- Gemini Nano-2: 56% MMLU, 3.25B

- Gemma4 E2B: 60.0% MMLU, 2.3B

- Gemma4 E4B: 69.4% MMLU, 4.5B

Sources:

- https://huggingface.co/google/gemma-4-E2B-it

- https://android-developers.googleblog.com/2024/10/gemini-nan...

I think most labs actively create synthetic data using existing model as part of the mix for the pretraining stage for their next model.

Would love to know exactly what the latest process is to keep slop out of training data.

Unfortunately large parts of the paper that he linked to from the Wharton school is entirely AI generated, and yet to be peer reviewed.

I realize that most researchers use AI to assist with writing, but when the topic of your paper is "cognitive surrender", I struggle to take any content in there seriously.

The crazy part is that none of this is unexpected.

This was exactly the reason why GPT-2 was restricted for general release in 2019.

Check out section 4 - https://cdn.openai.com/GPT_2_August_Report.pdf