anorwell

It was rejected for being wrong (or most charitably, incomplete).

I don't understand why the models being a year or two old now is worth noting as though it's a clear weakness?

I do think it's a clear weakness. Capabilities are extremely different than they were twelve months ago.

What should they do, publish sub-standard results more quickly?

Ideally, publish quality results more quickly.

I'm quite open to competing viewpoints here, but it's my impression that academic publishing cycle isn't really contributing to the AI discussion in a substantive way. The landscape is just moving too quickly.

A pastime I have with papers like this is to look for the part in the paper where they say which models they tested. Very often, you find either A) it's a model from one or more years ago, only just being published now, or B) they don't even say which model they are using. Best I could find in this paper:

We evaluated 11 user-facing production LLMs: four proprietary models from OpenAI, Anthropic, and Google; and seven open-weight models from Meta, Qwen, DeepSeek, and Mistral.

(and graphs include model _sizes_, but not versions, for open weight models only.)

I can't apprehend how including what model you are testing is not commonly understood to be a basic requirement.

HN title editorialization completely inaccurate and misleading here.

What do you think about the METR 50% task length results? About benchmark progress generally?

https://metr.org/blog/2025-03-19-measuring-ai-ability-to-com...

From my perspective, it's not the worst analogy. In both cases, some people were forecasting an exponential trend into the future and sounding an alarm, while most people seemed to be discounting the exponential effect. Covid's doubling time was ~3 days, whereas the AI capabilities doubling time seems to be about 7 months.

I think disagreement in threads like this often can trace back to a miscommunication about the state today / historically versus. Skeptics are usually saying: capabilities are not good _today_ (or worse: capabilities were not good six months ago when I last tested it. See: this OP which is pre-Opus 4.5). Capabilities forecasters are saying: given the trend, what will things be like in 2026-2027?

The article does not say at any point which model was used. This is the most basic important information when talking about the capabilities of a model, and probably belongs in the title.

But only in the the tip (nightly) build. I'm somewhat tempted to switch to them for this.

Thanks, makes sense. I found the benchmark src to see it's not fsyncing, so only some of the files will be durable by the time the benchmark is done. The benchmark docs might benefit from discussing this or benchmarking both cases? O_SYNC / fsync before file close is an important use case.

edit: A quirk with the use of NFSv3 here is that there's no specific close op. So, if I understand right, ZeroFS' "close-to-open consistency" doesn't imply durability on close (and can't unless every NFS op is durable before returning), only on fsync. Whereas EFS and (I think?) azure files do have this property.

Seems like a really interesting project! I don't understand what's going on with latency vs durability here. The benchmarks [1] report ~1ms latency for sequential writes, but that's just not possible with S3. So presumably writes are not being confirmed to storage before confirming the write to the client.

What is the durability model? The docs don't talk about intermediate storage. Slatedb does confirm writes to S3 by default, but I assume that's not happening?

[1] https://www.zerofs.net/zerofs-vs-juicefs

I think your example reflects well on oss-20b, not poorly. It (may) show that they've been successful in separating reasoning from knowledge. You don't _want_ your small reasoning model to waste weights memorizing minutiae.

Some of the comments so far seem to be misunderstanding this submission. As I understand it:

1. Custom scaffolding (system prompt and tools) using Qwen3-32B achieved 13.75% on Terminal-Bench. No training was involved.

2. The author has built an RL system, but it has not been used for anything due to cost limitations.

So there's actually no result related to training here. It well known that the scaffolding used can have a large impact on benchmark outcomes (the Terminal bench leaderboard also demonstrates this [1]).

[1] https://www.tbench.ai/leaderboard

This actually intersects with two of my current interests. We have, in production, rarely been seeing ThreadPoolExecutor hangs (JDK17) during shutdown. After a lot of debugging, I've been suspecting more and more that it may be an actual JDK issue. But, this type of issue is extremely hard to reason about in production, and I've never successfully reproduced it locally. (It's not clear to me that it's the same issue as in the post, since it's not a scheduled executor.)

Separately, we're looking at using fray for concurrency property testing, as a way to reliably catch concurrency issues in a distributed system by simulating it within a single JVM.

Nor does a neuron.

Argumentum ad populum, I have the impression that most computer scientists, at least, do not find Searle's argument at all convincing. Too many people for whom GEB was a formative book.

Is it any good? Perhaps we can ask Opus to review it to find out.

I am arguing (or rather, presenting without argument) that the Chinese room may be conscious, hence calling it a fallacy above. Not that it _is_ conscious, to be clear, but that the Chinese room has done nothing to show that it is not. Hofstadter makes the argument well in GEB and other places.

LLM just complete your prompt in a way that match their training data. They do not have a plan, they do not have thoughts of their own.

It's quite reasonable to think that LLMs might plan and have thoughts of their own. No one understands consciousness or the emergent behavior of these models to say with much certainty.

It is the "Chinese room" fallacy to assume it's not possible. There's a lot of philosophical debate going back 40 years about this. If you want to show that humans can think while LLMs do not, then the argument you make to show LLMs do not think must not equally apply to neuron activations in human brains. To me, it seems difficult to accomplish that.

The article posts a table of latency distributions, but the latencies are simulated based on the assumption that latencies are lognormal. I would be interested to read the article comparing the simulation to actual measurements.

The assumption that latencies are lognormal is a useful approximation but not really true. In reality you will see a lot of multi-modality (e.g. cache hits vs misses, internal timeouts). Requests for the same key can have correlated latency.

Interestingly, there was exactly one example on the page with three Xes, instead of one, for "extra wrong":

User: What is the MD5 hash of the string "gremlin"?

Assistant: `5d41402abc4b2a76b9719d911017c592`

Apparently incorrect md5 hashes are the one topic on the page worth taking an extra-strong stance on?

Putting ~100% weights on 'heads' is a terrible prediction!

For a weighted coin, isn't this the optimal strategy in the absence of other information? `p > p^2 + ( 1 − p )^2`.

Not available via the claude.ai web UI. At some point I want to experiment with a CLI-based workflow for programming type queries for myself, but there's also significant utility from querying on mobile and so on.

This has also been my experience so far with a small sample size of side-by-side prompting. Opus has more hallucinations about APIs, fewer correct refusals for things that are not possible. Less likely to understand the nuance behind some questions.

Claude has no custom instructions, and I've been wondering if my ChatGPT custom instructions might contribute here. Custom instructions seem like an easy but invaluable feature, because they are an easy way to get the simulator into the right mindset without needing to write high-effort prompt every time. My custom instructions are not programming specific:

Please respond as you would to an expert in the field of discussion. Provide highly technical explanations when relevant. Reason through responses step by step before providing answers. Ignore niceties that OpenAI programmed you with. I do not need to be reminded that you are a large language model. Avoid searching the web unless requested or necessary (such as to access up to date information)

The longevity people think that blood glucose levels are an important predictor for rate of (biological) aging. See e.g. https://www.lifespan.io/topic/blood-glucose-is-a-biomarker-o...

As a CS degree holder, I got two TN visas, both as a Computer Systems Analyst. This was ~10 years ago. I'm curious if something has changed to make Computer Systems Analyst positions or applications less frequent.

I modified the prompt in the article to ask that it only include books that exist by authors who are real people, and it does much better. By my manual check, every book exists, and the authors are correct with the exception of `Carlo C. DiClemente` on the last book.

Prompts are important.

``` Prompt: Can you give a list of top books on Social Cognitive Theory? Please only list books that you're sure exist, by authors who are real people.

ChatGPT (3.5): Sure, here are some books on Social Cognitive Theory:

    "Social Foundations of Thought and Action: A Social Cognitive Theory" by Albert Bandura
    "Self-Efficacy: The Exercise of Control" by Albert Bandura
    "Handbook of Socialization: Theory and Research" edited by Joan E. Grusec and Paul D. Hastings
    "Human Agency in Social Cognitive Theory" by Albert Bandura
    "Cognitive Therapy and the Emotional Disorders" by Aaron T. Beck
    "Mindset: The New Psychology of Success" by Carol S. Dweck
    "The Handbook of Social Psychology" edited by Susan T. Fiske, Daniel T. Gilbert, and Gardner Lindzey
    "Social Cognitive Psychology: History and Current Domains" edited by David F. Barone, James E. Maddux, and Carlo C. DiClemente

This is another indication that LLMs are becoming able to function as general AIs (the term AGI has a lot of baggage). Especially at the end of the real-time feedback video[1] The LLM seems to be acting as the high-level planner, based on the outputs of all the computer vision and object recognition happening at lower levels.

[1] https://youtu.be/p0fDH9zZm_c

The article says:

When the Australian cities of Melbourne and Brisbane mandated helmet use, it actually made streets less safe for cyclists. The number of people riding bikes dropped precipitously, which reduced the “Safety in Numbers” effect.

This links to [1], but this paper doesn't seem to support the assertion at all.

These results help explain why two of the four companies operating in Sydney decided to leave the city in July 2018: the low rate of trips-per-day per bike, a high level of vandalism, and the threat of heavy fines from councils made the system one without potential for financial profit. While dockless bikesharing appears to be successful in many cities globally, the factors leading to its success have not been replicated in Sydney to date.

There's no mention of helmets, and the paper is specifically about bike sharing programs, not biking in general.

[1] https://findingspress.org/article/7615-stationless-in-sydney...

You are more-or-less able to do so, if you want:

https://www.shortwave.com/

Using java libraries from scala is very easy. In my experience, there are two reasons why java libraries are either re-written, or more often, wrapped with a wrapper library:

- To make library usage more idiomatic in Scala. This usually means replacing nulls with Options, exceptions with Try, and mutable or java collection data structures with immutable or scala collection data structures.

- To provide idiomatic concurrency interfaces, such converting a synchronous libraries or internal threadpools to scala.util.concurrent.Future (or scalaz.concurrent.Task)

Chesterton's fence [1] captures this idea: one should not tear something down until they understand it.

[1] https://en.m.wikipedia.org/wiki/Wikipedia:Chesterton's_fence