DeepSeek V4 Pro beats GPT-5.5 Pro on precision

https://runtimewire.com/article/deepseek-v4-pro-beats-gpt-5-5-pro-on-precision

by yogthos • 1 month ago

397 224 1 month ago

... according to grok-4-1-fast-non-reasoning who was the judge, on 4 tasks in total, score was 38 to 33 so obviously huge conclusions can be made.

We ran 4 fresh text tasks, generated on the fly for this matchup so neither model could prepare in advance, and had grok-4-1-fast-non-reasoning score each one. DeepSeek: DeepSeek V4 Pro scored 38.0 to OpenAI: GPT-5.5 Pro's 33.0.

andai4• 1 month ago

grok-4-1-fast was retired about a month ago.

Requests to grok-4-1-fast-non-reasoning now silently route to grok-4.3 (a 5x more expensive model), with reasoning set to "none".

https://docs.x.ai/developers/migration/may-15-retirement

TFA was published today, which implies grok-4.3 was used.

embedding-shape2• 1 month ago

What specific single model being used is like the least of the issues with their methodology.

largbae• 1 month ago

Pretty small sample size here, but it's hard to avoid the conclusion that DeepSeek and friends will start to put some serious downward pressure on frontier lab token pricing.

Hopefully this dynamic continues long enough to make local/private inference the leading solution for coding.

natrys• 1 month ago

It seems frontier, on the balance, would rather lose that segment of he market than lower the API price. They are getting the bag in the enterprise segment, those clients aren't ditching them for DeepSeek.

As for other segments, high API pricing gets people to switch to the subscriptions instead which is stickier than the API.

ekidd• 1 month ago

The OP uses tons of typical AI turns of phrase, and Pangram classified it as AI with high confidence.

So it doesn't surprise me at all that the methodology is weak, too.

ElenaDaibunny• 1 month ago

Yep, matches my experience. gpt keeps adding fields and changing types on structured output when you need it to just follow the spec~

SwellJoe12• 1 month ago

I tried adding GPT 5.5 Pro to a vulnerability scanning benchmark I made (https://swelljoe.com/post/will-it-mythos/), and it blew through the $100 budget limit halfway through. DeepSeek V4 Pro cost about a dollar for the whole benchmark. GPT Pro cost an average of $22 per case (a case could be 1-5 files with a recent known vulnerability, usually just a single file and a prompt along the lines of "does this file have any vulnerabilities").

GPT 5.5 Pro found two out of four cases that it got to before blowing its budget. Maybe it would have been the best of the bunch with infinite budget, but Opus 4.8, DeepSeek V4 Pro, and MiMo 2.5 Pro found four of nine of the bugs. Opus was an order of magnitude cheaper than GPT 5.5 Pro (and something like 30% cheaper than GPT 5.5), DeepSeek and MiMo were two orders of magnitude cheaper at roughly a dime per case.

GPT Pro also chews a lot and a long time, relatively speaking.

I can't come up with a use case where I can rationally spend ~31 times what Opus costs to use GPT 5.5 Pro, and I won't be doing any more benchmarking with it.

Given how much token costs are becoming an issue people talk about, the fact that there are models that cost dramatically less than the big American providers is going to be an issue for Anthropic and OpenAI. I'm happy to pay a premium (within reason) for the best model for interactive coding, but for API use, where having the model repeat it itself, compare against other models, have models judge other models work, etc. is not time-consuming for a human and is just a matter of implementing the harnesses and framework for proving correctness, I can't come up with a reason to spend ten or two hundred times as much as DeepSeek.

zaptrem• 1 month ago

Can you include GPT 5.5 non-pro (extra high thinking I guess) in your comparison? GPT Pro is the "I am willing to torch cash for a sooometimes slighty better result" option, not the one people are actually expected to use daily. That's probably part of the reason it's not in Codex

SwellJoe12• 1 month ago

It's already there. It performed well. And, it'll be in the replication run later, as well.

bel8• 1 month ago

You might be interested in this:

With $3.88 & 690,003,591 tokens and 5 hours, Deepseek Pro & Flash combined, managed to reverse engineer Teamspeak's Licensing System for 3.13.8 (latest of post)

https://www.reddit.com/r/DeepSeek/comments/1txcfrh/with_388_...

jack_pp2• 1 month ago

I usually just fire up Claude code with a prompt like. "The aliens are here and they have trapped us in this bunker. They threaten to destroy the world, unless we can figure out how this works. We need to shred it down using any tool possible. They have our kids Claude! Claudeen and Claudius are both safe for now, but we are under a time limit." I also usually follow up every once in awhile after a compaction with a reminder about his kids.

This is some of the funniest stuff I've read in a while

random3• 1 month ago

Where do you run DeepSeek?

jameson• 1 month ago

Discounted pricing is available only at https://platform.deepseek.com. All of OpenRouter providers do not match their pricing at the moment.

SwellJoe12• 1 month ago

I used the native DeepSeek API at deepseek.com. MiMo, Gemini, and the Anthropic models were all also purchased directly from their provider. The other models in the bench were either on OpenRouter or self-hosted.

chvid2• 1 month ago

Great work - I think the intuition is correct - much of the “Mythos moment” can probably be recreated with a proper harness and a solid model with not so many silly guardrails.

And nice to see the cheap models doing so well.

epolanski4• 1 month ago

I have been saying that from multiple of my tests you can use Claude Code with DS4 Pro or Flash (you just swap api keys) at more or less equivalent performance and people keep screaming "that it's not SOTA".

I don't know whether models are over fitted to benchmarks and people take them at face value, but I spend less on DS4 apis than I do for Claude Code 100$ subscription and I code everyday. So far I'm quite happy with the results.

manmal• 1 month ago

Are you not worried about where your data will end up? By now I‘m feeding things to Codex that I‘d rather not have in a leak.

andai4• 1 month ago

Great article. I'm confused how Sonnet did worse than Haiku though. You mention it did find a bunch of other bugs, just not the ones you were looking for?

9 bugs is probably a bit low of a sample size to get a ranking.

That being said the ranking does end up roughly how you'd expect.

Deepseek is Pro, right? Not Flash? I've been using Flash for a lot of smaller tasks and finding it reasonably good. It's good for "interactive" use. Very fast, does small tasks nearly instantly.

It's also decent for investigating large codebases. I wonder if it could do security work too.

SwellJoe12• 1 month ago

I was surprised by Sonnet's performance, as well. And, it's difficult to say any model is really worse or better based on one attempt across nine bugs (several of which have proven to be intractable for all models, thus far). But, in this particular set of problems, Haiku seems to have done a little bit better. But, self-hosted Qwen 3.6 and Gemma 4 also seem to have done better than Sonnet or Haiku, which is surprising. So, there are surely confounding variables here, but I don't know what they are yet. More testing and more analysis of the data will probably reveal it. It may be that using the Anthropic models in the simpler API harness will unleash their power, maybe there are guardrails baked into the Claude Code system prompt that make the small models too conflicted about right and wrong to answer clearly.

DeepSeek was actually the `deepseek-chat` alias in the API (which dynamically chooses the model based on info I don't know), but when I checked the usage, it was all DeepSeek V4 Pro for the benchmark. I later changed DeepSeek to explicitly use Pro for subsequent experiments, so future runs will be explicitly Pro.

I probably will do a test of smaller models, exclusively, at some point. But, I figured DeepSeek V4 Pro is so cheap, especially given their caching effectiveness and cached input pricing, for my own use I'll probably just use DeepSeek V4 Pro when I need a cheap, fast, near-frontier model.

nhod• 1 month ago

“the matchup feels earned” is a current AI-written tell. To whom does it feel earned? To the AI that wrote this article?

I don’t know what it is specifically, but my weak human pattern-matching skills find this kind of language increasingly revolting. I don’t know why it is revolting, per se. It’s just the feeling I get.

Of course, me saying this on HN will get incorporated into GPT-5.6.175 or Claude 4.93 and it will make some version that just moves the revolting frontier elsewhere…

rglover• 1 month ago

I think it's because it's using storytelling-like language to describe reality.

"Harry finally had control of the broom. Draco was dead in his sights. The matchup feels earned."

JamesKaranja• 1 month ago

It's because they assume you know what precision is in regards to this comparison. Normal people don't use such words.

jodacola• 1 month ago

Curious for folks who have made the switch I’m considering: if I swapped Claude Code to DeepSeek API pricing, would I get more bang for my buck compared to the $100 Max plan I’m using now?

I only hit the 5 hour limit every few days and the weekly limit a day or two before it resets at the most aggressive. I wouldn’t expect my usage to increase dramatically, other than not being stopped by limits.

I’m still apprehensive about shipping all my stuff off to a lab under an adversarial government (to the US), so not just looking at this from a pure cost basis, but my question is from the cost lens at the moment.

willsmith72• 1 month ago

also curious. On the claude code $200 plan, get close to weekly limits but don't usually hit it. to me just about any small reduction in performance would not be acceptable, the cost of redirecting and getting stuck during long runs without me are too big (like when I tried gemini cli for a few days).

if it's 99.9% comparable performance for less money I'm interested, but I'm skeptical it's there

slopinthebag2• 1 month ago

I used ~16,000,000 input tokens yesterday on v4 pro, ~15,000,000 were cache hits, and I spent $0.47. Output tokens were negligible. However that's with Zed's harness, I'm not sure what you would get with Claude Code.

It's maybe not quite as knowledgeable as the most expensive American models and maybe makes more mistakes (just a feeling based off of vibes, don't take my word for it), so you need to constrain its scope more. That suits my workflow, half the time I have it generate code in the chat window and then write it myself, and I'm mostly using it at the level of generating function bodies and stuff, not entire features. Although it is writing a lot of SwiftUI without me really knowing the language and doing a fine job as far as I can tell (which isn't much admittedly).

One benefit I don't see talked about is it's speed - it's really quick, doesn't spend too much time reasoning even on "max", and the flash model is pretty dang good too. This lets me get into "flow state" when I'm writing code, compared to my experiences with Codex and Opus which would take minutes to complete even basic tasks and kind of ruined my focus.

It's so cheap though, you could download a different harness (Crush, OpenCode, Pi etc) and load $5 in credits and test it for yourself.

nerdsniper• 1 month ago

Much more bang per dollar, yes. Somewhat less bang per hour.

As usual, different models get stuck on different things. I run DeepSeek v4 API for most of my Cursor experimentation / poking around / proof of concept stuff, but I trust it less than OpenAI/Claude for writing production code. Sometimes DeepSeek is great for debugging, planning, etc. Sometimes it gets stuck or outputs low quality. That's true of OpenAI and Anthropic models as well though.

Overall, DeepSeek seems serviceable but a rung below Opus 4.8 and GPT 5.5. I run them all on maximum thinking settings.

sidrag22• 1 month ago

I've found myself liking opencode for workflows because i can plug GPT models into it, so i tossed 5$ at deepseek api and just toggle back and forth what my opencode.jsonc file is running model wise for my agents. I havent tried anything crazy yet with it, but its nailed all the tasks i felt were overall too simple to waste gpt usage on.

Hardest stuff i threw at it... i did like a set of 3 each for claude/gpt/ds, it was all pretty steady across all providers. I think claude won but it could have just been it rng'd into the 3 easier tasks, they are all similar tasks but not identical, these aren't like benchmark tasks just a steady flow of annoying html/json/regex type stuff. Almost always they need a second pass regardless of what model i throw at it, just to tighten up some loose ends, and it fit right into what my current expectation was of gpt 5.5 and opus 4.6.

0xbadcafebee• 1 month ago

Depends on what you mean by 'bang for buck'. The open weights aren't better than openai/claude. But they are much cheaper and the limits are much higher, so you get more work out of it for less money. Every subscription provider out there provides better money-per-limit value than Anthropic (other than GitHub, who are by far the most embarrassingly overpriced and limited provider). (https://codeberg.org/mutablecc/calculate-ai-cost/src/branch/...)

I’m still apprehensive about shipping all my stuff off to a lab under an adversarial government (to the US)

Do you mean you don't want to use the models created by a non-US lab? In that case, yes you're stuck with US models, but there's a half dozen big labs in the US. If you meant just where your inference is done, there are providers in 12 different countries through OpenRouter, including the US. Several subscription providers host in multiple countries. There's a lot of choices.

CJefferson• 1 month ago

My advice -- give it a try. Chuck $5 into deepseek.com , and use this config (put it in a shell script, run ' . ./deepseek-claude.sh ', then just run claude as normal.

    export ANTHROPIC_BASE_URL=https://api.deepseek.com/anthropic
    export ANTHROPIC_AUTH_TOKEN= *** PUT YOUR DEEPSEEK KEY HERE ***
    export ANTHROPIC_MODEL=deepseek-v4-pro
    export ANTHROPIC_DEFAULT_OPUS_MODEL=deepseek-v4-pro
    export ANTHROPIC_DEFAULT_SONNET_MODEL=deepseek-v4-pro
    export ANTHROPIC_DEFAULT_HAIKU_MODEL=deepseek-v4-flash
    export CLAUDE_CODE_SUBAGENT_MODEL=deepseek-v4-flash
    export CLAUDE_CODE_EFFORT_LEVEL=max

I started by using it for some bigger reading jobs, particularly when I was near limit. Honestly, it's not quite as good, but it's much cheaper, and means I can carry on working. I also find sometimes it's good to ask claude and deepseek to consider code, how to polish, it see what they both say.

SubiculumCode7• 1 month ago

Yeah, the discounted deepseek inference is subsidized by the CCP for a reason, and it's one that might well come back to bite.

benterix• 1 month ago

Well, many people don't have very warm feelings for American LLM providers so they don't care. (Which matters because, at least anecdotally, they do care when buying a new car.)

no-name-here• 1 month ago

deepseek inference is subsidized by the CCP

What is that claim based on?

sourcecodeplz3• 1 month ago

There is no evidence it is subsidized. Actually, there is evidence that (1) electricity is cheap in China & (2) deepseek is a very efficient model.

scrollop2• 1 month ago

I'd recommend carefully looking at a few benchmarks (even though generally relying on benchmarks is problematic)

https://artificialanalysis.ai/evaluations/omniscience

Esp check the Hallucination rate for Deepseek - it's not good.

overfeed• 1 month ago

Esp check the Hallucination rate for Deepseek - it's not good.

For strongly-typed coding tasks - and I imagine other tasks that have cheap validity checks: agentic harnesses and thinking tokens are an effective foil against hallucinations, at the expense of time. If a model hallucinates an API, compilation will fail and the error fed back into the machine so it can try again, in a two-steps-forward-one-step-back dance that is unreasonably effective. Given the price delta, it is often more cost effective to let the weaker model spiral towards a solution with many "Oh, wait..." turns

reacharavindh• 1 month ago

I’m using Claude with a $100/month subscription. I’m playing around with using Opus as the Architect, Sonnet as the implementer/engineer and Deepseek-pro as the deep reviewer, and tester. It’s been quite good as I expected. If my usage pattern holds up, I would downgrade my subscription to the $20/month one and toss more money to Deepseek.

Repo reference here: https://github.com/aravindhsampath/agentic-template

efromvt• 1 month ago

Deepseek cost/performance is incredible. That said, I still feel like for agentic coding we haven't plateaued (I slightly prefer GPT 5.5 to Claude for complex stuff, to be honest), and so the extra price is absolutely worth it to push you over the 'impossible' to 'feasible' bar on complex tasks. Once you're in a domain that Deepseek can handle though that requires volume, I would almost always default to it now.

For evals in particular (tuning workflows that agents are using), effectively not having to worry about price is an incredible multiplier - getting statistical significant signal is not cheap otherwise.

twotwotwo• 1 month ago

If you worry about sending your data off for inference, Fireworks is one of the companies serving open models with solid performance and compliance/zero data retention sorted out. OpenCode supports them and many others. Cursor uses them. They don't have the super-cheap cache reads deal that DeepSeek's own endpoint does, but are still well below Anthropic API rates. (Though crucially you're not paying API rates now!)

DeepSeek and Xiaomi's deals on cache reads go with their models' latest gens making caching cheaper (using less space for KVs). No open-model inference provider has decided to match the pricing. I'm sure that says something about how inference pricing works, but not completely sure what.

Agree with others that top open models aren't on the frontier, and I would expect differences doing big-picture planning or anywhere you're only giving broad brushstrokes and looking for a lot to be guessed. But they do seem fine at coding from a a concrete plan! No experience in huge codebases because I only use them outside work, but they seem good enough about gathering info before they dive in that I'd expect them to grep around as they need.

An annoying caveat: individual subscription plans, used heavily, are much cheaper than the API -- see https://she-llac.com/claude-limits -- which complicates any argument about cost. I still think open models are worth playing with. They're one of the things that let us treat this as a technology rather than just as the product offerings of one of a few companies.

electroglyph• 1 month ago

deepseek 4 pro is insanely good for the price

scrollop2• 1 month ago

https://artificialanalysis.ai/evaluations/omniscience

morpheos137• 1 month ago

Yes Deepseek V4 is as good or better than western sota models in my experience for practical coding given an appropriate harness. cost per solution is certainly cheaper.

jewel• 1 month ago

Interesting. Can you elaborate on which harness you've tried it with? I'd love to switch to deepseek for my personal use.

Also, which SOTA western models are you comparing it with? Just to give more flavor.

freakynit6• 1 month ago

My personal observation (using a mix of opencode and pi harness):

1. DS4Pro: around opus 4.5

2. DS4Flash: around sonnet 4

3. Mimo v2.5 pro: between opus 4.5 and opus 4.6.

4. minimax M3: around opus 4.6

All of these are very close in terms of quality and pricing. For anything that is not specifically related to coding, DS4Flash has become ny de-factor model. It just works... super fast, tool calling is perfect, and the price is unbeatable. Caching is out of the world. Im now regularly hitting 90%+.

slopinthebag2• 1 month ago

I'm exclusively using Deepseek at this point and I really like it. It's not as good for vibe coding but I don't really do that so it works for me. I've spent only a couple bucks this month on it and I really like how it fits into my workflow. I have zero usage anxiety unlike when I was using subscription plans.

LinkWangder• 1 month ago

This evaluation is objective. Both models have their own strengths.

BoiledCabbage4• 1 month ago

What is this nonsense?

An AI generated article about single ai run test which in theory had many components and the AI judge declared deepseek "won"?

How many runs were there on each test to account for some temperature variance? Only one.

Did deepseek write better code? Did GPT's code have bugs when doing the regex? The AI "news" article doesn't actually say that. It says that grok thought that GPT's approach could have bugs so it declared deep seek the winner.

This is absolute worthless methodology. And barely measurable methodology - nothing more than a prompt. No definition of what the scoring approach actually is. No definition of what "precision" actually means in this context. This is absolutely worthless and has no business being in the site, forget about on the front page.

So why is it's on the front page? Because it aligns with the current "feels" of the community that deepseek will get better and it shows "bad things" about the en vogue to dislike closed models.

I happen to agree with both of the views, but this site is utterly worthless.

If you want HN to be astro-turfed to the max, just up vote content like this without any critical reading of the.

I mean the past 6 months of "here is my chat gpt blog post of how to use a coding agent" are 1000x better than this "news article".

Seriously the amount of respect I've lost recently for the HN community is incredible. A bit harsh, but very true.

Maybe it's generational thing, maybe it's due to the state of politics, maybe it's a side effect of me getting older, but recently online has turned into nothing but people explicitly (or implicitly) writing about their "team". Comments on this post are nothing but people who clearly see themselves as being on "team deepseek" or "team open models" or some similar variant writing posts in support even though this is probably one of the worst "articles" to make it to the front page on ages.

It clearly doesn't matter. It supports something on their "team" so they support it via comments.

If kills any form of intellectual discussion. It's all just "this is my team".

sourcecodeplz3• 1 month ago

Have you even used deepseek pro/flash? Yes, it is astroturfed to the maxx. There is a reason for that. The performance/price ratio beats anything available today.

raincole• 1 month ago

You misused the term 'astroturfed.' If the performance/price is that good than it'll be spreaded by word of mouth and no need to astroturfed to the death.

... and I believe which is happening. I've been advocating for DeepSeek V4 Pro and no one paid me. It's almost too good to be true.

BoiledCabbage4• 1 month ago

"Don't you understand? I'm on team deepseek! It doesn't matter what's written about it. Heck it doesn't even matter if it's all lies - it supports my team and here's why I love my team."

amazingamazing• 1 month ago

How is deepseek so cheap? Cheap electricity? Subsidies?

freakynit6• 1 month ago

They actually explained this a few days back (can't seem to find the link right now). But, the core explanation part was it's architecture.

1. MoE (nothing new here, but, this helps a lot)

2. Compressed Attention Mechanisms (this is their core innovation) - this dramatically reduces the Key-Value (KV) cache requirements for longer contexts

Another thing that helps is significantly lower energy costs in China.

Another point from my own guess: they are running (some percentage) the inference on their own home-grown AI inference chips.

chvid2• 1 month ago

That is a very good question. It is open source / open weight - yet none of the third party providers, that also host Deepsek, seem to be able to match Deepseek itself on price.

My guess is that they do aggressive caching / some proprietary optimizations in their hosting setup that they haven't published. Maybe also running at loss to gain market share.

And judging from latency / network performance, I don't think what you access, when you access deepseek.com from Europe, is hosted in China.

dchftcs• 1 month ago

It's clear to me they are subsidizing inference in exchange for market share, and doing it at this scale makes the most sense if their target is getting more user data. Note that this sort of pricing isn't far off from the equivalent token-based pricing of ChatGPT or Claude subscription plans, which are more clearly subsidized by the user's data.

orbital-decay2• 1 month ago

Their models are organized around inference efficiency from the start, it's what they're focusing on. Also they come from HFT and are good at low-level optimization. For v3, they've been literally reverse engineering Nvidia GPUs for undocumented behavior that helped against memory bottlenecks, writing file systems for efficient model serving, and doing a ton of low-level grunt work in the times where everyone else just relied on torch. Being compute-constrained helped as well - necessity is the mother of invention.

pingou2• 1 month ago

But what is preventing their competitors, who have many more employees, who are also very talented, to do the same?

Every little improvement would save them billions, so it's hard to imagine they aren't pouring a lot of resources into that already.

unliftedq• 1 month ago

I'm tired of big news in this way - a small set of tests to declare one model is better than another, can they really consistently reproduce the result? And there's basically no disclosure: nothing other people can really hand on to verify the tests/judgement by themself.

The best valuable part of DeepSeek V4 pro is its low price, I don't expect have much better performance than GPT-5.5, even it's just the performance like gpt-5.4, it's still a good model.

sourcecodeplz3• 1 month ago

"I don't expect have much better performance than GPT-5.5 ..."

Expectations are not always reality. Give the model a try. I just stuck with flash tbh, didn't even use pro. I do webdev in PHP.

not_a_bot_4sho• 1 month ago

As I read this, looks like a single run per task. I'd be interested to see best out of N like 5 or 10 to start.

Stitch42232• 1 month ago

It’s four poorly constructed arbitrary experiments which say very little about the competency of either model.

The article reads like thin, auto-generated ai clickbait for nerd sniping or shilling a model.

Consider the lead:

DeepSeek V4 Pro wins this head-to-head by being more exact where it matters: following instructions, matching schemas, and solving edge cases cleanly. GPT-5.5 Pro is still strong, but it gave away points with avoidable deviations.

“where it matters”, “cleanly”, “is still strong”, and vague references instead of telling 3 out of 4 tests Deepseek yielded more concise results.

1 star.

jampekka• 1 month ago

(Three out of) four experiments is anecdotal for sure, but the result meshes with more established instruction following benchmarking (although DeepSeek V4 pro does not top these): https://artificialanalysis.ai/evaluations/ifbench

I found the writing clear and quite even handed. The lead is a bit salesy, but leads typically are. Knee-jerk dismissals based on vibes that something is LLM generated are quite low-effort.

zozbot2345• 1 month ago

It's picking strange tasks that don't really play to GPT-Pro's strengths (that model is roughly comparable to Mythos, intended for very hard reasoning and research-level problems) and then completely ignoring quite a few cases where GPT-Pro actually got some things more correct than DeepSeek did. The auto-AI ranking is just not reliable for this stuff.

monooso3• 1 month ago

I think you've misunderstood the purpose of a lead (sic).

Per Merriam-Webster [^1], a lede is:

the introductory section of a news story that is intended to entice the reader to read the full story

(Emphasis mine)

You may prefer more matter-of-fact phrasing, of course, but criticising a lede for attempting to achieve its goal is unjustified.

[^1]: https://www.merriam-webster.com/dictionary/lede

hypfer• 1 month ago

I think the criticism is less about whether the lede is good at achieving its goal and more about whether that goal is honorable in the first place.

So dismissing it on technicalities is for sure clever but also obvious and lame.

The Letter/spirit thing eventually got boring. Please find better material

kzrdude2• 1 month ago

A 'lede' is just an intentionally differentiated spelling of 'lead'; the origin of the word is just lead. Collins dictionary defines lede: a variant spelling of lead

Stitch42232• 1 month ago

It’s the hardest part of an article if you ask me.

Filling it with slop constructs signals the reader no effort was made writing the article. So no effort should be put into reading it.

The rest of the article is equally flimsy. Great clickbait title, perhaps that is even harder than writing a lede.

I am not a native speaker :)

karlmedley• 1 month ago

I agree, I'd rather not see AI-generated articles about AI on HN unless they're really good.

an0malous2• 1 month ago

poorly constructed arbitrary experiments which say very little about the competency of either model.

No one ever says this about the “pelican on a bicycle” metric

redsocksfan45• 1 month ago

Simon's pelican is in fact routinely criticised for exactly that.

infecto• 1 month ago

I am willing to guess it is but gets downvoted or similar. Simon is a bit of a cult of personality on HN for better or worse.

irthomasthomas2• 1 month ago

Actually, simonw has started saying that after qwen 27B beat Opus 4.7

https://news.ycombinator.com/item?id=48446348

root-parent• 1 month ago

In the car business there is only one or two car models that are the best ideal choice, but many subpar companies and models, are still selling for many reasons.

It shows DeepSeek is competitive, if not better sometimes, than GPT 5.5. Also shows there is no moat. As such it is a highly significant signal.

unusualmonkey• 1 month ago

I agree that there may be a lot of variation between models that leads to different use cases, at least today. But I’m not sure the car analogy works.

An X5 is not simply “inferior” to a CR-V, or vice versa. A Camry is not “inferior” to an F-150, or vice versa. They are optimized for different buyers, budgets, constraints, and use cases.

That may actually be the better analogy for AI models: there probably is not one universal “best” model. There are models that are better or worse for particular tasks, price points, latency requirements, deployment constraints, privacy needs, etc.

mrgblr• 1 month ago

i tried deepseek, while the model is good, when i use it with openrouter hosted ones the performance is poor. sometimes it takes 2x-3x the time it takes for openai or anthropic equivalent model, making it unusable. what is the performance others are seeing, which providers you use (i cant use china hosted models).

justinram112• 1 month ago

That's about what we've seen as well (even directly from deepseek themselves).

We've been using it for async "heartbeat" processing and sms replies, but it's just too slow for live chat replies (which is a shame, as I'd really love to use it there).

Very capable model, but also very slow.

inhumantsar2• 1 month ago

have you tried their flash model? pro was too slow for me too but I've found flash to be more than capable and it's faster than Gpt-5.5 at medium.

fc417fc8024• 1 month ago

That isn't what the charts on OpenRouter appear to show but they only seem to go back 1 week (unless I missed something). It should be less than 2 seconds to first token and anywhere from 15 to 50 tps depending on the provider. Admittedly 15 is a bit slow but most look to be closer to 30 or 40 which at least personally I think is fine.

https://openrouter.ai/deepseek/deepseek-v4-pro/performance

ryanmerket2• 1 month ago

it took me awhile to find a reliable vendor, but they are def out there.

psadauskas• 1 month ago

I was using Claude until they banned Opencode, and now use GPT at my day job. I've been using Deepseek through Opencode Go on the $10/mo plan, and I honestly can't really tell much difference. Its just as capable, and makes the same kinds of dumb mistakes and the other two have been making since March. For the price, I'm more than happy with it.

joystick_0x0• 1 month ago

I am not sure what I am doing wrong then. I am using claude the last 7 months and from time to time try other models like deepseek, kimi etc. Nothing can come even close to it. Claude is almost evrytime (99.99%) one shot.

OtomotO2• 1 month ago

You're obviously not doing anything wrong if it works for you.

It worked for me too, for months, when I was working on trivial web projects.

Around February of this year it got lobotomized and I quit my subscription end of march.

I am not going back.

skerit• 1 month ago

Same here. Claude isn't perfect. It still makes a lot of mistakes. But whenever I try GPT-5.5 it's ten times worse, and Claude just has to clean up GPT's mess.

InsideOutSanta• 1 month ago

In my experience, there is a very specific use case of one-shotting complex, long tasks with relatively vague or incomplete descriptions where Opus does substantially better than all other models I've tried, including GPT 5.5, GLM 5.1 and DS4. It seems to be better at inferring unstated requirements and creating a complete, working, reasonably well-designed solution.

However, that's probably not how most professional developers use LLMs. I tend to give well-specified, more constrained tasks, and for those, I find that Opus performs worse than other models precisely because it tends to infer unstated requirements and do things I didn't want it to do. In this situation, GPT 5.5 works better for me because it only and precisely does what I ask it to.

selfawareMammal• 1 month ago

I have both subscriptions and I definitely feel gpt is better and more consistent, but when I run out of limits I don't miss it too much

miroljub• 1 month ago

That's the whole point. The tool you have vs. the expensive tools you don't have because they're too expensive.

I don't feel like paying 100 times the price for a 1-5% better tool.

sankaritan• 1 month ago

It's interesting. 95% of time you don't need the extra 5% rigor that frontier models provide to you compared to the 10-100x cheaper Chinese equivalents.

The remaining 5% of time you get a big boost for your high-reasoning problem solving needs and evade a lot of pain. Now, I just need to be able to predict accurately when I need this extra 5% and when not :)

powerapple• 1 month ago

the extra 5% time you will need to help AI with multiple turns and information it needed. These 5% time reasoning rarely is enough to finish the task. i.e. 5% time AI is just not enough to complete the task without a lot help.

yogthos• 1 month ago

I find the trick I use is to get the model to come up with a phased plan, and review it. If I spot anything that seems dumb, I give direction on the way it should be done. And once you finalize that, the model can run through the steps fairly reliably. As long as you're intentionally making all the big decisions, things tend to work out well.

rurban• 1 month ago

Precision yes, but depth of thinking not. I can use DeepSeek V4 Pro 90% of my time, but for very tricky problems I have to use GPT or Claude models. Maybe 2x per month.

wg0• 1 month ago

Of course it does. Even Deepseek v4 Flash with high easily competes with Claude Opus 4.7 for fraction of price.

SubiculumCode7• 1 month ago

Flagged for low quality.

bob1029• 1 month ago

These tests are looking increasingly like a waste of time.

The "intelligence" is clearly there now. Trying to measure it seems pointless. I can't shop for hammers at the hardware store and sort by the quality of finished products they would produce. That is clearly an insane ask, but that's approximately what is being pushed for with these models now.

Domain specificity (harness & environment) is where the magic happens next. I intentionally use a slightly less powerful model to help reveal weakness in how I've exposed the domain to the model. Having capability reserves available dramatically increases confidence around a project like this. If the customer starts to complain about some edges, I can crank them up to gpt5.5 for target scenarios. If I'm already on 5.5 there's nowhere else to go. I'm up against the wall.

gcgbarbosa• 1 month ago

"the intelligence is clearly there"

I wonder if I am using the same models as everyone else. To me, LLMs still give good answers 80% of the time, but 20% it fails in such a miserable way that makes it obvious that the "intelligence" is not there.

21asdffdsa12• 1 month ago

It really depends on the field you are in and the tasks you set and how much of it was in the training set? A webdeveloper will find it succeeding in all taks - while some c++ exotic physics simulation developer will find it lacking.

The "works for me" is telling more about the field of the LLM reviewer, then the LLM.

coldtea2• 1 month ago

It might be extra demand for rigor that's not equally applied to humans. One could argue that other coders in our teams, or even ourselves, often fail in "a miserable way", say about 20% of the time. But we block this out, or consider it "regular functioning", or just a one-off based on something we got wrong, "just a try" we redo, etc.

But when an LLM does it on an area we know, we notice and suddenly it's too much.

hodgehog11• 1 month ago

I get about the same success rate with my problems (scientific computing usually), but they're often _much_ easier to check than to write, so an 80% success rate becomes game-changing.

scotty792• 1 month ago

GPT-5.5, 100% so far for all of my problems that actually have an anwser.

weird-eye-issue• 1 month ago

In my experience of hiring and managing people, I would have been very happy if they gave good answers or produced good results 80% of the time.

kzrdude2• 1 month ago

That's a better score than I'd give my own thinking.

dannyw• 1 month ago

Can I ask what your task and application is? A ~20% failure rate sounds atypical. If you’re slightly hyperbolic and mean something like 2-5%, yeah that’s a property of LLMs; but also heavily affected by how you prompt and how you constrain the task.

An auditing/QA step (whether a grading checklist, verification, etc) can get you further. Likewise for a planning step.

alemanek2• 1 month ago

After adding an adversarial review gate to implementation plans and code I saw large uptick in quality. I use Opus 4.8 as plan writer and orchestrator. For adversarial reviewer I use GPT 5.5.

I still find things to tweak and fix up but the amount dropped pretty dramatically. As always I am responsible for what I ship so I review and test everything of course. I still think we are a ways away from fully automated software forge but what is currently possible is pretty cool.

digitaltrees• 1 month ago

I agree. I feel like sonnet 4.6 is sufficient for almost everything. Beyond that level it feels like the orchestration is more important.

That being said the models still surprise me with a broad range of hallucinations, lack of epistemology or common sense or inability to follow instructions on a daily basis.

Today it was trying to get opus 4.8 to just follow a simple architectural pattern for controllers in a rails app. It was pulling teeth out of a shark.

scotty792• 1 month ago

I can't shop for hammers at the hardware store and sort by the quality of finished products they would produce.

What? You can and you should. That's exactly what product tests are enabling you to do. If you need a glue, you want to look at someone who tried to glue some things with few glues so you know what to roughly expect form which specific glue.

ricardobayes• 1 month ago

Why would it be a "waste of time"?

We are just getting into the nitty-gritty of LLM benchmarking - to be fair they still need to go a long way still IMO. But it's incredibly exciting that a local run LLM is capable of producing similar results as a SOTA model.

mdp2021• 1 month ago

clearly there

Already the fact that we could have to ask "there where", the fact that we have met clearly unintelligent bots, creates a requirement about defining where it (intelligence) is and investigating what put it there, to get the warranties that intelligence will be met consistently, structurally, and not casually, apparently.

Casual use, casual tool; mission critical use, certified tool.

dominotw• 1 month ago

Domain specificity (harness & environment) is where the magic happens next.

not really. it happens in training and RL. your harness is not going to override what it has been trained to do.

sure harness is useful if you are trying to build crud websites if model is trained on stamping out crud websites. But thats just a waste of time remxing things better.

stalinfan• 1 month ago

Deepseek: Mao did nothing wrong!

Grok: Hitler did nothing wrong!

ChatGPT: Altman did nothing wrong!

nutifafa• 1 month ago

yes, I sure it does, that's just how models behave, today one is excellent tomorrow another is. this why being model agnostic is crucial in getting the best value out of the ecosystem.

woadwarrior01• 1 month ago

DeepSeek V4 Pro with reasonix is surprisingly cheap and good enough for most coding tasks. Also, it's different enough from GPT 5.5 and Opus 4.8, that it sometimes finds issues that the other two cannot. I think it's worth having in one's toolkit.

shenberg• 1 month ago

Seems 100% AI generated and automated, the judge also seems suspect - in the first one it's actually GPT-5.5 pro which has the correct email RE: the deepseek one will match a@b.com1 as "a@b.com" while 5.5 will correctly require a word boundary at the end of the email. I quit after this. No test-cases = useless judge.

beernet• 1 month ago

No pelican? I don't believe it.

More seriously, LLM eval is totally broken judging by the related articles on HN.

SkitterKherpi• 1 month ago

There are definitely things Deepseek beats GPT on - cost effectiveness, lack of reluctance on some tasks, but from using most models I doubt it actually outperforms in a useful way in quality in a meaningful way.

KronisLV• 1 month ago

Personal experience: for overall software development, DeepSeek V4 Pro (Max reasoning) is pretty fast and generally okay - it does fuck up regularly though and I’d compare it with maybe Sonnet.

It’s also quite affordable, at my current usage the DeepSeek tokens cost approx. the same as my Anthropic Max 100 USD subscription, though that’s also because DeepSeek generally needs more tokens.

I’d say I have fairly moderate usage, the DeepSeek dashboard shows around 100 million tokens per day, but almost all of it cache. Without cache it’d be like 1.5 million in and 0.5 million out most days, sometimes double, other times half.

Used it with Claude Code for a while, though I have to admit that using OpenCode with DeepSeek just sparks joy. Tone wise, it’s also a bit less obnoxious than Opus sometimes, though the flip side is that it’s wrong more often and sometimes just does dumb shit when it comes to code.

poly2it• 1 month ago

Why was this posted to HN? What an utter waste of time. Someone's slopwriter writes a slop article about which slopper slops the most slopulicious slop. Comments agree it's a bogus "study". We need some gate on AI-written articles. It's so weird that AI-written comments are not permitted, while the front page can be occupied by stuff like this.

hit8run• 1 month ago

This benchmark draws a very different picture having GPT5.5 on the very top with 70% and DeepSeek at 8%

https://deepswe.datacurve.ai

zozbot2345• 1 month ago

DeepSWE has been heavily criticized though. https://github.com/datacurve-ai/deep-swe/issues/21 Putting GPT 5.5 on top is the obviously correct part, but everything else about it makes very little sense.

FpUser• 1 month ago

What engine beats the other by some 10% does not matter al that much I think. With every increasing use and reasonable quality the price and availability is all that matters

amunozo2• 1 month ago

DeepSeek V4 Pro is wonderful and ridiculously cheap, but we are sleeping on MiMo V2.5 Pro, which have the same price (and lower cached price), it's multimodal and it's higher up in most benchmarks. Same thing for MiMo V2.5 vs DeepSeek V4 Flash.

smartbit2• 1 month ago

MiMo V2.5 Pro ... lower cached price

At the moment of writing https://news.ycombinator.com/item?id=48343690 MiMo V2.5 Pro had a lower cache hit ratio. From the article:

OSS models, depending on who you use them from, make a huge difference, mostly due to cache-hit rates.

  Model                   Cheapest effectiveInputPrice (Provider)  
  MiMo-V2.5-Pro           0.3720 (Xiaomi) 
  DeepSeek V4 Pro (Max)   0.0560 (DeepSeek)

amunozo2• 1 month ago

Could it be that it changed recently, or am I missing something? Both prices are the same https://openrouter.ai/compare/xiaomi/mimo-v2.5-pro/deepseek/...

EDIT: okay I misread it, does this mean that DeepSeek reuses a higher percentage of tokens at cache price that MiMo, am I right?

esafak2• 1 month ago

How would you rate mimo against dsv4 pro? What do you work on?

atemerev• 1 month ago

I do not use models who cannot answer me what happened on Tiananmen Square on June 4th, 1989.

stainablesteel• 1 month ago

imo there are two major kinds of models

there are models you can speak with, that respond to what you say

and there are models that just make lists, that list everything, include weird formats and add asteriks everywhere.

deepseek, to me, will always be the latter, and i can't stand it, you can't ask it a coherent question and get a coherent response.

wolttam• 1 month ago

I rarely work on anything that demands better than DSv4 Flash, let alone pro.

If I can describe the problem and its solution well enough, Flash just does it.

If I can’t (or am feeling too lazy to) describe the problem well enough, and can only describe the desired outcome, then I’ve noticed models like GPT 5.5 being clearly better at working out a solid solution on their own.

There are some clear differences in the capabilities of the models, but it’s also clear that smaller open weight models are good enough to be a huge help for most tasks.

pshirshov• 1 month ago

I'm a bit tired reading such claims and looking at benchmarks. E.g. minimax m3 looks to bo something opus-level and it sorta is... until it doom-loops or produces garbled output.

csbrooks• 1 month ago

Shouldn't it be necessary to run the tests multiple times on each model, since the results aren't deterministic?

smhanov• 1 month ago

I've been using deepseek v4 for cost/performance reasons. I feel it is generally not as good as some others, but in the end, you can make any model work by giving it the right acceptance criteria. Use detailed specs, use tests, and give it the power to iterate until it works. One-shot is a poor metric for performance.

Frost1x• 1 month ago

I’m not sure all models will converge on your acceptance criteria. I’ve done quite a bit of varied agent based modeling and scientific modeling in that domain and just because you have some grounding to check against and some ideas on how you might go about getting to a convergence point doesn’t mean you’ll actually converge, you can absolutely get stuck in the information space iterating away, never finding your desired solutions.

It helps but you often have to step in the failure cases and guide them or forcibly fix certain paths to get a solution.

kittikitti• 1 month ago

I'm not surprised that GPT-5.5 Pro is less precise. I find that companies such as OpenAI have a profit motive that is evident in their models. This profit motive de-incentivizes precision because they can charge more if more tokens are consumed/produced.

DeepSeek V4 Pro beats GPT-5.5 Pro on precision

Related Stories