SWE-1.7 Reach Near GPT 5.5 and Opus Intelligence

by mekpro • 14 days ago

272 140 14 days ago

A company whose first demo was completely fraudulent announces that its model beats GPT-5.5, on its own benchmark? I’m gonna wait a little before I trust this.

This whole company seems to optimize for raising money and impressing VCs. Lying about their products, ignoring consumer market to target enterprise, bragging about how they work their employees like slaves, and writing these posts full of intimidating technical jargon...

giancarlostoro• 14 days ago

Would love to see these companies use benchmarks done by third parties.

anthonypasq2• 14 days ago

they are right there? it shows swe-bench multilingual and terminal bench

w4yai• 14 days ago

What happened ?

SubiculumCode2• 14 days ago

Link for this?

jeffnv• 14 days ago

https://news.ycombinator.com/item?id=40008109

oa335• 14 days ago

"A company whose first demo was completely fraudulent"

Could you expand on this?

achandra03• 14 days ago

To be fair it does seem like most AI startups are now like this (particularly when it comes to constantly mentioning how hard they work and ignoring consumer markets).

parineum• 14 days ago

it does seem like most AI startups are now like this

Remember when AGI was going to replace all jobs in 6 months? It's always been like that.

sigbottle• 14 days ago

I highly respect many people at cognition but yeah that's put a sour taste in my mouth.

I want to work in the AI space on actual AI research, at any part of the stack. Even if I'm developing training infra - as long as people are advancing knowledge of what intelligence could be.

But it seems like either it's big labs or grifters, that's it, and even the big labs, at least publicly, seem very grifty at times. Not like I have the technical chops probably, but still.

alansaber2• 14 days ago

This is inevitable when the primary incentive is to raise aggressively. Overall I dont find cognition blogs that jargony, there are definitely worse offenders

throwaw12• 14 days ago

Open source for the win!

Imagine how far community might have pushed if 2 past versions of 'morally superior' Anthropic and 'completely Open AI' open sourced their models for the community to build on top of them

spott• 14 days ago

Is this open source? I can't find a link to download the weights.

UncleOxidant3• 14 days ago

It's based on an open weight model (Kimi 2.7) so shouldn't it also be open weight?

akshaydeshraj2• 14 days ago

Not open source. Also, not available beyond it's own harness.

llmslave8• 14 days ago

These models are never as good, the benchmarks dont tell the full story

SubiculumCode2• 14 days ago

Funny, the cheerleading at HN for leading Chinese models, but a non Chinese lab (building on top of a Chinese model) gets dissed here.

llmslave8• 14 days ago

all the open source models are a waste of time relative to the bleeding edge from openai/anthropic

sosodev• 14 days ago

It's almost as if HN users aren't all the same.

mirekrusin2• 14 days ago

It's simple: close weights = not welcome.

haritha1313• 14 days ago

The reality is most people building their own models and providing that alongside SOTA ones don't really care about how great these models are. They just prove that 'hey we are smart enough to build our own models so you can trust us instead of going with a single provider like Claude via Claude Code', also a cheap alternative for cost sensitive/free users - at least this was the case for Windsurf, not sure if Devin Desktop still has that tier. They just need to hillclimb the benchmarks and show something reasonable enough there.

spate1412• 14 days ago

Benchmarks are just vibes with error bars... wake me up when it survives a week on a real codebase without hallucinating a package that doesn't exist.

achierius• 14 days ago

I've always had mixed feelings about Cognition. Obviously they have some very, very smart people working there (I even know a few), and they do make real products. But at the same time, they've made suspicious marketing claims more than once and even been caught making outright fabricated ones; and while they certainly seem to have shaped up from that, I still find their claims to be in a sort of grey area where they seem to avoid unfavorable comparisons and lean on their own benchmarks. Certainly when I've tried their models they have not been nearly as useful as comparable versions of Claude, GLM, etc. -- though I haven't had a chance to try SWE-1.7 yet.

nibbleyou2• 14 days ago

Unrelated: what's the point of "*equal contribution"? Why would someone specify this

edot• 14 days ago

Because papers are often referred to by the first author’s name, and often the first author is the primary researcher and therefore deserves the extra credit. When two or more primary authors are equally involved, they’ll often do a random ordering but annotate this so that no one thinks one did more than the others.

nibbleyou2• 14 days ago

Interesting. Thank you

tancop• 13 days ago

some journals and colleges actually have a policy to always use random order to help fight the "senior researcher gets all the credit" culture in academia.

theres a lot of cases where a prof forced their students to put them first even if they had an advisor role, or even credit someone for zero real work because they threatened to block submission and prevent the students from getting their degree.

yousif_1231233• 14 days ago

We need more models that optimize for coding and that can be cheaper than frontier models, like what SWE 1.7 and composer 2.5 are trying to do. I don't think there's an effort to make something GLM-5.2 level but focused only on coding.

UncleOxidant3• 14 days ago

Qwen was doing something like this with their coder models. But alas, they seem not to be releasing those anymore. Last one was Qwen3-coder-next.

yousif_1231233• 14 days ago

Its crazy that OpenAI and Anthropic themselves aren't doing that. No attempts at reducing inference cost for code as far as I know from them.

gsibble2• 14 days ago

I use this model. It's pretty good but not Opus 4.8 or Fable levels obviously. I'm really hoping we get more models like it (and better) soon. I run it locally and it's great that way.

jstummbillig2• 14 days ago

Defining what "coding" means now, and how quickly we fall off the capability cliff seems increasingly important.

Today my "coding" sessions often enough begin with real life problems, where I discuss domain or inter-domain things, ranging from business, economics, psychology, etc. Being able to do all of that with one model is something I am willing to pay a premium for.

Of course not having to pay the premium, because the routing is smart or whatever, would be great. I just don't want to have to think about it.

andriy_koval2• 14 days ago

Today my "coding" sessions often enough begin with real life problems

intuition is that your sessions consists of 10% of domain related reasoning, and 90% of code plumbing. Those 90% could be moved to cheap and efficient specialized and focused model.

tonyhart72• 14 days ago

but that is not model problem

most agentic coding app can use powerful model for planning/reasoning then use "budget" model to do ground work

gsibble2• 14 days ago

And make it run locally!

euleriancon• 14 days ago

This isn't as easy as it sounds. Every ML model is struggling to balance between generalization and test performance.

Taking a good model like GLM5.2 and just fine tuning it on coding can decrease real world performance due to mechanics like catastrophic forgetting. There is also other interesting behaviors were training on a broad training set can improve coding performance because there is positive transfer.

There is 100% an effort to make solid coding focused models, but it is very hard to do that without including capabilities across a broad set of adjacent tasks.

alansaber2• 14 days ago

As you say- LLMs are fundamentally good because of their generalism. Distillation, ablation, ft all tend to be hacky and in some way hurt the model

hedgehog2• 14 days ago

Heads up to anyone else curious, I installed the Devin CLI and SWE-1.7 is not currently available there.

_jx• 13 days ago

It is now, at least for me. Latest Devin CLI version: v3000.1.27 I don't think SWE is geo-restricted.

hedgehog2• 13 days ago

Looks like now it shows up but it's not available to try on a free plan.

taf2• 14 days ago

Not finding anything about this while searching huggingface: https://huggingface.co/search/full-text?q=SWE-1.7 i assume this is another closed source model?

mirekrusin2• 14 days ago

Open weight models should have GPL-like license where it says if you train model on it, it needs to be open weight as well.

joecot• 14 days ago

Yes, and not only that but you can't even access it via API, you can only use it in Devin (formerly Windsurf).

I'm an OpenCode user, but I'll fall back to Claude Code if I want to use Opus end to end for something, given my company has a subscription. But I'm not using yet another tool and subscription for a model that isn't even winning.

pants2• 14 days ago

Kinda funny that their "cost-vs-performance" chart looks the same as the one for Composer 2.5[1], except that it includes Composer 2.5 at a completely different spot.

What are the chances that CursorBench ranks Cursor's model highest, and Cognition's bench ranks Cognition's model highest? Both are to be RL'd from Kimi as a base model, BTW.

I'd posit that it's not deliberate deception, but for both companies their training data and benchmarks come from the same dataset (Devin/Cursor interaction logs) so they naturally overfit.

1. https://cursor.com/blog/composer-2-5

culi• 14 days ago

I think it's also telling that they left out the usual hallmarks of the Pareto distribution: GLM 5.2, Qwen 3.7, Minimax M3, and Mimo 2.5

https://arena.ai/leaderboard/code/webdev/pareto

petesergeant3• 14 days ago

they left out ... GLM 5.2

They did not.

bluelightning2k• 14 days ago

Good observation.

I actually started typing the same point that the chances are actually high because of train/eval overlap then realised you answered your own question with that same observation.

It is interesting though!

Perhaps in some way this means we should decide which eval set aligns best with our taste?

Back to the blog post. This is an excellent write up of an excellent technical achievement.

I have a lot of respect for the Cognition/Devin (always "Windsurf" to me) and Cursor teams.

I found it interesting - but justified - that they referred to themselves as a foundation lab rather than a dev tools company.

swyx• 14 days ago

agent lab, not foundation lab

oofbey• 14 days ago

Agreed on the likely mechanism. I'm not sure "overfitting" is even the right description. These things are of course absurdly complicated, and evaluating their quality down to a single number involves a lot of judgement and trade-offs. I think it's more "you get what you measure" which is true in human organizations too. Define a KPI and people work hard to make it go up, even if it's not quite right or has bad side-effects.

meowface• 14 days ago

At this point I barely put any value in any of the benchmarks. I just use the models for coding (and related things like software product design/planning/ideation/etc.) tasks and judge them subjectively, and also see how others judge them subjectively on HN and Twitter.

fallinditch• 14 days ago

I'm looking forward to trying this out. I've been using SWE 1.6 quite a lot for grunt work alongside Opus for higher level planning and tricky stuff - a good combo.

As a (former) Windsurf user I'm pretty happy with the progress of the Cognition/Devin ecosystem after they took over Windsurf, now known as Devin Desktop.

ryandvm• 14 days ago

Okay, let's give software engineers a break for a bit and focus on obsoleting other high-linguistic context occupations.

silvertaza• 14 days ago

Like diplomacy? heh

cromka• 14 days ago

Some of these models could be particularly small, depending on the market they'd target...

FergusArgyll• 14 days ago

I know you didn't mean this but have you ever seen Meta's model which plays (the game) Diplomacy? really cool

https://ai.meta.com/research/cicero/

znsnjanwnwwn• 14 days ago

And to do that you’ll need development so until we’re all out of a job they’ll keep pushing. Once automating is automated it’s done.

blauditore• 14 days ago

Any day now... Just a bit more space in the context window, trust me bro.

yieldcrv• 14 days ago

software and agentic workflows will obsolete those things

RL environments building on top of each other will get these models there

needs people doing software development lifecycles to figure it out and implement

spate1412• 14 days ago

Feels like they discovers that if you build your own benchmark, you can win it

londons_explore• 14 days ago

Pretty sure most benchmarks are being gamed by people training on the test set deliberately or accidentally anyway.

ahk-dev• 14 days ago

The benchmark debate is fair, but I think the more interesting signal is how quickly coding models are becoming a category of their own rather than just smaller frontier models. More specialization, more competition on cost, and probably a lot more benchmark gaming along the way :)

petesergeant3• 14 days ago

I think it's a bit odd to show the API prices for competitors when that's not how most people pay for them. I do like that it's provisioned by Cerebras though. I think I'd have leant towards focusing on the TPS.

akshaydeshraj2• 14 days ago

Would have been worth a consideration if it could have been used beyond it's own harness. Unfortunately, doesn't seem to be the case.

https://x.com/theodormarcu/status/2074896486047834380

esafak• 14 days ago

Ugh, that changes everything. If I wanted an arranged marriage I could go back to Claude Code.

2001zhaozhao2• 14 days ago

Harness-wrapper tools that support multiple harnesses and allow sharing workspace features (skills, slash commands, etc.) between them will be meta.

RussianCow7• 14 days ago

Ironically, Devin Desktop is one of those tools. It supports any harness that supports ACP (which is most of them)—you can use Claude Code, Codex, OpenCode, etc from the Devin Desktop UI.

I'm currently experimenting with OpenSpec[0] as the "framework" and using different subscriptions for different parts of the spec-driven process: Opus via Claude Code for exploration, Devin SWE for building, and GLM 5.2 via the Z.ai Coding Plan for verification. I don't love having to mix and match harnesses, but in practice it's barely more effort than switching models.

[0] https://openspec.dev/

mrinterweb• 14 days ago

I really don't want harness lock-in. I am trying to decouple myself from Claude Code now. I love the model of OpenRouter and being able to switch models at will let's your harness focus on your personal tooling and you can easily switch to the flavor of the month LLM with a single slash command instead of rewiring your entire workflow to use a harness to use a model.

I like Cerabras, but I really wish they would make more of their hosted models generally available.

Mitchem• 14 days ago

While I am skeptical of the results here, I am very excited for this new trend of making models faster. Running capable models at 1k TPS is more valuable for me than running better models at 30 TPS. I can only imagine the trend continues to move from "let's only make models smarter" to just incremental intelligence gains but with step improvements in speed.

lnenad4• 14 days ago

Why? I'm personally on the opposite end. Less babysitting/higher quality means more time goes back to me/the user. 1000tps of bad code means you have to keep validating the output and circling back.

anthonypasq2• 14 days ago

id rather iterate multiple times than wait 15 minutes to notice it made a mistake.

unshavedyak2• 14 days ago

So i agree with you, but there's no SOTA model that i don't have to babysit. I'm not going to just throw a large pile of code in there unreviewed, and so what i want is faster iteration on code in logical, reviewable chunks. Ie just like i'd normally write myself; small, logical commits.

Faster iteration means i mentally checkout less and am more involved with the code being created.

My hope is that in the far far future, we can get LLMs so fast that i can work in my IDE like normal and the LLM will just be an extension of autocomplete. I can state a goal, rough out functions, code, etc, and it'll just work around me like a very fast pair programmer / autocomplete.

The chat interface is an intermediate step that frankly i hate. The faster it is the less i wait.

Now for vibe-slop i'm making on the side, yea i don't care about speed. But that's not something i'm employed to do or anything i truly care about. It's a different workflow entirely.

aunty_helen• 14 days ago

High tps is good for deeper agent thinking loops and openclaw etc. I was running cerebus recently doing some data heavy tasks, it managed to crash the server I was submitting posts to. 6 hour task down to ~1hr

holoduke• 14 days ago

Indeed. For me opus 4.8 is good enough. If only it would be 100 times faster. You could run it in self verification loops much much faster. It sometimes takes 15 minutes for me to complete a simple task. For example configuring AWS agentcore and deploying an agent on it. Takes forever with Claude with constant issues it tries to solve.

anentropic3• 14 days ago

https://devin.ai/pricing

Apparently 'free' on the $20/mo Devin plan (presumably within some quota still)

and that is "via Cerebras at 1000 TPS" according to the announcement

I live on Opus 4.8 High and their benchmark scores SWE-1.7 slightly higher ... if at all realistic that sounds like a great deal ... too good to be true?

yousif_1231233• 14 days ago

The 1000 TPS shows for me as "SWE 1.7 Lightning" and took 14% of my daily quota in one prompt on the $20 plan.

But the normal speed one seems to be free or with very generous limits.

RussianCow7• 14 days ago

The "Lightning" (Cerebras) variant isn't free, only the regular one, which runs closer to 50 TPS in my experience with SWE 1.6.

anentropic3• 14 days ago

Oh, the free plan said "Slow" so I thought maybe the others had the fast version :)

settled• 14 days ago

I used SWE-1.5 and 1.6 when it was Windsurf (before Devin Desktop), it's not that bad (grunt work, tests, can actually plan and implement some medium level stuff) but you get a much much better value and better models (GPT-5.4^) going with a Codex subscription (plus you get resets).

That company truly subsidized its user base to the extreme before, the $15/mo subscription was the best value on Earth paired with weekly deals reducing credits for premium models. Now it's barely any messages for paid models, completely watered down.

chris_st• 14 days ago

FWIW, Cognition has all the Sonnet/Opus/Fable models, and all the GPT ones, as well as GLM, Kimi, and Gemini.

2001zhaozhao2• 14 days ago

Wait Devin has a CLI?

Time to support it in my agent IDE just like Cursor's...

hudo• 14 days ago

How do i use it from opencode/openrouter?!

RussianCow7• 14 days ago

You cannot; you must use either their Devin Desktop app or the Devin CLI.

kgeist• 14 days ago

On artificialanalysis.ai, Kimi 2.7 Code is way worse than GLM 5.2 at everything (general intelligence, coding, agentic tasks).

But here, both Kimi 2.7 and its derivative SWE-1.7 are ahead of GLM 5.2. This tells me the benchmarks they use are cherry-picked.

petesergeant3• 14 days ago

This tells me the benchmarks they use are cherry-picked.

Which benchmarks would you have chosen instead, and why?

cogman10• 14 days ago

It honestly seems like there's not a great way to currently benchmark AI.

The ideal way to run these benchmarks would be to give a 3rd party the model to run in an isolated environment so the prompts don't make their way back to the AI engineers.

That seems doable for open weight models, but not for private models.

bearjaws• 14 days ago

Likely benchmaxxed. You see it in Qwen and other smaller models all the time.

qingcharles• 14 days ago

Composer 2.5 is worse than both; I use it all day for simple stuff. It's Kimi 2.6 in a new outfit.

godzillabrennus• 14 days ago

Cognition... oh what a ride... We were customers when they acquired Windsurf, stopped offering customer support, raised prices, dismantled the brand, and raised prices again. We are not customers anymore. Benchmarks are not the only thing to worry about when you are using models.

Bnjoroge2• 14 days ago

I’ve unfortunately had to temper my excitement with Cognition’s models/products given the amount of unwarranted hype they created with Devin on first release, but hopefully this is good.

inglor• 14 days ago

I work with Devin daily (as well as Claude and a few others) and I can attest it's not a cheap product but it's a good one and it saves me a bunch of time.

Bnjoroge2• 13 days ago

Yea I cant convince myself to pay api pricing unless my employer is doing it

amarant• 14 days ago

Very thankful someone is doing this work! I suspect it will be thankless work for a while: we're not far enough into diminishing gains territory for anything other than the absolute best being worth considering for most people, but I reckon we will be pretty soon. If a year from now swe 2.0 or whatever reaches fable 5 parity for a fraction of the cost, that'll be very attractive indeed!

messia2• 14 days ago

And yet when I use swe it feels like massive shit

messia2• 14 days ago

if you are still using these products in 2026 you are really a shit engineer

modeless• 14 days ago

What is the actual per token price? The benchmarks look similar to Grok 4.5 also released today and priced at $2/M input tokens and $6/M output tokens.

tonyhart72• 14 days ago

and what is actual intelligence per dollar benchmark ???? its useless comparing token/dollar while some model inherently generate more thinking output and cost more despite lower cost

RussianCow7• 14 days ago

The regular one (not the fast variant) is free but slow. The "Lightning" variant (which uses Cerebras and gets supposedly 1000 TPS) costs $12.50/M output, $2.5/M input, $1/M cached input. So it's quite a bit more expensive than SWE 1.6.

bobtheborg• 14 days ago

I like to use SWE-1.6 for quick help with git. For instance:

review the top stash and tell me what's in it (grouped appropriately)

1.6 does this fine nearly instantly.

1.7 tried for 17s before I killed it

luciana1u• 14 days ago

SWE-1.7: the benchmark where AI agents finally learned to write code that passes tests, but only because they also wrote the tests

retinaros• 14 days ago

it is obvious at this stage that most of the gains are in distillation post training and having good RL simulations. the moat of the private labs are just their capability of stealing open research and data and locking the few bit they innovate on the top of it.

It will work until they IPO.

SWE-1.7 Reach Near GPT 5.5 and Opus Intelligence

Related Stories