After 2 Weeks of Testing, What Do Developers Think About Claude 3?

https://favtutor.com/articles/claude-3-developers-feedback/

by thunderbong • 2 years ago

35 17 2 years ago

I’m not fully decided but I haven’t seen Claude edging out ChatGPT in my tests. Every query I’ve had over the last week or so I’ve sent to both of them (manually writing the question in one, then copying it to send to the other as well, no fancy tricks) and the results have been very similar.

I have used ChatGPT’s responses in some cases and Claude’s in others. I never had a “oh wow, X is way better” case with either of them and sometimes I even sent the same replies to both for a few rounds of back and forth since they made the same mistake/misunderstanding.

However, I will say that ChatGPT’s UI is better than Claude’s. The chat-like interface with the conversation list on the left hand side feels better, the larger chat text input box and wider UI of ChatGPT feels better as well. Claude is very narrow for no good reason. Also the model selector for Claude is annoyingly placed, I’ve accidentally clicked it on multiple occasions. Lastly there is no app for Claude, there is no voice chat (that I’ve seen), nor can you modify the prompt/response which I didn’t think I’d miss as much as I do.

After this month is up I think I’ll stop paying for Claude unless something changes but I’ll continue to double-ask until then.

patrickhogan1• 2 years ago

Agree on Claude UI. Its selling point is, I have a bigger context window. But the chat UI is this tiny little box that doesn’t extend. Small nit which I’m sure they will fix.

xerox13ster• 2 years ago

You could easily make the box expandable with a home rolled browser extension by changing the input field’s properties. You could even ask Claude to write it for you.

joshstrange2• 2 years ago

I could and I might but if Claude 3 Opus and ChatGPT 4 are both pretty much on par with each other in my tests and I prefer ChatGPT's UI (and ChatGPT 5 is probably coming before Claude 4) then I'll probably just keep using ChatGPT.

Also I don't think Claude 3 is going to write me an iOS app to interact with it and do things like voice chat (or rather I'm not going to jump through those hoops even if it could), that's another place where ChatGPT excels.

gorbypark• 2 years ago

My very unscientific test is to ask new models I'm trying to build a basic React Native component using a specific library. "Please make me a react native component, using typescript and react native reanimated that creates an animation of a circle that bounces when it hits the ground. It should loop indefinitely."

Both GPT4-turbo and Claude 3 Opus produce a generally acceptable output. Claude 3 gave me a ball that bounces off the "ground" and "roof", but in a smooth and continuous motion. GPT4-turbo gives me a ball that bounces on the ground then "snaps" back up to the top and does it again. Mistral large gives me garbage output. It imports a bunch of function from react-native-reanimated that don't exist and produces nothing usable.

Technically GPT4-turbo's response is a bit more faithful to what I asked, I suppose. However, Claude 3 Opus animation looks much more "polished" even though I never asked for the ball to bounce off the roof.

fakedang• 2 years ago

In my experience, Mistral Medium seems to give better output than Mistral Large. And they have the audacity to charge 75% of GPT4.

rschoultz• 2 years ago

Developers worldwide have tested it…

Claude does seem to be explicitly blocking people living in GDPR jurisdictions to sign up. I don’t know what to make of that, but this statement from the first paragraph rings false.

ndr_• 2 years ago

Well, there is two Claudes: first, there is claude.ai. Second, there is the Anthropic API. (And there is also AWS Bedrock, but it doesn’t offer Opus). Signing up for the Anthropic API from within the EU works without hassle.

The reason for the geo-block against the EU may also be rooted in the AI act, rather that GDPR. That would explain why the end-user service claude.ai is blocked off, and foundation model access remains available.

To use Claude 3 through Gradio and/or HuggingFace, see https://huggingface.co/spaces/ndurner/claude_chat

antman• 2 years ago

I have tested it in huge contexes and it leads. It is great. Code writing? Don’t know. Refactoring? Miles ahead from GPT4. No laziness, true large context, YMMV

dorkwood2• 2 years ago

Every time I've mentioned having problems with GPT4 I've been told that I mustn't be prompting it correctly. Now suddenly Claude 3 comes out and everyone is admitting GPT4's shortcomings.

fragmede2• 2 years ago

Both things can be true though

dTal• 2 years ago

I was impressed by the IKEA demo until I looked more closely and realized it was complete gibberish. Every single part is number "101339" and it keeps talking about "attaching the hammer-shaped piece"; it gets worse from there. Meanwhile the article breathlessly claims perfect performance. Given the clear lack of even cursory critical analysis I would take the other less verifiable claims with a small mountain of salt.

BatmansMom• 2 years ago

Saw this too. The multiplayer game seemed impressive though

cheema33• 2 years ago

I have been using Claude 3 since it came out. And I am somewhat surprised to find anybody who claims that GPT-4 is about equal. For software dev tasks, for me, there is no comparison. I always have both tools open and I feed them the same exact question. Claude has been winning nearly 100% of the time.

The only downside for me is that Claude's limit of questions you can ask per day is much more limited than ChatGPT-4's. You exhaust your limit much faster if you feed it a lot of tokens/data. Which I often do.

After 2 Weeks of Testing, What Do Developers Think About Claude 3?

Related Stories