Model card and evaluations for Claude models [pdf]

https://www-files.anthropic.com/production/images/Model-Card-Claude-2.pdf

by famouswaffles • 3 years ago

60 25 3 years ago

www-files.anthropic.com

Besides coding, not much better than v1.3 by the looks of it. and of course, behind GPT-4 on most things but much better than 3.5

version_five2• 3 years ago

While we're still in the honeymoon phase with llms, I think it's hard to "rank" them, at least ones with different goals. Is Anthropic just trying to make a GPT-4 competitor? I'll be interested once we have a real sense of the strengths and weaknesses of different approaches and choices to see how the main contenders differ. I don't expect it will just boil down to one being better than another.

The28thDuck• 3 years ago

I imagine models will be chosen over another for very specific use cases, at least until we can ad-hoc generate LLMs specific to what we want.

thatguymike• 3 years ago

200k context, 2023 data cutoff… nice!

ramesh313• 3 years ago

Like all models, Claude can be jailbroken, and our work to make Claude more helpful, harmless, and honest is ongoing.

Is "harmless" a good metric to be judging an AI by? I find all of this "evil AI" doom and gloom stuff pathetic. Who's out there actually training models to do whatever they want without the nonsensical Care Bear ethical limitations? And moreso, do you think your model will keep up with the people who are doing that if you aren't? We are only hamstringing ourselves with this stuff.

It's literally the equivalent of inventing the first digital calculator, and programming it not to display the results of 80084 + 1.

version_five2• 3 years ago

There's a lot of vocal hangers on in the AI community that hijack the discussion towards "harms" and whatnot. Not to say it's not worth thinking about but it's run amok to the point that models are neutered (contrained within what a very specific political alignment calls ethics) before they're even used for anything, because someone speculates that somehow somewhere someone could do something bad with it. It's a dumb approach and hopefully as AI tools become more widely available and the activists move on, we can focus on building good AI and then putting guardrails in place for specific applications as required, rather than this "only the enlightened can be trusted with it" approach. Imagine if this is what had happened with general purpose computing. There are those that wish it had and see this as an opportunity to claw back some of their power.

flangola72• 3 years ago

It's not "doom and gloom" whatsoever, I have yet to see an internally coherent and persuasive argument that AI will not be the most powerful technology ever created. 95% of the "arguments" are little more than dismissive emotional outbursts and bold claims made with no evidence, much like the claims that airplanes and computers will never take off of yesteryear.

The single largest problem of the tech industry all along is the pervasive and perverse "shoot first, ask questions never" attitude, especially for ethics. It's a breath of fresh air and a small relief to for once see industry leaders take the stance that potential consequences should not only be considered, but assessed and mitigated before ever breaking ground. When you're building a bridge you don't deploy a single construction truck until you have a very comprehensive plan that thoroughly theorizes, assesses, and mitigates the possible harms and failure modes. A bridge can harm far fewer people than a large internet entity, which is yet still less harmful than resourceful cognitive agents.

blueboo• 3 years ago

You’re conflating the doomers, with their apocalyptic forecasts, with the more prosaic concerns of measuring and controlling the behaviour of the models outputs.

It’s reasonable to measure and control tendencies to counsel people into hurting themselves or others. It’s also reasonable to measure and control the tendency to present harmful stereotypes as fact.

Maybe these mitigations go too far for your taste. The power of these tools and the diversity of a mass audience indicate a thoughtful process.

cubefox2• 3 years ago

To present the claim that certain things are "harmful stereotypes" as fact is a popular tactic of activist censors. Not even articles in scientific journals are safe when they arrive at the "wrong" results. Optimizing language models for political correctness instead of truth, whatever it may be, teaches them lying and deception.

KaoruAoiShiho2• 3 years ago

Who do I have to email or beg on twitter to get approved for Claude api? Can anyone help me out please? I'll pay a couple hundred bucks for someone to hook me up.

bkrausz• 3 years ago

Send an email to support@anthropic.com and mention my username...I'll see what I can do :)

KaoruAoiShiho2• 3 years ago

Oh my, sending email immediately thanks.

exo-pla-net• 3 years ago

Does this perform better than GPT-4 on any metric?

doctoboggan• 3 years ago

Context length is much higher than GPT-4

ramesh313• 3 years ago

Does this perform better than GPT-4 on any metric?

Nothing will. You need a dozen H100s just to run inference for GPT-4 [0]. The point is that smaller models can still be very useful.

[0] https://www.semianalysis.com/p/gpt-4-architecture-infrastruc...

SparkyMcUnicorn• 3 years ago

I think it's better at retaining memory with longer contexts, creative writing, and (when desired) longer responses.

As an example for creative writing: 2 is a slight degradation from 1.3 for this specific prompt, but Claude 2 is still better than any OpenAI model.

Write a love letter. It needs to be written as if it was from Linus Torvalds, rudeness and all.

jackson1372• 3 years ago

GRE writing: 5 for Claude vs 4 for GPT-4

Bar exam: 76.5 for Claude vs 75.7 for GPT-4

RoddaWallPro• 3 years ago

This is a nitpick, but their "Claude 2 on 200k Context Data" graph doesn't actually extend to 200k, only 100k. Would be curious to know if that _is_ actually the graph for 200k with the wrong axis, or if its the 100k graph.

malisper• 3 years ago

This is a nitpick, but their "Claude 2 on 200k Context Data" graph doesn't actually extend to 200k, only 100k

It does extend to 200k. The chart is logarithmic. You can see the little 2 in the bottom right.

kibibu• 3 years ago

Is there a word to describe these documents written by industry groups that give the impression of academia without needing to go through the discomfort of peer review? These OpenAI, Microsoft, DeepMind and Anthropic reports are formatted like academic papers, use academic conventions,and are (often) uploaded to an academic preprint server, but do not have to justify their claims in any meaningful way to external reviewers.

sergiotapia• 3 years ago

If we are able to use the 200k context window via API, that's a good use case for us to switch from OpenAI.

We need a large context to answer organization wide questions across a lot of our customer's data.

Model card and evaluations for Claude models [pdf]

Related Stories