jafitc

"thanks to lobbying". if you use "for" it sounds like you are adressing the OP and he did the lobbying that you don't like.

bigger change here might not be model quality, but debuggability.

once you hide the reasoning, remove the knobs, and let the model choose its own effort, it gets much harder to tell whether the model got worse or just got harder to inspect.

that’s a real shift. less tool, more black box.

subprime mortgages sprinkled on top of prime ones, treated as prime ones. because they were printing money. subprime code sprinkled on the backbone of software we use everyday. because they are printing code. reckoning

that's a great story!

I think you should consider trimming that file.

Exclude movies with very low number of rating or potentially very low scores too.

The long tail reduction would be significant

"People are really bad at understanding just how big LLM's actually are. I think this is partly why they belittle them as 'just' next-word predictors"

https://nitter.net/jam3scampbell/status/1748200331215835561

Deepinfra Mixtral is $0.27 / M tokens as per their website

Important to note that this model excels in reasoning capabilities.

But it was on purpose not trained on the big “web crawled” datasets to not learn how to build bombs etc, or be naughty.

So it is the “smartest thinking” model in weight class or even comparable to higher param models, but it is not knowledgeable about the world and trivia as much.

This might change in the future but it is the current state.

Do you think the ISIS is bound by the words “non-commercial” in a license file when they have the source anyway?

It was available even before this, all they changed is that law abiding citizens can put apps in the App Store and charge money for it.

(More importantly law abiding companies can build on and fine tune it in hopes of profit).

This "vibe" check that it's even better than GPT-4 Turbo is not what its Elo rating shows on the Chatbot Arena based on not 1 but thousands of user votes. GPT-4 (Turbo) is in a league of its own still.

This is based on users choosing the better from 2 models at a time, and calculating an ELO rating from who-beats-who.

BYOT - bring your own tests style.

Gives a better picture of real-world performance and more robust against contamination.

They collected over 6000 and 1500 votes for Mixtral-8x7B and Gemini Pro.

While ELO ratings are widely used to rank performance in Chess or among sports teams, here's a disclaimer by the makers of the leaderboard:

---

Please note Arena is a "live eval" and pretty much a sampling process to estimate models capability.

That's why we show the confidence intervals through bootstrapping. Statistically, these models (e.g., GPT-3.5, Mixtral, Gemini Pro) are very close and only looking at their ranking can be misleading.

https://twitter.com/lmsysorg/status/1735729398672716114

https://twitter.com/lmsysorg/status/1735751052287226059

from the announcement tweet: https://twitter.com/rasbt/status/1735293149965062476

---

So, we've been quietly building something new for running AI experiments and deploying models ...

Our Lightning AI Studios let you switch between different machines and GPUs flexibly in the same environment without any setup steps.

Everything can be accessed via your browser and supports

  - VSCode
  - Jupyter Notebook
  - a regular terminal
  - a control pane for multi-node jobs 
  - ... many, many collaborative and extra features

It's basically what I've been using internally as a productivity tool for the last few months to run AI experiments.

(*there's also a demo video in the linked tweet)

---

A persistent GPU cloud environment.

Code online. Code from your local IDE. Prototype. Train. Serve. Multi-node. All from the same place.

No credit card. 6 Free GPU hours/month.

Important note: Bing balanced mode (default) uses GPT 3.5

Only Precise and Creative modes use GPT-4

https://twitter.com/emollick/status/1732495030143549541

Also see:

An Opinionated Guide to Which AI to Use: ChatGPT Anniversary Edition

https://www.oneusefulthing.org/p/an-opinionated-guide-to-whi...

All I can say is it’s really fast

It’ll never be completely gone.

But you’ll need it in less and less everyday scenarios and time goes on

Just like we need to write less and less assembly by hand

OpenAI provides “instruct” version of their models (Not optimized for chat)

Isn’t that the first sentence?

Brain is just neurons and synapses at the end of the day.

The whole universe might just be a stochastic swirl of milk in a shaken up mug of coffee.

Looking at something under a microscope might make you miss its big-picture emergent behaviors.

The fact that the makers of such LLM make a post about it shows that they have incentive to cater to even these kind of use cases

These are not actual tests they used for themselves.

Some third party did these tests first (in article and spread on social) to which the makers of Claude are responding.

I knew it’s a weird test right when I first encountered it.

Interesting that the Claude team felt like it’s worth responding to.

Language can be ambiguous.

But these LLMs were fine tuned on realistic human question and answer pairs to make them user friendly.

I’m pretty sure the average person wouldn’t prefer an LLM whose output is always playing grammar Nazi or semantics tai chi on every word you said.

There has to be a reasonable “error correction” on the receiving end for language to work as a communication channel.

We already know LLMs are good at summarizing.

Question is how good they are are retaining minute details from extremely long context, say 200k tokens.

That’s the frontier Claude and now GPT-4 Turbo are pushing

Interestingly human memory works the other way.

We tend to remember out of place things more often.

E.g. if there was a kid in a pink hat and blue mustache at a suit and tie business party, everybody is going to remember the outlier.

Imagine an assembly that you didn’t make, but was passed down to you by aliens.

Now we have to tinker with it to learn instead of read Textbooks

Link from thread https://dev.to/zvone187/gpt-4-vs-claude-2-context-recall-ana...

My experience matched this as well.

GPT-4 Turbo is more watered down on the details with long context

But also it’s a newer feature for OpenAI, so they might catch up with next version

There are services online that can help you out. Google is your friend.

desperate times, desperate measure...ment practices

the new model is live

Bard currently is GPT-3.5 class model. Of course it's faster than GPT-4. (YMMV on specific examples of course)