vvrm

For FAANG engineers this will likely mean moving to Vancouver, Zurich or Singapore with their job, salary, rsus and taxes.

This story plays out so often that here should be a law about it: Supply lags demand, prices soar, everybody hears about it, everybody pours in, supply surges, demand normalizes, supply overshoots demand, prices collapse. Already happened to software engineering, data science. Keeps happening to hardware production every few years. Sounds like AI research is headed that way too.

This one’s a bit worse than meta’s usual sins: enabling political ads to manipulate the user is one thing. But enabling naughty people to generate naughty pictures of innocent bystanders because they happened to be in the field of view of an idiot talking to their glasses is a whole different level. Would be surprised if this is legal.

The British introduced electricity, railways, capitalism and a thousand other things we take for granted behind those saffron-tinted glasses.

Some people love being a slave, but that is not a requirement for having electricity, railways or capitalism. Just look at any other country.

naive, childish, saffron-tinted glasses.

A brain with less self-righteousness has more space for common sense and logic.

CV Raman won the Nobel prize in science, Tagore won the Nobel prize in literature, Ramanujan etc, the names are numerous.

So golden age of India was when the country with a seventh of the world's population won 2 nobel prizes over 5 decades ?

The British rule also was the largest and most stable unification of India till the modern times. After 1850s there were almost no pockets of military resistance against the British rule.

Mughal and Gupta empires lasted over 3 centuries, Mauryan empire a little under 1.5 centuries. By comparison, east india company rule lasted a century and the British crown's rule less than that. So again completely incorrect.

The third golden age which no one wants to admit (left or right) is the British Golden age.

There's your hint: if people on both sides of the aisle don't "want to admit" something, maybe it doesn't make sense. Not to mention a slap in the face of billions of Indians.

The British age declined with WW1 and WW2, and ended with Indian independence.

Thank god for that decline, otherwise Indian taxpayers would have been funding Brexit and the crumbling British economy right now.

My oversimplified summary has been

This is not a summary, it's a lazy opinion backed by little research.

Useful idea, but needs a lot more work IMO e.g.

- Looks like there's just 7 abstracts right now.

- Most of the abstracts are written by the author of the paper, so might not be as unbiased as an actual "community-written" abstract.

- There's no stated guidelines for the "community-written" abstract e.g. should it be less biased than the original abstract, should be shorter than the original, should it be more accessible to a less AI crowd or all of the above.

- There's no way to upvote/downvote some abstracts e.g. the "attention is all you need" paper has two abstracts and one of them is clearly worse than the other.

AI rescued Nvidia when nobody was buying their shovels for digging crypto gold. If it wasn’t for ChatGPT, Nvidia story would have been very different right now. He is probably just hoping that this dream never ends. Otherwise there’s no way to justify the current valuation in the long term.

DPO is pretty much strictly better than RLHF + PPO

Out of genuine curiosity, do you have any pointers/evidence to support this. I know that some of the industry leading research labs haven't switched over to DPO yet, in spite of the fact that DPO is significantly faster than RLHF. It might just be organizational inertia, but I do not know. I would be very happy if simpler alternatives like DPO were as good as RLHF or better, but I haven't seen that proof yet.

Another problem with the title: the article is about DPO, which doesn’t do reinforcement learning. So not RLHF. I guess RLHF has more of a name recognition than DPO.

Because a salesman’s skills complements those of a researcher. Salesman sells what the researcher built and brings in money to keep the lights on. Researcher gets to do what they love without having to worry about the real world. That’s a much sweeter deal than a micromanaging PI.

The fine-grained results look like:

- 1444x faster for single character prefixes

- 252x faster for two character prefixes

- 55x faster for three character prefixes

- ~20x faster for 4 and 5 character prefixes

- <= 5x faster for longer prefixes

I used to work on a production auto-complete system operating at over 100k peak QPS. For prefixes of length one and two we would not even bother hitting the server, just from a quality perspective, not because of latency/throughput considerations. Btw, up until 3 characters, you could store everything in an in-memory hash map. 20x speedup on length 4 and 5 prefixes is still very impressive, but not quite 1000x speedup either.

I have been training a natural intelligence model for 3 years now and she still doesn’t get nuance. Things are either good or bad in her book: nothing in between. My plan is to let her train with binary good/bad labels till the age of 5 and then start smoothing the labels after that. Wonder if that works for your AI.

In this day and age a motivated 18+ year old adult has more than enough tools to actively pursue education outside a college setting while having a day job that pays. And this learning does not need to end after four years. So for most students college is just a transaction: a bachelor’s degree in exchange for some money and four years of their life. No point in passively receiving education (i.e. being indoctrinated) at college, when there are better learning tools outside.

Brought to you by the same creative minds that conceived CUDA (which means trash in Hindi).

another victim of the rapid pace of improvement in smartphones

It's not just the pace of improvement, but also the marketing spin. I find the strengths of smartphone camera and ILCs pretty complementary. Smartphone cameras work pretty well outdoors where there is enough light. DSLR and mirroless are hard to beat indoors in low light conditions. Coincidentally it is also easier to find your ILC indoors at home when you need it, rather than lugging it around on a hike. When we didn't have kids, we used to spend more time outdoors and so most of our memorable pictures are from a phone. Now that we have restless young kids and are spending more time indoors, almost all of the memorable photos are from a mirrorless camera. But the marketing spin makes it seem like ILCs are completely redundant.

The problem with over-hiring is not that there's a lot of people doing nothing. The problem is that there's a lot of people who need to do something to write in their annual reviews. Since there's only so many useful things to do, people start doing non-useful things e.g. over-engineering simple things. These non-useful things then slow down the development of useful things by

  1. Creating needless technical complexity.
  2. Creating needless compatibility issues.
  3. Distracting the focus.

Hiring 2X more people is not going to result in 2X more value creation, but laying off half the staff is not going to 2X the productivity either.

Thanks for laying out the plan. I was trying to understand the cost of each of these steps below and started wondering about the following:

rough steps:

1. collect a very large dataset, see: https://www.lesswrong.com/posts/6Fpvch8RR29qLEWNH/chinchilla... . scrape, de-duplicate, clean, wrangle. this is a lot of work regardless of $.

Pile seemed quite clean and manageable to me (I was able to preprocess it ~8 hours for a simple task on consumer grade hardware). Is Pile clean and rich enough for LLM training too ?

2. get on a call with the sales teams of major cloud providers to procure a few thousands GPUs and enter into too long contracts.

It seems like the standard instructGPT model itself is based on a 1 billion param GPT model. Wouldn't that fit on a 24GB RTX 3090 ? Might take longer, maybe not enough opportunity for hyper-parameter search, but still possible right ? Or is hyper-parameter search on a thousand machines in parallel the real magic sauce here ?

3. "pretrain" a GPT. one common way to do this atm is to create your own exotic fork of MegatronLM+DeepSpeed. go through training hell, learn all about every possible NCCL error message, see the OPT logbook as good reference: https://github.com/facebookresearch/metaseq/blob/main/projec...

Sounds like a good opportunity to learn. No pain, no gain :-)

4. follow the 3-step recipe of https://openai.com/blog/chatgpt/ to finetune the model to be an actual assistant instead of just "document completor", which otherwise happily e.g. responds to questions with more questions. Also e.g. see OPT-IML https://arxiv.org/abs/2212.12017 , or BLOOMZ https://arxiv.org/abs/2211.01786 to get a sense of the work involved here.

Maybe somebody would open source the equivalent datasets for this soon ? Otherwise the data collection seems prohibitively expensive for somebody trying to do this for fun: contract expert annotators, train them, annotate/reannotate for months ?

LLMs for the most part learn P(someone on internet said A | internet specific context B) given ginormous amounts data. There’s no other type of A, B with that much training data at hand.

+1 if your input and output are both sequences, FSTs can be your best friend…Unless you prefer the company of the flaky cool kids from deep learning (CTC, RNN-T, …)

+1 Awesome introduction to computational geometry. Couldn’t help but mention this fun fact: Voronoi diagrams were used to fight the 1854 Cholera outbreak in London

https://plus.maths.org/content/uncovering-cause-cholera

+1 for including the sample size in the title. Good way of distinguishing from click-bait.

Microsoft spent almost the same on nuance: https://news.ycombinator.com/item?id=26778780