rlanday

Claude Code also costs $20/month for the lowest paid tier.

SOTA models are already capable of outperforming any human on earth in a dizzying array of ways, especially when you consider scale.

So why are so many people still employed as e.g. software engineers? People aren’t prompting the models correctly? They’re only asking 10 times instead of 20? They’re holding it wrong?

Yann LeCun is 65. It is extremely unlikely his future research contributions will amount to anything close to what he has already done.

There’s a design problem here if the driver can’t be self-contained in such a way that it’s possible to roll back the kernel to a known good state.

A developer is much more likely to be able to fix their computer and/or restore from a backup than a typical user is. A significant problem is cascading failures, where one bozo installing malware either creates a business problem (e.g. allowing someone to steal a bunch of money) or is able to disable a bunch of other computers on the same network. It is not that common for macOS to be implicated in these sorts of issues. I know people have been saying for a long time that it’s theoretically possible but it really doesn’t seem that common in practice.

I think if someone wants to criticize Microsoft after experiencing their buggy products for 20 years straight, that is not “baseless,” although I accept that taking responsibility for literally anything our products do goes against the core values of our profession.

> The Windows ecosystem typically deployed in corporate PCs or workstations is often insecure, slow, and poorly implemented

Yes, but that's not because of Windows itself

Come on. There’s a reason Windows users all want to install crappy security products: they’ve been routinely having their files encrypted and held for ransom for the last decade.

Yes, the original Parkinson’s Law paper contains an equation:

https://en.wikipedia.org/wiki/Parkinson's_Law > The growth was presented mathematically with the formula x = (2k^m + P)/n, in which k was the number of officials wanting subordinates, m was the hours they spent writing minutes to each other.

(check the original paper for details since this obviously doesn’t explain what n is)

The reason we have 3204 2B allocations is that each of the 2B contains info (latent space dimension).

I think the author is more correct than you are. It is not necessarily the case that we need 3,204 dimensions to represent the information contained in the tokens; in fact, the token embeddings live in a low-dimensional subspace; see footnote 6 here:

https://transformer-circuits.pub/2021/framework/index.html

We performed PCA analysis of token embeddings and unembeddings. For models with large d_model, the spectrum quickly decayed, with the embeddings/unembeddings being concentrated in a relatively small fraction of the overall dimensions. To get a sense for whether they occupied the same or different subspaces, we concatenated the normalized embedding and unembedding matrices and applied PCA. This joint PCA process showed a combination of both "mixed" dimensions and dimensions used only by one; the existence of dimensions which are used by only one might be seen as a kind of upper bound on the extent to which they use the same subspace.

So some of the embedding dimensions are used to encode the input tokens and some are used to pick the output tokens (some are used for both), and everything else is only used in intermediate computations. This suggests that you might be able to improve on the standard transformer architecture by increasing (or increasing and then decreasing) the dimension, rather than using the same embedding dimensionality at each layer.

Secondly, his description of each layer's function as adding information to the original vector misses the mark IMO--it is more like the original input is convolved with the weights of the transformer into the output. I am probably missing the mark a bit here as well.

I think his description is basically correct given how the residual streams work. The output of each sublayer is basically added onto the input. See https://transformer-circuits.pub/2021/framework/index.html

Lastly, his statement that the embedding vector of the final token output needs all the info for the next token is plainly incorrect. The final decoder layer, when predicting the next token, uses all the information from the previous layer's hidden layer, which is the size of the hidden units times the number of tokens so far.

I think the author is correct. Information is only moved between tokens in the attention layers, not in the MLP layers or in the final linear layer before the softmax. You can see how it’s implemented in nanoGPT: https://github.com/karpathy/nanoGPT/blob/f08abb45bd2285627d1...

At training time, probabilities for the next token are computed for each position, so if we feed in a sequence of n tokens, we basically get n training examples, one for each position, but at inference time, we only compute the next token since we’ve already output the preceding ones.

That's some major scapegoating of flight crew, when their airline clearly has not maintained the plane properly and caused engine failure in the first place and airport failed to provide adequate ground control support.

Most disasters result from multiple things going wrong together, hence the importance of addressing them individually.

This is hard to believe given the quality of most software, but at some point, one might expect a decline in product quality to harm revenue and profitability, which would reduce or eliminate the need for headcount growth.

To be fair, I don’t think he’s been responsible for any world wars.

It can be transmitted between family members too. I would hazard a guess that, given the risk factors for hepatitis B include being an IV drug user who shares needles, there’s probably a correlation between the mother and/or the father having hepatitis B and not bringing their kid back to get vaccinated later.

You want a “dumb sorted list” of every video on YouTube?

That’s not good behavior but I think it’s not really the same phenomenon. That’s more of a coordinated action/tragedy of the commons problem where people want to bury their heads in the sand and do nothing because the problem can be mostly ignored for now.

Absolutely not. Quite the contrary. They originally dropped the charges against him in agreement for 16 hours of community service and forfeiting his $10,000 bond. He inflamed racial and political tensions for attempted personal gain, which is quite a serious crime.

https://www.nytimes.com/2019/03/26/arts/television/jussie-sm...

I hated the vaccination thing sooo much. I was already annoyed that the US is becoming a communist country where all the most valuable goods (e.g. vaccines) are given to powerful politicians (members of Congress) instead of letting people in the private sector buy them with their own money. The fact that people started lying to jump the line ahead of all the honest people in addition to the people who would’ve paid for them with their hard-earned money pissed me off royally. It made me feel like a sucker for working and paying taxes. I identify very strongly with the author of this article.

You can move from one float to the next, which is a meaningfull operation on discreet numbers like integers, but not continuous numbers like rational and irrational numbers.

What do you mean by continuous? Obviously if you take a number line and remove either the rational or irrational numbers, you will end up with infinitely many holes.

The thing that makes floating point numbers unique is that, for any given representation, there are actually only finitely many. There’s a largest possible value and a smallest possible value and each number will have gaps on either side of it. I think you meant that the rationals and irrationals are dense (for any two distinct numbers, you can find another number between them), which is also false for floating-point numbers.

Hong Kong has been indisputably part of China since July 1, 1997.

If you’re lucky, the train conductors and the air traffic controllers won’t be on strike at the same time though!

They’re not capable of understanding a brilliant language

Maybe your language isn’t so “brilliant” if some of the smartest programmers in the world still can’t figure it out.

My initial reaction was, “it can’t possibly be as big as the mistake you’re making if you’re using TensorFlow!”

It’s long been possible to decode H.264 using JavaScript (which obviously uses way more power than the native decoder). This is why browsers only restrict audio playback rather than actually autoplaying video.

Gil Scott-Heron is right - but also not. I was reminded of this watching First Man - and I realised that if 1960s USA had not funded a Moonshot, then would it have looked at itself and said "Hey our police force is used to keep 1/5th of our population in poverty through violence and our financial policies do the same. Let's fix that"?

Well, we have 2020s America as a comparison. Whether or not you think we’ve recognized the problem, it certainly hasn’t been solved to the extent that e.g. we actually flew people to the moon and back.

You seem to think that Facebook managed to get 2.74 billion monthly active users by providing a product that people don’t significantly value. 3.21 billion if you also count Messenger, WhatsApp, and Instagram.

Does this difference in pay for remote workers based on city actually exist yet? I feel like a bunch of people are e.g. just telling Google they’re going to work out of Mountain View or New York and counting on never being asked to actually go to the office.

2008 was, again, a demand crash: a lot of people could no longer afford to pay their mortgages and speculators stopped buying properties under the assumption that they would be guaranteed to be able to resell them at a profit after a relatively short period of time.

Worked, past tense. I’m perfectly happy paying New York taxes on the income I earned while actually working out of the New York office.

New York has a rule about “convenience of the employer.” The name is misleading and doesn’t make any sense, but basically if there was no way to do the work in New York (e.g. your company sent you to New Jersey to hook up someone’s internet), it’s not considered New York-sourced income, but otherwise it is.

The issue with the current situation is that in principle I could be doing the work in New York (it doesn’t matter where I do it), but the office is closed and I don’t live in New York. So I have strong reasons for not doing my work in New York, but in principle I could be doing it there.

I guess there’s a further clause that you have to spend at least one day out of the year in New York to be taxed there, which unfortunately I have done (despite no longer traveling there for any work-related purpose).