ajb117

Holy marketing

I get what you're saying, but I think confidence is actually more important than enjoyment.

A lot of people confuse enjoying [x] with enjoying being good at [x]. This is why so many students switch subjects later on in life; when a field suddenly doesn't come naturally to them, they seek to play to their strengths elsewhere. Problems occur when they quit too early, and building confidence early on is important for stopping this.

In my experience, when you think you're bad at something, it's almost impossible to enjoy doing it, which makes preliminary mastery actually the first step to enjoyment and therefore downstream success.

I'm curious about the AI research job market in fields that aren't LLMs and RL. They're all the rage now, but what about researchers in fields like vision or graphs or more fundamental topics like optimization?

Besides the the title being a bit exaggerated, it's been long-acknowledged in the GNN community that graph rewiring often helps learning; see [1, 2] for example. Also, at first skim, I'm surprised there's no discussion on oversmoothing/oversquashing in this paper. It seems like what they're calling "overfitting" on the graph structure could be interpreted as the GNN's node representations saturating/starving because of poor graph topology.

[1] https://arxiv.org/abs/2111.14522 [2] - https://arxiv.org/abs/2210.02997

I don't know much physics, but is there a reason we should be hoping for superconductivity specifically rather than a cheap material that's just a very very good conductor? Also, even if LK-99 were superconductive, that wouldn't imply it would be useful for computing, right? Surely there are materials that are better conductors than silicon but have mechanical and other physical quirks preventing us from using then for chips.

Makes me wish python had a pipe operator like Julia's |> and R's %>%

Besides Nvidia cards usually being faster and having higher memory bandwidth, Ada cards also have FP8 cores. I'm not sure how well Apples' M-series chips handle low/mixed precision tensors, but I wouldn't be surprised if Nvidia cards perform better with them.

Well at least in the U.S. a lot of used 3090s are out for less than $800 on eBay which is a steal for 24GB VRAM. The 4090 is about $1600 and it's still a consumer-level card; gamers are buying it more often than the 4080. For ML workloads, it's actually a pretty good deal considering how expensive Nvidia's higher-end, non-consumer offerings are. Lambda Labs did some benchmarking: https://lambdalabs.com/blog/nvidia-rtx-4090-vs-rtx-3090-deep...

But yeah the M2 MacBooks are incredible for local LLMs for their price. Nvidia doesn't have any consumer-level priced accelerators with that much memory.

I'm pretty sure you can't buy TPUs, but people usually buy GPUs instead. If you're building a personal rig, these days, you can get an Nvidia RTX 3090 for about $720 USD on ebay used, which is pretty cheap for 24GB VRAM. There's also the A6000 with 48GB VRAM but that'll cost about $5000 on Amazon. Of course, there's new cards that are faster with more VRAM like the 4090 and RTX 6000, but they're also more expensive.

Of course, this is all pretty expensive still. If your models are small enough you can get away with even older GPUs with less VRAM like a GTX 1080 Ti. And then of course there's services like Google Collab and vast.ai where you can rent a TPU or GPU in the cloud.

I'd check out Tim Dettmers' guide for buying GPUs: https://timdettmers.com/2023/01/30/which-gpu-for-deep-learni...

I think that's more in line with transfer learning, a variant of fine-tuning. If I'm reading this article correctly, they're fine-tuning the LMs end-to-end.

My guess is that it's because they've already done RLHF on top of the standard next token prediction. In other words, they can't cheaply fine tune ChatGPT without undoing the RLHF objective by training on next token prediction with post-2021 data, and then retraining with RLHF to make sure it still gives good human-like output.

I mention the "undoing RLHF" since it's not uncommon for fine-tuned models to increase in error in the original training objective after being fine-tuned with a different one. I think people saw this happen in BERT.

Also ChatGPT is almost certainly huge.

I honestly don't know how people can stand Gnome/Unity. I've been running Xubuntu LTS with the Plank dock and have had nothing but smooth sailing save Nvidia drivers.