eugenhotaj

This is because everyone is training with synchronous sgd. all gpus need to synchronize on each gradient step so tail latency will kill you.

This is pretty cool. I had the same idea but in zig: https://github.com/EugenHotaj/zig_gpt2

Not fully finished yet, haven't gotten around to implementing bpe encoding/decoding and only some ops use BLAS.

In my experience processes like these rarely work out as intended and usually add layers of bureaucracy for marginal benefit. It’s usually senior engineers or middle managers looking for “org wide impact” so they can get promoted.

This post would sound so dumb if it didn’t come from the almighty pg.

The issue is not that ChatGPT will kill things off, the issue is that ChatGPT 4.0 will kill things off. If you don’t think that’s a real possibility, you’re sleeping.

Some of these are so odd. When Kylie Jenner launched her company she could have sold celery and still made $1B+. It wasn’t because of the effectiveness of her small team.

My experience is exactly the opposite in almost all cases. Most software is much more complicated than it needs to be. Reads like the author is just butthurt at feedback they received about their work.

I’ve likely had it all my life, but really started noticing about two years ago during the pandemic. Now I can’t unhear it. Went to a doctor a couple of times but nothing they tried really helped. Thankfully mine is not too bad and I’m mostly able to ignore it.

The network communication overhead would be way too high to make this useful. At least for current methods of training large models.

Source: trust me bro

Now someone just needs to pipe the output into stable diffusion.

Just take a look at big tech salaries on levels.fyi. Very senior engineers (L8/L9) easily make 10x the salary of entry to mid-level engineers (L3/L4/L5). For a lot of tech companies, L5 is a terminal level meaning most engineers will spend the rest of their careers there. Clearly the market thinks 10x engineers exist.

Let me guess, someone pushed a bad config.

This is neat for toy problems but I don't see it working well for "real" pipelines. The magical DAG creation is going to be super hard to wrap your head around and even worse to debug.

This reminds me of an internal Google tool for doing async programming in Java (ProducerGraph or something). The idea was that you'd just write annotated functions and the framework would handle all the async stuff. Wasted many thousands of engineering hours while giving an even worse experience than manipulating futures directly.

Since the bigger network contains the smaller network, it is perfectly capable of achieving the same performance, so the only reason why this does not happen is that SGD cannot find it.

This is maybe true in the limit of infinite data, but not true in any practical sense, and I don't think it has anything to do with SGD. E.g. polynomial basis functions also have this property, but you can't use an arbitrarily large polynomial order or you'll eventually overfit. You can get a closed-form solution for polynomial regression problems, so no SGD involved.

If your pipeline uses only “classic” ml models, then this won’t make too much sense. It’s mostly applicable to NNs.

I also don’t have any data, but I feel like these types of tactics don’t account for higher order effects of pushing people to more extreme platforms, and in general seem extremely short sighted.

For kernel SVMs, one needs to keep around part of the training data (the support vectors) right? With DNNs, after training, all you need are the model parameters. For very large datasets, keeping around even a small part of your training data may not be feasible.

Furthermore, number of parameters do not (necessarily) grow with the size of the training data, can be reused if you get more data, can be quantized/pruned/etc. There's not really an easy way to do these things with SVMs as far as I understand.

True, but it sounds like you’re just shifting computation from training to inference. And I’m not sure that’s a very good trade off to make, you’re likely to predict on much more data than you trained on (e.g. ranking models at google, fb, etc)

Fair enough, but the number of support vectors for non trivial problems is still pretty large (as I understand but could be wrong), e.g. 20-30% of the dataset. Having to iterate over 30% of say imagenet on each batch of predictions seems unfeasible.

Don’t kernel SVMs need a full pass through the data they were trained on to make predictions? How is that faster?

It's also strange because what they've trained is essentially a recommender system. However, at inference time you're not trying to rank 2 different articles, but the same article with 2 different titles, which I guess you could do, but I'm not sure how well this correlates to your actual ranking on HN.

I guess something more interesting would be to take into account the current front page and see how your article would rank against those, or something along these lines.

In any case, neat idea.

Hi HN!

I’ve been working on the pytorch-generative library in my spare time for a couple of months now as a way to dive deeper into recent work on neural autoregressive generative modeling. The goal of the library is to provide clean, high quality implementations of recent SOTA work as the code for a lot of papers is hard to read and understand. I’m also trying to see what parts can be abstracted and easily reused to make future implementations and experiments easier.

Please take a look! I’m happy to answer any questions and would love any feedback.

I disagree, PAC bounds are usually way too loose to be useful in practice when it comes to NNs. I think part of the reason is that the way we measure model complexity for NNs does not correspond very well with their ability to generalize, as we've seen with recent papers like deep double descent [1].

Anecdotally, I was of the main authors of the AdaNet framework [2] where we used (approximate) Rademacher complexity (closely related to PAC) to bound the complexity of our learned ensemble models. We never got good results using the complexity measure and would basically turn it off for any practical problem we solved.

[1] https://openai.com/blog/deep-double-descent

[2] https://github.com/tensorflow/adanet

Misleading -- you can just tuck in your fu manchu

I'm particularly proud of this meta approach and I am actually thinking this could become huge: the same thing can be done for hyperparameter optimization in machine learning tasks.

There is already a substantial field of Machine Learning/Meta Learning which focuses on exactly this. For example, this paper [1] from NeurIPS 2015 does exactly what you suggest.

[1] https://papers.nips.cc/paper/5872-efficient-and-robust-autom...

I would expect it to (but I haven't thought about it too deeply so I could be extremely wrong). My thinking is as follows:

At the end of the day, all we're doing is maximum likelihood estimation. So we're trying to find model parameters which define a probability distribution where our observed data is the most probable. In the original GPT-2, this observed data is the text from quality outgoing links on Reddit. Since this data is so diverse, there will not really be any special structure that the model can pick up on, besides whatever structure exists in the English language.

However, when we fine-tune on RapGenius, the observed data is now songs. These songs have a certain structure to them such as stanzas, rhyming, etc. In order to maximize the likelihood of this data, the model must learn to model the structure.

Finally, if we further fine-tune on Beatles lyrics, the model is again trying to find parameters which maximize the likelihood of the data. So the model will try to match both the lyrics and the structure of Beatles songs. It's likely that the structure of Beatles songs is pretty similar to the other songs from RapGenius, so mostly what will change are the lyrics. Also, changing the lyrics seems to be the most straightforward way to maximize the likelihood since by definition we want these particular lyrics to be the most likely.

That being said, this is all just conjecture. It would be interesting to try out both methods and see if you get better results doing this two step fine-tuning vs the original fine tuning (or just fine tuning on RapGenius then conditionally sampling Beatles songs as @gwern suggested).

I've seen some research [1] where the authors use beam search with an explicit diversity penalty to get around the repetition problem. They seem to get good results.

[1] https://arxiv.org/pdf/1610.02424.pdf

Hi, author here.

Yea I did the delimiting you mentioned when "training" a bigram model. For GPT-2 I was mostly interested in how well the model would be able to pick up signals from the raw data so I didn't do any kind of preprocessing at all (it's also not very fun ;)). I think it's interesting that the model was able to pick up titles, authors, starts/ends of songs on it's own.

I didn't try generating specific songs but that's a good idea. Having the delimiters would probably improve things but feeding in "On the Run\nJohn Lennon" would work as well with the current approach.

Using RapGenius corpus is also something interesting that I didn't think about. The goal of the post was to generate Beatles lyrics not song lyrics in general. To that end, I'd like to see what you get if you first fine tune on RapGenius to learn general things like song structure, rhyme, etc, then fine tune even further on the Beatles corpus. I suspect you'd get much nicer, less memorized songs.

Really cool stuff, thanks for sharing!