kingcai

https://jcaip.github.io/ I actually recently updated this for the first time in 5 years!

This is cool. Kudos to the team too - I feel like this is what engineerings all about.

I've recently been in a rut and while I still have a lot further to go I've been feeling better over the last couple of days and here's what helped:

- going to therapy

- taking PTO

- going outside

- hanging out with friends

- cooking a meal

- going on a date

- going to LA and getting sun

- reconnecting with old friends

I'm not out yet but I feel like with some time and effort I can be. How did I do it? I think there are really two parts of this - helping others and asking for help. I've always enjoyed the first but never done the second and that was really holding me back.

ML training is not as easily parallelizable as the other problems that have been explored. I'm not familiar with SETI but I know this to be true for folding@home.

As you mentioned, ML training can be parallelized but this requires either model/data parallelism.

Data parallelism means spreading the data over many different compute units and then synchronizing gradients somehow. The heterogeneous nature of @home computing makes this particularly challenging, as you will be limited by the smallest compute unit. I've personally only ever seen data (and model) parallel done on a homogenous compute cluster (i.e. 8x GPUS)

For model parallelism, we split the model across different compute units. However, this means that you need to synchronize the different parts of the model together, which can get very expensive when you do it across the internet. If you have 8xGPUS on one machine, your latency is limited by PCIe instead of TCP/IP in a distributed @home cluster.

But I would say it's not impossible, someone clever could definitely figure it out.

I graduated in 2020 and was considering doing a PhD in NLP as I had already been doing research at a lab. I ended up joining a series A startup, trying to turn research into products.

I don't regret it much, as I was tired of being a broke student. Also I think the problems you run into in industry are more interesting. However, I do think my skills stagnated a bit and I learned more when doing research.

I like this post a lot, even if it's a somewhat contrived example. In particular I like his point about switch statements making it easier to pulled out shared logic vs. polymorphic code.

There's so much emphasis on writing "clean" code (rightly so) that it's nice to hear an opposing viewpoint. I think it's a good reminder to not be dogmatic and that there are many ways to solve a problem, each with their own pros/cons. It's our job to find the best way.

Hi, I wrote this blog post about startup equity a couple of years ago that answers a lot of the questions you asked, I hope you find it helpful.

https://jcaip.github.io/Startup-Equity-TLDR/

You negotiate around dilution by asking for additional equity grants, like you would for a raise / promotion. But in general dilution is a good problem to have because it means you are raising more money and growing.

In general I would say expert knowledge is more valuable. However, I think of knowledge as a pyramid - how high you can go is determined by how wide/strong your foundation is.

In my experience I've found that trying to develop deep expertise requires a solid understanding of many different fundamentals. For example, when I was doing NLP research, I had to learn about distributed systems in order to diagnose problems when training with multiple GPUs, or about dynamic libraries for fixing CUDA errors.

RTO = return to office

Very uncommon for people to do ML locally nowadays. Almost everyone uses either cloud or has an cluster they ssh into. NVIDIA is still king for ML, very few people use AMD / M1, although hopefully that will change.

If you want to buy a GPU for ML this is a good resource: https://timdettmers.com/2023/01/30/which-gpu-for-deep-learni...

Basically buy the NVIDIA GPU with the largest amount of VRAM that's in your budget.

I work on FOSS (pytorch) that I used to use a lot during research. It feels like I've come full circle and I'm giving back.

Also I'm well compensated and surrounded by smart and kind people.

It's somewhat common in the industry for your employer to give you a stipend to buy equipment / tools / learning materials you like.

I would get him the gaming console, but otherwise maybe nice headphones?

Absolutely brilliant!

My total comp is ~300k, 200 of which is in cash.

I think it's really hard to think up of curriculum for a "programming" class. It'd be hard to standardize across different professors. Also while I do think that more programming experience would benefit most new grads, it's hard for me to say that it would be more beneficial than say better interpersonal working skills or even specialized knowledge.

FWIW at the uni I attended, projects were a huge part of some classes (usually ~50%) and there were some SWE specific classes where you learn best practices / tooling. You would need to be able to program in order to graduate. I feel like this is the norm and not the exception, but I may be wrong.

There were a lot of CS clubs at my school that hosted workshops and tried to teach people some of this stuff. Maybe you can find a club (or start one) at your school.

Startups are also not profitable yet and are reliant on fundraising in order to continue to operate. The fundraising market is bleak currently, so there's pressure to reduce burn.

I found reading more code helped me to become a better programmer. It helps a lot that reading code (at least for me) is much faster than writing code, so you can do more of it.

Old techniques are just as good as the new ones :) If you have a specific problem you are trying to solve, I'd recommend just reading papers and looking up things you don't understand. Otherwise, if you're just looking for general knowledge, a class could be helpful.

It's an unavoidable part of the job, but it's definitely not most of the work.

I feel it's kind of like writing tests in SWE - something you do because it's beneficial even if it's not enjoyable.

My best advice is to find people who do what you want to do and try to learn as much as possible from them. If you're interested in doing ML/DL research I think the best way to get into the field is to reach out to professors. I studied ML/DL (books, projects, classes, reimplementing papers) for several years in undergrad, but discussing and debating ideas is the one thing that took my understanding to a much deeper level. A good professor will also point out gaps in your knowledge that you might be missing.

A second bit of advice: Programming (and execution) skills are IMO heavily undervalued by people looking to get into ML. The faster you can write code, debug, and implement new things, the easier it is to produce good research.

Some books I liked: PR & ML (Bishop), Deep Learning Book (Goodfellow), AI: A Modern Approach (Norving), Elements of Statistical Learning (Friedman)

Other users advice about traveling, relaxing, and exercising is spot on, but probably not what you were looking for. Besides these are things you should do for a lifetime, not just a summer.

I know you mentioned working at a research lab but have you tried more academic research? You seem to value learning and mentorship a lot, so it could be a lot of fun to pick a professor and work with them for a summer. Given your background, you could probably convince any STEM professor to work with you, and I would heavily recommend exploring stuff outside your area of expertise.

Sure. I don't work in the ecommerce space, but I think the big problem that industry is attempting to solve right now is - "how can we take off the shelf language models and use them to do things that make us money?". This is a super broad problem and there's many answers to this question. It can be as complicated as creating intelligent chatbot, but also as simple as adding multi-lingual support to an app via cross-lingual training.

The example I gave of multi-modal learning was really just highlighting a dichotomy in the techniques that we use in machine learning today. FWIW I am a couple of years removed from working heavily with tabular data, so do take this with a grain of salt. But there are essentially two different modeling approaches for two different types of datasets. On the one hand, you have deep learning (BERT, language models, CV models) which does well on raw data like text or images. These usually work by mapping the raw data to dense embeddings, which are the output of neural models. On the other hand, you have decision trees / forests (think XG boost) that work great on tabular data - spreadsheets or other data of that nature.

But what do you do if you have a spreadsheet of data and one of the columns is raw text data but the other columns are say sparse boolean features? How can you incorporation the extra information from the spreadsheet into your language model? I think this is a common problem in industry that there's not a clear solution for right now.

For academia: I expect that FAANG will continue to produce larger and larger language models as well. Probably new improvements in multi-task learning to keep on increasing model size - this will probably take ideas from relational / contrastive learning. Few-shot /zero-shot learning is also something that's come a long way the last couple of years. There will probably be a bunch of secondary papers about language models as well - hypothesis on how they work, explanation of corner cases, ways to deal with bias and fairness.

For industry: Feels like “making BERT / some other language model do things” is a common job nowadays. On the more engineering side - I think we’ll see more tools to quickly and efficiently fine-tune language models, especially tools that allow a human in the loop.

Overall it feels like we’re getting to a point where there’s a pretty standardized approach to simple NLP problems like text classification - no more real feature engineering, just throw BERT at the problem. I expect for this trend to continue - with more and more of a focus on dataset creation and validation and less of an emphasis on model architecture.

I also think there will be a rise in multi-modal language models - combination of language and vision models for example. But I think the more interesting application will be combining dense language model representations with sparser tabular data. Think of trying to predict a users likelihood to buy a product given a review of another product (dense embedding of text), but also their clicks over the last 2 hours. (sparser tabular data) - this feels like a much more common problem people have.

To stay updated: read papers (arxiv-sanity.com is a lifesaver) and watch talks (usually just on youtube or a lot of uni reading groups are public on zoom nowadays).

I saw this quote on a tombstone and it really resonated with me.

  What is success?
  To laugh often and much;
  to win the respect of intelligent people and the affection of children;
  to earn the appreciation of honest critics and endure the betrayal of false friends;
  to appreciate the beauty; to find the best in others;
  to leave the world a bit better, whether by a healthy child, a garden patch Or a redeemed social condition;
  to know even one life has breathed easier because you have lived.

I think because it expressed something that I've felt but could never quite articulate - that there isn't one big well-defined reason for living but a bunch of little reasons that are all worthwhile.

I've used Snorkel quite a bit at work, usually combined with transformers models.

It has worked quite well for us. The snorkel public package is a bit out of date now, as I think they're building a SaaS solution and focusing more on that. But aside from that it's quite easy to use. Other downside is that lot's of cool ideas are present in the papers but not fully implemented (not complaining though!). Also thinking of a diverse set of heuristics can be hard.

We use snorkel a lot for bootstrapping text classifiers. Our classification models don't require much domain expertise, as it's pretty easy to tell if a text sample is classified correctly, so the main advantage is just avoiding labeling costs and quicker prototyping. We find that we can usually use embedding similarity as a good heuristic. I wrote up a little bit about this approach here if you're curious: https://cultivate.com/why-cultivate-uses-embeddings-for-rapi...

Happy to answer any additional questions you have too :)

I did a little analysis on this back in 2020 (pre-covid) when I was job searching: https://jcaip.github.io/Startup-Equity-TLDR/

Hope it's helpful!