mitchellgoffpc

Some neat results from the last six months or so:

- Significantly-improved diffusion models (DALL-E 2, Midjourney, Stable Diffusion, etc)

- Diffusion models for video (see https://video-diffusion.github.io/, this paper is from April but I expect to see a lot more published research in this area soon)

- OpenAI Minecraft w/VPT (first model with non-zero success rate at mining diamonds in <20min)

- AlphaCode (from February, reasonably high success rate on solving competitive programming problems)

- Improved realism and scale for NeRFs (see https://dellaert.github.io/NeRF22/ for some cool examples from this year’s CVPR)

- Better sample efficiency for RL models (see https://arxiv.org/abs/2208.07860 for a recent real-world example)

https://www.deeplearningbook.org/ and http://incompleteideas.net/book/the-book-2nd.html are excellent resources for supervised and reinforcement learning, respectively, and some knowledge of statistics and probability go a long way. But I think by far the most important thing is to just start training models, even very small ones, and developing an intuition for what works and what the failure modes are.

- Get really comfortable with matplotlib or your graphing library of choice. Plot your data in every way you can think of. Plot your models' outputs, find which samples they do best and worst on.

- Play around with different hyperparameters and data augmentation strategies and see how they affect training.

- Try implementing backprop by hand -- understanding the backward pass of the different layers is extremely helpful when debugging. I found Karpathy's CS231n lectures to be a great starting point for this.

- Eventually, you'll want to start reading papers. The seminal papers (alexnet, resnet, attention is all you need, etc) are a good place to start. I found https://www.youtube.com/c/YannicKilcher (especially the early videos) to be a very useful companion resource for this.

- Once you've read some papers and feel comfortable with the format, you'll want to try implementing something. Important tricks are often hidden away in the appendices, read them carefully!

- And above all, remember that machine learning is a dark art -- when your dataloader has a bug in its shuffling logic, or when your tensor shapes get broadcast incorrectly, your code often won't throw an error, your model will just be slightly worse and you'll never notice. Because of this, 90% of being a good ML researcher/engineer is writing tests and knowing how to track down bugs. http://karpathy.github.io/2019/04/25/recipe/ perfectly summarizes my feelings on this.

It’s possible, though not quite as clean as you’d like:

  from IPython.display import display, Markdown as md
  display(md(f'*pi = {math.pi}*'))

I was shooketh

Huge thanks to the node team for adding this, I’ve been wanting fetch in node for years now! Installing node-fetch for every project was getting kind of old haha

The lisp folks are probably wondering why it took us so long to figure out the whole “all configuration is code” thing haha

Yeah, the stock driver assists in most cars still need a LOT of work. I drove from San Diego to LA last weekend with a comma, and openpilot handled the entire trip without a single disengagement until I got off the highway. I’d be really impressed to see a stock LKA manage the same (although I’ve heard great things about super cruise!)

Man, I saw that thing moving sideways right after liftoff and I thought it was a gonner! Huge congrats to SpaceX for landing with an offset engine like that on their first try!!!

Shoot you’re right, I read “warning light” and just assumed. Thanks for the correction! On the other hand, it’s even weirder that Boeing didn’t bother fixing it if it was just a software patch...

Yep. The "AOA sensor disagreement" light is built into every cockpit, but it won't turn on unless the airline purchased a specific premium add-on. Boeing claims they initially didn't realize the light was bundled as an add-on, and then once they did realize, they decided it wasn't critical to the airplane's safety and just kept selling it as an add-on, which is... kind of alarming, to put it mildly. https://www.nytimes.com/2019/05/05/business/boeing-737-max-w...

The success of Tesla as a business will have far more impact on our progress against global warming than CA’s regulatory goals will. Maybe some don’t see it that way, but I’m quite certain Musk does, and frankly I agree with him.

I think https://www.alexirpan.com/2019/10/29/openai-rubiks.html is a pretty good take. tldr: unfortunately, even tiny discrepancies between the simulation and reality (like the size of the cube's bezels or a few extra parts in the robotic hand) can significantly degrade your model's performance in the real world, even with tons of domain randomization during training. So, either we need to improve our simulations to randomize a lot more details, or we need to improve our neural nets to generalize more effectively.

My guess would be radar data overlaid onto the cameras, but not 100% sure

Inter-frame jumpy-ness can be smoothed out slightly by the control algorithms, on the assumption of object permanence, as long as the underlying vision NN doesn’t miss things too often.

I’m kind of impressed by how smooth it is, actually. If you watch videos of state-of-the-art object localization NNs, they tend to be EXTREMELY jumpy. These neural nets usually operate on only a single frame at a time, at least in the lower layers, so their predictions tend to jump around a lot from frame to frame (especially when the camera is moving!)

They use both radar and cameras, but primarily cameras. Lines, road markings, stop lights, and street signs can only be detected with vision, but cars+pedestrians can (usually) be verified by both radar and cameras. In fact, they’ve mentioned that they use radar data to help train the camera neural nets to predict distances more accurately. However, radar alone isn’t sufficient for detecting obstacles — it sometimes misses vehicles with a lot of ground clearance, and sometimes registers false positives when driving underneath bridges and overpasses, resulting in phantom braking.

STRONGLY disagree. I’m just a hobbyist, but trying to read Keras models can be a god damn nightmare if the author has to do anything even slightly non-standard. Keras seems to REALLY want you to believe that you can just throw a bunch of layers together and call .fit and everything will just work, but it never seems to be that simple unless you’re training on MNIST or ImageNet.

I agree with this to some extent, but there are real advantages to having all your code in same framework, and PyTorch is significantly easier to iterate on and debug compared to TensorFlow (in my experience). Hopefully PyTorch will start offering better support for tensor processors like google’s TPUs, but from the sound of it, OpenAI is primarily using Azure for their training infrastructure and I don’t think Microsoft currently offers anything except GPUs.

Yeah probably true, I did say “approaches” though! My guess is that short interest will eventually drop below 5% over the next couple years if Tesla continues to perform well and grow their revenue stream, but we’ll see.

The base Model 3 costs $40k in the US, and even without any extra options at all, it’s a great car to drive IMO! (Although it IS a lot more expensive in foreign markets)

Well, short interest in Tesla has decreased significantly over the past six months, from over 40% of the float in June 2019, to under 20% in December 2019. As more and more short sellers choose (or are forced) to close their positions, they drive the value of the stock higher and make it harder for other shorts to maintain their positions. With 20% of the float still sold short, I personally don’t think $TSLA has hit its peak yet, although the stock price will probably recede as short interest approaches normal levels (1-2% of the float)

For the loser now will be later to win

For the times they are a-changin'

I mean, there’s only so long Fiat can keep handing billions of dollars to Tesla, so either they’ll eventually go bankrupt or they’ll start producing good electric cars!

Sure, you could definitely argue that the EU’s strict environmental regulations make European auto manufacturers less competitive. On the other hand, these regulations are designed to promote social good, not the wellbeing of the manufacturers, and I think the laws are pretty much working in that regard by allowing Tesla to expand more quickly and forcing other automakers to take EV production more seriously. To be fair though, EU lawmakers probably didn’t foresee how long it would take for European automakers to jump on the EV bandwagon, and how much these regulations would help foreign competitors gain a foothold on their continent.

Highly recommend it! I love pytorch so much, it's basically numpy with automatic backprop and CUDA support. It evaluates eagerly by default, which makes debugging a lot easier since you can just print your tensors, and IMO it's much simpler to jump between high-level and low-level details in pytorch than in tensorflow+keras. Just as one example, activation functions in pytorch are applied by calling a python function on your layer, instead of passing a string argument with the function name to the layer constructor, so you write

  layer = F.relu(Linear()(input))

  layer = Dense(activation_fn='relu')(input)

Agreed. Tensorflow kinda reminds me of OpenGL in that its dependence on global flags causes some really annoying bugs, especially when you're using third-party libraries or pre-trained models. `enable_eager_execution`, `enable_tensor_equality`, and `enable_v2_tensorshape` have all completely broken my code at one point or another.

Yep, same exact thing keeps happening to me. I kinda can’t believe it never occurred to Apple to put haptic feedback in the touchbar, at least then you’d feel it when you accidentally press a button instead of wondering if you’re going insane because you keep unintentionally lowering the screen brightness

I 100% agree with you, but definitely check out pock.dev. Saw it on HN a little while ago, it shows your dock in the touchbar and you can customize it to add music/brightness/volume controls as well. At least for me, it actually makes the touchbar useful since I can hide the on-screen dock and still see at a glance if I have notifications, which apps are open, and what Spotify is playing (I’ve heard bettertouchtool can do something similar too)

I would ask for evidence to support your claim, but I think Newton’s Flaming Laser Sword probably applies in this case.

Maybe so, but Deepmind isn’t training AlphaZero chess any further! Leela will hopefully just keep on getting better and better :)