yaroslavvb

You can use eigenvalue solver to find singular values of any matrix but not vica versa. Implementing eigenvalues needs complex numbers

Eigenvalue are more flexible than singular values -- If you have procedure for computing eigenvalues of C, squared singular values of X are eigenvalues of C=XX'. Left and write singular vectors of X and eigenvectors of XX' and X'X. For rotation rotation matrix, singular values are all 1. Meanwhile eigenvalues tell you angles of rotation https://math.stackexchange.com/questions/4874616/obtaining-a...

I had similar experience, Claude made report of MRI for achilles tear, it measured the gap, but it was completely hallucinated. Achilles tendon is black on the MRI, it instead measured 13mm distance between two completely different things (looked white), the radiologist looked at and saw no gap at all

its real, I tried it a couple of weeks ago, asked some questions and got answers - https://x.com/yaroslavvb/status/2062692318415867983

This project gave me motivation to build the deep learning next token prediction integration for JetBrains because I was using PyCharm at the time. (Eventually, it wasn't continued because it was kind of expensive to host)

I don't hold grudge, GPT-2 wasn't that great of a model, so releasing it would be more of a publicity value. But the blog post already had that purpose.

Thanks for the share! Didn't realize eleutherai launched around same time

This was around the time I trained Transformer-XL (outside of OpenAI) with Ben Mann (https://yaroslavvb.medium.com/scaling-transformer-xl-to-128-...) . Originally we wanted to release train and release the weights as a kind of GPT-2.5, but our OpenAI friends pushed us to keep weights closed.

Modern AGI discourse defies the voice of reason

I used to look at all TensorFlow questions when I was on the TensorFlow team (https://stackoverflow.com/tags/tensorflow/info). Unclear where people go to interact with their users now....Reddit? But the tone on Reddit is kind of negative/complainy

Stricter (but not looser) standards can be imposed on state level. Canada has no binding national drinking water law, they leave it to territories/provinces to decide how to implement guidelines.

Balancing protection against water bills - https://www.epa.gov/newsreleases/epa-announces-it-will-keep-...

I've used this feature before to make my chats discoverable through search engines. I had to manually click it each time I shared, it didn't toggle on.

Researchers like to talk about and show off their work outside the company. If you don't let them, they get unhappy and leave.

The difference is that for some assets you can calculate value based on fundamentals. IE, humans need shelter, hence estimate future value of shelter (real estate) based on migration trends and other factors. How do you estimate future value of bitcoin? Lack of predictability is probably why serious investment funds don't go into crypto

Realistic simulation of neurons is expensive. Back in my grad school days we ran Genesis and could afford at most 10k neurons - each neuron needs a lot of work to model the corresponding differential equations. However, it's unclear how to translate this into requirements for artificial neural networks -- the type of computation is too different.

A different metric is a more relevant goalpost -- number of synapses. If each of 125 trillion synapses in the brain can adjust its strength independently of others, it loosely corresponds to a parameter in a neural network. So if we get 100 trillion parameter networks training but still no human intelligence, we'll know conclusively that the bottleneck is something else. Currently training 1T parameter networks seem feasible

It has a name too -- крякозяблики (kryakoziabliki)

Haha, me too. I guess it's a good learning exercise. I've been using Mathematica for 20 years (my name still shows up on https://stackoverflow.com/tags/wolfram-mathematica/topusers), but in a time crunch I would do this in Python.

SF has programs that supplies methadone to homeless, ie, www.sfstreetmed.org. This doesn't fix underlying behavioral disorders.

1. You don't have to sell the stock -- one could fund an ultra-wealthy lifestyle by renting expensive things, using stock as collateral. As your stock goes up in value, so does the value of things you have access to.

SPSA seems to be missing. OpenAI's "evolutionary strategies" paper used a form of SPSA. https://en.wikipedia.org/wiki/Simultaneous_perturbation_stoc...

On 64 core Xeon E5 this example gives 1.8x increase in user time, but 8x decrease in wall-clock time

I was an engineer in Google Books in 2011 and got an impression that this effort turned out to be not as useful as expected. This took away energy from the legal fight. Having access to every printed book seemed amazing in principle, but it didn't translate to amazing demand for the service. My theory is that it didn't lower the barriers to knowledge much -- there's not that much useful information left in offline-only storage.

Can you elaborate on the performance-per-dollar part? IE, I'm seeing GCP providing V100 at $2.48/hour and a TPUv3 at $8.00/hour

Divide by 3 to get pre-emptible price

It means that there are other factors besides diet. Here's a relevant study https://www.ncbi.nlm.nih.gov/pubmed/707372

This kind of scaling was rigorously shown for a related metric called "gradient diversity" in https://arxiv.org/abs/1706.05699

I've been using Mathematica since 1995 and Jupyter/colab for 5+ years. Most recently I've been using them both in parallel. While Jupyter is probably the future in terms of mass adoption, there are still some areas where Jupyter is lagging.

1. Mathematica has an easy way of sharing notebook. I just run "deploy" command which turns notebook into publicly accessible webpage, hosted by wolfram, here's an example -- https://www.wolframcloud.com/objects/user-eac9ee2d-7714-42da...

2. Mathematica has more active community. Mathematica-specific questions are likely to be answered within an hour by experts on https://mathematica.stackexchange.com/

3. Mathematica has better tools for simple interactivity. I like to throw in "Manipulate" for a simple graph with a draggable constant, or go to http://demonstrations.wolfram.com/index.php for an idea for more complicated demonstration to use in a presentation

4. Mathematica has more options for advanced visualization, and interfaces are more uniform since graph drawing, 3D drawing, and other kinds of visualizations are developed within a single system. Some examples https://www.wolfram.com/language/11/new-visualization-domain...

Underlying implementations often have a way to disable parallelism, ie, OMP_NUM_THREADS=1 or MKL_NUM_THREADS=1

You could specify computation in high level API like TensorFlow, and then have the framework pick the best implementations available (ie, CuDNN for GPU, MKL for CPU, something custom for TPU)