patrick0d

Inspired by parameter golf and speedrun approaches I make the case for picking loss functions like a wallclock for LoRA on AI safety targets. The result when I tried it was a functional distillation of an Sparse AutoEncoder into a 5.3MB probe. I have a technical writeup below about it if anyone is interested.

https://www.lesswrong.com/posts/PagGF8roBJmjLunsX/competitiv...

paper looks nice! i think what they found was that they can recover the input sequence by trying all tokens from the vocab and finding a unique state. they do a forward pass to check each possible token at a given depth. i think this is since the model will encode the sequence in the mid flight token so this encoding is revealed to be unique by their paper. so one prompt of 'the cat sat on the mat' and 'the dog sat on the mat' can be recovered as distinct states via each token being encoded (unclear mechanism but it would be shocking if this wasn't the case) in the token (mid flight residual).

anyone who spends 10 years writing performance reviews and rebuilding a service they already rebuilt last year, all to collect RSUs, was never going to land us on mars

Well don't do it and instead of using an off the shelf library that is known to work while the rest of the development team isn't reinventing the wheel. Doing it for fun and education is fine of course.

I am an AI interpretability researcher and have a new proposal for a way to measure the per token contribution of each head and neuron in LLMs. I found that the normalisation that happens in every LLM is avoided by modern attribution methods despite it having a large impact on the model's computation.

Here is the full preprint paper and the code I used. https://github.com/patrickod32/landed_writes Happy to some insight from any interested people and would like to know if other people here have been working on anything similar. This seems like a real gap in the research to me.

Maybe eating honey has never happened in the history of human nature, and it's fool-hardy.

the average human certainly can not run that fast. treadmills in gyms usually cap at 20km/h and how many people on the street do you think could handle it? (very few of course)

this vibe check is more insightful to me than the popular evals. nice job!

they could have made the law:

if you collect users data

you must ask first

add a yes or no button on a banner so they can pick

but instead the eu citizens were let down by the legislators

this should be the real headline. I am certain and beyond any doubt that I speak for all of us hackernews readers when I say that I have no frame of reference on the value of cryptocurrencies without comparing first with the Malaysian Ringgit

What's your definition?

All you need to do to see some change is pick a goal and commit a long time towards it. So if you have a long time maybe you can help with this project that you would like to see.

I said a good teenager might run 10.7 and you have found a list of teenagers (I don't know how old they are) that can do just that. Very few people sprint after they leave college but if they spent time becoming stronger and more physically mature then 10.7 would be quite achievable - even a good highschool runner can run that time.

My original point being that people can run this fast and that if they do they would only be a few percent slower than the pinnacle of the sport. Even from the list you shared there most of the runners in those state finals ran 10.7 or better. Imagine an olympic final that had extra lanes for the fastest teenagers from Californian highschools. We would see a horde of 60kg teenagers that could finish the race less than a second (<10.7s) behind the fastest man in the world (9.80s in the previous olympics) over a 100m race.

It is a good time and I gave the example of a teenager since they will likely be less physically developed, weaker and less experienced than a pro. In a place where sprinting is popular like in the US 10.70 is a pretty achievable for someone who spends time at the sport.

After a quick google, the men's Scottish 100m record was set in 1980 and has not been broken since so I would think that they do not devote a lot of attention to sprinting, and in 2022 only 4 runners broke 11 seconds in the 100m championships final.

height and weight, fast- and slow-twitch muscle mass, cardiovascular conditioning, flexibility and elasticity, and probably more.

When we try to figure out how to measure someones speed (say to compare Usain Bolt to another high quality sprinter) we will look at factors like height, stride length and strength. But in terms of all the factors that need to be measured to describe the function of someones running speed we also have to think of things like the speed of gravity, how many legs they have, the viscosity of air etc. Usain Bolt can't change any of these features of course so if he wants to run 3% faster he might need to be 3x as strong as another runner - since that is one of the few parameters in the equation he can influence.

I think this explains the distribution of the top speed runners performances. The olympic 100m champion won in 2021 with a time of 9.80 and a good teenager might run a time of 10.70. If you saw this on a running track as the olympic champion would cross the line when the amateur was about 93 meters into the race. That's pretty close when you consider that one runner is a professional that might be able to lift weights 3x as heavy as the teenager, they have been training for years and years with professional help and they execute the race with superior form.

This is something that I like to remember when we try to compare and measure peoples skills and how they are likely distributed in society.

Those people who were suffering from curable blindness that had been forgotten by society sure are marketed, businessed and entertained by their restored sight. He's doing good things to improve peoples lives and entertaining millions along the way.

The worry is not that chatgpt will take over the world. It is that a future system will be unaligned with human interests and once it is created by gradient descent (the internals of the system are not understood by anyone - they're just matrices) there will be no guarantee that humanity will be safe. By looking at the power of gpt4 we have no clear idea of how fast it will continue to improve.

There is a book called "The perfect machine" about building this observatory.I found it very interesting since you might think it's not as hard as it is to make a piece of glass a few meters across and make it reflective.

The parts of chemistry that work are just rebrandings of physics and the parts of physics that work are just rebrandings of math. Nanotech is a subset of physics with connections to solid state physics, quantum mechanics, materials science, optics etc. I don't get why people say nanotech is charlantry when the semiconductors chip manufacturing industry is top down nanotech manufacturing.

Why not take the 100 billion from somewhere other than frontier physics research. I always consider these articles small minded since the people writing them don't consider projects outside of their domain that eat up more funding than something like LHC++. It's such a zero sum situation where there is a 100 billion grant for grabs and the author wants it spent on one project in place of this one.

The more research we do the better and 100 billion spent over approx 20 years by over a dozen countries is a far cry from putting all our eggs in one basket.

He's referring to Carlsen's prodigious fantasy football skill.

We should add treads and a cannon while we're at it. Is outer space counted when we say "all terrain"?

Eddington said:

"The law that entropy always increases—the Second Law of Thermodynamics—holds, I think, the supreme position among the laws of Nature. If someone points out to you that your pet theory of the universe is in disagreement with Maxwell’s equations—then so much the worse for Maxwell’s equations. If it is found to be contradicted by observation—well these experimentalists do bungle things sometimes. But if your theory is found to be against the second law of thermodynamics I can give you no hope; there is nothing for it but to collapse in deepest humiliation."

But hey there's a lot we still don't know about the universe. I mean these unbreakable laws are only a few hundred years old for us.

Because the universe is expanding the older and farther away light is the redder it looks to us. So by collecting this infrared light we can see farther back in time than we could with any other instrument before.

There's a book called "Skunk Works" by Ben Rich about the making of different spy planes. There's some mentions in the book about satellites and what existed at the time how projects were carried out.