jengels_

Sure, but completeness is a much higher bar than being able to find at least some things we weren’t looking for. And I’m reasonably optimistic that we’re going to make SAEs much better in the future, I agree they’re definitely imperfect right now

It's a super interesting direction! That's one of the long term goals of interp research: deconstruct model behavior into circuits of features, and then turn those circuits into code (that we can maybe even formally verify!).

I'm one of the first authors on this paper, happy to answer any questions :)

I feel like un-supurvised methods like Anthropic's SAEs can be argued to find things we're not looking for (their most recent work is from a couple days ago: https://transformer-circuits.pub/2024/scaling-monosemanticit...). And we can get some sense of how "much" of the model they're recovering by looking at their downstream reconstruction loss.

Pretty efficient! E.g. a recent paper describes a system to do fully private search over the common crawl (360 million web pages) with an end to end latency of 2.7 seconds: https://dl.acm.org/doi/10.1145/3600006.3613134

Try it out! We have simple scripts you can play with and comparisons with GPT-3.

ThirdAI (thirdai.com) | Houston, Bangalore, or remote | Full time | Senior Software Engineers

We are a startup building high performance machine learning systems on CPUs. Our core technologies are hash-based algorithms that accelerate neural network training and high dimensional near neighbor search. We are a new and well funded company (we just raised $6M of seed funding), and already have more customer interest than we know what to do with. We pride ourselves on a collaborative, open, and idea heavy culture. Our perks are great, including 401K matching, free lunches, and substantial equity.

For location, remote works, as long as you can travel some days of the month to our office in central Houston or Bangalore.

We are looking for engineers with 3+ years of experience building and shipping high performance systems. Machine learning, C++ and Rust experiences are a big plus, and definitely tell us about any AI publications you have, we’re interested! If building a new set of machine learning algorithms from scratch sounds fun to you, get in touch! Feel free to email our head of engineering at vaidhy [at] thirdai.com. Our website is under active reconstruction using Hugo (that’s how early stage we are), but feel free to find out more about us at https://www.thirdai.com

ThirdAI (thirdai.com) | Houston or remote | Full time | Senior Software Engineer, VP Product, VP Engineering

We are a startup building hash-based algorithms that accelerate neural network training and high dimensional near neighbor search. We are a new and well funded company (we just raised $6M of seed funding), and are seeking to grow our team quickly. We pride ourselves on a collaborative, open, and idea heavy culture. We offer 401K matching, a competitive salary, substantial equity, and lots of great perks.

For our senior software engineering roles, we are looking for engineers with 3+ years of experience building and shipping high performance systems. Machine learning and C++ experience is a big plus, and definitely tell us about any AI publications you have, we’re interested!

For our open VP roles, we’re looking for technical leaders with a track record of leading teams of engineers and growing new products. Experience in high performance computing and systems engineering is highly preferred.

For location, we prefer if you work or commute some days of the week to our office in central Houston, but we are flexible for outstanding candidates.

We’re trying to scale rapidly, and already have more customer ideas and interest than we know what to do with. If building a new set of machine learning tools from scratch sounds good to you, get in touch! Feel free to email our CTO at tharun [at] thirdai.com, or apply on our website: https://www.thirdai.com/careers/