mlepath

This is an awesome project! Thanks for taking time to document this. What's next on your plate? How do we follow you?

All companies that are doing something with the current iteration of "AI" are underwater. Sam Altman says that the pro tier of ChatGPT is losing money, Adobe is losing tons on firefly, ... This is pretty typical for silicon valley though, we always burn investor money to corner the market and then tech usually catches up. Most enterprises don't need to be first adopters.

In ML everything is a tradeoff. The article strongly suggests using dot product similarity and it's a great metric in some situations, but dot product similarity has some issues too: - not normalized (unlike cosine simularity) - heavily favors large vectors - unbounded output - ...

Basically, do not carelessly use any similarity metric.

Awesome project! I love the idea of not sending my data to a big company and trust their TOS.

The effectiveness of coding assistant is directly proportional to context length and the open models you can run on your computer are usually much smaller. Would love to see something more quantified around the usefulness on more complex codebases.

The author appears to be confused about the difference between research and production. In research more generic approaches typically win because they get resourced much better (plus the ever-growing compute and data have helped, but there is no guarantee that these will continue).

On production side of "AI" (I don't love the term being thrown around this loosely as true AI should include planning, etc not just inference) the only question is how well do you solve the one problem that's in front of you. In most business usecases today that problem is narrow.

LLMs drive a minuscule (but growing) amount of value today. Recommender systems drive huge amount of value. Recommender systems are very specialized.

This reinforce the idea that there is no real strategic advantage in owning a model

For these models probably no. But for proprietary things that are mission critical and purpose-built (think Adobe Creative Suite) the calculus is very different.

MS, Google, Amazon all win from infra for open source models. I have no idea what game Meta is playing

Wait, I thought every pornsite already had age verification, that modal that pops up and says "are you sure you are over 18?"

As ML practitioner I am not a big fan of the concept of singularity and wholly agree that IF it happens it will be more like sunrise than a light switch, but...

I think thinking about it IS useful. Problems like value alignment are really difficult. No one alive knows what to do about value alignment and even if it takes us 500 years to reach real singularity it may take that long to solve value alignment problem

I worked at Meta, I am not sure the hiring was ever "diverse".

DEI always seemed like an activity they did for show. This changes nothing honestly.

Great work!

Do you see any artifacts from having trained on synthetic data? Is there a natural benchmark dataset (real tables in the wild)?

In my experience synthetic data can only take you so far, it has all the quirk the dataset creator can think of but the real value is usually in patterns they cannot. Vision took a huge leap forward with ImageNet dataset release

This is really awesome. Tested it on both Nvidia and Mac GPUs.

Interested to know how debugging in a real application would work since WASM is pretty hard to debug and GPU code is pretty hard to debug. I assume WASM GPU is ... very difficult to debug.

The process-centric taxonomy in this paper is one of the most structured frameworks I’ve seen for anomaly detection methods. It breaks down approaches into distance-based, density-based, and prediction-based categories. In practice (been doing time series analysis professionally for 8+ years), I’ve found that prediction-based methods (e.g., reconstruction errors in autoencoders) are fantastic for semi-supervised use cases but fall short for streaming data.

The first rule of programming with LLMs is don't use them for anything you don't know how to do. If you can look at the solution and immediately know what's wrong with it, they are a time saver otherwise...

I find chat for search is really helpful (as the article states)

At University I once talked to a former Soviet professor about publications and he mentioned how they proliferated. It was uncommon to have a lot of publications until recently and now everyone has a really high count.

I think we came to any metric that is used for rewards loses its value as a metric. That includes citations, sadly.

I have had a somewhat similar journey but 14 years instead of 25 and I always wonder how it would be different today.

We were lucky enough to grow up with the industry and progressively learn more complexity. The kids out of school today are faced with decades worth of complexity on day one on the job.

This is a great application of various domains of ML. This reminds me of Vesuvius Challenge. This kid of thing is accessible to beginners too since the data by definition are pretty limitted.

Hello, fellow Metamate ;)

I have worked with Jetson Orin platform, and honestly Nvidia has something that is really easy to work with there. The Jetsons are basically a full GPU (plus some stuff) at very low power. If I were tasked with building a robot it would likely be the first place I look.

Yea, people have a really hard time dealing with data leakage especially on data sets as large as LLMs need.

Basically if something appeared online or was transmitted over the wire should no longer be eligible to evaluate on. D. Sculley had a great talk at NeurIPS 2024 (same conference this paper was in) titled Empirical Rigor at Scale – or, How Not to Fool Yourself

Basically no one knows how to properly evaluate LLMs.

Naive question, do people no longer respect robots.txt?