tkellogg

how is this self promotion? What am I promoting? This post is literally rooted in curiosity, to the core.

Whelp, it took an hour for this to quickly rise on the front page and then get shadow banned. I guess that says something.

FYI https://mcp.so/ is the exact same thing as was posted. Not sure why they directed to the github instead of the actual site..

Somewhere in 2024 I noticed that "AI" shifted to no longer include "machine learning" and is now closer to "GenAI" but still bigger than that. It was never a strict definition, and was always shifting, but it made a big shift last year to no longer include classical ML. Even fairly technical people recognize the shift.

The R1 paper did it as well. Agreed, it's always very interesting.

it's just an example, but it's great to see smolagents in practice. I wonder how well the import whitelist approach works for code interpreter security.

Additionally, in the following days, I've seen evidence suggesting that the SFT part might not even be necessary. I'd argue that work wouldn't have happened if R1 wasn't released in the open.

Ah! My bad, I edited the article to simply quote Francois. Thanks for catching this, Simon.

Author here. I do believe it's going to be exponential (not yet), but that's out of scope for the article. However, if someone has a good explainer link for that, please put it here and I'll link it into the post.

I think their thought process is unconvincing, although I think they're probably correct.

A much better paper is, "Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models". They took it much further and trained an LLM such that for every inference, they could see exactly which document from the training dataset it referenced to answer a particular question. ngl This paper AGI-pilled me.

https://arxiv.org/abs/2411.12580

Right, there's already some very encouraging trends (this study out of Nigeria). Clearly AI can lead to laziness, but it can also increase our intelligence. So it's not a simple "better" or "worse", it's a new thing that we have to navigate.

https://blogs.worldbank.org/en/education/From-chalkboards-to...

I added Phi-4 to my reasoning model collection because it seems to exhibit reasoning behavior, it stopped to consider alternatives before concluding. I assume this is related to their choice in training data:

Chain-of-Thought: Data should encourage systematic reasoning, teaching the model various approaches to the problems in a step-by-step manner.

https://github.com/tkellogg/lrm-reasoning/blob/main/phi4.md

I feel like this conversation isn't complete without referencing Hillel Wayne's "Are We Really Engineers?"

https://www.hillelwayne.com/post/are-we-really-engineers/

Entropix gives you a framework for doing that sort of thing. The architecture is essentially to detect the current state, and then adjust sampler settings or swap in an entirely new sampler strategy.

You absolutely could experiment with pushing it into a denial, and I highly encourage you to try it out. The smollm-entropix repo[1] implements the whole thing in a Jupyter notebook, so it's easier to try out ideas.

[1] https://github.com/SinatrasC/entropix-smollm

Honestly, this is going to be huge for healthcare. There's an incredible amount of waste due to incumbent tech making interoperability difficult.

lolololol, no that's me. I don't use AI at all while writing, which is why I make these mistakes

OP here: I definitely agree that shallow learning is an issue, and that it's an intoxicating effect. I've done it a few times — spent a few minutes learning a new topic, only to realize when I put it into practice that I'd been lied to.

But that's why it's critical to engage kids in this. There's a skill in using AI. Resisting the urge to take it at it's word, yet still using it for what it's good at. You can't build a skill without practice.

Yeah, that's a good point. On the other hand, the model is out there, irrevocably so, and you can use knowledge distillation techniques to train other weaker models. So yeah, we're not fully independent, but the situation is significantly improved.

I'll choose to take that as a compliment

I've had the same experience. I thought I was the weird one, but, my god, LangChain isn't usable beyond demos. It feels like even proper logging is pushing it beyond it's capabilities.

There's 2 ways mastodon can scale (or not):

1. Per-server

2. Network-wide

Moderation load is per-server scaling, mostly. I'd argue that if the load gets to be too much, moderators do less moderating and people decide to migrate to other servers. That's kind of a clean scaling strategy, tbh.

However, adding servers isn't a great story. There's n^2 network connections (worst case) that need to be made to service all subscriptions. That's definitely a scaling problem, although probably addressable via an architecture inspired by gossip protocols.

That's great! I submitted another story for a Python library

https://news.ycombinator.com/item?id=40619672

I think CodeCarbon (the python library) might have a better method for estimating carbon usage. They seem to be digging it up on a per-power station basis. Would love to see the join forces.

I’m not connected to the authors, but the detail seems legit. They seem to have a database of energy mix per power station.

I think Claude is on AWS

Perplexity is also out for me. That uses `gpt-3.5-turbo` via the API, so I assume the API might be having trouble too, but `llm` seems to work fine with `gpt-3.5-turbo`

The OpenAI philosophy is that adding modes improves everything. Sure, it’s astronomically expensive, but I tend to think they’re on to something.

not OP, but for the "unique identifiers", you can think of it like the footnote style of markdown links. Most of these models are fine-tuned to do markdown well, and a short identifier is less likely to be hallucinated (my philosophy anyway), so it usually works pretty well. For the examples, something like this can work

ID: 1

URL: https://example.com/test.html

Text: lksdjflkdsjlksjkl

And then tell it to use the ID to link to the page.

This is obviously a joke, but I wonder if there's something to it. A recommendation engine to surface interesting jobs to a candidate, and quality candidates to an employer. It seems like recruiters/sourcers are inefficient at doing it. The trouble is getting critical mass.

what even is “a model”? I’m not sure there is a technical definition that corresponds to how it’s used by the tech public

- single interconnected neural network (LLM attention layers break this, autoencoders complicate this)

- single training pass (LLMs have multiple passes, GANs have a single but produce multiple models)

I’m immediately skeptical of the methods. When I was a kid, doctors would shove antibiotics on you if you even coughed. The families where that wasn’t the case had a definite vibe, so this is screaming confounded variables to me. Who knows, maybe they did it right, but I’m skeptical.