HN user
https://blog.quipu-strands.com/
Cool project! I tried something similar a while ago [1] - I wanted to load up an embedding model and semantically order texts, all in the browser.
So I pull ONNX weights from HuggingFace (MPNet, MiniLM), use Transformers.js to embed, and use a clusterer from scikit-learn (running on pyiodide - it was a surprise to me that this worked flawlessly) on the page - all client-side.
Oh wow, they are still around! I used to keep a Knoppix disk around for times when my desktop would run out of space and wouldn't boot (if I remember right). Boot it with Knoppix, delete some files, and then boot it normally!
You usually see this kind of analyses in conference papers, esp. if they have a datasets track. The NeurIPS Datasets & Benchmarks (D&B) track is a good example. But you will have to monitor the proceedings yourself closely - there is little chance of being accidentally exposed to them, because most blogs, announcements and popular media only mention a handful of the popular ones, e.g., Tau^2. For ex., across the years 2022, 2023 and 2024, 900+ papers were accepted in the D&B track [1] - of course, not all of them are LLM-related. I find them interesting because they often focus on specific system behaviors, and like you said, study them scientifically, so you can draw authoritative conclusions (or at least know specifically what part of a model's behavior you now know about, and what parts you don't).
[1] https://blog.neurips.cc/2025/09/30/reflecting-on-the-2025-re...
These "You're right to push back" scenarios are scary for me. I mostly code ML implementations, and some of the errors Claude Code (CC - have only used Opus 4.7) makes are very sneaky, and if you don't have sufficient experience in the area (I see this with people entering ML and writing their implementations with CC), you wouldn't know when to question CC and will let errors or future pitfalls silently slip into your code. A recent example was when there was data leakage in a model calibration step, which it refused to see as an error, till I wrote a detailed reason, and then it agreed that there was a "subtle leakage".
Reg PRs - for the ones with complex requirements what I am seeing is that time to initial PR is very short, and a ping-pong between the reviewer and developer begins, because in my cases (not all) the developer vibe-coded parts, and they didn't really understand the requirements deeply or their code, and it takes multiple iterations for them to fix it. You can argue this is a human problem but this is the net effect I'm seeing.
I am not sure but for complex cases it seems to me that the earlier sum of moderately long PR time + moderately long review time has been replaced by very short PR time + even longer review time. I am not sure if there's a net gain in these cases. Sometimes even if the code is functionally correct, it's verbose enough (e.g., too many intermediate functions) that I think they will impact future reviews.
I should go back to this (like some of the other comments suggest) - I think there is potential in suggesting multiple lines, and I would have really loved it to work, but it is clear that their ux testing was poor.
I am not particularly bothered by the speed but the AI suggestion clutter is quickly becoming an issue for me :( I type fragments of a line and it will suggest the next 5-6 lines. They may not be outright wrong, but they might not represent the way I like to do things. Pressing Esc. and refocusing on what I was going to type in anyway is a disruptive experience.
They are great for specialized use-cases: (a) where the problem is not hard enough (you don't need reasoning), or (b) diverse enough (you don't need a world model), (c) you want cheap inference (and you can make it happen hardware-wise) and (d) you either have enough data or a workflow that accumulates data (with fine tuning with enough data you can sometimes beat a premier model while ensuring low latency - ofc, assuming (a) and (b) apply).
I make it sound like a rare perfect storm needs to exist to justify fine tuning, but these circumstances are not uncommon - to an extent (a), (c) and (d) were already prerequisites for deploying traditional ML systems.
I notice you mentioned dspy - do you also support prompt optimization?
This is an amazing quote - thank you. This is also my argument for why I can't use LLMs for writing (proofreading is OK) - what I write is not produced as a side-effect of thinking through a problem, writing is how I think through a problem.
I once modeled user journeys on a website using fancy ML models that honored sequence information, i.e., order of page visits, only to be beaten by bag-of-words (i.e., page url becomes a vector dimension, but order is lost) decision tree model, which was supposed to be my baseline.
What I had overlooked was that journeys on that particular website were fairly constrained by design, i.e., if you landed on the home page, did a bunch of stuff, put product X in the cart - there was pretty much one sequence of pages (or in the worst case, a small handful) that you'd traverse for the journey. Which means the bag-of-words (BoW) representation was more or less as expressive as the sequence model; certain pages showing up in the BoW vector corresponded to a single sequence (mostly). But the DT could learn faster with less data.
I use Claude Code within Pycharm and I see the git diff format for changes there.
EDIT: It shows the side-by-side view by default, but it is easy to toggle to a unified view. There's probably a way to permanently set this somewhere.
Thank you. Your comment about LLMs to semantically parse diverse data, as a first step, makes sense. In fact come to think of it, in the area of prompt optimization too - such as MIPROv2 [1] - the LLM is used to create initial prompt guesses based on its understanding of data. And I agree that UMAP still works well out of the box and has been pretty much like this since its introduction.
[1] Section C.1 in the Appendix here https://arxiv.org/pdf/2406.11695
I was not aware this existed and it looks cool! I am definitely going to take out some time to explore it further.
I have a couple of questions for now: (1) I am confused by your last sentence. It seems you're saying embeddings are a substitute for clustering. My understanding is that you usually apply a clustering algorithm over embeddings - good embeddings just ensure that the grouping produced by the clustering algo "makes sense".
(2) Have you tried PaCMAP? I found it to produce high quality and quick results when I tried it. Haven't tried it in a while though - and I vaguely remember that it won't install properly on my machine (a Mac) the last time I had reached out for it. Their group has some new stuff coming out too (on the linked page).
Thanks for the example. Yes, true, this is for expensive functions - to be precise functions that depend on data that is hard to gather, so you interleave the process of computing the value of the function with gathering strategically just as much data as is needed to compute the function value. The video on their page [1] is quite illustrative: calculate shortest path on a graph where the edge weights are expensive to obtain. Note how the edge weights they end up obtaining forms a narrow band around the shortest path they find.
Timefold looks very interesting. This might be irrelevant but have you looked at stuff like InfoBax [1]?
You don't - the way I use LLMs for explanations is that I keep going back and forth between the LLM explanation and Google search /Wikipedia. And of course asking the LLM to cite sources helps.
This might sound cumbersome but without the LLM I wouldn't have (1) known what to search for, in a way (2) that lets me incrementally build a mental model. So it's a net win for me. The only gap I see is coverage/recall: when asked for different techniques to accomplish something, the LLM might miss some techniques - and what is missed depends upon the specific LLM. My solution here is asking multiple LLMs and going back to Google search.
Love awk. In the early days of my career, I used to write ETL pipelines and awk helped me condense a lot of stuff into a small number of LOC. I particularly prided myself in writing terse one-liners (some probably undecipherable, ha!); but did occasionally write scripts. Now I mostly reach for Python.
I'm curious to know if Anthropic mentions anywhere that they use speculative decoding. For OpenAI they do seem to use it based on this tweet [1].
Wouldn't this be an optimization problem, that's to say, something like z3 should be able to do - [1], [2]?
I was about to suggest probabilistic programming, e.g., PyMC [3], as well, but it looks like you want the optimization to occur autonomously after you've specified the problem - which is different from the program drawing insights from organically accumulated data.
[1] https://github.com/Z3Prover/z3?tab=readme-ov-file
[2] https://microsoft.github.io/z3guide/programming/Z3%20Python%...
Hadn't seen this before, very nice read, thank you!
Thank you for your kind comment!
Aside from secondmind [1] I don't know of any companies (only because I haven't looked)... But if I had to look for places with strong research culture on GPs (I don't know if you're) I would find relevant papers on arxiv and Google scholar, and see if any of them come from industry labs. If I had to take a guess on Bayesian tools at work, maybe the industries to look at would be advertising and healthcare.I would also look out for places that hire econometricists.
Also thank you for the book recommendation!
This is the definitive reference on the topic! I have some notes on the topic as well, if you want something concise, but that doesn't ignore the math [1].
[1] https://blog.quipu-strands.com/bayesopt_1_key_ideas_GPs#gaus...
Active Learning is a very tricky area to get right ... over the years I have had mixed luck with text classification, to the point that my colleague and I decided to perform a thorough empirical study [1], that normalized various experiment settings that individual papers had reported. We observed that post normalization, randomly picking instances to label is better!
Evals somehow seem to be very very underrated, which is concerning in a world where we are moving towards (or trying to) systems with more autonomy.
Your skepticism of "llm-as-a-judge" setups is spot on. If your LLM can make mistakes/hallucinate, then of course, your judge llm can too. In practice, you need to validate your judges and possibly adapt to your task based on sample annotated data. You might adapt them by trial and error, or prompt optimization, e.g., using DSPy [1], or learning a small correction model on top of their outputs, e.g., LLM-Rubric [2] or Prediction Powered Inference [3].
In the end, using the LLM as a judge confers just these benefits:
1. It is easy to express complex evaluation criteria. This does not guarantee correctness.
2. Seen as a model, it is easy to "train", i.e., you get all the benefits of in-context learning, e.g., prompt based, few-shot.
But you still need to evaluate and adapt them. I have notes from a NeurIPS workshop from last year [4]. Btw, love your username!
[2]https://aclanthology.org/2024.acl-long.745/
I noticed the conflation of terms too but it seems to arise out of the original Microsoft announcement!
https://devblogs.microsoft.com/commandline/edit-is-now-open-...
You're right! My bad. Thank you for pointing it out! Leaving my comment up for info on bayesopt.
Note the website (ai-contest.com) that the post links to seems to have been hijacked by a gambling site.
For the use-cases where Genetic Programming was popular, I would recommend looking at Bayesian Optimization (bayesopt) as an alternative today (I know I keep recommending the area - but I hope I do when it is relevant :-)). This is mostly because IMHO it has a principled foundation that has been productively developed further in the past few years. Here's a good book on the topic [1], and I've a tutorial as well [2]. Interestingly one of the books I had encountered when reading up on Genetic Algo. years ago was by Melanie Mitchell [3]!
Bayesopt or Genetic Programming, or any search algorithm that can operate over non-differentiable objective functions are very useful in practice. For ex, when performing model selection in the space of hyperparameters, when your model is not differentiable such as a traditional Decision Tree [4]. Or exotic use-cases like molecule discovery [5].
You can try out bayesopt using the botorch or hyperopt libraries. The latter only implements a specific bayesopt algo. which was/is popular but it seems to have been bettered of late [4].
[2] Part 1 https://blog.quipu-strands.com/bayesopt_1_key_ideas_GPs
[3] Found a free copy online https://www.boente.eti.br/fuzzy/ebook-fuzzy-mitchell.pdf
[4] "... Analysis of the Black-Box Optimization Challenge 2020" https://proceedings.mlr.press/v133/turner21a.html
[5] ChemBO is an example but there are others https://proceedings.mlr.press/v108/korovina20a.html
The people who founded Ponoc seemed to have creative differences with Miyazaki. They wanted to make a movie [1] that they felt Ghibli won't greenlight [2] - but there seems to have been no deep seated animosity or desire to rip-off. Incidentally I just borrowed this movie from the local library a few hour ago because the cover art reminded me of Ghibli but I noticed it wasn't a Ghibli production. Some searching online led me to the cited article.
[1] Mary and the Witch’s Flower https://imdb.com/title/tt6336356/
[2] https://otakuusamagazine.com/hayao-miyazaki-says-he-wont-see...