HN user
https://lgessler.com
You're saying it's important to have up-to-date facts stored in parametric knowledge? It seems to me like that's grown less and less important as agentic capabilities have grown. Even if a frontier model doesn't know something, if it's out there, it can easily find it through tool use.
Accuracy isn't a meaningful metric here without reference to a specific task.
Not that this really takes away from the substance of the article, but the first two paragraphs are giving heavy Claude smell. Semicolons, em dashes, "That sequencing matters"... I guess I'm just a little surprised that anyone could be arsed to take on a hardware project like this but can't be arsed to write their own introduction.
Raphaël Millière has a very useful term for this kind of vacuous dismissal, the redescription fallacy (https://arxiv.org/pdf/2401.03910, page 9):
Recent debates have been clouded by a misleading inference pattern, which we term the “Redescription Fallacy.” This fallacy arises when critics argue that a system cannot model a particular cognitive capacity, simply because its operations can be explained in less abstract and more deflationary terms. In the present context, the fallacy manifests in claims that LLMs could not possibly be good models of some cognitive capacity because their operations merely consist in a collection of statistical calculations, or linear algebra operations, or next-token predictions. Such arguments are only valid if accompanied by evidence demonstrating that a system, defined in these terms, is inherently incapable of implementing . To illustrate, consider the flawed logic in asserting that a piano could not possibly produce harmony because it can be described as a collection of hammers striking strings, or (more pointedly) that brain activity could not possibly implement cognition because it can be described as a collection of neural firings. The critical question is not whether the operations of an LLM can be simplistically described in non-mental terms, but whether these operations, when appropriately organized, can implement the same processes or algorithms as the mind, when described at an appropriate level of computational abstraction.
I'll be really interested to hear qualitative reports of how this model works out in practice. I just can't believe that a model this small is actually as good as Opus, which is rumored to be about two orders of magnitude larger.
Is Java or Haskell any closer to human language?
Has everyone always nailed their implementation of every program on the first try? Of course not. Probably what happens most times is you first complete something that sorta works and then iterate from there by modifying code, executing, observing, and looping back to the beginning. You can wonder about ultimately how much of your time/energy is consumed by the "typing code" part, and there's surely a wide range of variation there by individual and situation, but it's undeniable that it is a part of the core iteration loop for building software.
I don't understand why GP's comment is so controversial. GP is not denying that you should maybe think a little before a key hits the keyboard as many commenters seem to suppose. Both can be true.
I know this is mostly about keyword substitution but it still tickles me that you still write f(x) in this language and not (x)f given that Korean is SOV but I guess that's just how you notate that no matter what cultural context you're in. Hadn't ever considered that the convention of writing a function before its arguments might have been a contingency of this notation being developed by speakers of SVO languages.
Let's be real here, regardless of what Boris thinks, this decision is not in his hands.
Novels are fictional too. So long as they're not taken too literally, archetypes can be helpful mental prompts.
If you're really just doing traditional NER (identifying non-overlapping spans of tokens which refer to named entities) then you're probably better off using encoder-only (e.g. https://huggingface.co/dslim/bert-large-NER) or encoder-decoder (e.g. https://huggingface.co/dbmdz/t5-base-conll03-english) models. These models aren't making headlines anymore because they're not decoder-only, but for established NLP tasks like this which don't involve generation, I think there's still a place for them, and I'd assume that at equal parameter counts they quite significantly outperform decoder-only models at NER, depending on the nature of the dataset.
I recommend having a look at 16.3 onward here if you're curious about this: https://web.stanford.edu/~jurafsky/slp3/16.pdf
I'm not familiar with Whisper in particular, but typically what happens in an ASR model is that the decoder, speaking loosely, sees "the future" (i.e. the audio after the chunk it's trying to decode) in a sentence like this, and also has the benefit of a language model guiding its decoding so that grammatical productions like "I like ice cream" are favored over "I like I scream".
In my (poor) understanding, this can depend on hardware details. What are you running your models on? I haven't paid close attention to this with LLMs, but I've tried very hard to get non-deterministic behavior out of my training runs for other kinds of transformer models and was never able to on my 2080, 4090, or an A100. PyTorch docs have a note saying that in general it's impossible: https://docs.pytorch.org/docs/stable/notes/randomness.html
Inference on a generic LLM may not be subject to these non-determinisms even on a GPU though, idk
Sure, this is a common sentiment, and one that works for some courses. But for others (introductory programming, say) I have a really hard time imagining an assignment that could not be one-shot by an LLM. What can someone with 2 weeks of Python experience do that an LLM couldn't? The other issue is that LLMs are, for now, periodically increasing in their capabilities, so it's anyone's guess whether this is actually a sustainable attitude on the scale of years.
I'm a professor at an R1 university teaching mostly graduate-level courses with substantive Python programming components.
On the one hand, I've caught some students red handed (ChatGPT generated their exact solution and they were utterly unable to explain the advanced Python that was in their solution) and had to award them 0s for assignments, which was heartbreaking. On the other, I was pleasantly surprised to find that most of my students are not using AI to generate wholesale their submissions for programming assignments--or at least, if they're doing so, they're putting in enough work to make it hard for me to tell, which is still something I'd count as work which gets them to think about code.
There is the more difficult matter, however, of using AI to work through small-scale problems, debug, or explain. On the view that it's kind of analogous to using StackOverflow, this semester I tried a generative AI policy where I give a high-level directive: you may use LLMs to debug or critique your code, but not to write new code. My motivation was that students are going to be using this tech anyway, so I might as well ask them to do it in a way that's as constructive for their learning process as possible. (And I explained exactly this motivation when introducing the policy, hoping that they would be invested enough in their own learning process to hear me.) While I still do end up getting code turned in that is "student-grade" enough that I'm fairly sure an LLM couldn't have generated it directly, I do wonder what the reality of how they really use these models is. And even if they followed the policy perfectly, it's unclear to me whether the learning experience was degraded by always having an easy and correct answer to any problem just a browser tab away.
Looking to the future, I admit I'm still a bit of an AI doomer when it comes to what it's going to do to the median person's cognitive faculties. The most able LLM users engage with them in a way that enhances rather than diminishes their unaided mind. But from what I've seen, the more average user tends to want to outsource thinking to the LLM in order to expend as little mental energy as possible. Will AI be so good in 10 years that most people won't need to really understand code with their unaided mind anymore? Maybe, I don't know. But in the short term I know it's very important, and I don't see how students can develop that skill if they're using LLMs as a constant crutch. I've often wondered if this is like what happened when writing was introduced, and capacity for memorization diminished as it became no longer necessary to memorize epic poetry and so on.
I typically have term projects as the centerpiece of the student's grade in my courses, but next year I think I'm going to start administering in-person midterms, as I fear that students might never internalize fundamentals otherwise.
Fermentation is completely stopped in a regular fridge, you need higher temperature for fermentation.
My understanding as a hobbyist brewer and fermenter is that this is not true. Fermentation is greatly slowed at lower temperatures, but you should have things happening above freezing. Lager beers, for example, go from pure sugar to beer at 35F. And kimchi matures at fridge temperatures in ways that I'm pretty sure are caused by fermentation.
I think this is both a harmful and irrational attitude. Why focus on some trivial mechanical errors and disparage the authors for it instead of the thing that is much more important, i.e., the substance of the work? And in dismissing work for such trivial reasons, you risk ignoring things you might have otherwise found interesting.
In an ideal world would second-language speakers of English proofread assiduously? Of course, yes. But time is finite, and in cases like this, so long as a threshold of comprehensibility is cleared, I always give the benefit of the doubt to the authors and surmise that they spent their limited resources focusing on what's more important. (I'd have a much different opinion if this were marketing copy instead of a research paper, of course.)
With all respect and love to the OP, I must admit that I laughed out loud when I saw the AWS architectural diagram and wondered whether this might be a joke. Personally, I'd have implemented this as a few dozen lines of Python living as a cron job (or even as a long-lived process with a schedule), but I'm no pedigreed engineer.
I think GP is saying that just lines on the same plot would have been less deceptive, whereas the plot that is actually there has fake data points. Readers are used to the former being purely imaginative, whereas the latter almost always means there's real measurement involved.
Extended (or rather, distended) Markdown flavors always make me queasy, and I don't love the snippet from Code Hike. In my mind, Markdown is fundamentally intended to be lightweight--something for a Reddit post or a blog comment. Heavily extending its syntax instead of choosing a saner syntactic basis strikes me as ill-advised. But one can't deny the advantages for adoption to be had from using Markdown as a starting point, and the project appears to be very popular, so perhaps worse is better.
We hope that our BTS will light the way up like dynamite[9] for future research on Korean NLP.
This has led to work showing that models can do better sometimes if you decompose these into their constituent characters, e.g.: https://aclanthology.org/2022.emnlp-main.472.pdf
I'll add mine to the pile: I needed a visualization of the classic Towers of Hanoi puzzle for a course I'm teaching and Claude banged it out (in pure JavaScript, no less) in like 3 minutes of me typing: https://lgessler.com/files/towers.html
The first stab was basically correct, but then I needed to prompt him several times to correct an issue where the disk that's currently picked up was being rendered as a vertical strip instead of as a disk, and then I just told him to convert the React into pure JS and IIRC it worked first try.
I'm afraid I feel that this piece confuses more than it clarifies.
First, saying a model "scours through its vast training data" is misleading at best: at inference time, LLMs no longer have direct access to their training data; they only have access to it insofar as it's been encoded in its parameters.
Second, saying "Every instance of an answer in the training data is like a vote" doesn't give the full picture. First of all, there are embedded contexts where "votes" can be negated: consider saying "the Earth is flat." vs. "We know it's false that the Earth is flat." or "Only a fool thinks the Earth is flat." All three contain the substring "the Earth is flat.", but both humans and LLMs are able to use context to understand that the latter two sentences are doing the opposite of endorsing the proposition that the Earth is flat. You could even imagine an extended satirical bit with "the Earth is flat" embedded within it where it is clear to a reader that all its content is intended to be taken as farcical, and I'd wager that an LLM would in many cases recognize this. So the voting metaphor breaks down here--it makes you think that the LLM is just keeping a tally of propositions, but really, it is doing something a bit more sophisticated.
I don't disagree with the premise, of course. LLM overhype is real. But we should be skeptical for the right reasons. Anna Rogers and Sasha Luccioni have a paper I really like: https://openreview.net/pdf?id=M2cwkGleRL
IMHO reviews are almost indistinguishable from noise at the AI conferences I'm familiar with these days anyway, so I don't see much of a value add.
GP wrote "simple to use API". You can attribute many qualities to TensorFlow, but this is not one of them.
I liked Yoav Goldberg snarky's quote tweet:
next paper: transformers can solve any problem but on some of them they may compute indefinitely and never provide an answer
(and you cannot tell in advance which is which!!)
Most LLMs determine their token inventories by using byte-pair encoding, which algorithmically induces sub-word tokens from a body of text. So even in English you might see a word like "proselytization" tokenized apart into "_pro", "selyt", "iz", "ation", and non-English languages will probably (depending on their proportional representation in the training corpus) also receive token allocations in the BPE vocabulary.
Here's actual output from the GPT-4o tokenizer for English and Hindi:
>>> [enc.decode([x]) for x in enc.encode("proselytization")]
['pros', 'ely', 't', 'ization']
>>> [enc.decode([x]) for x in enc.encode("पर्यावरणवाद")]
['पर', '्य', 'ावरण', 'वाद']A division of the NIH (NCBI) maintains a repository of open access publications in life sciences called PubMed Central (https://en.wikipedia.org/wiki/PubMed_Central) and this article was in a qualifying journal.
Yep, came here to say this. The big thing about the results here that might not be obvious to someone not in AI is that the models being trained in this paper are very many orders of magnitude smaller than the LLMs we've all heard so much about recently, and they're also being trained on specific tasks instead of general language modeling.
So I'm not expecting this will find its way into a LLaMA near me any time soon, but maybe this is an interesting result for people working in the specific domains represented in the evaluations.