davidatbu

So would you take these claims seriously if they came from OpenAI (since Codex is a pretty lean CLI app)?

If so, I think it would be in the spirit of HN to discuss the subject matter of the blogpost (increasingly autonomous coding towards the end goal of RSI) as if the blog post was indeed from OpenAI. OpenAI is, by all accounts, going through a very similar process anyways.

By frontend, do you mean wasm in the browser? I'd have expected multithreading there, especially in wasm, to be extremely unergonomic.

Maybe you mean to refer to concurrency?

Note that something that helped the misinformation was that, on Twitter, there were Kimi employees expressing their surprise that the base model was Kimi K2.5, and their indignation that Cursor didn't credit Kimi. They later deleted their tweets (what I infer from that is that some employees were not aware of some pre-existing agreement or understanding between Cursor and Kimi until the drama happened).

Thanks for correcting the title I misremembered. Fwiw, the article did culminate with LSTMs: https://karpathy.github.io/2015/05/21/rnn-effectiveness/

---------------------

EDIT: It looks like you deleted the part of your post I quoted below. So feel free to ignore my question about it, I guess.

---------------------

Not sure what you mean by

Shows how much you know

Do you mean that the fact that I misremembered a word on the title suggests that I know very little about Karpathy's contributions to the field of neural networks?

I can spare a minute :). This isn't exhaustive because this is just stuff I know of, obviously.

- At Stanford, Led research on the first (to my knowledge) crop of joint image/text models. Super widely cited work.

- At Tesla, led their whole self driving effort for a while, came up with critical techniques that allowed them to make progress (e.g., the concept of "auto labelling": using a much larger NN to generate training data with which to train smaller models that could fit in the on-device compute. IIRC, Elon said they would not have been able to make progress without this insight).

I'm not sure his educative efforts for the mold of what you're looking for, but if so, the course he designed at Stanford (and availed online):for neural networks, as well as his blog posts, (most famous of which, to my knowledge, is "the unreasonable effectiveness of LSTMs"), made a huge impact on educating a generation of tinkerers and researchers.

Thank you!

Gotcha. I'm genuinely curious: by "impressive", are you referring to coverage? I'd be grateful if you could say a few words about it could be more impressive (e.g, if you indeed meant to talk about coverage, say what functionality/edge cases aren't covered as of now)

Ah, I just learnt that you don't. Jarred's comment saying exactly that: https://news.ycombinator.com/item?id=48133806

Do you have some examples?

Not OP. For this particular use case, I think performance is a primary concern.

But if you mean in general, I also totally feel that languages that let you represent more invariants statically are better fit for LLMs. I'd love to see experimentation with LLMs with dependent types and managed effects.

Fwiw, that's not the stated motivation for the rewrite experiment. In fact, the Rust rewrite is slower to compile than the zig code when compiled with their internal fork of zig (tho it is faster when OG zig is used).

I don't want to infringe upon your right to speculate. I just want to point out that your statement is at best a speculation.

I'm pretty sure that they have decided that backwards-compat is not the best path for Mojo. Matter of fact, the following is the _last_ item on the roadmap on the home page:

Supporting more of Python's dynamic features like classes, inheritance, and untyped variables to maximize compatibility with Python code.

What's more, note how it says "to maximize compatibility" not "to achieve full compatibility."

Well, of the top of my head, both chatgpt.com and Gemini have text on their home page to the effect of "AI can make mistakes". I'll bet a few bucks such copy can be found in other places, including the terms of service.

Do you do rolling deploys?

I'd love to see a link to these emails, if you have one handy!

I don't think pypi or npm allow replacing existing packages?

Are you saying that VSCode runs tsserver in its own NodeJS process? Or are you saying that VSCode uses the NodeJS it ships to run tsserver in a different process?

Just want to say that as an AI engineer, you and the Latent Space folks are doing work that is extremely useful to me. Without y'all, I'd be forced to doom scroll on X to catch up on the latest developments.

I wanted to explicitly highlight the utility of what you do because of surrounding comments that suggest/imply otherwise.

Thank you Simon!

Super interesting! Was this C# or something? is there a write-up/mini-blogpost about this somewhere?

How is enshitification (the gradual degredation of service and products for commercial gain) even related to what's being discussed (the gradual obsoletion of a certain set of skills of an SWE)?

I doubt that Meta (the company that sponsors the work on pyrefly) is looking forward to selling a product based on Python typing (assuming that's what's "what's being glazed in the article").

Koka, Roc-lang come to mind.

IIUC, triton uses Python syntax, but it has a separate compiler (which is kinda what Mojo is doing, except Mojo's syntax is a superset of Python's, instead of a subset, like Triton). I think it's fair to describe it as a different language (otherwise, we'd also have to describe Mojo also as "Python"). Triton's website and repo describes itself as "the Triton language and compiler" (as opposed to, I dunno, "Write GPU kernels in Python").

Also, flash attention is at v3-beta right now? [0] And it requires one of CUDA/Triton/ROCm?

[0] https://github.com/Dao-AILab/flash-attention

But maybe I'm out of the loop? Where do you see that flash attention 4 is written in Python?

Fwiw, Chris has mentioned both of those as lessons he took from Swift that he'd like to avoid for Mojo.

Yeah the rate of progress in AI definitely makes it seem like that from the outside for me too.

But having never written cuda, I have to rely on authority to some extent for this question. And it seems to me like few are in a better position to opine on whether there's a better story to be had for the software-hardware boundary in ML than the person who wrote MLIR, Swift-for-Tensorflow (alongside with making that work on TPUs and GPUs), ran ML at Tesla for some time, was VP at SiFive, ... etc.

Good point. But the overall point about Mojo availing a different level of abstraction as compared to Python still stands: I imagine that no amount of magic/operator-fusion/etc in `torch.compile()` would let one get reasonable performance for an implementation of, say, flash-attn. One would have to use CUDA/Triton/Mojo/etc.

I assume HLL=Higher Level Language? Mojo definitely avails lower-level facilities than Python. Chris has even described Mojo as "syntactic sugar over MLIR". (For example, the native integer type is defined in library code as a struct).

Whether it's 1, 2, or N kernels is irrelevant.

Not sure what you mean here. But new kernels are written all the time (flash-attn is a great example). One can't do that in plain Python. E.g., flash-attn was originally written in C++ CUDA, and now in Triton.

So I think a demonstrative example of your claim would be if you knew someone who is as accomplished with regards to compilers, language design, tackling really hard long term projects, but not as good at self promotion, and elaborate on what the lack of that skill-set caused.

The only other person I know of who has started and lead to maturity multiple massive and infrastructural software projects is Fabrice Bellard. I've never ran into him self promoting (podcasts, HN, etc), and yet his projects are widely used and foundational.

It seems to me like the evidence points to "if you tackle really hard, long term, and foundational software projects successfully, people will use it, regardless of your ability to self promote."

Chris's claim (at least with regards to Triton) is that it avails 80% of the performance, and they're aiming for closer to 100%.

Yeah this is slightly confusing for me as well. Even in this very podcast, being a superset of Python was mentioned as a goal (albeit a long term one).