terafo

Dynamic libraries are a dumpster fire with how they are implemented right now, and I'd really prefer everything to be statically linked. But ideally, I'd like to see exploration of a hybrid solution, where library code is tagged inside a binary, so if the OS detects that multiple applications are using the same version of a library, it's not duplicated in RAM. Such a design would also allow for libraries to be updated if absolutely necessary, either by runtime or some kind of package manager.

This article specifically talks about PC laptops and discusses changes in them.

Having access to the text and being trained on the text are two different things.

There are LLM's that can process 1 million token context window. Amazon Nova 2 for one, even though it's definitely not the highest quality model. You just put whole book in context and make LLM answer questions about it. And given the fact that domain is pretty limited, you can just store KV cache for most popular books on SSD, eliminating quite a bit of cost.

Yes

MLA uses way more flops in order to conserve memory bandwidth, H20 has plenty of memory bandwidth and almost no flops. MLA makes sense on H100/H800, but on H20 GQA-based models are a way better option.

https://www.youtube.com/watch?v=b2F-DItXtZs

Because you have to do inference distributed between multiple nodes at this point. For prefill because prefill is actually quadratic, but also for memory reasons. KV Cache for 405B at 10M context length would take more than 5 terabytes (at bf16). That's 36 H200 just for KV Cache, but you would need roughly 48 GPUs to serve bf16 version of the model. Generation speed at that setup would be roughly 30 tokens per second, 100k tokens per hour, and you can server only a single user because batching doesn't make sense at these kinds of context lengths. If you pay 3 dollars per hour per GPU, it's $1440 per million tokens cost. For fp8 version the numbers are a bit better: you need only 24 GPUs, generation speed stays roughly the same, so it's only 700 dollars per million tokens. There are architectural modifications that will bring that down significantly, but, nonetheless, it's still really really expensive, but also quite hard to get to work.

Why mention Microsoft twice?

There was. Now second gen of that goes for $15.

Overwhelming majority of flops is indeed spent on matmuls, but softmax disproportionately uses memory bandwidth, so it generally takes much longer than you'd expect from just looking at flops.

t5 is an architecture, t5x is a framework for training models that was created with that architecture in mind, but can be used to train other architectures, including decoder-only ones(there is one in examples).

To quote their official response "If the WSE weren't rectangular, the complexity of power delivery, I/O, mechanical integrity and cooling become much more difficult, to the point of impracticality.".

Not right now.

Because SRAM stopped getting smaller with recent nodes.

This thing targets training, which isn't affected by tiny accelerators inside CPUs.

No, it's comparable to 230Mb of SRAM on Groq chip, since both of them are SRAM-only chips that can't really use external memory.

I would say that Bradley is actually more valuable, since it can serve wider range of missions, while having higher crew survival rate and being more maneuverable.

they have 24,700,000 left of fighting age

Without equipment, logistics and ammo to support it it's a dead weight. Also, it's very interesting that you omitted Gulf War, which would be the most similar conflict in terms of power dynamics. 4th by strength military in the world in war against large coalition of countries that is led by USA.

Wrong. Shells, artillery, drone components, engineering vehicles, tanks, APCs, jets, long-range missiles, anti-air defenses. 10x that and Ukraine starts winning again. 10x manpower won't do that.

"Escalation management" through "slowly boiling the frog" combined with lack of coherent strategy for the war, partially because some western politicians are afraid of Russia's tantrum in case of Ukrainian victory(using nukes if Crimea is lost, for example), and that are balancing aid so it's enough for Ukraine not loosing, but not enough for Ukraine winning. Equipment that would've been sufficient to win the war in 2022(before Russia constructed massive defensive lines) was delivered in 2023, largely in the second half of 2023. And there's not that many signs that it might change.

Ukrainan Air Force has HARMs, but they are VERY limited in their capabilities due to them being employed from soviet-era jets. Basically area where target resides have to be pre-programmed on the ground, rocket then flies to that area and lock on any radar it finds there. But what previous commenter missed is that even if Russian air defenses are suppressed, their planes outclass Ukrainian ones. For example, air to air missiles that UAF has available need to be guided by planes radar all the way through, also that missiles have shorter range than something like R-37, which is fire and forget with VERY high range. Western air to air missiles are much better than what Ukraine has right now, but they can't be fired from Su-27 or MiG-29, they require something like F-16 or Gripen, but while a bunch of European countries agreed to transfer them, Ukrainian pilots and ground crews don't know how to operate them, and need to be trained, which happens right now. If there were trained beforehand it would've changed current situation on the front lines VERY significantly.

For video to be a major contributor to a game size is a very rare occurrence. Games are 120GB+ in size because texture sizes continue to grow.

Jetson Nano is a lot more expensive. But at the same price point as Coral you can buy Orange Pi 5 with 4 gigs of ram, 6 tops NPU(not supported as good as Coral though, but is generally more open and has support of much more frameworks) and have a full-blown OS there. SBC is $66, and camera is another $15. And you can buy it right now, not "pre-order" something "coming soon". Coral supply history is a very sad tale. The hardware was impressive 4 years ago, when it first came out. But now we have no second generation in sight, no supply and no communication from Google that they are still interested in this project.

Vision Pro isn't really supposed to be used outside.

Attention is shared. It's ~30% of params here. So ~2B params are shared between experts and ~5B params are unique to each expert.

Yes, but you wouldn't want to do that. You will be able to run that on a single 24gb GPU by the end of this weekend.

v4 chips are basically the same in bf16 performance as A100(slower in int8).

They are mostly incompatible.

I don't have to imagine.