stygiansonic

From a brief reading of what Fusion does: https://openrouter.ai/docs/guides/features/plugins/fusion

Looks like Fusion calls a bunch of models and then uses an LLM to synthesize the results, and pass to another model for final output.

Fugu looks like it's doing something different? Using an LLM earlier on in the flow as an orchestrator to decide which other LLMs to call. More coordinator than simply synthesizing results, and more "agentic".

It's interesting because it's all exposed behind a single OpenAI compatible endpoint (Responses API?) and so then presumably someone could use this for one of their single agents. Now you have agent-of-agents, nested in some sense. The token usage increases accordingly!

See also: https://en.wikipedia.org/wiki/Metcalf_sniper_attack

(Perpetrators also not caught)

  The jury found that Ding stole trade secrets relating to the hardware infrastructure and software platforms that allow Google’s supercomputing data center to train and serve large AI models. The trade secrets contained detailed information about the architecture and functionality of Google’s custom Tensor Processing Unit chips and systems and Google’s Graphics Processing Unit systems, the software that allows the chips to communicate and execute tasks, and the software that orchestrates thousands of chips into a supercomputer capable of training and executing cutting-edge AI workloads. The trade secrets also pertained to Google’s custom-designed SmartNIC, a type of network interface card used to facilitate high speed communication within Google’s AI supercomputers and cloud networking products.

Neat experiment that gives a mechanistic interpretation of temperature. I liked the reference to the "anomalous" tokens being near the centroid, and thus having very little "meaning" to the LLM.

That paper doesn’t seem to be about security vulnerabilities in MiG but rather using it to improve workload efficiency

Wonder why they haven’t gotten an in house pizzeria yet to reduce the signal on this side channel leak

From the article it appears to be something they invented:

Gemma 3n leverages a Google DeepMind innovation called Per-Layer Embeddings (PLE) that delivers a significant reduction in RAM usage.

Like you I’m also interested in the architectural details. We can speculate but we’ll probably need to wait for some sort of paper to get the details.

Interesting! If Knuth is not the original author then they’ve been lost to the sands of time

Great article and nice explanation. I believe this describes “Algorithm R” in this paper from Vitter, who was probably the first to describe it: https://www.cs.umd.edu/~samir/498/vitter.pdf

The article mentions this union, not sure if it meets your definition of success: https://www.alphabetworkersunion.org/our-wins

When subtlety proves too constraining, competitors may escalate to overt cyberattacks, targeting datacenter chip-cooling systems or nearby power plants in a way that directly—if visibly—disrupts development. Should these measures falter, some leaders may contemplate kinetic attacks on datacenters, arguing that allowing one actor to risk dominating or destroying the world are graver dangers, though kinetic attacks are likely unnecessary. Finally, under dire circumstances, states may resort to broader hostilities by climbing up existing escalation ladders or threatening non-AI assets. We refer to attacks against rival AI projects as "maiming attacks."

Sorry to hear this

A lot of my teenage years were spent building and playing with PCs and a lot of the knowledge and interest came from reading each and every issue of boot and maximum pc

+1

Jumping into an unknown codebase (which may be a library you depend on) and being able to quickly investigate, debug, and root cause an issue is an extremely invaluable skill in my experience

Acting as if this isn’t useful in the real world won’t help. The real world is messy, documentation is often missing or unreliable, and the person/team who wrote the original code might not be around anymore.

I wrote about something similar, which was motivated by an issue I saw caused by an (incorrect) expectation that a Java hashmap iteration order would be random: https://peterchng.com/blog/2022/06/17/what-iteration-order-c...

Yeah, ops comment makes it seem like they are building racks of RTX 4090s, when this isn’t remotely true. Tensor Core performance is far different on the data center class devices vs consumer ones.

Thanks for writing this.

Is this concept (dice room puzzle, doomsday argument) at all related to the st Petersburg paradox? https://en.m.wikipedia.org/wiki/St._Petersburg_paradox

Reminiscent of a scene from Billions: https://www.reddit.com/r/Billions/comments/czlg4u/need_help_...

Thanks - I added my contact info (I don’t comment a lot on HN, mostly just read) but will drop you a line

This is probably using their excess capacity, but not necessarily that their GPUs are idle.

For LLMs/large models the huge cost is memory ops to load each layer weights during the forward pass. This is why doing inference at batch size 1 is extremely wasteful: you pay all the mem ops cost and don’t use enough compute FLOPs to justify.

You want a high enough batch size so that compute:mem ops is close to the ratio for that GPU. This is usually done by batching together multiple user requests.

At times of low usage there is excess capacity because batch size is below this “optimal” ratio. So they can slot in these “relaxed SLA” requests for little marginal increase resource usage on their end.

Basically have a queue of these requests that you use to “top off” your batch size when you can.

Edit: also you may not be able to get optimal batch size depending on when the requests arrive, eg you don’t want to wait forever to fill up a batch. So again having a queue of outstanding/delayed requests to serve allows for smoothing things out and increasing compute utilization

A simplified explanation, which I think I heard from Karpathy, is that transformer models only do computation when they generate (decode) a token. So generating more tokens (using CoT) gives the model more time to “think”.

Obviously this doesn’t capture all the nuance.

Op probably referring to an M series MacBook since it has a unified memory architecture and the same memory space used by both cpu and gpu

This is a much better set of general terms vs the original, which currently is heavily oriented towards LLMs/Gen AI

It seems like he (re)joined OpenAI almost exactly 1 year ago: https://twitter.com/karpathy/status/1623476659369443328

Also who would cover the legal liability in case of damages?

https://developer.apple.com/contact/request/download/alterna...

One of the requirements for third party App Store use by developers is maintaining a standby letter of credit of 1M euros from an A rated financial institution

Edit: was mistaken, this is a requirement for the operator of a third party App Store.

For further reading this is called the Variability hypothesis, that male ability has higher variance (but not necessarily a higher mean) than female ability: https://en.m.wikipedia.org/wiki/Variability_hypothesis

(Not making a judgment as to whether I believe this is true or not)

This seems unethical:

Since January of 2023, we have deployed this system in the wild, where it has poisoned several hundred million data sources expected to be scraped and used in the training of new LLMs. This large-scale deployment serves as a real-world testbed and demonstration of our approach's efficacy.

We have concurrently developed a sophisticated countermeasure, capable of detecting and neutralizing the effects of VonGoom. Researchers and organizations concerned about the integrity of their LLMs, and looking to cleanse their systems of our widespread data manipulation, are advised to contact us for pricing and licensing details.

This assumes that they will publish a paper that has substantive details about Q*

Well, this was probably the best weekend to do it.

See this blog post (linked in original article) from a security firm that was one of Okta's customers, and alerted them to the breech, when they detected suspicious activity on their network shortly after sharing the HAR file with Okta:

https://www.beyondtrust.com/blog/entry/okta-support-unit-bre...