taupi

They see their competitors as Claude Code, Codex, and Cursor, not ghostty or something.

Glad you found it interesting!

What you described is the goal of Attainable SOL, but using GPU utilization as the metric rather than throughput. We're answering "for a given model and workload, have you optimized this well enough?", where "optimized" includes hyperparameter tuning. So if someone hasn't tuned batch size, parallelism, or other knobs well for their workload, the gap between their current utilization and the Attainable SOL is what tells them there's still room to improve.

We're motivated by the fact that reaching 100% Compute SOL is impossible -- no model can run at the hardware's theoretical maximum -- but we want to provide a realistic target for optimization. And we've noticed that different model architectures have different realistic ceilings. For example, MoE models run at much worse utilization due to their sparsity. We don't expect you to retrain an MoE model in order to get a higher utilization, and no hyperparameter tuning can bring you close to 100%, so the maximum attainable SOL should be lower for that model.

AMD support is on the roadmap, but we mentioned it for now to highlight that AMD calculates their utilization metric the same way -- it's not just NVIDIA.

There's a new "Effective Load" metric that we've looked at -- it's derived from Power, which has the same problems we mentioned here: https://news.ycombinator.com/item?id=47925149

It's useful as a rough heuristic, but tends to overestimate utilization. We've also noticed that power-derived metrics have a lag time behind true utilization, the controller that regulates it has a delayed response time. This especially becomes important for spiky workloads like real-time inference.

Any tool (like nvtop) that only queries NVIDIA's NVML library does not have access to the detailed metrics that we draw upon, and therefore has to use proxies for efficiency.

Part of me wonders if they're not releasing it for safety reasons, but just because it's too expensive to serve. Why not both?

Right now authentication doesn't work if you're working on a remote machine and try to authenticate with Google, FYI. You need an alternate auth flow that gives the user a link and lets them paste a key in (this is how Claude Code does it).