HN user
startup ml engineer
How big is this market, self-hosting a model that requires 64 GPUs, H100 or better, with good interconnects between nodes?
I suspect the overlap of those that can afford it, and those that have the talent to manage it, is a fairly thin slice of the Venn diagram. Even the large corps are gonna be getting it from the inference vendors, or more likely Bedrock and friends.
I can also use the models anytime, and for a lot of time, until i’m anywhere close to salary for an engineer like that
release-please[0] allows you to do a manual version override in a commit, which would allow you to decrement the major version upon reverting a breaking change
I think that could be simplified, so the tool can tell that a commit is reverting a breaking change and thus the version should be decremented, but at least there's an escape hatch.
With a lot of the adult-oriented sets (ICONS and others), especially anything plant-based, they go out of their way to point out how they're using existing pieces in an unusual way. One example is the cherry blossoms on the bonsai tree are actually frogs, but cast in a pink color for the first time, called out in the instruction manual as you're building it.
I love it, knowing about these little details. Also fun to share with friends that inquire about the various LEGO on display in the house. This, and all the fancy mechanics (e.g. typewriter, nintendo), engender a ton of respect and awe for the designers.
Kind of the opposite, I was deep in the R world a decade ago and there was a huge trend of replacing Java dependencies with C/++ ones because the JVM was such a pain to manage. The community eagerly adopted the replacements about as soon as they existed.
I experienced a flavor of this, too. We had some outages, management said no more daytime deploys, so we had after-hours “deploy parties” whose scope and participant count increased weekly. The smarter managers said it was temporary, but couldn’t say how we’d move back towards continuous deployment. If anything went wrong in any service, you’d end up with a dozen or so folks on a zoom call for 3 hours. We did this once or twice a week.
Went on for about a year, worse each week, before i left.
I don’t know how big the market is, but seems pretty commercial-friendly to this old magic player. I have a big box of cards from a few decades ago I’ve held onto. I’ve thought about selling them, but it seems i either take them to a shop and get lowballed, or spend hours meticulously researching each card and then figuring out how to sell it for what it’s worth. taking a pile of photos and having the ID and valuation automated could go a long way! Hard to sell to individuals like me, but i would think a card marketplace would find it invaluable?
What sorts of sites are you thinking of? To me, “most useful to a programmer” evokes docs and blogs and github issues and forum posts. I suppose some forums might be AI-resistant (login wall), but the others are trivially AI accessible.
Yeah this sounds pretty reasonable really, like instead of using a CMS directly they’re having Claude file PRs to make the same changes. As someone who likes static sites and change control, it actually sounds like an improvement.
It is, but it’s the only way for a company to succeed and scale over time. A pet approach works well in the early days, but you can’t become a VC-backed success without drastically reducing bus factors throughout the company.
That could be an incentive to keep companies small, but high-scale companies do have unique benefits to society.
My experience as well. It sounds nice at first, but since it’s tied to org flattening these “player-coaches” end up with 15-20 reports, which is way too many for even a pure manager.
I noticed it was especially bad for on-call and incident response; these managers get pulled in to all the incidents because of their status and supposed involvement, but are not particularly useful in those rooms, adding even more cooks to the already crowded kitchen.
This is what i often do, but i have never been able to get many coworkers onboard. In my experience I’d say less than 5% of all software folk i’ve worked with are willing to do an interactive rebase; everyone else finds it too scary
most teams we talk to can't even tell you how many GPUs are in use right now
how can this be? isn’t this a trivial metric to pull from any clouds monitoring service?
to get the good ones (H100+) you generally have to reserve them, a fixed cost you pay monthly and can’t pretend to not know
Even before LLMs, Data Science was being replaced by more specialization, IME.
Data Engineers took over the plumbing once they moved on from Scala and Spark. ML Engineers took over the modeling (and LLMs are now killing this job too, as it’s rare to need model training outside of big labs). Data analysts have to know SQL and python these days, and most DS are now just this, but with a nicer title and higher pay.
Once upon a time I thought DS would be much more about deeper statistics and causal inference, but those have proven to be rare, niche needs outside soft science academia.
not even google thinks this will happen, given their insistence on only offering TPU access through their cloud
plans now open in a separate file tab, and if you don’t accept it, it just…disappears so you can’t discuss it!
you can’t really buy H100s except in multiples of 8. If you want fewer, you must rent. Even then, hyperscalers tend to be a bit inflexible there; GCP only recently added support for smaller shapes, and they can’t yet be reserved, only on-demand or spot iirc.
We have common words for those two flavors of “fast” already: latency and throughput. S3 has high latency (arguable!), but very very high throughput.
how would you explain how hard he fought to NOT buy twitter?
people seem to forget he was legally forced to buy Twitter after he tried for months to get out of his joke bid, primarily through claiming he was misled about the extent of bots on the platform
I used to like allowing squashing or fast-forward merges. Most PRs would be squashed, because most developers write terrible commit messages and use merge where they could rebase. But, if you had a well-crafted set of commits, we could retain them.
I’ve recently switched to using conventional commits and release-please everywhere, but that pretty much forces us into a squash-only world, since even the devs who write nice commit messages don’t want to make each commit a conventional commit; much nicer to do it as the PR title, and more visible
Calumet, in the Keweenaw Peninsula (just north of Michigan Tech) was nearly made the capital, instead of Lansing, because of this!
for decades, big tech contributed relatively little in the way of python ecosystem tooling. There’s Facebooks Pyre, but that’s about it. Nothing for package/dependency management, linting, formatting, so folks like those at Astral have stepped up to fill the gap.
why is type checking the exception? with google and facebook and astral all writing their own mypy replacements, i’m curious why this space is suddenly so busy
that’s not really what’s new or special about pixi, is it? poetry (poethepoet) and uv can both do variations of this.
From the outside, pixi looks like a way to replace (conda + pip) with (not-conda + uv). It’s like uv-for-conda, but also uses uv internally.
Better task running is cool, but it would be odd to use pixi if you don’t otherwise need conda stuff. And extra super duper weird if you don’t have any python code!
anyone have any tips for testing actions locally, rather than workflows?
Despite the name, act is really only for the latter. You can try to test a local action by putting it in a workflow and calling that, but if you do a checkout in your workflow that will overwrite the mount of your local code into the act container, meaning you’ll get the version from the remote branch instead. Depending on the action, you may not be able to comment out the checkout step while testing.
That’s one of the arguments in the substack post. BNPL is like credit cards, but more specific, and that specificity allows for more accurate pricing of the loan risk.
Not OP, but i think python is very far above R for engineering stuff. I built my early career on R and ran R user groups. R is great for one-off analyses, or low-volume controlled repetition like running the same report with new inputs.
For engineering stuff i want strong static analysis (type hints, pydantic, mypy), observability (logfire, structlog), and support (can i upload a package to my cloud package registry?).
For ML stuff, i want the libraries everyone else uses (pytorch, huggingface) because popularity brings a lot of development and documentation and obscure github issues the R clones lack.
Userbase matters. In R, hardly any users are doing any engineering; most R code only needs to run successfully one time. The ecosystem reflects that. The python-based ML world has the same problem, but the broader sea of python engineers helps counterbalance.
money that you need soon shouldn’t be in the broader market.
for retirement savings, balance should shift to bonds or similar as you near retirement.
same principle applies to saving for other large events, like a down payment on a house. decrease risk as even nears.
no need to manage this yourself; every roboadvisor will do this for you rather cheaply
Because of the TPUs, or due to other factors?
What even is an AI data center? are the GPU/TPU boxes in a different building than the others?
I’ve always thought it was weird GPU stuff in python doesn’t use asyncio, and mostly assumed it was because python-on-GPU predates asyncio. But I was hoping a new lib like this might right that wrong, but it doesn’t. Maybe for interop reasons?
Do other languages surface the asynchronous nature of GPUs in language-level async, avoiding silly stuff like synchronize?
Not the point of the post, but i’m surprised to see less than a million queries per day. Is that all of Kagi, or just some subset?