HN user
I work on infrastructure for AI/ML.
Interested in virtualization, cryptography, distributed computing, hardware architecture.
contact info is my username @gmail.com
KV cache is, well, a cache that can fill up and trigger eviction. You require enough space to execute at least 1 fwd pass of 1 request at your context length. KV cache hits reduce TTFT by avoiding prefill. You don’t get to skip decode.
MoE is kinda related in terms of lower usage requirements vs a dense model of same total param size, but I think your mental model is a bit off.
“It depends”: what’s your prior experience, what kind of roles interest you, how big is the gap between what you have + a little ML knowledge/side projects?
I’d argue there’s a big need for people with solid fundamental CS, sysadmin, infra skills who can bridge the gap into ML practitioner/researcher understanding. Applications or inference generally are probably easiest to break into, especially if you already have service knowledge. If you want to work on distributed training or kernel/model optimization, you probably need to prove your chops there.
Neoclouds, startups in the AI space, maybe hw vendors are probably good places to look.
as someone in the space this ticks a lot of boxes: kubernetes-native, strong isolation, python sdk (ideal for ML scenarios). devmapper is a nice ootb approach.
Glancing at the readme, is your business model technical support? Or what's your plan with this?
Anything interesting to share around startup time for large artifacts, scaling, passing through persistent storage (or GPUs) to these sandboxes?
Curious what things like 'Multi-node cluster capabilities for distributed workloads' mean exactly? inter-VM networking?
Oh nice! I just had an excuse to try mojo via max inference, it was pretty impressive. Basically on par with vllm for some small benchmarks, bit of variance in ttft and tpot. Very cool!
ouch, last “recent update” in 2023. Any idea what happened?
Larger memory, weaker comms. You can optimize for this by doing things like increasing batch size/data parallelism vs sharding schemes with more comms.
At scale training won’t be able to avoid comms entirely, while many models can fit in a single MI300 for serving.
can't speak to GCP specifically but usually the issue is they are host-attached and can't be migrated, so need to be wiped on VM termination or migration -- that's when you lose data.
Reboots typically don't otherwise do anything special unless they also trigger a host migration. GCP live migration has some mention of support though
GCP mentions data persists across reboots here https://cloud.google.com/compute/docs/disks/local-ssd#data_p...
note that stop/terminate via cloud APIs usually releases host capacity for other customers and would trigger data wipe, a guest initiated reboot typically will not.
yeah, the system/application distinction feels somewhat superficial. The “multiple user space” inside a container thing sounds interesting (not sure what that means exactly), but maybe more similar to a Kubernetes pod, except maybe instead of different rootfs there’s another isolation mechanism?
The "first" link (after the home button) on bbchallenge is the header bar link to https://bbchallenge.org/story which cites Aaronson in the first sentence (double first!). I would not describe it like OP for someone trying to find the actual link ;)
"One Collatz Coincidence", the 2nd story on the blog, also mentions Aaronson
that's kind of a thing with https://docs.rs/no-panic/latest/no_panic/ or no std and custom panic handlers.
not sure what the latest is in the space, if I recall there are some subtleties
Brightline https://www.gobrightline.com/
Never had the chance to use it, but one of the more exciting US rail developments I’ve seen recently.
you seen kubie? `kubie ctx` is great for avoiding the first issue
Supabase would be a good pick. Vercel and Cloudflare both do SQL DBs now too, curious how they are. Neon seemed interesting, haven’t heard much about them in a while.
None of those integrated of course, but easier than going big cloud or self hosted
great talk! I’m curious about an approach like this combined with CUDA checkpoint for GPU workloads https://github.com/NVIDIA/cuda-checkpoint
What was your takeaway? Any added insights/reading?
Isn’t there an argument we’re simply better at brain statistics and modeling than current AI? Forget architectural limitations. What is the nature of the extrapolation? How do individuals balance their experiences and determine likely outcomes?
have you seen systemd-analyze?
This is the classic Dropbox criticism, no?
Moreover, the pleasure has nothing to do with self hosting or not, it’s just a pleasant and whimsical UX while being technically solid.
They allegedly tried chromium without luck.
Fair question. This is where I’m struggling and part of the reason I ask the question.
On some things, it’s likely I neither have a chance nor would they want me interfering. I respect that.
On others, it’s clear they want help or would accept it, but don’t have the time, energy, or inclination to do it themselves. I specifically asked about several of my ideas — they’d both accept help cleaning/decluttering, and my dad would be happy with a meal service if he likes the food (he treats my mom like one, sadly).
My family didn’t grow up with traditionally strong emotional ties and I think we all wish we were a bit closer. I get what you’re saying but I’m not sure it’s letting go so much as seeing where they struggle and trying to help. My mom does actively ask for tech help, ironically, running websites with some online sales etc. She’s not great at finding solutions to help herself, or following through when she knows they’re useful (struggled to install a dash cam she bought which was basically “plug into car charger and go”).
From the description, I’d think this is all in cluster, not public.
Huh! Does this account for the difference between what we perceive as "room temperature" and our body temperature? I suppose that delta in temperature depends on medium the body is surrounded by and effectiveness of heat transfer (e.g. similar issue as evaporation, but maybe not the same direction?)
Does it work with lower temps? I’ve always found it odd that human body temp is close to 100F but getting into 90 degree water feels hot. I wonder if it’s the evaporative effect like you said and if it reverses when it’s hot enough. Like if it’s 110 and I get in a 90 degree tub, does it feel good? That’s below the temperature my body naturally wants to be at, why doesn’t it feel “cool”?
The article mentions avoiding overcommit and oom score adjust. You can avoid overcommit by always specifying requests == limits and can use priority class for oom score adjust.
There are definitely improvements like memory qos/pressure handling, but not sure what you mean about those specifics, they can be handled.
You can always oom the whole node, I don’t think(?) the fact that there’s a non root oom matters? So you could arguably set no pod limit, set priority class critical, and let another pod in the workload cgroup get killed.
Doesn’t this still exist? You Force Touch the space bar and get a cursor
I think iOS has http proxy settings in the wifi configuration for a given network? Haven’t tried recently.
Signal validity out of band, not in band with the data.
Could you expand on this?
Yeah, I recently went down this path. It’s all doable but frankly I’m not a nation state target and getting locked out after a kernel update or similar would be far more annoying.
Instead I’m leaning toward separate boot and root disks, with a root/data disk encrypted with LUKS with a detached header. dm verity on a read only root with a separate data partition also seems simple/appealing. Of course, these all allow attacks full secure boot/tpm/etc avoid, but it’s a balance.
That’s not really the point though - it’s more like saying the NYC metro area isn’t only Manhattan. Brooklyn would be the 3rd largest city in America by population.
I agree it’s kind of irrelevant and the original point was correct: DC is an urban core without the rural split most other states have.