shayonj

that's very interesting! I was noticing page vault storm on live migrations as well and I wonder if that's what you were running into / mentioning here regarding the lock contention

you can do things that aren't practical with 100-200ms startup: speculative parallel execution (fork 10 VMs, try 10 approaches, keep the best), treating code execution like a function call instead of an infrastructure decision, etc.

i am not following, why isn't it practical?

Cool project. +1 on userfaultfd for the multi-node path. Wrote about how uffd-based on-demand restore works wrt to my Cloud Hypervisor change [1] if you are curious.

I think the the main things to watch are fault storms at resume (all vCPUs hitting missing pages at once) and handler throughput if you're serving pages over the network instead of local mmap. I think its less likely to happen when you fork a brand new VM vs say a VM that has been doing things for 5 mins.

Also interestingly, Cloud Hypervisor couldn't use MAP_PRIVATE for this because it breaks VFIO/vhost-user bindings. Firecracker's simpler device model is nice for cases like this.

[1] https://www.shayon.dev/post/2026/65/linux-page-faults-mmap-a...

That's a good shout. I will update/port it back from here - https://github.com/cloud-hypervisor/cloud-hypervisor/pull/78..., but quite fast

This is going to be an interesting space to watch I think and big part of offering sandbox as a service basically for enterprise and saas needs.

Yeah, it's hard to hit the right balance with nuance around these and you're spot on. What I meant to get at was the specific difference in default modes where gVisor's systrap intercepts syscalls via seccomp traps and handles them entirely in a user-space Go kernel, so there's no hardware isolation boundary in the memory/execution sense. A microVM puts the guest in a VT-x/EPT-isolated address space, which is a qualitative difference in what enforces the boundary (perhaps?)

Whereas yeah, you can run gVisor in KVM mode where it does use hardware virtualization, and at that point the isolation boundary is much closer to a microVM's. I believe the real difference then becomes more about what's on either side of that boundary where gVisor gives you a memory-safe Go kernel making ~70 host syscalls, a microVM gives you a full guest Linux kernel behind a minimal VMM. So at least in my mind it comes down to a bit of around different trust chains, not necessarily one strictly stronger than the other.

Heya! nice to see you here. In retrospect it feels like CI companies and environments are very well suited for sandboxes since a lot of the problems overlap around ephemeral workloads, running untrusted code, fast cold starts, multi-tenancy isolation. Also, loved Buildkite at a past job! Looking forward to following cleanroom

It touches in the gvisor section around the trade-off that the surface area for gvisor is smaller. There are trade offs. It’s not dishonest.

That’s a good shout! I have been curious as well and did some experiments. Also left out GPU sandboxing from the post as well. Maybe will reflect in a part II post.

Wasmer looks v cool. I must check it out

That is a good call out and I missed to consider the options you pointed. When I am back on keyboard I will add an updated note with a link to your comment. Thank you!

Very cool to see this and something I have been curious about myself and exploring the space as well. I'd be curious what are some parallels and differentiations between this and NVIDIA's stdexec (outside of it being in Rust and using Future, which is also cooL)

Being able to debug and diagnose difficult problems and distributed systems still remains a key skill, at least until Opus or some other model gets better at it.

I think being intentional about learning while using AI to be productive is where the stitch is, at least for folks earlier in their career. I touch that in my post here as well: https://www.shayon.dev/post/2026/19/software-engineering-whe...

yeah, got confirmation of

Increased API Error Rates in Region: eu-west-1 where s3 Service is affected.

This! Only thing worse than your main queue backing off is you dropping items from going into the DLQ because it can’t stay up.

v cool and impressive!

more to do with the deeply unfussy work of actual context engineering.

I agree. There is something to be said about autonomous systems producing valid outcomes within the current constraints of RL

Sadly, its not the first time I have noticed unexpected and odd behaviors from Aurora PostgreSQL offering.

I noticed another interesting (and still unconfirmed) bug with Aurora PostgreSQL around their Zero Downtime Patching.

During an Aurora minor version upgrade, Aurora preserves sessions across the engine restart, but it appears to also preserve stale per-session execution state (including the internal statement timer). After ZDP, I’ve seen very simple queries (e.g. a single-row lookup via Rails/ActiveRecord) fail with `PG::QueryCanceled: ERROR: canceling statement due to statement timeout` in far less than the configured statement_timeout (GUC), and only in the brief window right after ZDP completes.

My working theory is that when the client reconnects (e.g. via PG::Connection#reset), Aurora routes the new TCP connection back to a preserved session whose “statement start time” wasn’t properly reset, so the new query inherits an old timer and gets canceled almost immediately even though it’s not long-running at all.

"juice wasn't worth the squeeze" - adding that to my vocab

I was kinda surprised the lack of CAS on per-endpoint plan version or rejecting stale writes via 2PC or single-writer lease per endpoint like patterns.

Definitely a painful one with good learnings and kudos to AWS for being so transparent and detailed :hugops:

Here is another take on deletes through by just updating the row groups in Parquet file through multi part upload and UploadPartCopy - https://www.shayon.dev/post/2025/285/mutable-atomic-deletes-...

I think ideally you could map retention of cold data to file objects itself and using key space naming strategy and lifecycle rules, expire the data that is not needed, thus saving on the storage costs (as much as possible hopefully)

Yeah , I poorly phrased it - I meant in an ideal situation with the benefits of parquet like columnar file structure. I very much understand that it’s not possible on parquet today for the reasons you mentioned and others.

Yeah. Or just sub out the data with null bytes. Something like that could be nice too.

Yeah, it mentions in few areas - compared to OLTP or similar workloads, this will definitely be slow

Thanks! looks like i messed up some css on my last frontend refresh a bit.

That's fair. I had an EXPLAIN ANALYZE which shows the time it takes for planning and execution. And sending the query directly to the table outperforms that.

More to come soon.

This is v cool! thanks for sharing

That’s a good distinction and thank you! AGI is indeed orthogonal to LLMs today.

Fair! I was going for more generic example for the intro and eventually segue into the greater point and questions the post is trying to make. It does touch on few other examples like the AlphaFold, later on.