HN user
(author here) I'm one of the maintainers of HashiCorp's Nomad, so that example was likely inspired by the separation of duties that's part of our security model. In that environment, there's a subset of task (ex. container) configuration that's controlled by the cluster admin and a subset that's controlled by the job author deploying onto the cluster.
Author here! The motivating example of this post is frankly pretty lousy in retrospect (and was even so soon after writing, given the friendly reminder from Giovanni Campagna that `socket` wasn't one of the io_uring opcodes). At best this is an interesting limitation of seccomp. Maybe relevant if you were using gVisor?
I should have just used on-disk mode from the start, but only now know better.
Yeah, I saw the recent post about reducing rqlite disk space usage. Using the on-disk sqlite as both the FSM and the Raft snapshot makes a lot of sense here. I'm curious whether you've had concerns about write amplification though? Because we have only the periodic Raft snapshots and the FSM is in-memory, during high write volumes we're only really hammering disk with the Raft logs.
Do you find it in the field much with Nomad? I've managed to introduce new Raft Entry types very infrequently during rqlite's 10-years of development, only once did someone hit it in the field with rqlite.
My understanding is that rqlite Raft entries are mostly SQL statements (is that right?). Where Nomad is somewhat different (and probably closer to the OP) is that the Raft entries are application-level entries. For entries that are commands like "stop this job"[0] upgrades are simple.
The tricky entries are where the entry is "upsert this large deeply-nested object that I've serialized", like the Job or Node (where the workloads run). The typical bug here is you've added a field way down in the guts of one of these objects that's a pointer to a new struct. When old versions deserialize the message they ignore the new field and that's easy to reason about. But if the leader is still on an old version and the new code deserializes the old object (or your new code is just reading in the Raft snapshot on startup), you need to make sure you're not missing any nil pointer checks. Without sum types enforced at compile time (i.e. Option/Maybe), we have to catch all these via code review and a lot of tedious upgrade testing.
it requires discipline on the part of the end-users too.
Oh for sure. Nomad runs into some commercial realities here around how much discipline we can demand from end-users. =)
[0] https://github.com/hashicorp/nomad/blob/v1.8.2/nomad/fsm.go#... [1] https://github.com/hashicorp/nomad/blob/v1.8.2/nomad/fsm.go#...
This architecture is roughly how HashiCorp's Nomad, Consul, and Vault are built (I'm one of the maintainers of Nomad). While it's definitely a "weird" architecture, the developer experience is really nice once you get the hang of it.
The in-memory state can be whatever you want, which means you can build up your own application-specific indexing and querying functions. You could just use sqlite with :memory: for the Raft FSM, but if you can build/find an in-memory transaction store (we use our own go-memdb), then reading from the state is just function calls. Protecting yourself from stale reads or write skew is trivial; every object you write has a Raft index so you can write APIs like "query a follower for object foo and wait till it's at least at index 123". It sweeps away a lot of "magic" that normally you'd shove into a RDBMS or other external store.
That being said, I'd be hesitant to pick this kind of architecture for a new startup outside of the "infrastructure" space... you are effectively building your own database here though. You need to pick (or write) good primitives for things like your inter-node RPC, on-disk persistence, in-memory transactional state store, etc. Upgrades are especially challenging, because the new code can try to write entities to the Raft log that nodes still on the previous version don't understand (or worse, misunderstand because the way they're handled has changed!). There's no free lunch.
Totally agreed. I wrote up my thoughts on this recently: https://blog.0x74696d.com/posts/mise-en-place/
GKE clusters run on VMs, not on bare metal.
I'm the lead developer for Joyent of ContainerPilot, which is the tool at the core of our Autopilot Pattern implementation examples. The lifecycle events you recognize in Distelli are definitely similar. And Chef's new tool Habitat has a supervisor that was independently developed but ended up having interesting parallels with ContainerPilot. So there's a universal idea lurking under there, which is why we called Autopilot a "Pattern" rather than a tool in itself.
But it's not clear to me from a casual glance at the the docs whether Distelli lives inside the container during those hooks? That's part of the distinction of the Autopilot Pattern is making the higher-level orchestration layer as thin as possible.
(As far as the root, some of it is derived from my experiences as a perhaps-foolishly-early adopter of Docker in prod at my previous gig at a streaming media startup. The rest is derived from both principals with which Joyent's own Triton infra is built and our experiences speaking with enterprise devs and ops teams.)
It’s not just deployment, but provides facilities for sharing code and extracting telemetry from your running apps. We believe that operations should be coupled with development, that developer tools should also be operational tools. For this reason, we’re keen to ensure that telemetry — metrics, logging, analytics, distributed tracing — is built-in from the beginning.
I think this is the most important problem IOPipe is trying to solve. Portability between platforms is a good goal but achievable with some discipline about avoiding the proprietary bits of the platform. Making "serverless" software operable to the same (or better) degree than traditional software deployment is the key.
Sending a swarm of unguided relativistic projectiles across the universe seems profoundly rude.
Yeah, that plus gvp. But if you dig into both they're just shell scripts under the hood (nice ones, though!) and given that I've got a makefile or shell script to build the container, run tests, etc. then adding a third-party tool is just one tiny bit more overhead.
Makefiles, build: docker build Ugh, more Linux-Only. Don't follow this please.
Meh. I get your sentiment here, but I build and deploy on Linux and I post about the stuff I work on. I don't post about the Windows stuff that I haven't worked on in years. And I assume a level of intelligence in my readers that they can translate whatever might be generally applicable into their own platform in the same sense that I don't grumble about Raymond Chen's `Old New Thing` not being directly applicable to my own work but still enjoy it.
The claim is that by not also solving versioning, go get is an incomplete tool. Stating that it was never intended to solve versioning is kind of beside the point.
You've summed up the entire reasoning behind my post more succinctly than I could. Thanks.
now I have to check what VCS feature it uses, wether it uses a shell script, make , or build tool Z to fetch dependencies
The core of my work is on the operations side rather than on development (although I do a good bit of that as well), so my perspective is that this initial overhead is a one-time cost for using a library, and that I can live with that.
I think it's time go devs acknowledge the fact that there is an serious issue here, instead of resorting to the ususal "you don't need that in go".
Encountering this and a lot of "well this is the way Google does is so we should too" was a big part of what inspired this post.
Author here. Happy to answer any questions about our setup.
Nice. If I was starting this project today (it's pretty mature at this point) that'd definitely be something I'd look into.
That being said, when Kinesis previewed it had only Java bindings for KCL. Not sure if that's still the case, but that'd be a limiting factor for our shop unfortunately.
The frequency isn't configurable. We execute the ingest daily for a 24 hour window that's IIRC 4 hours old. This works fine for us in practice.
Meh. It's my personal blog. I don't care enough about that. =)
Oh? Edited to say "The long story short of this is that it’s like a hosted Kafka." Thanks for the correction!
Yeah, using the Cloudfront CDN logs is a very similar approach. If we'd known about Snowplow and it was production-ready when this service were written, it would certainly be worth looking at.
That being said, this service does double-duty for us: when an analytics ping comes in from the video player we're also using that same data to write "bookmarks" for resuming play. This data has different liveness requirements than the analytics and can't be done as a nightly batch. So we'd end up having to write our own Collector (in the Snowplow architecture) anyway to perform that fan-out of incoming events.
Weird. It's #000 on #FFF, unless my meager CSS skills have failed me. Maybe it's just too thin or something? I'll see what I can do but probably not today. /points to the markdown link in sibling comment.
I host on Github pages, so the original markdown is available at https://github.com/tgross/tgross.github.io/blob/master/_post... Does that help?
Also, what kind of corporate firewall lets you have access to Github but not tech blogs? We're hiring. Just sayin'
Author here. Happy to answer any questions.