Cap'n Proto 1.0

https://capnproto.org/news/2023-07-28-capnproto-1.0.html

by kentonv • 3 years ago

720 218 3 years ago

hgsgm4• 3 years ago

Tell me about your uses of capn proto.

bkiran• 3 years ago

I'm using Cap'N Proto in a message broker application(LcuidMQ) I'm building for serialization. It has allowed me to created client applications rather quickly. There are some quirks can be difficult to wrap your head around, but once you understand it is really solid.

There are some difference between the language libraries and documentation can be lacking around those language specific solutions. I'm hoping to add blog articles and or contribute back to the example of these repositories to help future users who want to dabble.

Check out my repo here for how I use it across Rust and Python, with Golang coming soon: https://github.com/lucidmq/lucidmq

maccam912• 3 years ago

If any cloudflare employees end up here who helped decide on Capn Proto over other stuff (e.g. protobuf), what considerations went into that choice? I'm curious if the reasons will be things important to me, or things that you don't need to worry about unless you deal with huge scale.

matlin• 3 years ago

The lead dev of Cloudflare workers is the creator of Cap'n Proto so that likely made it an easy choice

mikesurowiec• 3 years ago

The post states "they used Cap’n Proto before they hired me"

saghm2• 3 years ago

The article says they were using it before hiring him though, so there must have been some prior motivation:

In fact, you are using Cap’n Proto right now, to view this site, which is served by Cloudflare, which uses Cap’n Proto extensively (and is also my employer, although they used Cap’n Proto before they hired me)

coolsunglasses• 3 years ago

I don't work at Cloudflare but follow their work and occasionally work on performance sensitive projects.

If I had to guess, they looked at the landscape a bit like I do and regarded Cap'n Proto, flatbuffers, SBE, etc. as being in one category apart from other data formats like Avro, protobuf, and the like.

So once you're committed to record'ish shaped (rather than columnar like Parquet) data that has an upfront parse time of zero (nominally, there could be marshalling if you transmogrify the field values on read), the list gets pretty short.

https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-... goes into some of the trade-offs here.

Cap'n Proto was originally made for https://sandstorm.io/. That work (which Kenton has presumably done at Cloudflare since he's been employed there) eventually turned into Cloudflare workers.

Another consideration: https://github.com/google/flatbuffers/issues/2#issuecomment-...

hgsgm4• 3 years ago

Aside from CF Workers using capn proto, how is it related to capn proto or sandstorm?

kentonv28• 3 years ago

Here's a blog post about Cloudflare's use of Cap'n Proto in 2014, three years before I joined: https://blog.cloudflare.com/introducing-lua-capnproto-better...

To this day, Cloudflare's data pipeline (which produces logs and analytics from the edge) is largely based on Cap'n Proto serialization. I haven't personally been much involved with that project.

As for Cloudflare Workers, of course, I started the project, so I used my stuff. Probably not the justification you're looking for. :)

That said, I would argue the extreme expressiveness of Cap'n Proto's RPC protocol compared to alternatives has been a big help in implementing sandboxing in the Workers Runtime, as well as distributed systems features like Durable Objects. https://blog.cloudflare.com/introducing-workers-durable-obje...

hblanks• 3 years ago

To summarize something from a little over a year after I joined there: Cloudflare was building out a way to ship logs from its edge to a central point for customer analytics and serving logs to enterprise customers. As I understood it, the primary engineer who built all of that out, Albert Strasheim, benchmarked the most likely serialization options available and found Cap'n Proto to be appreciably faster than protobuf. It had a great C++ implementation (which we could use from nginx, IIRC with some lua involved) and while the Go implementation, which we used on the consuming side, had its warts, folks were able to fix the key parts that needed attention.

Anyway. Cloudflare's always been pretty cost efficient machine wise, so it was a natural choice given the performance needs we had. In my time in the data team there, Cap'n Proto was always pretty easy to work with, and sharing proto definitions from a central schema repo worked pretty well, too. Thanks for your work, Kenton!

fullung• 3 years ago

Albert here :) Decision basically came down to the following: we needed to build a 1 million events/sec log processing pipeline, and had only 5 servers (more were on the way, but would take many months to arrive at the data center). So v1 of this pipeline was 5 servers, each running Kafka 0.8, a service to receive events from the edge and put it into Kafka, and a consumer to aggregate the data. To squeeze all of this onto this hardware footprint, I spent about a week looking for a format that optimized for deserialization speed, since we had a few thousand edge servers serializing data, but only 5 deserializing. Capnproto was a good fit :)

Timon32• 3 years ago

Congrats on the release! It must be very exciting after 10 years :)

If you don't mind the question: will there be more work on implementations for other languages in the future? I really like the idea of the format, but the main languages in our stack aren't supported in a way I'd use in a product.

kentonv28• 3 years ago

This is indeed the main weakness of Cap'n Proto. I only really maintain the C++ implementation. Other implementations come from various contributors which can lead to varying levels of completeness and quality.

Unfortunately I can't really promise anything new here. My work on Cap'n Proto is driven by the needs of my main project, the Cloudflare Workers runtime, which is primarily C++. We do interact with Go and Rust services, and the respective implementations seem to get the job done there.

Put another way, Cap'n Proto is an open source project, and I hope it is useful to people, but it is not a product I'm trying to sell, so I am not particularly focused on trying to get everyone to adopt it. As always, contributions are welcome.

The one case where I might foresee a big change is if we (Cloudflare) decided to make Cap'n Proto be a public-facing feature of the Workers platform. Then we'd have a direct need to really polish it in many languages. That is certainly something we discuss from time to time but there are no plans at present.

Timon32• 3 years ago

That's completely understandable, thank you for the answer! I'd love to try and help with at least one implementation for those languages, but there's a good chance that it would end up like the existing implementations due to lack of time.

Anyway, thank you for making it open source and for working on it all this time!

thegagne• 3 years ago

if we (Cloudflare) decided to make Cap'n Proto be a public-facing feature of the Workers platform.

How likely is this? What would be the benefits and use-cases of doing this? Would it be a standardized JS offering, or something specific to Workers that is deserialized before it hits the runtime?

doctorpangloss• 3 years ago

Hmm, the main weakness of Cap'n Proto is that you have to already know so much stuff in order to understand why it makes all the great decisions it does. The weakness you're talking about matters to me, sure, I don't use Cap'n'Proto because it lacks the same tooling as gRPC, but it is better than gRPC from an ideas point of view.

I am not going to write those language implementations, I have other stuff I need to do, and gRPC is good enough. But the people who love writing language implementations might not understand why Cap'n Proto is great, or at least not understand as well as they understand Golang and Rust, so they will rewrite X in Golang and Rust instead.

Anyway, the great ideas haven't changed in whatever it is, almost 10-15 years you've been working on this, they've been right all along. So it is really about communication.

A comment on HN that really stuck with me was like: "Man dude, this is great, but try to explain to my team that it's Not React. They won't care."

I'm just a guy, I don't know how to distill how good Cap'n Proto is. But "The Unreasonable Effectiveness of Recurrent Neural Networks" is the prototype. What is the unreasonable effectiveness of Cap'n Proto? In games, which I'm familiar with, entity component systems, user generated content and their tooling have a lot in common with Cap'n Proto. "The Unreasonable Effectiveness of ECS" is deterministic multiplayer, but that is also really poorly communicated, and thus limits adoption. Maybe you are already facing the same obstacles with Cloudflare Workers. It's all very communications related and I hope you get more adoption.

bsder• 3 years ago

There are people who have tried to write the RPC layer without it simply being a wrapper around the C++ implementation, but it's a LOT of code to rewrite for not a lot of direct benefit.

Feel free to take a crack at it. People would likely be rather cooperative about it. However, know that it's just simply a lot of work.

CodesInChaos• 3 years ago

I find it surprising how few protocols (besides Cap'n Proto) have promise pipelining. The only other example I can think of is 9p, but that's not a general purpose protocol.

https://capnproto.org/news/2013-12-13-promise-pipelining-cap...

giovannibonetti• 3 years ago

Redis transactions [1] also apply pipelining, but AFAICT there is no practical way to use them for implementing generic RPC.

[1] https://redis.com/ebook/part-2-core-concepts/chapter-4-keepi...

ackfoobar• 3 years ago

Does the pipelining in Redis allow you to have the second command depend on the result of the first command?

jayd16• 3 years ago

As neat as it is I guess it's hard optimize the backend for it compared to explicitly grouping the queries. I imagine a bespoke RPC call that results in a single SQL query is better than several pipelined but separate RPC calls, for example.

But even still, you would think it would be more popular.

kentonv28• 3 years ago

If you're thinking strictly about stateless backends that just convert every request into a SQL query, then yeah, promise pipelining might not be very helpful.

I think where it shines is when interacting with stateful services. I think part of the reason everyone tries to make everything stateless is because we don't have good protocols for managing state. Cap'n Proto RPC is actually quite good at it.

dan-robertson• 3 years ago

Without knowing how exactly capnproto promise pipelining works, when I thought about it, I was concerned about cases like reading a directory and stating everything in it, or getting back two response values and wanting to pass only one to the next call. The latter could be made to work, I guess, but the former depends on eg the number of values in the result list.

kentonv28• 3 years ago

In the actual implementation, when making a pipelined call on a result X, you actually say something like "Call X.foo.bar.baz()" -- that is, you can specify a nested property of the results which names the object that you actually want to operate on.

At present, the only operation allowed here is reading a nested property, and that seems to solve 99% of use cases. But one could imagine allowing other operations, like "take the Nth element of this array" or even "apply this call to all elements in the array, returning an array of results".

sly010• 3 years ago

Not being able to do N+1 queries would be a feature in my book.

catern• 3 years ago

I didn't know 9p had promise pipelining!

Or more specifically, it seems to have client-chosen file descriptors, so the client can open a file, then immediately send a read on that file, and if the open fails, the read will also fail (with EBADF). Awesome!

This is great, but "promise pipelining" also needs support in the client. Are there 9p clients which support promise pipelining? For example, if the user issues several walks, they're all sent before waiting for the reply to the first walk?

Also, it only has promise pipelining for file descriptors. That gives you a lot, definitely, but if for example you wanted to read every file in a directory, you'd want to be able to issue a read and then walk to the result of that read. Which 9p doesn't seem to support. (I actually support this in my own remote syscall protocol library thing, rsyscall :) )

dontlaugh2• 3 years ago

io_uring supports that too, although not a network protocol.

mananaysiempre2• 3 years ago

Last time I checked (a couple of years ago) they wanted to use eBPF to handle this sort of problem. Did they end up doing something simpler?

mananaysiempre2• 3 years ago

There is also CapnP’s moral ancestor CapTP[1]/VatTP aka Pluribus developed to accompany Mark Miller’s E language (yes, it’s a pun, there is also a gadget called an “unum” in there). For deeper genealogy—including a reference to Barbara Liskov for promise pipelining and a number of other relevant ideas in the CLU extension Argus—see his thesis[2].

(If I’m not misremembering, Mark Miller later wrote the promise proposal for JavaScript, except the planned extension for RPC never materialized and instead we got async/await, which don’t seem compatible with pipelining.)

The more recent attempts to make a distributed capability system in the image of E, like Spritely Goblins[3] and the OCapN effort[4], also try for pipelining, so maybe if you hang out on cap-talk[5] you’ll hear about a couple of other protocols that do it, if not ones with any real-world usage.

(And I again reiterate that, neat as it is, promise pipelining seems to require programming with actual explicit promises, and at this point it’s well-established how gnarly that can get.)

One idea that I find interesting and little-known from the other side—event loops and cooperatively concurrent “active objects”—is “causality IDs”[6] from DCOM/COM+ as a means of controlling reentrancy, see CoGetCurrentLogicalThreadId[7] in the Microsoft documentation and the discussion of CALLTYPE_TOPLEVEL_CALLPENDING in Effective COM[8]—I think they later tried to sell this as a new feature in Win8/UWP’s ASTAs[9]?

[1] http://erights.org/elib/distrib/captp/index.html

[2] http://erights.org/talks/thesis/index.html

[3] https://spritely.institute/goblins/

[4] https://github.com/ocapn/ocapn

[5] https://groups.google.com/g/captalk/

[6] https://learn.microsoft.com/openspecs/windows_protocols/ms-d...

[7] https://learn.microsoft.com/windows/win32/api/combaseapi/nf-...

[8] https://archive.org/details/effectivecom50wa00boxd/page/150

[9] https://devblogs.microsoft.com/oldnewthing/20210224-00/?p=10...

cyberax• 3 years ago

I find it surprising how few protocols (besides Cap'n Proto) have promise pipelining.

Pipelining is a bad idea. It reifies object instances, and thus makes robust implementation much harder. You no longer make stateless calls, but you are running functions with particular object instances.

And you immediately start getting problems. Basically, Client Joe calls Service A and then pass the promised result of the call to Service B. So that Service B will have to do a remote call to Service A to retrieve the result of the promise.

This creates immediate complications with security boundaries (what is your delegation model?). But what's even worse, it removes the backpressure. Client Joe can make thousands of calls to Service A, and then pass the not-yet-materialized results to Service B. Which will then time out because Service A is being DDoS-ed.

dannyobrien• 3 years ago

I'm excited by Cap'n Proto's participation in the OCAPN standardization effort. Can you speak to if that's going to be part of the Cap'n Proto 2.0 work?

https://github.com/ocapn/ocapn

kentonv28• 3 years ago

Sadly, the person leading that participation, Ian "zenhack" Denhardt, recently and unexpectedly passed away.

For my part, I'm a fan of OCapN, but I am not sure how much time I can personally commit to it, with everything on my plate.

I wish I had better news here. This was a tragic loss for all of us.

the_common_man• 3 years ago

That's tragic. Was he also not helping maintain sandstorm? Would appreciate a blog post or a note about him there

bryanlarsen• 3 years ago

The guy making Tempest? That is tragic and deserves an HN front page. Is there an obituary or good link you can submit and we can upvote? Or would Ian have preferred not to have such?

omginternets7• 3 years ago

Kenton, I'm @lthibault on GitHub. I was working closely with Ian on the Go capnp implementation, and I would be happy to take over this initiative. Can you point me in the right direction?

Also, are you on Matrix or Telegram or something of the sort? I was hoping I could ping you with the occasional question as I continue work on go-capnp.

batch12• 3 years ago

I know this isn't new, but I wonder if the name is an intentional nod to Star Trek Voyager or is there another reference I'm not aware of.

https://memory-alpha.fandom.com/wiki/Captain_Proton

azornathogron• 3 years ago

Given that it's billed as a "cerealization protocol", I always assumed it was a reference to Cap'n Crunch cereal.

kentonv28• 3 years ago

Huh, that reference actually never occurred to me.

The name Cap'n Proto actually originally meant "Capabilities and Protobufs" -- it was a capability-based RPC protocol based on Protocol Buffers. However, early on I decided I wanted to try a whole different serialization format instead. "Proto" still makes sense, since it is a protocol, so I kept the name.

The pun "cerealization protocol" is actually something someone else had to point out to me, but I promptly added it to the logo. :)

mi_lk• 3 years ago

What does capacity-based mean in this context?

IshKebab2• 3 years ago

Great achievement. To be honest I wouldn't recommend Capnp. The C++ API is very awkward.

The zero copy parsing is less of a benefit than you'd expect - pretty unlikely you're going to want to keep your data as a Capnp data structure because of how awkward it is to use. 99% of the time you'll just copy it into your own data structures anyway.

There's also more friction with the rest of the world which has more or less settled on Protobuf as the most popular binary implementation of this sort of idea.

I only used it for serialisation. Maybe the RPC stuff is more compelling.

I really wish Thrift had taken off instead of Protobuf/gRPC. It was so much better designed and more flexible than anything I've seen before or since. I think it died mainly due to terrible documentation. I guess it also didn't have a big name behind it.

mgaunard• 3 years ago

You mean flatbuffers, not protobuf.

It has established itself as the de-facto standard, with a few other places using SBE instead.

In any case the main problems with binary serialization are:

- schemas and message version management

- delta-encoding

If you ignore these, flat binary serialization is trivial.

No library provides a good solution that covers the two points above.

kentonv28• 3 years ago

What part of the industry are you in where flatbuffers is seen as the de facto standard? Personally I've never randomly encountered a project using flatbuffers. I see protobuf all the time.

(I've randomly run into Cap'n Proto maybe 2-3 times but to be fair I'm probably more likely to notice that.)

dtech6• 3 years ago

Protobuf is very widely used, I just had to Google flatbuffers...

Rapzid3• 3 years ago

I find MessagePack to be pretty great if you don't need schema. JSON serialization is unreasonably fast in V8 though and even message pack can't beat it; though it's often faster in other languages and saves on bytes.

alfalfasprout• 3 years ago

Except messagepack is really slow...

imtringued• 3 years ago

MessagePack is obsolete. Use CBOR now.

kentonv28• 3 years ago

I do agree that the API required for zero-copy turns out a bit awkward, particularly on the writing side. The reading side doesn't look much different. Meanwhile zero-copy is really only a paradigm shift in certain scenarios, like when used with mmap(). For network communications it doesn't change much unless you are doing something hardcore like RDMA. I've always wanted to add an optional alternative API to Cap'n Proto that uses "plain old C structures" (or something close to it) with one-copy serialization (just like protobuf) for the use cases where zero-copy doesn't really matter. But haven't gotten around to it yet...

That said I personally have always been much more excited about the RPC protocol than the serialization. I think the RPC protocol is actually a paradigm shift for almost any non-trivial use case.

foobiekr2• 3 years ago

One thing about google proto is that, at least in many languages, every message throws off a ton of garbage that stresses the GC. On the send side, you can obviously re-use objects, but on the receive side no.

cmrdporcupine6• 3 years ago

I've always been excited about zero copy messages in the context of its potential in database systems; the thought of tuples working their way all the way from btree nodes in a pager, to query results on the network without copies seems fantastic.

But every time I've tried to prototype or implement around this model I've run into conceptual blocks. It's a tricky paradigm to fully wrap one's head around, and to squeeze into existing toolsets.

nvarsj• 3 years ago

fbthrift is still alive and kicking.

s17n• 3 years ago

It’s a testament to the subtlety of software engineering that even after four tries (protobuf 1-3, capn proto 1) there are still breaking changes that need to be made to the solution of what on the surface appears to be a relatively constrained problem.

kentonv28• 3 years ago

Of course, nothing is ever "solved". :)

I assume you are talking about the cancellation change. This is interesting, actually. When originally designing Cap'n Proto, I was convinced by a capabilities expert I talked to that cancellation should be considered dangerous, because software that isn't expecting it might be vulnerable to attacks if cancellation occurs at an unexpected place. Especially in a language like C++, which lacks garbage collection or borrow checking, you might expect use-after-free to be a big issue. I found the argument compelling.

In practice, though, I've found the opposite: In a language with explicit lifetimes, and with KJ's particular approach to Promises (used to handle async tasks in Cap'n Proto's C++ implementation), cancellation safety is a natural side-effect of writing code to have correct lifetimes. You have to make cancellation safe because you have to cancel tasks all the time when the objects they depend on are going to be destroyed. Moreover, in a fault-tolerant distributed system, you have to assume any code might not complete, e.g. due to a power outage or maybe just throwing an unexpected exception in the middle, and you have to program defensively for that anyway. This all becomes second-nature pretty quick.

So all our code ends up cancellation-safe by default. We end up with way more problems from cancellation unexpectedly being prevented when we need it, than happening when we didn't expect it.

EDIT: Re-reading, maybe you were referring to the breaking changes slated for 2.0. But those are primarily changes to the KJ toolkit library, not Cap'n Proto, and is all about API design... I'd say API design is not a constrained problem.

binary1322• 3 years ago

I always liked the idea of capnp, but it bothers me that what is ultimately a message encoding protocol has an opinion on how I should architect my server.

FWIW, gRPC certainly has this problem too, but it’s very clearly distinct from protobuf, although pb has gRPC-related features.

That entanglement makes me lean towards flatbuffers or even protobuf every time I weigh them against capnp, especially since it means that fb and pb have much simpler implementations, and I place great value on simplicity for both security and maintenance reasons.

I think the lack of good third-party language implementations speaks directly to the reasonability of that assessment. It also makes the bus factor and longevity story very poor. Simplicity rules.

insanitybit15• 3 years ago

How does the serialization layer impact your rpc choice?

cmrdporcupine6• 3 years ago

Cap'N'Proto comes with a (quite good) RPC facility. Based on asynchronous promises and grounded in capabilities.

You don't have to use it. You could just use it just as a 'serialization' layer but if you're writing services you could be missing half the advantage, really. And if you're writing in C++ you'll end up having to use their KJ library anyways.

If you take the whole package the zero copy, capability-security, and asynchrouny (a word I just coined!) all fit together nicely.

cmrdporcupine6• 3 years ago

Part of the problem with cap'n'proto whenever I've approached it is that not only does it have an opinion on how to architect your server (fine, whatever) but in C++ it ends up shipping with its own very opinionated alternative to the STL ("KJ") and when I played with it some years ago it really ended up getting its fingers everywhere and was hard to work into an existing codebase.

The Rust version also comes with its own normative lifestyle assumptions; many of which make sense in the context of its zero-copy world but still make a lot of things hard to express, and the documentation was hard to parse.

I tend to reach for flatbuffers instead, for this reason alone.

Still I think someday I hope to have need and use for cap'n'proto; or at least finish one of several hobby projects I've forked off to try to use it over the years. There's some high quality engineering there.

kentonv28• 3 years ago

Yes, it's true, the C++ implementation has become extremely opinionated.

I didn't initially intend for KJ to become as all-encompassing as it has. I guess I kept running into things that didn't work well about the standard library, so I'd make an alternative that worked well, but then other parts of the standard library would not play nicely with my alternative, so it snowballed a bit.

At the time the project started, C++11 -- which completely changed the language -- was brand new, and the standard library hadn't been updated to really work well with the new features.

The KJ Promise library in particular, which made asynchronous programming much nicer using the newly-introduced lambdas, predated any equivalent landing in the standard library by quite a bit. This is probably the most opinionated part of KJ, hardest to integrate with other systems. (Though KJ's even loop does actually have the ability to sit on top of other event loops, with some effort.)

And then I ended up with a complete ecosystem of libraries on top of Promises, like KJ HTTP.

With the Workers Runtime being built entirely in that ecosystem, it ends up making sense for me to keep improving that ecosystem, rather than try to make things work better across ecosystems... so here we are.

dtech6• 3 years ago

While I never used Cap'n Proto, I want to thank kentonv for the extremely informative FAQ answer [1] on why required fields are problematic in a protocol

I link it to people all the time, especially when they ask why protobuf 3 doesn't have required fields.

[1] https://capnproto.org/faq.html#how-do-i-make-a-field-require...

kccqzy3• 3 years ago

This is some very valuable perspective. Personally, I previously also struggled to understand why. For me, the thing that clicked was to understand protobuf and Cap'n proto as serialization formats that need to work across API boundaries and need to work with different versions of their schema in a backwards- and forwards-compatible way; do not treat them as in-memory data structures that represent the world from the perspective of a single process running a single version without no compatibility concerns. Thus, the widely repeated mantra of "making illegal states unrepresentable" does not apply.

chubot3• 3 years ago

Rich Hickey (creator of the Clojure language) has a good talk "Maybe Not" that touches on these issues, with a nice way of explaining it

https://www.youtube.com/watch?v=YR5WdGrpoug

The capnproto link explains it concretely in terms of a message bus example, which is useful.

But more abstractly you can think of the shape of data (aka schema, names and types) and field presence (optional/required) as separate things

https://lobste.rs/s/zdvg9y/maybe_not_rich_hickey

First, being valid or invalid with respect to a static type system is a GLOBAL property of program -- writing a type checker will convince you of that. And big distributed systems don't have such global properties: https://news.ycombinator.com/item?id=36590799

If they did, they'd be small :) Namely you could just reboot the whole thing at once. You can't reboot say the entire Internet at once, and this also holds for smaller systems, like the ones at say Google (and I'm sure Cloudflare, etc.).

So the idea is that the shape/schema is a GLOBAL property -- you never want two messages called foo.RequestX or two fields called "num_bar" with different types -- ever, anywhere.

But optional/required is LOCAL property. It depends on what version of a schema is deployed in a particular binary. Inherently, you need to be able to handle a mix of inconsistent versions running simultaneously.

---

To be pedantic, I woulds say "making illegal states unrepresentable" DOES apply, but you can't do it in a STATIC type system. [1] Your Maybe<T> type is not useful for data that crosses process boundaries.

A distributed system isn't a state machine.

1. Lamport showed us one important reason why: the relative order of messages means that there is no globally coherent state. You need something like Paxos to turn a distributed system back into a state machine (and this is very expensive in general)

2. The second reason is probably a consequence of the first. You can think of deploying a binary to a node as a message to that node. So you don't have a completely consistent state -- you always have an in-between state, a mix of versions. And presumably you want your system to keep working during this time period :)

And that coarse-grained problem (code versioning and deployment) implies the fine-grained problem (whether a specific message in a field is present). This is because protobufs generate parsers with validation for you -- or they used to!

---

tl;dr Think of the shape of data (aka schema) and field presence (optional/required) as different dimensions of data modeling. Maybe<T> mixes those up, which is fine in a single process, but doesn't work across processes.

---

[1] A very specific example of making illegal states unrepresentable without static types - my Oils project uses a DSL for algebraic data types, borrowed from CPython. The funny thing is that in CPython, it generates C code, which doesn't have any static notion of Maybe<T>. It has tagged unions.

And in Oils we first generated dynamically typed Python at first. Somewhat surprisingly, algebraic data types are STILL useful there.

Now the generated code is statically typed with MyPy (and with C++), and we do pleasant type-driven refactorings. But using algebraic data types were still extremely useful before static typing. They made illegal states unrepresentable -- but you would get the error at runtime.

3cats-in-a-coat9• 3 years ago

Can't we extend this argument to eliminating basically all static typing? And frankly that'd not even be wrong, and is why Alan Kay defined OOP as one that's dynamically typed and late bound, and we went against it anyway to keep relearning the same lessons over and over.

kentonv28• 3 years ago

The argument is really more like: Always defer validation until the point where the data is actually consumed, because only the consumer actually knows what is valid.

Which is definitely a counterpoint to the oft-stated argument that you should validate all data upfront.

Either way though, you can still have types, the question is just when and where (in a distributed system, especially) they should be checked.

insanitybit15• 3 years ago

Can't we extend this argument to eliminating basically all static typing?

No, because static typing exists in all sorts of places. This argument is primarily about cases where you're exchanging data, which is a very specific use case.

kccqzy3• 3 years ago

Do you need to make different versions of a program exchange information even though they do not agree on the types? No? Then this argument cannot be extended this way.

mrkeen• 3 years ago

It's up to you.

It's easy to imagine any statically typed language having a general-purpose JSON type. You could imagine all functions accepting and returning such objects.

Now it's your turn to implement the sum(a,b) function. Would you like to allow the caller to pass anything in as a and b?

chubot3• 3 years ago

See my sibling comment, e.g. with respect to Rich Hickey's framing - https://news.ycombinator.com/item?id=36911033

nly4• 3 years ago

Avro solves this problem completely, and more elegantly with its schema resolution mechanism. Exchanging schemas at the beginning of a connection handshake is hardly burdensome

dtech6• 3 years ago

If by "solving" you mean "refuse to do anything at all unless you have the exact schema version of the message you're trying to read" then yes. In a RPC context that might even be fine, but in a message queue...

I will never use Avro again on a MQ. I also found the schema resolution mechanism anemic.

Avro was (is?) popular on Kafka, but it is such a bad fit that Confluent created a whole additional piece of infra called Schema Registry [1] to make it work. For Protobuf and JSON schema, it's 90% useless and sometimes actively harmful.

I think you can also embed the schema in an Avro message to solve this, but then you add a massive amount of overhead if you send individual messages.

[1] https://docs.confluent.io/platform/current/schema-registry/i...

kentonv28• 3 years ago

Exchanging schemas at the beginning of a connection handshake is hardly burdensome.

I dunno, that sounds extremely burdensome to me, especially if the actual payload is small.

And how exactly does exchanging schemas solve the problem? If my version of the schema says this field is required but yours says it is optional, and so you don't send it, what am I supposed to do?

throwboatyface• 3 years ago

Disagree. Avro makes messages slightly smaller by removing tags, but it makes individual messages completely incomprehensible without the writer schema. For serializing data on disk it's fine and a reasonable tradeoff to save space, but for communication on the wire tagged formats allow for more flexibility on the receiver end.

The spec for evolving schemas is also full of ambiguity and relies on the canonical Java implementation. I've built an Avro decoder from scratch and some of the evolution behaviour is counter-intuitive.

alphanullmeric• 3 years ago

Rustaceans in shambles

oftenwrong• 3 years ago

Typical provides "asymmetric" fields to assist with evolution of types:

https://github.com/stepchowfun/typical#asymmetric-fields-can...

To help you safely add and remove required fields, Typical offers an intermediate state between optional and required: asymmetric. An asymmetric field in a struct is considered required for the writer, but optional for the reader. Unlike optional fields, an asymmetric field can safely be promoted to required and vice versa.

skybrian5• 3 years ago

Yeah, it only works for migrations in fairly closed systems where you can upgrade or delete all the old data, though.

AtNightWeCode• 3 years ago

Very good point.

A gotcha along the same path. Deserialization of things not needed as what you get with generated clients. An aspect of interfaces in Go I really like. Remotely type what I use. Skip the rest. Not fun to have incidents caused by changes to a contract that is not even used by a service. Also hard to find.

throw14082020• 3 years ago

From the FAQ [1]

The right answer is for applications to do validation as-needed in application-level code.

It would've been nice to include a parameter to switch "required message validation" on and off, instead of relying on application code. Internally in an application, we can turn this off, the message bus can turn it off, but in general, developers would really benefit from this being on.

[1] https://capnproto.org/faq.html#how-do-i-make-a-field-require...

kaba0• 3 years ago

That FAQ answer has a very nice parallel with Hickey’s video of a similar topic: https://m.youtube.com/watch?v=YR5WdGrpoug&feature=youtu.be

insanitybit15• 3 years ago

Any plans to improve the Rust side of things? The API could definitely use some more work/ docs around it.

dwrensha• 3 years ago

I intend to continue work on capnproto-rust, at my own pace and according to my own priorities.

Are there any particular pain points that you want to call attention to?

shdh• 3 years ago

We have a great plethora of binary serialization libraries now, but I've noticed none of them offer the following:

* Specification of the number of bits I want to cap out a field at during serialization, ie: `int` that only uses 3 bits.

* Delta encoding for serialization and deserialization, this would further decrease the size of each message if there is an older message that I can use as the initial message to delta encode/decode from.

jeffbee• 3 years ago

`int` that only uses 3 bits.

CBOR approximates this, since it has several different widths for integers.

an older message that I can use as the initial message to delta encode/decode from.

General-purpose compression on the encoded stream would do something toward this goal, but some protocol buffers library implementations offer merge functions. The question is what semantics of "merge" you expect. For repeated fields do you want to append or clobber?

Dylan168072• 3 years ago

CBOR has different integer widths, yes, but 1/2/3/5/9 bytes is not particularly flexible.

And if someone is asking for exact bit sizes and deltas, they probably don't want a format that embeds every key in every message.

no_circuit• 3 years ago

Take a look at FAST protocol [1]. It has been around for a while. Was created for market/trading data. There appears to be some open source implementations, but I don't think in general they'd be maintained well since trading is, well, secretive.

[1] https://en.wikipedia.org/wiki/FAST_protocol

IshKebab2• 3 years ago

Most formats use varints, so you can't have a 3-bit int but they will store a 64-bit int in one byte if it fits. Going to smaller than a byte isn't worth the extra complexity and slowness. If you're that space sensitive you need to add proper compression.

By delta compression you mean across messages? Yeah I've never seen that but it's hard to imagine a scenario where it would be useful and worth the insane complexity.

wichert• 3 years ago

zserio [1] has the former at least. It isn't intended for the same use cases as protobuf/capnproto/flatbutter though; in particular it has no backward or forwards compatibility. But it's great for situations where you know exactly what software is used on both ends and you need small data and fast en-/decoding.

[1] http://zserio.org/doc/ZserioLanguageOverview.html#bit-field-...

wiml• 3 years ago

One thing I liked about Ada, the small amount I used it, is it has actual subtypes: you could define a variable as an integer within a specific range, and the compiler would (presumably) choose an appropriate underlying storage type for it.

synthetigram• 3 years ago

After exploring a few constant access serialization formats, I had to pass on Capn Proto in favor of Apache Avro. Capn has a great experience for C++ users, but Java codegen ended up being too annoying to get started with. If Capn Proto improved the developer experience for the other languages people write, I think it would really help a lot.

vicaya• 3 years ago

Worker should really adopt Apache Arrow, which has a much bigger ecosystem.

https://github.com/apache/arrow

Timothycquinn• 3 years ago

Congrats in 10 years! Question: Can Cap'n Proto be used as an alternative to Python Pickle library for serializing and de-serializing python object structures?

kentonv28• 3 years ago

If your goal is to serialize an arbitrary Python object, Pickle is the way to go. Cap'n Proto requires you to define a schema, in Cap'n Proto schema language, for whatever you wan to serialize. It can't just take an arbitrary Python value.

emtel• 3 years ago

I used capnproto years ago as the network serialization format for a multiplayer RTS game. Although the API can be quite awkward, it was overall a joy to use and I wish I was able to use it in more projects.

richardfey• 3 years ago

Amazing to see Cap'n Proto come this far! I wonder how easy it would be to swap it for gRPC, and still have advanced load balancing support for it.

hiddencost• 3 years ago

For context: Kenton ran Google's in house proto system for many years, before leaving and building his own open source version.

AceJohnny2• 3 years ago

I believe he was the one who open-sourced protobufs.

omginternets7• 3 years ago

I have some very unfortunate news to share with the Cap'n Proto and Sandstorm communities.

Ian Denhardt (zenhack on HN), a lead contributor to the Go implementation, suddenly and unexpectedly passed away a few weeks ago. Before making a request to the community, I want to express how deeply saddened I am by this loss. Ian and I collaborated extensively over the past three years, and we had become friends.

As the de facto project lead, it now befalls me to fill Ian's very big shoes. Please, if you're able to contribute to the project, I could really use the help. And if you're a contributor or maintainer of some other implementation (C++, Rust, etc.), I would *REALLY* appreciate it if we could connect. I'm going to need to surround myself with very smart people if I am to continue Ian's work.

RIP Ian, and thank you. I learned so much working with you.

------

P.S: I can be reached in the following places

- https://github.com/lthibault

- https://matrix.to/#/#go-capnp:matrix.org

- Telegram: @lthibault

- gmail: louist87

pja• 3 years ago

It seems @zenhack maintained the Haskell bindings as well.

omginternets7• 3 years ago

I think the Haskell project was a complete implementation, not just bindings. But yes, Ian was truly prolific.

jcalabro• 3 years ago

Oh gosh, I didn't know that. Thank you for sharing :( I really loved his blog. That's awful.

omginternets7• 3 years ago

Indeed. His blog was outstanding. It might be a good idea to mirror his blog before the domain expires, as it's a real treasure-trove.

freedomben• 3 years ago

I've had a couple people suddenly taken from me, and it is soul crushing. Every time it happens it reminds me of how fragile life is, and how quickly things can change. I've started trying to enjoy the small things in life more, and while I don't neglect the future, I also try to enjoy the present.

He has left an amazing legacy that has touched a lot of people. RIP Ian.

omginternets7• 3 years ago

Every damn day, something comes up that makes me go, "Oh, I should ask Ian about tha-"

It really sucks. And I know exactly what you mean.

doh• 3 years ago

That is really sad news. Ian was an inspiration. Sorry for your loss and the loss of the whole community. He will be greatly missed.

yoz• 3 years ago

I’m so sad to hear this. I didn’t know him but hugely admired the work he did on Tempest (his recent fork of Sandstorm, attempting to revive the project). Thank you for letting us know.

unixhero• 3 years ago

Congratulations to kentonv and the team

jFriedensreich• 3 years ago

i love how the main reference for workerd can be just one capnp file.

https://github.com/cloudflare/workerd/blob/main/src/workerd/...

this changed my world how i think about computing on the web.

if there was just a good enough js library as for lua and you could directly send capnp messages to workerd instead of always going through files. I guess one day i have to relearn c++ and understand how the internals actually work.

zX41ZdbW2• 3 years ago

We support CapNProto as an import/export format in ClickHouse.

It was contributed by Marek Vavruša from CloudFlare in 2017: https://github.com/ClickHouse/ClickHouse/pull/1387

Since then, we extended and improved the support: added it for export (initially, it was only for import) and improved the performance.

About strange stuff in the library - it uses a non-obvious approach for exception handling, and their C++ code feels like they too much focusing on some non-orthodox approaches.

Xeoncross• 3 years ago

[2013] Show HN: Cap'n Proto, by the ex-maintainer of Protocol Buffers: https://news.ycombinator.com/item?id=5482081

xvilka• 3 years ago

I am trying to revive C implementation [1]. Any help is welcome.

[1] https://github.com/opensourcerouting/c-capnproto

up2isomorphism• 3 years ago

Some of these “high perf” RPC libraries never get one key point, if I really need something to be fast, the most important aspect is that it must be simple.

jviotti• 3 years ago

If you are interested in learning more about binary serialization (for Cap'n Proto and others), I wrote two free-to-read papers extensively looking at the scene from a space-efficiency point of view that people here might be interested in:

- https://arxiv.org/abs/2201.02089

- https://arxiv.org/abs/2201.03051

Not something that aims to compete with Cap'n Proto (different use cases), but I've been also working on a binary serialization format that is pure JSON-compatible with a focus on space-efficiency: https://www.jsonbinpack.org.

Benchmarks here: https://arxiv.org/abs/2211.12799

Cap'n Proto 1.0

Related Stories