Cap'n Proto beta release 0.1 is ready for Real Work

http://kentonv.github.io/capnproto/news/2013-06-27-capn-proto-beta-release.html

by kentonv • 13 years ago

105 19 13 years ago

kentonv.github.io

vosper• 13 years ago

Link to the previous discussion on HN in April:

https://news.ycombinator.com/item?id=5482081

Great to see this project is moving forward, can you comment on whether (and which of) the criticisms and suggestions from the original HN discussion have been integrated or have influenced development for this release?

kentonv7• 13 years ago

Oh boy, there were so many suggestions, I'd have to read through them all to remember which ones I ended up incorporating. There's so much text, though...

Off the top of my head, I remember that multiple people suggested that text should be allowed to contain NUL characters (not just at the end), which I agreed to. That's not terribly significant, though.

People involved on the mailing list have helped me with several design decisions. One contributor convinced me that my original syntax for unions was pretty stupid and helped me come up with something much better.

At another point, I implemented "inline" structs and lists (which are embedded into their parent without a pointer). This turned out to be overcomplicated for not all that much benefit and I was encouraged to ditch it. A week of work lost, but it was the right decision.

Contributors hacking on implementations in other languages made clear that they weren't interested in writing their code generators in Haskell, so the compiler now has a plugin system allowing code generators in any language.

This is a very incomplete list, obviously... I'd love to have more people telling me what I'm doing wrong. :)

Scaevolus• 13 years ago

Using colons to denote types is still ugly and feels unnecessary.

Your last annotation example confuses the @field and $annotation syntax-- string $0 :Text $qux;

simcop23872• 13 years ago

Off the top of my head, I remember that multiple people suggested that text should be allowed to contain NUL characters (not just at the end), which I agreed to. That's not terribly significant, though.

This seems contrary to what the docs[1] say: > Text is always UTF-8 encoded and NUL-terminated.

Which is right?

[1] http://kentonv.github.io/capnproto/language.html

jcarden• 13 years ago

Nice!

RyanZAG• 13 years ago

Can anybody explain how/why this type of data transfer is secure? By the sound of it, the main benefit is that we can just read a large file by using mmap on it. Are there not a lot of security considerations to doing that...? Especially when feeding this through into dynamic languages like Python.

kentonv7• 13 years ago

Cap'n Proto doesn't allow you to transmit arbitrary native structs. It still has a defined message format, with pointers that can be (and are) validated by the receiver. It's just that the format is one that happens to be very convenient for typical CPUs to access, in particular without the need for any copying or decoding.

In practice, the current implementation hasn't undergone security review yet, so I wouldn't recommend using it on data you don't trust. But it's intended to be secure, and will be by version 1.0.

DannoHung2• 13 years ago

With respect to how imports and the dynamic API: When/If you do get around to implementing the parser in C++, Could you consider making the import statement relative to the path of the file that is currently being dynamically imported? In the Protocol Buffer runtime, it's relative to the path of the top level file being imported which makes sub-files have to know about the path they're located at in order to import anything else.

kentonv7• 13 years ago

Actually, that's already how it works. :) Imports are relative to the importing file unless they start with a '/', in which case the import path is searched.

The protobuf approach was somewhat necessitated by the fact that various parts of the protobuf implementation expected file names to be canonical, e.g. so that it could tell if two descriptors were for the same file by comparing the names. This mean it was necessary for the compiler to know where the top of the source tree was and dealing with relative paths would have been pretty ugly. This worked fine for Google-internal usage, but was problematic for a lot of open source users.

Cap'n Proto does not use file names as canonical identifiers. Instead, it requires every file to assign itself a unique ID (a random 64-bit number). This introduces some other problems, but on the whole I think it comes out a lot nicer.

DannoHung2• 13 years ago

Woo! Awesome!

ocharles• 13 years ago

Now I'm just eagerly waiting a Haskell and Perl library. Suppose that means I should get off my lazy ass and start contributing!

kentonv7• 13 years ago

Yes, please! :D

boothead• 13 years ago

Have a look at alphaheavy's protobuf implementation as a starting point.

enigmo• 13 years ago

Or just send some pull requests!

I think the core of our protobuf library should support many different encoding formats. There is often no need to define and redfine types to support multiple wire formats: Protobuf, Thrift, Cap'n Proto, etc. One definition is enough for most.

https://github.com/alphaHeavy/protobuf

_pmf_• 13 years ago

Allowing to write additional code generators as plugins in a language agnostic way is really nice.

Cap'n Proto beta release 0.1 is ready for Real Work

Related Stories