dweis

I believe that the reason for this limitation is that not all languages can represent open enums cleanly to gracefully handle unknown enums upon schema skew.

These are simply tradeoffs. For example, one can make very fast protobuf parsers/serializers if you prioritize that over the ergonomics of the generated code.

Yes, they are both varint encoded on the wire. Refer to https://developers.google.com/protocol-buffers/docs/encoding...

Minor nit, but not necessarily. For basically all values of SomeMessage, dec should fail to parse due to improperly encoded UTF8 data for field 2 (modulo some proto2 vs. proto3 and language binding implementation differences).

Change field 2 to a bytes field instead of a string field and then yes.

I don't think this is the case, or at least, I'd expect it to be a bug.

Protocol Buffers should generally be non-destructive of the underlying data. That means even if it encounters the wrong wire type for a field, it should simply retain that value in the unknown field set rather than discard it.

How do you handle binary rollbacks and rollouts safely with an "everything is required" approach? Do you force binaries to roll out in a strict order with appropriate soak time at each layer? How does that affect developer velocity?

Hi there,

I'm an actual author of Protocol Buffers :)

I think Sandy's analysis would benefit from considering why Protocol Buffers behave the way they do rather than outright attacking the design because it doesn't appear to make sense from a PL-centric perspective. As with all software systems, there are a number of competing constraints that have been weighed that have led to compromises.

- D

P.S. I also don't believe the personal attacks to be warranted or productive.

Thanks for pointing these out! This sort of archaeology is always interesting. I tend to have to do it for Protocol Buffers too to understand various design decisions.

Disclaimer: I have written and designed many parts of the Protocol Buffer libraries.

https://tools.ietf.org/html/rfc1832.html#section-6 has a good encoding example of XDR.

Contrasting that with Protocol Buffers is enlightening as it clearly demonstrates some differences in design goals and where tradeoffs are made. Feel free to correct my interpretations as I may have missed something!

1. XDR appears to have no equivalent of a Protocol Buffer field number => the format is not self describing. That is, one must have a schema to properly interpret the data

2. XDR appears to encode lengths as fixed width based on the block size => faster to read/write but larger on the wire than using a varint encoding

3. XDR's string data type is defined as ASCII => it does not support modern unicode outside of the variable-length opaque type

1 would seem to present difficulty for modern distributed systems as one cannot control the release process of every distinct binary in the ecosystem to ensure that they are all schema equivalent at the same time. This can be remediated by propagating the schema as a header to the underlying data for consumption on the other side, but that bloats the wire format. Header compression could help with this but may be problematic for systems with severely constrained networks that constantly reestablish new connections (ex. mobile).

1 also has implications for data persistence. One cannot ever remove or reorder an XDR struct member else they will incorrectly parse data that was written in the older format. This is in contrast with Protocol Buffers, where one can remove or reorder message members whenever they'd like, as long as they take care not to reuse a tag number (and the newer `reserved` feature can help with that).

2 is just a performance tradeoff: binary size or (de|en)code performance?

3 has implications for memory constrained systems. Ex. on Android we eagerly parse string fields to avoid doubling the allocation overhead (first as raw bytes, then as a String object). If we required all string datatypes in Protocol Buffers to be defined as bytes fields (the variable-length opaque data type equivalent), we wouldn't be able to provide this optimization.

Overall, XDR looks like a good fit for inter-process communication in a homogeneous environment. Protocol Buffers looks like it's a good fit for cross language communication across heterogenous and unversioned environments. Directly comparing the two, XDR is much more verbose on the wire (particularly if we mitigate the versioning issues by serializing the schema in a header) whereas it's likely significantly faster to (en|de)code. i.e. there's a tradeoff for networking/storage costs vs. CPU performance.

Scott makes a bunch of provocative declarations in his post but I think many of them betray a lack of background to appropriately understand the tradeoffs involved. As illustrated above, Protocol Buffers makes a bunch of design affordances for compactness on the wire which XDR does not accommodate. He also believes Google's RPC system to be "unarguably shitty" even though it has never been open sourced due to dependency issues (what is open sourced as part of Protocol Buffers is a shim, gRPC is the future here). His impression of why Facebook built Thrift is similarly misinformed as Protocol Buffers was not open source when Thrift was written.

We use LevelDB.

Yes.

CMU's 15-440 Distributed Systems course did this over a year ago. See http://www.cs.cmu.edu/~dga/15-440/F12/

The professor, David Andersen, catalogued his experience on Google Plus: https://plus.google.com/115237092509505721130/posts/DKVcPLbq... https://plus.google.com/115237092509505721130/posts/AQqe7Xay... https://plus.google.com/115237092509505721130/posts/d2RE6gbj... https://plus.google.com/115237092509505721130/posts/3PFz3RUe...

This already existed for NYC: http://moredrunk.com