In Depth Talk on Performance and Serialization (Protobuf, json, Thrift, etc.)

http://www.infoq.com/presentations/Dealing-with-Performance-Challenges-Optimized-Data-Formats

by peterb • 14 years ago

31 9 14 years ago

trjordan• 14 years ago

Direct link to slides: http://qconsf.com/dl/qcon-sanfran-2011/slides/SastryMalladi_...

I didn't see anything in the slides about compressing (gzip) any of the non-binary formats (json, xml, etc). It would be interesting to see those in a comparison like this too.

Also would have been neat to see tnetstrings <or insert favorite format here>. ;)

BlueZeniX2• 14 years ago

I seriously wonder how it's possible that in his tests JSON got smaller than MsgPack.

MsgPack is so close to JSON in structure, but very compact (1 byte type header, small numbers type and payload combined) it doesn't make any sense.

stock_toaster3• 14 years ago

Consider for example a message with a double in it. A double in msgpack is 9 bytes (according to their spec[1]). In json it is just however many bytes it takes to represent the "number" in ascii. So if the number happens to be a very small double, such as 3.0, it may be just 3 bytes to store the double in json (depending on the encoder?), as apposed to 9 bytes for msgpack. Something similar could be said for large intergers too. '3' is only one byte in json, but would be 5 bytes in msgpack when trying to encode an int32.

That something similar is occurring to the messages in the talk, is the only explanation I could think of anyway... Looking at the thrift description, it does appear that there are int32s and doubles in the messages.

[1] http://wiki.msgpack.org/display/MSGPACK/Format+specification...

saurik2• 14 years ago

One thing that is important to point out that is that JavaScript (and thereby JSON) doed not separate integers from floating-point numbers; so if you have the double 3.0 it will be represented in JSON as "3", not "3.0". If MessagePack does not have a mechanism for dropping down to int32 for integral doubles, then you will actually see a 9:1 difference for this specific case (although if you are often storing integers in your doubles maybe you should be using int32 anyway, in which case 3.1 may have been a better general example than 3.0 ;P).

BlueZeniX2• 14 years ago

True.

The msgpack serializer was probably used in a suboptimal way. The integer 3 can be encoded by msgpack in one byte as well. I never use doubles which is probably why I didn't think of this case.

Integers are actually always <= bytes in msgpack than the JSON ascii representation.

Anyway - benchmarking is hard.

chad_walters• 14 years ago

I recommend that anyone interested in this topic look closely at the raw data provided in the deck on slides 38-41.

In Depth Talk on Performance and Serialization (Protobuf, json, Thrift, etc.)

Related Stories