MessagePack: Fast and Compact Serialization

http://msgpack.org/

by tilt • 15 years ago

50 15 15 years ago

msgpack.org

xal• 15 years ago

We use this as data exchange format for our RPC services at Shopify. Incredibly fast, stable, easy to use.

mdonahoe• 15 years ago

Is this actually better than gzipped JSON?

roel_v• 15 years ago

Depends on how you define 'better', I guess - at the very least it has to be several magnitudes faster, with no overhead for compression, parsing, etc.

frsyuki4• 15 years ago

You can use type-conversion APIs with MessagePack if you use C++, Java or C#. It converts deserialized objects (=dynamically-typed) into statically-typed objects using templates (C++) or reflection + dynamic code generation (Java and C#). You don't have to write boring type-checking codes.

Note that this is a characteristic of the implementation, not of the data format. Some libraries for JSON like JSONIC (Java) provides these APIs.

rabidsnail• 15 years ago

This appears to be mmap-friendly. Since arrays have length values it's possible to jump around in a large messagepack-serialized structure without having to scan it. In JSON if you want to get to element 2 of an array you have to scan to the end of element 1. If there's a length value at the beginning of element 1 (as it looks like there is with msgpack) then you can seek straight past it.

Ralith• 15 years ago

This sounds an awful lot like yet another ProtocolBuffers. What, if anything, new does it bring to the table?

ericmoritz• 15 years ago

This is schemaless. Protobufs require a .proto file that describes the data being serialized. A .proto file serves a similar purpose as the `struct` keyword in C.

MessagePack uses type prefixes to describe the data. For instance an unsigned 8-bit int starts with a `0xcc` followed by a byte.

frsyuki4• 15 years ago

Schemaless (dynamic typing) is the most obvious difference.

I wrote about the pros/cons of schema here: http://news.ycombinator.com/item?id=2836242

frsyuki4• 15 years ago

Compatibility with JSON is a characteristic of MessagePack. If you're already using JSON, you can use MessagePack as a faster/smaller substitution of JSON.

Since Protocol Buffers brings their original semantics, you must change lots of codes.

zdw• 15 years ago

Does it have it suitably low memory/CPU requirements to run on embedded devices?

How well does it work on ARM/MIPS?

frsyuki4• 15 years ago

It works very well at least on ARM.

I've used MessagePack on iPhone 3GS for the format of a dictionary file (> 100MB). I utilized mmap and zero-copy deserialization feature of MessagePack. And I received (and merged) some patches sent by those who use it on iPhone4/iPad2.

I've never tried to run it on MIPS. But it probably works if you use gcc.

(I'm author of MessagePack)

dimva• 15 years ago

In node.js, JSON is actually faster than msgpack: https://github.com/aikar/wormhole/issues/3

Also, due to V8's poor C++ API, msgpack doesn't support deserialization of 64 bit integers.

mraleph• 15 years ago

> Also, due to V8's poor C++ API, msgpack doesn't support deserialization of 64 bit integers.

JavaScript _does_not_ have a number type wide enough to hold 64-bit integer. V8 API has nothing to do with it.

erluser• 15 years ago

Would be nice if suuport for RPC (and generally) for Erlang was better. It's "currently" in alpha.

jstephens• 15 years ago

For perl it's fast as long as you have a small payload. Here's my benchmarks using progressively bigger data structures.

    perl: 5.008008
    Storable: 2.21
    JSON::XS: 2.25
    Data::MessagePack: 0.34

    ==== Size ====

    .-----------------+-----------+---------+---------.
    | src             | storable  | json    | msgpack |
    +-----------------+-----------+---------+---------+
    | 1               | 4         | fail    | 1       |
    | 3.14            | 8         | fail    | 5       |
    | {}              | 7         | 2       | 1       |
    | []              | 7         | 2       | 1       |
    | [('a')x10]      | 37        | 41      | 21      |
    | {('a')x10}      | 14        | fail    | 11      |
    | +{1,+{1,+{}}}   | 29        | 14      | 7       |
    | +[+[+[]]]       | 19        | 6       | 3       |
    '-----------------+-----------+---------+---------'
    ==========================

    Testing: theirs:HASH (4995 bytes)

    ==== Serialize ====

    Benchmark: running json, mp, storable for at least 1 CPU seconds...
          json:  1 wallclock secs ( 1.14 usr +  0.00 sys =  1.14 CPU) @ 26947.37/s (n=30720)
            mp:  1 wallclock secs ( 1.11 usr +  0.00 sys =  1.11 CPU) @ 70446.85/s (n=78196)
      storable:  1 wallclock secs ( 1.06 usr +  0.00 sys =  1.06 CPU) @ 18442.45/s (n=19549)
                Rate storable     json       mp
    storable 18442/s       --     -32%     -74%
    json     26947/s      46%       --     -62%
    mp       70447/s     282%     161%       --

    ==== Deserialize ====

    Benchmark: running json, mp, storable for at least 1 CPU seconds...
          json:  1 wallclock secs ( 1.02 usr +  0.00 sys =  1.02 CPU) @ 23424.51/s (n=23893)
            mp:  1 wallclock secs ( 1.02 usr +  0.00 sys =  1.02 CPU) @ 35136.27/s (n=35839)
      storable:  1 wallclock secs ( 1.06 usr +  0.00 sys =  1.06 CPU) @ 25357.55/s (n=26879)
                Rate     json storable       mp
    json     23425/s       --      -8%     -33%
    storable 25358/s       8%       --     -28%
    mp       35136/s      50%      39%       --


    ==========================

    Testing: xdbi:HASH (24138 bytes)

    ==== Serialize ====

    Benchmark: running json, mp, storable for at least 1 CPU seconds...
          json:  1 wallclock secs ( 1.06 usr +  0.00 sys =  1.06 CPU) @ 2414.15/s (n=2559)
            mp:  1 wallclock secs ( 1.07 usr +  0.00 sys =  1.07 CPU) @ 8373.83/s (n=8960)
      storable:  1 wallclock secs ( 1.06 usr +  0.00 sys =  1.06 CPU) @ 7244.34/s (n=7679)
               Rate     json storable       mp
    json     2414/s       --     -67%     -71%
    storable 7244/s     200%       --     -13%
    mp       8374/s     247%      16%       --

    ==== Deserialize ====

    Benchmark: running json, mp, storable for at least 1 CPU seconds...
          json:  1 wallclock secs ( 1.11 usr +  0.00 sys =  1.11 CPU) @ 4842.34/s (n=5375)
            mp:  2 wallclock secs ( 1.13 usr +  0.00 sys =  1.13 CPU) @ 4324.78/s (n=4887)
      storable:  1 wallclock secs ( 1.05 usr +  0.00 sys =  1.05 CPU) @ 4654.29/s (n=4887)
               Rate       mp storable     json
    mp       4325/s       --      -7%     -11%
    storable 4654/s       8%       --      -4%
    json     4842/s      12%       4%       --


    ==========================

    Testing: pdf:HASH (35239 bytes)

    ==== Serialize ====

    Benchmark: running json, mp, storable for at least 1 CPU seconds...
          json:  1 wallclock secs ( 1.04 usr +  0.00 sys =  1.04 CPU) @ 9845.19/s (n=10239)
            mp:  1 wallclock secs ( 1.05 usr +  0.00 sys =  1.05 CPU) @ 5119.05/s (n=5375)
      storable:  1 wallclock secs ( 1.10 usr +  0.00 sys =  1.10 CPU) @ 7518.18/s (n=8270)
               Rate       mp storable     json
    mp       5119/s       --     -32%     -48%
    storable 7518/s      47%       --     -24%
    json     9845/s      92%      31%       --

    ==== Deserialize ====

    Benchmark: running json, mp, storable for at least 1 CPU seconds...
          json:  1 wallclock secs ( 1.08 usr +  0.00 sys =  1.08 CPU) @ 3110.19/s (n=3359)
            mp:  1 wallclock secs ( 1.07 usr +  0.00 sys =  1.07 CPU) @ 2790.65/s (n=2986)
      storable:  1 wallclock secs ( 1.08 usr +  0.00 sys =  1.08 CPU) @ 3554.63/s (n=3839)
               Rate       mp     json storable
    mp       2791/s       --     -10%     -21%
    json     3110/s      11%       --     -13%
    storable 3555/s      27%      14%       --


    ==========================

    Testing: globallib:HASH (197319 bytes)

    ==== Serialize ====

    Benchmark: running json, mp, storable for at least 1 CPU seconds...
          json:  1 wallclock secs ( 1.09 usr +  0.00 sys =  1.09 CPU) @ 1760.55/s (n=1919)
            mp:  1 wallclock secs ( 1.06 usr +  0.00 sys =  1.06 CPU) @ 791.51/s (n=839)
      storable:  1 wallclock secs ( 1.05 usr +  0.00 sys =  1.05 CPU) @ 1279.05/s (n=1343)
               Rate       mp storable     json
    mp        792/s       --     -38%     -55%
    storable 1279/s      62%       --     -27%
    json     1761/s     122%      38%       --

    ==== Deserialize ====

    Benchmark: running json, mp, storable for at least 1 CPU seconds...
          json:  1 wallclock secs ( 1.71 usr +  0.00 sys =  1.71 CPU) @ 301.75/s (n=516)
            mp:  1 wallclock secs ( 1.03 usr +  0.00 sys =  1.03 CPU) @ 404.85/s (n=417)
      storable:  1 wallclock secs ( 1.12 usr +  0.00 sys =  1.12 CPU) @ 544.64/s (n=610)
              Rate     json       mp storable
    json     302/s       --     -25%     -45%
    mp       405/s      34%       --     -26%
    storable 545/s      80%      35%       --

MessagePack: Fast and Compact Serialization

Related Stories