frsyuki

ACK of network transfer is available ("require_ack_response" option). This option ends up choice of at-most-once semantics vs. at-least-once semantics. You need to choose and you can choose.

Fuentd provides "buffer_type file" to buffer records on disk. Shutting down won't loose data. If you need to choose memory buffer for performance reasons, fluentd enables "flush_at_shutdown" option by default.

You would also want to use <secondary> feature. This lets you to write a buffer chunk to another storage if the primary destination is not available "retry_limit" times.

Those concerns would be solved by the document: http://docs.fluentd.org/articles/out_forward#buffered-output...

Did you guys see this video?: http://vimeo.com/53261709 Github uses MessagePack for their internal RPC protocol.

Here is a benchmark result of msgpack, json, bson and protobuf: https://gist.github.com/4348013

An important point we should consider here is that actual performance depends on implementation.

Disclaimer: I'm authoer of MessagePack for C++/Ruby and committer of one for Java.

As for strings, JSON has to allocate memory and copy to deserialize strings because strings are escaped.

MessagePack does't have to allocate/copy because the serialized format of strings is same as the format in memory. But it depends on the implementation whether actually it doesn't allocate/copy.

C++ and Ruby implementations try to suppress allocation and copying (zero-copy). But Java implementation doesn't support zero-copy feature so far (we have plan to do so. Here is "TODO" comment: https://github.com/msgpack/msgpack-java/blob/master/src/main...).

As for the other types, C++ implementation (and new Ruby implementation which is under development) has memory pool which optimizes those memory allocations petterns. But it's hard to implement such optimizations for Java because JVM (and Dalvik VM) doesn't allow to hook object allocation.

We're using MessagePack in a Rails application to log user behaviors and analyze them.

Compared with other serialization libraries such as Protocol Buffers, Avro or BSON, one of the advantages of MessagePack is compatibility with JSON. (In spite of its name, BSON has special types which cause incompatibility with JSON)

It means we can exchange objects sent from browsers (in JSON format) between servers written in different languages without losing information.

I will not use MessagePack with browsers but it's still useful to use it with web applications.

MessagePack includes a concept named "type conversion" to support types which are not supported by its wire format. With the concept, we can serialize/deserialize user-defined classes as well as strings with encodings.

So far, MessagePack for Java, C++ and D implement the concept.

Although the original blog post focuses on JavaScript and browsers, MessagePack itself doesn't mainly focus on them.

A major use case of MessagePack is to store serialized objects in memcached. A blog post written by Pinterest describes this use case (http://engineering.pinterest.com/posts/2012/memcache-games/). They use MessagePack with Python which is faster than one with JavaScript. They could store more objects in a server without performance declination (e.g. gzip).

It's true that MessagePack is not always faster than JSON (e.g. within browsers), and it's not always smaller than other serialization methods (e.g. with gzip compression). So we should consider that which serialization methods should I use for "my" case.

There are also general tendency which is helpful to select MessagePack or JSON:

    MessagePack is faster to serialize binary data such as thumbnail images.
    MessagePack is better to reduce overheads to exchange small objects between servers.
    JSON is better to use it with browsers.

We're also using Fluentd as well as original JSON-based logging libraries.

Fluentd deals with JSON-based logs. JSON is good for human facing interface because it is human readable and GREPable.

On the other side, Fluentd handles logs in MessagePack format internally. Msgpack is a serialization format compatible with JSON and can be an efficient replacement of JSON.

I wrote plugin for Fluentd that send those structured logs to Librato Metrics (https://metrics.librato.com/) which provides charting and dashboard features.

With Fluentd, our logs became program-friendly as well as human-firnedly.

Spitting out sprintfs means parsing texts. JSON might be slow since it's text, but binary based formats should be faster especially the log includes many integers.

There are binary-based serialization formats such as MessagePack [1] or BSON [2] that doesn't need escapeing overhead.

You can simply replace JSON with MessagePack since MessagePack is compatible with JSON.

[1] Why Not MessagePack? http://blog.andrewvc.com/why-arent-you-using-messagepack

[2] Performant Entity Serialization: BSON vs MessagePack (vs JSON) http://stackoverflow.com/questions/6355497/performant-entity...

You can use type-conversion APIs with MessagePack if you use C++, Java or C#. It converts deserialized objects (=dynamically-typed) into statically-typed objects using templates (C++) or reflection + dynamic code generation (Java and C#). You don't have to write boring type-checking codes.

Note that this is a characteristic of the implementation, not of the data format. Some libraries for JSON like JSONIC (Java) provides these APIs.

Compatibility with JSON is a characteristic of MessagePack. If you're already using JSON, you can use MessagePack as a faster/smaller substitution of JSON.

Since Protocol Buffers brings their original semantics, you must change lots of codes.

Schemaless (dynamic typing) is the most obvious difference.

I wrote about the pros/cons of schema here: http://news.ycombinator.com/item?id=2836242

It works very well at least on ARM.

I've used MessagePack on iPhone 3GS for the format of a dictionary file (> 100MB). I utilized mmap and zero-copy deserialization feature of MessagePack. And I received (and merged) some patches sent by those who use it on iPhone4/iPad2.

I've never tried to run it on MIPS. But it probably works if you use gcc.

(I'm author of MessagePack)

(Note: I'm author of MessagePack)

I've described about the difference between MessagePack and BSON:

"Performant Entity Serialization: BSON vs MessagePack (vs JSON)" http://stackoverflow.com/questions/6355497/performant-entity...

Technically, there are 2 important differences:

- Statically typed or dynamically typed

- Type mapping between language's type system and serializer's type system (Note: these serializers are cross-language)

The most understandable difference is "statically typed" vs "dynamically typed". It affects that how to manage compatibility of data and programs. Statically typed serializers don't store detailed type information of objects into the serialized data, because it is explained in source codes or IDL. Dynamically typed serializers store type information by the side of values.

- Statically typed: Protocol Buffers, Thrift, XDR

- Dynamically typed: JSON, Avro, MessagePack, BSON

Generally speaking, statically typed serializers can store objects in fewer bytes. But they they can't detect errors in the IDL (=mismatch of data and IDL). They must believe IDL is correct since data don't include type information. It means statically typed serializers are high-performance but you must strongly care about compatibility of data and programs.

Note that some serializers have original improvements for the problems. Protocol Buffers store some (not detailed) type information into data. Thus it can detect mismatch of IDL and data. MessagePack stores type information in effective format. Thus its data size becomes smaller than Protocol Buffers or Thrift (depends on data).

Type systems are also important difference. Following list compares type systems of Protocol Buffers, Avro and MessagePack:

- Protocol Buffers: int32, int64, uint32, uint64, sint32, sint64, fixed32, fixed64, sfixed32, sfixed64, double, float, bool, string, bytes, repeated, message [1]

- Avro: int, long, float, double, boolean, null, float, double, bytes, fixed, string, enum, array, map, record [2]

- MessagePack: Integer, Float, Boolean, Nil, Raw, Array, Map (=same as JSON) [3]

Serializers must map these types into/from language's types to achieve cross-language compatibility. It means that some types supported by your favorite language can't be stored by some serializers. Or too many types may cause interoperability problems. For example, Protocol Buffers doesn't have map (dictionary) type. Avro doesn't tell unsigned integers from signed integers, while Protocol Buffers does. Avro has enum type, while Protocol Buffers and MessagePack don't have.

It was necessary for their designers. Protocol Buffers are initially designed for C++ while Avro for Java. MessagePack aims interoperability with JSON.

I'm using MessagePack to develop our new web service. Dynamically typed and JSON interoperability are required for us.

[1] http://code.google.com/apis/protocolbuffers/docs/proto.html#...

[2] http://avro.apache.org/docs/1.5.1/spec.html#schema_primitive

[3] http://wiki.msgpack.org/display/MSGPACK/Format+specification

The article doesn't describe but one of the biggest difference is scalability. MongoDB supports horizontal scaling as a built-in feature. It's known that MySQL can be scaled out by sharding but you lose ACID with it. Of course scalability is required only when the service goes well. But it's important to considerate it if you intend to be successful.

It can be said scaled MySQL is a no-SQL. MongoDB is the New MySQL with this point of view.

This slide was published by @acmurthy at Hortonworks, which is a new company formed by core Hadoop committers from the Yahoo! Hadoop software engineering team. http://www.hortonworks.com/

RITE is exactly what I want to need.

The problem is that distributed systems (like message queues or notification services) are required to be fast/scalable but very hard to program/debug/test. One solution is embedding plugin mechanism in a carefully programmed framework. There are examples such as Tokyo Tyrant's Lua plugin or Apache Solr's plugin mechanism. But rather than Lua or Java, I want to use Ruby because it's syntax and semantics are very suitable to write plugins.

RITE will make it possible. Ruby may be new standard of embedded languages.

I'll use postfix for that case.

For yet another point of view, how about using external SMTP server directly? You are sending messages as follows:

  app -> local MTA -> /var/spool/mail <- users read mails

  app -> gmail's SMTP server -> users

Interesting. OMeta can create a new parser that inherits from an existing parser. It's unique. But we have to not just parse but also evaluate. OMeta's external DSL is excellent for parsing but seems difficult to use for evaluating.

There are other parser libraries based on PEG such as pyPEG (Python), LPeg (Lua) or Parsec (Haskell). They use internal DSL rather than external DSL. I have used Parslet (Ruby) in my project and it's worth to try. http://kschiess.github.com/parslet/

Message Pack vs similar utilities: http://gist.github.com/290425

It basically uses TCP. Raw unreliable UDP is also supported on Ruby and Java. TCP or UDP will be enough choices for most of applications. Practically, MessagePack-RPC mainly focuses on cross-language communication, like Ruby frontend <-> C++ backend. So portability is rather important.

It's true that network latency is dominator rather than serialization. But low CPU/memory usage is still important for CPU/memory-bound applications. Especially, zero-copy feature is effective to raise maximum throughput. As for RPC performance, parallelism will be important for high-load systems. It's described at http://msgpack.wordpress.com/2010/05/08/messagepack-rpc-for-... On the other hand, some uses serialize/deserialize directly on search engine or caching to pack objects.