Oidbs: An Open Source MQTT Driven Benchmark Suite for IoT Data

by jinmingjian • 4 years ago

54 7 4 years ago

Thanks for all the 3 upvotes:) I am the developer of the OIDBS, and glad to answer any question here.

Kuinox• 4 years ago

I have a WIP, more or less working, MQTT Client & Server, is there a way for me to benchmark my implementation with your benchmark suite ?

jinmingjian4• 4 years ago

Great. The OIDBS is using a built-in MQTT client. So, if you want o benchmark your server, just run the `import` command like as shown in the quick start video:

``` MODELS_ROOT=CURRENT_OIDBS_DIR import DIR_TO_DATA_SOURCE -n nyct_lite -d ```

here,

CURRENT_OIDBS_DIR is a hacky ENV var, for finding the models, should be removed in the future.

-n: is for the built-in model name, only the OIDBS repsect two models: nyct_lite and nyct_strip, you can get the two model sources from the article

-d: is for importing data only without injecting schemas. This is common if you are benching against a MQTT broker or server as you said. In fact, with this opt enabled, you can import any data source iff the data source is CSV or JSON. The benchmark logic is just to read and send the message line by line. That is, the files in the nyct_lite and nyct_strip directory is can be any CSV or JSON format. And the files'name is not important, but the model name still to be one of above two. Because the code checks the name.

(Ok, OIDBS may let the model name to be optional for such case, I will record this issue. thanks!)

If you have any problem, you can ask details in the community. I will help you as possible.

jeffrallen• 4 years ago

Reading about load testing mqtt reminded me of these articles describing some mqtt competitive coding I did with a couple of friends:

http://blog.nella.org/mqtt-code-golf/ https://atilaoncode.blog/2013/12/05/go-vs-d-vs-erlang-vs-c-i... https://atilaoncode.blog/2014/01/08/adding-java-and-c-to-the...

jinmingjian4• 4 years ago

Interesting reading, thanks for sharing. This is quick summary:

The articles compare the performance of different MQTT brokers implemented in different languages, including D, C, Erlang, Go, Java and C++. The basic conclusion should be D impl is the winner in the loadtest case, but C (Mosquitto) are the total winner for two case: loadtest + pingtest. C++ impl is not good. And the author said he "really don’t want to write C++ again unless I have to".

Basically, the conclusion of the article is nice. For MQTT brokers, the primary case sub-to-pub is just a layer-7 forwarding. This is an IO intensive scenario, if your MQTT parser is not written too badly.

So, for the impls done in system level languages, like C or D should archive the similar performance. Here, C++ impl should be an exception, although I haven't looked into the exact reason.

These blog articles, in fact, are quite old. The 100k message/second got with hundreds of connections is top ranked. However, the current Intel 12th Gen notebook can run at 250k mps with just 1 connection. I have tested (and read) several open-source MQTT clients and brokers. These MQTT brokers including impls done in C, Java and Erlang, can run at 1 million to 2.x million mps in a single modern Xeon socket but hard to go upward.

The truth here is that, the free lunch given by the efficient modern kernel TCP stack is just at 1-2M mps. If you want a faster broker, you need to make this kind layer-7 forwarding faster in the computational part, i.e. faster parsing and parallelizing. Traditional optimization tricks comes back again, like cache-friendly coding. The result is that we reached around 8 million mps in the same modern box even with message parsing, data routing and data dump all done in the first version of JoinBase(https://joinbase.io/).

The free lunch from the Moore's Law is truly over. We should be prepared for this.

xs83• 4 years ago

Where is the code for Joinbase? It looks like a solution to a problem I am currently in the process of analysing.

I can't see any way of installing it from the website right now?

jinmingjian4• 4 years ago

sorry, JoinBase now is just freeware, but not open-sourced. If you are interesting, you can request here: https://joinbase.io/request/.

Oidbs: An Open Source MQTT Driven Benchmark Suite for IoT Data

Related Stories