Printf Oriented Message Protocol

https://github.com/Parrot-Developers/libpomp

by Davidbrcz • 2 years ago

106 42 2 years ago

lukevp• 2 years ago

I don’t really get the advantage of this over either json (for human readability), protobuf (for type safety and schemas at the parsing layer), etc. it seems like it mostly has disadvantages and would be difficult to implement efficiently because there are not consistent characters to tokenize.

jitl• 2 years ago

It’s binary and there’s no code generating step like with Protobuf. Protobuf is hugely complicated, I can see using this for IPC between processes I own. I’m not sure what you mean about tokenization, but printf format strings are pretty well understood.

klodolph• 2 years ago

I read the page and missed the part where it’s binary. I had to read it a second time to find that information.

The reason why I missed it the first time is because this information is introduced in the second half of a paragraph, where the first half is about a different topic. This is a bad way to divide paragraphs.

packetlost2• 2 years ago

It's far, far simpler, than Protobuf, more efficient than JSON, and uses existing infrastructure that many people will be at least loosely familiar with. I think it's a pretty good idea!

shadowgovt4• 2 years ago

Key things it lacks that protobuf offers:

- bindings to other languages: I'm seeing C/C++ and Python supported. I could, hypothetically, use anything with a C-native interface to wrap that C library and bind to other languages, but that's work I'll have to do. And that binding is going to be as easy-to-use as a wrapper around a C-native library.

- versioning (or some other way to deal with drift between serialization and deserialization). In practice, protocol buffers are often used to do both long-term storage serialization and short-term on-wire transmission. Both use cases always eventually involve two services that cannot be kept in lockstep sync on releases (either because you have to read older files or because you needed to release your backend before your frontend, do a rolling release, whatever).

This is simpler than protobuf, which is cool (and I'd recommend looking over its implementation to figure out how to write something like this), but if you put it into implementation in a non-toy distributed-system project it will not scale. It may have other applications though; I can definitely see it as the backbone for a simple videogame messaging protocol.

Still, no harm in writing a thing and putting it out there; I'm happy it exists even if I won't use it.

dhagrow• 2 years ago

Seems like a better comparison would be any schemaless binary format, like msgpack, bson, or the many others. This adds some basic type validation, which you would have to do with a separate library, like msgspec if you're using Python.

0xbadcafebee3• 2 years ago

It's intended to be a barebones IPC between 2 processes. It's not for Google to use for 100-gigabit control planes, or for web development.

(But fwiw, if you did want to inspect this traffic as human-readable, it would be insanely easy to create either a packet sniffer for this or add support to tcpdump/wireguard. It's just dumping a printf format code and some raw data onto the wire, so you just pick up a new frame, read the format code, snip the data from the frame, pass it to printf() with the format code and the data. It's like 12 lines of code, no library needed other than libc. As a hacker I find this way more exciting than using a bloated standard to pass utf-8 strings)

unwind• 2 years ago

This is innovative and kind of brilliant, I'm impressed!

- Binary message formatting for efficiency and data integrity (float to/from string is messy)

- Familiar printf()-style format strings that the compiler can check[*] gives type safety!

- No preprocessing and/or code generation from message templates or anything like that.

Really nice!

Edits: markdown and grammar.

[*]: This uses an extension I think to mark the argument as a format string

lelanthran• 2 years ago

This is innovative and kind of brilliant, I'm impressed!

As someone who is not the author of this blog post, I'm glad you are impressed: I wrote pretty much the same thing around 7 years ago: https://www.lelanthran.com/chap2/content.html

The difference with mine is that I intended for the library to be endian-independent, so usable over a network transmission to a different endian machine.[1]

Great minds ... (everyone knows how the rest of it goes).

[1] I also eschewed using `%` for the format string specifiers, because (in my words at the time):

Our parameterised functions must take a specification that tells it how each field should be written, very similar to the printf and scanf family of functions. While it is indeed possible to reuse the standard format specifiers as our own field specifiers it might not be a good idea to do so as this would break The Principle of Least Astonishment.1 This is because anyone who is changing the code and who sees a string literal with well-known format specifiers such as %02x and %c would naturally (but incorrectly) assume that all format specifiers are supported. We do not wish to confuse the reader.

thechao• 2 years ago

You can use `funopencookie` to create a binding of a socket to a `FILE*`; in that case you could literally use fprintf and fscanf to send messages...

shawn_w• 2 years ago

`fdopen()` is a lot simpler.

pjmlp• 2 years ago

Not really inovative, this used to be common in C libraries for network programming, back when C paint was still rather fresh.

fisian• 2 years ago

To mention a usecase for this library: it is used as part of the software Parrot ships with their drones.

Davidbrcz• 2 years ago

And like all of Parrot's code, it's terrible.

Source: I worked on Parrot's code. And you can make your mind by looking at the other projects in Parrot dev group...

0xbadcafebee3• 2 years ago

Show me code written by a corporation that isn't terrible and I'll show you one developer's pet project

jbverschoor• 2 years ago

How is the string length determined? How does this handle packed bits/ and odd sized numbers, for example a 5 bit digit and 3 flags

rangerelf• 2 years ago

I think that would be up to you; just use a union struct with that data and send/receive the corresponding int.

edflsafoiewq• 2 years ago

Strings are serialized as (type code) (uint16 length) (string contents) (null byte).

In the API, the string length is determined the normal way in each language: strlen() in C, len() in Python. The trailing NUL allows the C decoder to return just a pointer to the string (without allocating).

So internal null bytes and strings longer than 0xFFFF are functionally prohibited.

tonyg• 2 years ago

Feels similar to the dbus serialisation format.

immibis• 2 years ago

IIRC Cube 2: Sauerbraten uses a similar concept for network, though the format strings aren't similar to printf (just a simple list of sized types) and the types/format aren't transmitted on the wire.

lanstin• 2 years ago

The format (%ii) semantics aren't platform independent that seems like an obvious error.

And code gen and protocol descriptions bring a certain useful discipline to modifying existing protocols.

But it has that funky C style so I bet people using C like it.

chrchr• 2 years ago

It reminds me of Perl's 'pack' and 'unpack'.

https://perldoc.perl.org/perlpacktut

MathMonkeyMan• 2 years ago

or python's 'struct'

https://docs.python.org/3/library/struct.html

naberhausj• 2 years ago

Also reminds me of the huge shortcoming that these "simple" serialization libraries have. In most use-cases you're eventually going to need some way to add/remove fields from the payloads.

JSON and protobuf make that easy. You can start emitting the new field and then update the receiving end at your leisure.

With these libraries you have to write your own versioning system. For a small performance improvement over JSON parsing it's pretty much not worth it. There's almost always going to be a piece of lower hanging fruit.

sltkr• 2 years ago

This is a valid concern, but it's possible to work around in a similar way that you do with protobufs: never redefine the format of any message type used in production, but instead, introduce a new message type whenever you need to change the format.

For example, imagine you are sending message type 1 with format "%d %d" and later you realize you actually need three instead of two ints. You introduce message type 2 with format "%d %d %d" and update your readers to support both types. Once those are deployed in production, you update your writers to send the new message.

This is kind of the opposite of what happens with protobufs, where unsupported fields are silently dropped by the receiver, so you can update the sender before the receiver. But this is arguable less safe, since if the receiver drops some fields, it might not interpret it the way the sender intended. In that case, it might be more sane if the receiver outright refuses to process a message it doesn't understand.

inetknght• 2 years ago

I've used libpomp to interact with Parrot's Anafi drones.

This library takes the idea of modern type safety and throws it away. Instead, the library leans in on `printf()` instead, which is known to be unsafe. And it does it in a memory-unsafe language.

...on a drone. Where safety needs to be important (even a small drone can do significant damage).

It's neat and all. But that's a killer anti-feature in my opinion. I wouldn't use it on my drones.

nurettin• 2 years ago

How the knowledge of the format string is shared is out of the scope of this library. It can simply be a shared header with defines.

They don't force you to share the format strings at connection time. In fact, it is just a thin layer around sockets and you just override the message handler. I like it! Might be improved further with something like libfmt which is also used by spdlog.

touringmachine• 2 years ago

This is very clever. Furthermore, lmao.

khimaros• 2 years ago

i wonder if there is anything equivalent to this for rust?

Printf Oriented Message Protocol

Related Stories