Pg-Emoji

https://github.com/truthly/pg-emoji

by JoelJacobson • 6 years ago

48 24 6 years ago

github.com

jszymborski• 6 years ago

So, am I right in my understanding that this is meant to add support for emojis not present in UTF8?

alex_duf• 6 years ago

I think it's a tongue in cheek project that isn't meant to be used in any production system.

I'm pretty sure you can already put emojis in the text fields of postgres. (or at least I'd be surprised if you couldn't)

Ancapistani5• 6 years ago

I think it's a tongue in cheek project

That would certainly make more sense than anything I’ve been able to glean from it.

I’ve definitely used Postgres text columns to store user-provided text values that included emoji in the past. They’re part of my standard test case for any user input.

JoelJacobson4• 6 years ago

The idea is to encode binary strings in a visually shorter form than e.g. hex, and also make it easier to visually detect differences. It’s also possibly easier to remember a bunch of emojis than a hex string.

Ancapistani5• 6 years ago

I don’t think so.

Without digging into the source - and I intend to do that, if someone more familiar with this doesn’t chime in - it appears that it’s targeted at reducing resource consumption.

UTF-8 can encode emoji fine. Consider (“ grinning face with smiling eyes”), which is `\xF0\x9F\x98\x81` in bytes. That’s four bytes. From the pg-emoji Readme:

A lookup-table is constructed from the first 1024 emojis from [https://unicode.org/Public/emoji/13.1/emoji-test.txt], where each emoji maps to a unique 10 bit sequence.
The input data is split into 10 bit fragments, mapped to the corresponding emojis.

If my understanding is correct thus far, then instead of storing four bytes for each emoji, you’d only need 10 bits.

I don’t know where this would be worthwhile.

I’m further confused by the purpose of `to_text()` and `from_text()`. Their example shows a string composed of mostly Latin characters being encoded into a string of emoji and back.

JoelJacobson4• 6 years ago

I’m further confused by the purpose of `to_text()` and `from_text()`. Their example shows a string composed of mostly Latin characters being encoded into a string of emoji and back.

This is meant to be used if you want to pass some text containing escape characters or perhaps JSON. Note also that the first emoji is a checksum, which might be useful if you want to make sure a user correctly copy/pasted a string, as opposed to sending a raw text string (without checksum).

MrStonedOne• 6 years ago

this is not an emoji support system or a system for storing emojis.

Its a system for encoding data as emojis. "this is a string" => some emojis. ie baseemoji or base1024

sfeng• 6 years ago

I believe it is an encoding for data into base 1024, using emoji as the symbol set. Similar in concept to Base64, meant to encode data into a format which can be sent anywhere ASCII is acceptable. This, one could think, would allow you to do the same thing more efficiently but with systems which accept emoji.

jfk133• 6 years ago

I'd expect that most places where emoji are accepted and reliably preserved, you could also use things like Han ideographs, which would give you a much larger symbol set to work with.

cmeacham98• 6 years ago

It's a novelty fun project, it serves no practical purpose. It's an encoding scheme similar to base64 or URL encoding, but one that does nothing useful.

jeltz2• 6 years ago

A similar technique but with a different set of emojis is used by the Element chat client to verify signatures for end to end encryption.

https://matrix.org/docs/spec/client_server/latest#sas-method...

jasperry• 6 years ago

I was disappointed when I saw it wasn't emojis drawn to look like Paul Graham.

sillysaurusx• 6 years ago

I've been waiting for this moment for literally years.

Long ago, I made http://github.com/strayptr/memes

If you scroll down to Kappa, you'll see a Lambda, which is pg in the style of twitch.tv's Kappa emote.

https://cloud.githubusercontent.com/assets/12214175/7581578/...

mathiasrw• 6 years ago

This is genius.

Data encoded in base1024 (here using the 1024 safe chars represented by emojis) gives much more efficient storage usage.

16 kB raw data encoded in base64

    ceil(16*1024/3)*4 = 21848 bytes long ~= 21.8kB.

16 kB raw data encoded in base1024

    ceil(16*1024/9)*10 = 18210 bytes long ~= 18.21kB.

So base64 needs about 19.7% more data storage than base1024 and both can be used anywhere utf8 is supported.

Let the baseEmoji revolution begin...

Pg-Emoji

Related Stories