Over-engineering an emoji webcam filter with a neural network

https://jott.live/markdown/images_as_emoji

by auraham • 4 years ago

79 18 4 years ago

helsinki• 4 years ago

I suspect calculating the average RGB of each emoji and comparing the distances between the average RGB of subsections of the image would have resulted in better emoji choices and better output (simpler, too).

I mean, I get that maybe it’s not the point of this article, but I think the end result would have been a better ‘mosaic’.

thfuran• 4 years ago

Rgb isn't a great color space to use with distance measures. CIELAB is better, though really perceptually uniform color difference measures are still nontrivial in that space as well.

williamcotton• 4 years ago

Take a look at the ‘12’ on the bottle of scotch. Higher resolutions than the dimensions of the emoji pixels would otherwise warrant are achieved by selecting emoji that work akin to subpixel antialiasing.

Llamamoe2• 4 years ago

I'd reckon there's probably room for about 2x2 subpixels' worth of "antialiasing", making the search space 4x3=12 dimensional (probably in Lab colorspace). Is that doable? If yes I'd expect the approach to be close to optimal possible.

auraham2• 4 years ago

I think there is an algorithm for determining the dominant color in an image (in this case, in a emoji). I do not its name, though.

https://stackoverflow.com/questions/3241929/python-find-domi...

https://dev.to/producthackers/creating-a-color-palette-with-...

liuliu• 4 years ago

I think author can try to encode location of the emoji and incorporate better scoring function (GAN)? But that is obviously really "over-engineered" :)

ipsum22• 4 years ago

Forgive me if I'm misunderstanding how it works, is the model is predicting the original emoji from an augmented emoji, and doing that for every 36x36 patch in the webcam image?

brrrrrm• 4 years ago

author here! I grabbed them from here IIRC https://emojipedia.org/apple/ (there are a bunch of sites that have these)

edit: yep - predicting the original based on the augmented version (as a hack to generalize for unseen inputs)

ipsum22• 4 years ago

I've updated my comment to clarify the question

taylorius• 4 years ago

Nearest neighbour algorithm though, no?

shreyshnaccount• 4 years ago

Yep, isn't that used to find dominant colours in images fast anyway?

bluelightning2k• 4 years ago

You mention a lookup table would have been tedious, but surely an easier method.

This might be a case of "to a hammer everything looks like a nail".

The solution in my familiarity would be to align the emoji in a grid, take one screenshot and write a script to iterate over the pixel data to figure out the most dominant color and how dominant. So the build of the lookup set would be automatic.

Less interesting for sure but easier (at least for a JS dayjob skillset).

Similar to a below comment, I know this isn't the point. Thanks for an interesting write up

auraham2• 4 years ago

I think a (N, D) array can be used as a lookup table. Here, N is the number of emojies and D is the dimensionality of the color space (say 3 for RGB). Each emoji is converted to a D-dimensional vector by computing the dominant color. Then, we can take a D-dimensional vector from an input image and compute the closest emoji by using cosine similarity. To minimize the loss function, we can increase the number of dominant colors, so the table would be (N, DxM). In this case, each emoji is represented by M D-dimensional vectors, each one encoding a dominant color.

Llamamoe2• 4 years ago

Do you have any good(preferably light on dry math) resources on these kinds of multidimensional search problems?

humanizersequel2• 4 years ago

Hey, Bun in the wild!

MuffinFlavored• 4 years ago

does Bun just wrap node.js?

humanizersequel2• 4 years ago

I believe Bun is meant to replace Node. Could be wrong though, I am not a Javascript understander by any stretch of the imagination.

lebaux• 4 years ago

ai already lost my guys

Over-engineering an emoji webcam filter with a neural network

Related Stories