Superpack: Pushing the limits of compression in Facebook’s mobile apps

https://engineering.fb.com/2021/09/13/core-data/superpack/

by detaro • 5 years ago

91 21 5 years ago

engineering.fb.com

pella• 5 years ago

important: "We may someday consider open sourcing Superpack."

mshockwave• 5 years ago

I hope they'll push out a paper as well

tentacleuno3• 5 years ago

Didnt Facebook open-source their other stuff? I don't think they were very good with outside contributions, it was just "here's the code".

umanwizard3• 5 years ago

There’s a spectrum. Some stuff is explicitly intended for outside consumption and a lot of effort goes into documentation, marketing, polish, etc. (e.g. PyTorch). Some other stuff like you said is just thrown over the wall.

warning26• 5 years ago

The fact that Facebook, something that really shouldn't even need an app, requires this kind of compression seems like more of an unintentional commentary on how bloated their terrible app is than anything else.

Kayou• 5 years ago

Yeah, and I think it really shows that WhatsApp, which wasn't developped by Facebook originally, doesn't benefits from the compression. The app seems very optimized originally, but maybe I'm wrong.

dirkf• 5 years ago

Also see the stark contrast with their Facebook Lite app. Makes you wonder what difference all those megabytes in the full app actually make...

burnte2• 5 years ago

I don't even have the app on my phone, just a shortcut to the mobile site, and it does everything I need.

tentacleuno3• 5 years ago

How is Messenger? Last time I remember, they were rendering it useless on the mobile site in favour of downloading an app (more data).

I haven't used Facebook in a long time, though.

umanwizard3• 5 years ago

Yep, this 100%. I worked on Facebook iOS app perf/reliability/efficiency for a couple of years. The company is structurally and politically incapable of making a lean app (thousands of engineers have their performance reviews tied to launching new features, not deleting old ones). So they rely on a relatively small number of engineers making heroic technical fixes in order to make it possible for the app to even boot. Similar situation on Android.

The status quo is clearly dysfunctional but I would be surprised if the company ever figures out how to fix it.

ermir2• 5 years ago

A few years back the FB app reached Android’s limit for the number of classes defined in the codebase (2^16), so the FB engineers wrote a hack that ran before the app to somehow bypass this limit. Apparently they don’t delete anything and just keep adding stuff to the codebase.

What kind of app needs 65536 classes?

ermir2• 5 years ago

https://m.facebook.com/nt/screen/?params=%7B%22note_id%22%3A...

Found the old explanation, it seems I had misremembered some details but it’s still super insane.

int3• 5 years ago

The method limit is the more constraining one.

Also, Google eventually lifted this limit on newer versions of Android because other apps were running into the same issue. It's not just FB :)

tentacleuno3• 5 years ago

Bear in mind that number also includes classes declared in dependencies.

benreesman• 5 years ago

Drive-by associations of the morality of FB’s business (which I personally found objectionable enough to leave a few MM on the table over) and the caliber of the engineers who work there are suspect.

The folks are every bit as good as the hiring practices would indicate. These people meet a problem, and in the average case haven’t read the relevant textbook. So they read it, and solve the problem. Sometimes they invent something new. I personally had LSM in production before Dean et al.

Criticize the business all you want, I might even back you, but the typical HN smartass has no idea the world of hurt they would be in if they had to code against an FB new hire.

umanwizard3• 5 years ago

The comment you’re responding to didn’t mention Facebook’s hiring practices at all.

The folks are every bit as good as the hiring practices would indicate. These people meet a problem, and in the average case haven’t read the relevant textbook. So they read it, and solve the problem. Sometimes they invent something new.

Give me a break. There are some really bright people, but a lot are mediocre, just like at any other company with tens of thousands of employees.

I’m not sure when you worked there. The only way I can make sense of your post is if it was a very long time ago, or you were in a very unusual team.

alimbada• 5 years ago

Now if Google could do some optimisation of their apps on the Apple App Store, that'd be great. Currently, the latest Gmail update sits at ~328MB, the total of all nine Google apps on my phone sits at 2226MB (averaging ~247MB). It's ridiculous.

lifthrasiir• 5 years ago

So basically the compressor is aware of opcode structure so uses multiple data streams and contextual encoding for operands. The article doesn't say much about the use of SMT solver, but my guess is that it transforms opcodes in a way that they run exactly identically as the original (verified by the solver) but compress better. Well-known stuffs in the data compression community, but also hard to execute efficiently.

Superpack: Pushing the limits of compression in Facebook’s mobile apps

Related Stories