Dismissing Python Garbage Collection (2017)

I'm surprised this wasn't vanished into the memory hole for containing harmful terms like "master process", even if it was from way back when in 2017.

bluehark• 6 years ago

Follow-up blog post a year later: https://instagram-engineering.com/copy-on-write-friendly-pyt...

lolc• 6 years ago

It's nice to see a hack turn into an API extension that addresses the underlying problem.

vincent-manis2• 6 years ago

This article is an excellent argument against reference counting.

joosters• 6 years ago

As the article says, reference counting was not the problem. They got their improvements by disabling the garbage collector sweeps, but reference counting was still being used.

vincent-manis2• 6 years ago

My point was that CPython does reference counting, resulting in the Copy-On-Read behavior described in the article. That isn't something that can be easily changed, not without writing a whole new CPython system (or switching to another Python implementation). Their solution to the problem was the cleanest, under the circumstances.

mhh__2• 6 years ago

Is python a good arena for any language features? Just as I wouldn't use C++ to show off garbage collection (Conservative M+S), Python seems like a bad place to show off reference counting because the compiler can't spend a practically infinite amount of time on data flow analysis to elide RC ops.

ogogmad• 6 years ago

Why couldn't they use PyPy? The GC for that has higher throughput.

yorwba• 6 years ago

The more important question is whether PyPy's GC uses intrusive data structures to maintain the metadata for each object. If it does the same thing as CPython, just faster, then the process would run out of memory faster as the GC writing to COW pages converts them into owned memory.

nuclear_eclipse• 6 years ago

The easy answer is C extensions. PyPy does not support C extensions, and thereby does not support a wide range of native, accelerated libraries. Facebook infra is heavily dependent on wrapping C/C++ libraries (like Thrift) for use in many languages, and not having C extensions (or Cython) would cut off a large portion of our shared codebase.

Doxin• 6 years ago

Pypy does support C extensions, but it has to somewhat emulate the cpython API so it's rather slow and doesn't work for 100% of modules.

If your C library uses FFI or Ctypes it'll work perfectly fine under pypy.

nayuki• 6 years ago

Only half similar, but Java has a no-collection memory management strategy called "Epsilon GC": https://blogs.oracle.com/javamagazine/epsilon-the-jdks-do-no...

rurban• 6 years ago

Now just imagine if someone would replace the horrible refcnt with a real copying GC. At least 2x faster I would assume.

Dismissing Python Garbage Collection (2017)

Related Stories