kmavm

Crudely? Because you can't grep a sequence of latent states for variants of "If I kill all the puny humans, I can <achieve my current goal>."

Hi, Klaudia and Dawid! Any clue how 4.7 does?

I was an intern at SGI in the summer of 1998, when we shipped the latest minor version of IRIX, 6.5. I worked on a test suite for IRIX's pthreads implementation, and got to ship a teeny, tiny bit of real code that fixed a real-time hold-off in pthread_mutex_t. (IRIX is a hard RTOS, you see.) As things happened, the dot-dot releases of that minor version would be the last releases of IRIX to roll off the software assembly lines before SGI put it in maintenance mode for these last darn-near-30-years.

In 2000, I was the 20th-or-so full-time engineer at VMware, where I worked for 9 years. Then was at Facebook from 2009 to 2016, where I worked on the search backend (now replaced), HHVM (which still runs the Big Blue Application, a shrinking portion of the Meta Empire), and started FAIR in 2015 (which finally seems to have turned around the "open" sign with Yann's departure).

In 2016 I started at Slack as Chief Architect, where I mostly did not write a ton of code. I worked on a job queue scheduler which I would not be surprised to find has been replaced. And after that I was mostly encouraging/advising people doing Real Work.

All of which is to say, it is quite possible that the last code I've worked on professionally that is out there running on customer machines ... is that libpthread mutex bug fix from when I was barely old enough to drink.

It's an easter egg on the website that usually goes unnoticed. It's our first time on the front page of HN, so it's a little overutilized right now. Capital-C clears it.

Hi Fil! Congrats on all the amazing progress on Fil-C.

We needed to port all the user-level fork(2) calls to vfork(2) when working on uCLinux, a port of Linux to MMU-less microcontrollers[1]. It used to be that paging MMUs were kinda expensive (those TLBs! so much associativity!!!), and the CPU on your printer/ethernet card/etc. might not have that much grit. Nowadays not so much.

Still. A hard-and-fast use for vfork(2), as requested perhaps.

[1] http://www.ibiblio.org/lou/old/ViewStation/

Amazing. I can practically smell that owl it looks so darned owl-like.

From the article it doesn’t seem as though photorealism per se was a goal in training; was that just emergent from human preferences, or did it take some specific dataset construction mojo?

Your LLM of choice generates insecure code. The good folks at Vidoc (https://www.vidocsecurity.com/) built this browser extension to help.

(Disclaimer: I'm an investor in Krea AI.)

When Diego first showed me this animation, I wasn't completely sure what I was looking at, because I assumed the left and right sides were like composited together or something. But it's a unified screen recording; the right, generated side is keeping pace with the riffing the artist does in the little paint program on the left.

There is no substitute for low latency in creative tools; if you have to sit there holding your breath every time you try something, you aren't just linearly slowed down. There are points that are just too hard to reach in slow, deliberate, 30+ second steps that a classical diffusion generation requires.

When I first heard about consistency, my assumption was that it was just an accelerator. I expected we'd get faster, cheaper versions of the same kinds of interactions with visual models we're used to seeing. The fine hackers at Krea did not take long to prove me wrong!

It's "out of favor" because it completely failed as a research program. Let's not equivocate about this; it's nice to understand heuristic search, and there was a time when things like compilation were poorly understood enough to seem like AI. But as a path towards machines that succeed at cognitive tasks, these approaches are like climbing taller and taller trees in the hopes of getting to the moon.

Slight correction: HipHop for PHP was cleanroom, including rewriting large families of native extensions to work with its C++ runtime, although it eventually developed workalikes for the PHP dev headers to ease development. Source: I worked on HHVM, its JIT successor that initially shared its source tree and runtime.

The idea that "virtualization" began with Zen in 2004 is rather difficult to read as an early VMware employee. Before QEMU independently discovered it, VMware was JIT'ing unrestricted x86 to a safe x86 subset from 1999 on[1]. Hardware support for trap-and-emulate virtualization came to the market in the early 'aughts after VMware had proven the market demand for it.

[1] https://www.vmware.com/pdf/asplos235_adams.pdf

When I was at VMware in the 'aughts, VESA often saved us as an unaccelerated option for guests that didn't yet have a driver for our virtual display. Was there really no VESA driver for the 9x family? Or does QEMU's BIOS not do it or something?

I was Chief Architect at Slack from 2016 to 2020, and was privileged to work with the engineers who were doing the work of migrating to Vitess in that timeframe.

The assumption that tenants are perfectly isolated is actually the original sin of early Slack infrastructure that we adopted Vitess to migrate away from. From some earlier features in the Enterprise product (which joins lots of "little Slacks" into a corporate-wide entity) to more post-modern features like Slack Connect (https://slack.com/help/articles/1500001422062-Start-a-direct...) or Network Shared Channels (https://slack.com/blog/news/shared-channels-growth-innovatio...), the idea that each tenant is fully isolated was increasingly false.

Vitess is a meta-layer on top of MySQL shards that asks, per table, which key to shard on. It then uses that information to maintain some distributed indexes of its own, and to plan the occasional scatter/gather query appropriately. In practice, simply migrating code from our application-sharded, per-tenant old way into the differently-sharded Vitess storage system was not a simple matter of pointing to a new database; we had to change data access patterns to avoid large fan-out reads and writes. The team did a great write-up about it here: https://slack.engineering/scaling-datastores-at-slack-with-v...

This is accurate: Slack is exclusively using Hack/HHVM for its application servers.

HHVM has an embedded web server (the folly project's Proxygen), and can directly terminate HTTP/HTTPS itself. Facebook uses it in this way. If you want to bring your own webserver, though, FastCGI is the most practical way to do so with HHVM.

Hi! We didn't do a great job writing about this at the time, but Slack's migration into HHVM took place in 2016. We've been gradually increasing the coverage of Hacklang (Facebook's gradually typed descendant of PHP) since then, and are now 100% Hacklang.

We decided early on to colocate most aspects of the back-end, in part because we anticipated shared channels[1], but also because provisioning even virtual hardware for each team would be prohibitively expensive: we have over 600,000 organizations in Slack today[2], too many to make hard-partitioning most resources economical.

[1] https://www.zdnet.com/article/slack-brings-shared-channels-t... [2] https://sec.report/Document/0001628280-19-004786/

A lot of VC funds are required to liquidate and return proceeds to LPs in the event of an exit, whether through IPO or acquisition.

I'm chief architect at Slack, and we migrated to Hack from PHP 5 throughout 2016.

The toolchain for HHVM is all installed as a single big deliverable, which gives you the language engine and supporting runtime libraries itself, an in-address-space web server (Facebook's Proxygen), a debugger in the form of hhvm -a, and the Hacklang toolchain accessed via hh_client and appropriate editor/IDE integrations.

I share your intuition that there is actually a glittering core of "stuff-that-makes-you-successful" hiding in the incidental complexity of PHP, and we wrote this blog post trying to put some substance behind that intuition: https://slack.engineering/taking-php-seriously-cf7a60065329

You're asking about Hack, and the only response I see so far is about HHVM. So!

I've seen Hack used in a multi-tens-of-MLOC codebase to gradually insert types. It made huge differences in the kinds of changes that were possible; you can, e.g., rename a class or method, or change the order of its arguments, with confidence comparable to that in a C++ codebase. Most developers hack-ified everything they could get their hands on, and did all new work in Hack, without any external encouragement.

I'm not able to make sense of why it would not just be region-based memory-management. (http://www.sciencedirect.com/science/article/pii/S0890540196...). The usual way this fails is that everything ends up bubbling up to the top-most context, which is basically uncollectable. So you leak to death unless you limit yourself in awareness of this possibility.

I was hoping that their might actually be a linear type system hiding out in there, but alas, such is not the case.

FAIRie here. GA is essentially guess-and-check. SGD doesn't require guess or even check; if your error function is differentiable, and you didn't screw up the chain rule, you know what direction to head in to make it go down. For large models with lots of parameters, finding a setting for each parameter that makes the model as a whole go down by random choice has complexity n * f(n) for some monotonically increasing f, while SGD really finds you your next model, with decent guarantees it will be better, in O(n) time.

The theoretical hand-wringing about SGD for neural nets is that their loss surface isn't convex. It turns out this doesn't matter. The loss surface is a high-dimensional egg-carton, and you need to get winning-the-lottery-while-struck-by-lightning unlucky to find a significantly shallow local minimum. There are lots of saddle points, and you need to do something to drive out of those, but stochasticity seems sufficient in practice.

So, convolution is by itself an attempt to exploit translation-invariance in the visual world, and typical deep convnets end up picking up a certain amount of scaling tolerance (though I would not call it invariance) by having features that are sensitive to larger and larger patches of the input as you go up the hierarchy of features. This is not real scale-invariance, and many people run a laplacian pyramid of some sort at test time to get it real scale-invariance when eking out the best possible numbers.

Rotation-invariance is probably not really a thing you want. The visual world is not, in fact, rotation-invariant, and the "up" direction on Earth-bound, naturally-occurring images has different statistics than the "down" direction, and you'd like to exploit these. Animal visual systems are not rotation-invariant either; an entertainingly powerful demo of this is "the Thatcher Effect" (https://en.wikipedia.org/wiki/Thatcher_effect).

Reflection across a vertical axis, on the other hand, often is exploitable, at least in image recognition contexts (as opposed to, say, handwriting recognition). If you look at the features image recognition convnets are learning they are often symmetric around some axis or other, or sometimes come in "pairs" of left-hand/right-hand twins. As far as I know nobody has tried to exploit this architecturally in any way other than just data augmentation, but it's a big world out there and people have been trying this stuff for a long time.

The win here wasn't about scale, though. A tiny fraction of wikipedia users are logged in.

The win here was about individual page load time. And page load time is just as important, if not more so, for something new trying to vigorously grow as it is for the big sites.

(Disclaimer: HHVM alum.)

Thanks for the kind words. For context, I'm an HHVM alum who has been at Facebook for almost six years now (wow time flies).

From my point of view, most of what has changed is resources and the immediacy of our survival-level concerns. Four years ago Google had declared nuclear war on us, we had far fewer users, we were not profitable, there were constant fires to put out with basic production operations stuff we've gotten better at, and we were enormously more under-staffed. I was working on HHVM already, but it was in a million little pieces spread across Drew's, Jason's, and my desks. The tools we're open sourcing over the last two years mostly did not yet exist, and if they existed, it was in some primordial form. We also have gotten much, much better imho at being good stewards of our open source projects; HHVM's predecessor system, the HipHop compiler, was also open source, but we people were spread way too thin to be able to respond to bug reports, pull requests, get FB's latest code into public hands, build binary packages for popular distros, etc. on a timely basis. Huge props are due to all of the technical people on our open source teams.

As far as I know, you could try tracelets with any dynamic language.

And yes, you could try to share tracelets whose bodies are identical, but, unless their successor tracelets are also identical, you'd need to "dynamicise" the dispatch so you go down Path 1 when you're really tracelet 1 and Path 2 when you're really tracelet 2. We normally chain tracelets together by either falling through (if it's unconditional and the successor tracelet could be placed right next to it) or with jmp or branch instructions.

Hi, I'm one of the paper's co-authors.

As usual, the abstract really isn't enough to draw big conclusions from. The concept of a latent type applies to arbitrary expressions, not just variables. All the values flowing through a PHP program implicitly share the same union type; they can be floats, strings, arrays, etc. The latent types are the narrower types that can actually flow through the program in practice.

To be clear, it is more general than just things like:

  $a = 0; // $a is an int!

  g(foo() . bar());

By sheer coincidence, Michael Jordan did an AMA yesterday, and made this case better than I could. http://www.reddit.com/r/MachineLearning/comments/2fxi6v/ama_...

I agree dropout is awesome. Buddies? :)

It is arguable whether Hessian free methods, contrastive divergence or auto-encoder based training methods qualify as 'breakthroughs' but they have definitely equipped invigorated researchers in this broad area with their capabilities.

The scientific revolution in computer vision right now is due to deep convnets, trained in a supervised way using backprop and SGD. All of the systems we're talking about that have started blowing away records are members of this family, and were trained this way. If Alex Krizhevsky had not entered ImageNet 2012, we would not be having this conversation (in part because I probably would never have gotten curious enough about it to leave my home territories of systems and programming languages). Second order methods, unsupervised pre-training, RBMs, graphical models, etc. etc. etc. were exciting for those inside the field, definitely provided encouragement to those optimistic about deep models, and still might prove important, but they have had little impact and visibility to skeptical people outside, in the way that entering a computer vision competition and murdering all the computer vision systems did.

Hey, nice to make your acquaintance, and thanks for offering your contact info. Mine can be backed out my HN profile as well. I actually work in Menlo Park; the AI group is split across New York and Menlo Park, with a very small European contingent for now.

WRT the Glorot and Bengio paper, it’s true that there was a lot of excitement surrounding unsupervised pre-training of DNNs, but this mostly preceded the current wave of successes. The big differences between the architectures that are working on image processing today and that paper are:

1. Moar data. The datasets this paper was looking at were on the order 10^5 or so images. 10^6 is a different ballgame.

2. Convolution. Sharing weights really is special. This means there are far fewer parameters to learn in the early parts of the network, and so pre-training seems less necessary.

3. RelU activations. The survey of activation functions uses only smoothly differentiable ones whose gradients get tiny as you are far from zero. RelU has fewer problems with the gradient getting tiny or huge at idiosyncratic points, and also has the virtue of sparsifying the gradients as you backprop (since anything that landed in the negative tail has zero gradient).

So yeah, we really do do entirely unpretrained learning of low-level features, straight from RGB values between 0 and 256, and it works! Isn't that cool??