Speeding up PHP with the HipHop VM

https://www.facebook.com/notes/facebook-engineering/speeding-up-php-based-development-with-hiphop-vm/10151170460698920

by kmavm • 14 years ago

203 118 14 years ago

facebook.com

38252• 14 years ago

Why is the blog on facebook?

dkoch• 14 years ago

HipHop is created by Facebook engineering staff.

nthitz• 14 years ago

HipHop was developed by Facebook?

spitfire• 14 years ago

I thought it was DJ Kool Herc.

kmavm7• 14 years ago

We work for Facebook, and make a lot of announcements on the engineering blog. We also have a group blog on WordPress-in-HHVM running here: http://www.hiphop-php.com/wp/

38252• 14 years ago

Didn't realize that until I actually went to the page. I need to learn to keep my pie hole shut more often.

maratd4• 14 years ago

Things make sense now. I remember a while back Facebook invested in an experiment with PHP on PyPy. They didn't pursue it, even though it produced impressive results. It seems their own in-house JIT has better performance?

kmavm7• 14 years ago

HHVM was pretty far along when we started talking to the PyPy folks; it was already able to run the site, and hosting internal development at Facebook. Our interest in PyPy wasn't an immediate, drop-everything-and-change-to-PyPy kind of interest. It was a research project, which had a positive outcome. Making a production-ready, PyPy-based system for PHP would still be an enormously big undertaking, though.

PyPy is taking a radically different approach from what HHVM is doing (and really from what almost all other dynamic language systems are doing), and it's a fascinating system. Part of what's exciting about it is that it seems like it should be applicable to other languages with less effort than most other JITs, and we wanted to understand its potential for a language like PHP. We asked Maciej Fijalkowski (hi, Maciej, if you're reading!) to help us do a research prototype to see what the first few roadblocks would look like, and Maciej did a great job. Just because we didn't scrap our current project and shift all our resources to PyPy should not be seen as a negative reflection on PyPy at all.

bascule• 14 years ago

I'm kind of surprised you didn't pursue the JVM and InvokeDynamic for this purpose instead of PyPy

fomojola• 14 years ago

I'm curious if they were still evolving HPHPc at the same time as they were evolving HHVM: the chart shows HPHPc as having flat performance over time, while HHVM was getting better over a 7-month period. Could HPHPc have achieved the same performance gains if the same effort was expended?

kmavm7• 14 years ago

Hey, I'm an HHVM engineer.

We just normalized HPHPc performance to make the graph easier to read. HPHPc actually got considerably faster over the same period as well; since both systems share a runtime, many changes helped both.

HPHPc probably got about 20% faster over 2012, even though nobody was actively working on it, mostly through happy side effects of work that was directed at HHVM.

nbm3• 14 years ago

I'm not on the HPHP team, but when discussing these results internally, they mentioned that HPHPc performance was improving during this time as well (as you would expect we would need to keep running the site efficiently), and the graph showed point-in-time comparisons of their performance, not from a baseline.

As to whether it would have been possible to get better performance on HPHPc, I'll let people with more knowledge answer.

The benefit of having a single runtime for development and production (and production-like staging environments) is huge, though. There may also be benefits in the push process, which while fast when compared to most other companies, which will continue to deliver benefits for increased speed.

olaf• 14 years ago

"So, when you combine XHP with HipHop PHP you can start to imagine that the performance penalty would be a lot less than 75% and it becomes a viable approach." http://toys.lerdorf.com/archives/54-A-quick-look-at-XHP.html

debacle3• 14 years ago

That's from 2 years ago. What is its relevance now?

maratd4• 14 years ago

For those who want to get their hands dirty:

CentOS 6.3 x64: https://github.com/facebook/hiphop-php/wiki/Building-and-ins...

Ubuntu 12.04 x64: https://github.com/facebook/hiphop-php/wiki/Building-and-ins...

snissn• 14 years ago

Instructions for Ubuntu 12.04 x64: http://pastie.org/5456298

bretthellman• 14 years ago

Will FB ever invest and move off PHP? Hiring talent at FB has to be getting harder and PHP probably isn't helping.

nbm3• 14 years ago

PHP is used for a subset of development at Facebook. I'm not sure what the split is, but I would not be surprised if PHP is the primary language used by less than 60% of Software Engineers at Facebook. The rest code in C++, Java, Python, and then a bunch of other languages that are less used.

The PHP development environment at Facebook is unlike that found at any other company I'm aware of, and common PHP pitfalls and legacy code are actively "linted" out of the system (as an automated part of the code review process). It still wouldn't be my choice of language to use outside of Facebook, but I have no issue working on the PHP code base here (although I rarely venture into it).

Over time, much logic and processing has been offloaded to backend services, with PHP increasingly playing a (largely parallelised) dispatch, post-process, and render role. That will likely continue.

With all that said, it is questionable whether it would be an investment to move off of it - which language/environment would you think a good target would be, and why?

propercoil• 14 years ago

re-writing the code of any big project is insane. You are going to face those walls you faced in the past - again. Using this logic you would change the whole codebase every few years.. ridiculous

untog• 14 years ago

I think you underestimate the number of PHP developers out there. There aren't many on Hacker News because we're all too cool for that stuff, but in the developer world in general there are a ton.

MichaelGG• 14 years ago

I think his point is that real top-tier developers are going to despite working with such a grotty language as PHP, so it hurts their hiring process that way. Whereas if they worked in a superior language (an ML, Haskell, a LISP...) they'd attract talent. Who wants to deal with a language such such inane design decisions all day?

I'd imagine people signing up to work at FB do so despite the fact they use PHP, not because of it.

skeletonjelly• 14 years ago

Where do you get your estimates from?

res0nat0r• 14 years ago

I'm sure having some PHP code at a company like Facebook isn't deterring any serious candidates. Working for a company with almost a billion users who've created software like Cassandra, Thrift, HipHop and Haystack and working on writing loosely coupled systems which will run across thousands of machines; analyzing 10s of PB of data every day with Hadoop, there are definitely very specific and challenging tasks which would attract tons of talent. I think having a PHP codebase there isn't indicative that your new job would be sitting and writing crappy PHP code templates all day and wanting to kill yourself because it is so boring. Just my 2c.

michael_miller• 14 years ago

First, it's really hard to do a large rewrite of software. Netscape, for example, made the mistake of trying to rewrite their browser from scratch, and it killed them [1].

Second, what evidence to you base your statement "hiring talent at FB has to be getting harder"? If you're talking about market cap / upside, Google has had no problems hiring (well, relative to other tech companies, anyway), despite a market cap 3.5x+ what FB is today.

Lastly, FB is not all PHP. A lot of services are written in C++ or Java, and use Thrift to talk to the frontend [2]. Thrift allows a more gradual movement off PHP, by moving isolated components to other languages individually.

[1] http://www.joelonsoftware.com/articles/fog0000000069.html [2] http://www.quora.com/Facebook-Engineering/What-is-Facebooks-...

timdorr• 14 years ago

What version of PHP does HPHP match? 5.3 or 5.4? I ask because I've become very accustomed the short array syntax ([1,2,3]) and other goodies in 5.4.

kmavm7• 14 years ago

It is closer to 5.3, though we've adopted some 5.4 features: traits, our closures' treatment of $this, and f()[$x] syntax.

Edit: notably not short array syntax, at least yet.

dkhenry3• 14 years ago

This makes me sad. That's one of the nicest features of 5.4 not needing to type array() all over the place.

nicholassmith3• 14 years ago

Amazed at all the 'engineer X policy or feature request slackers'. Can't say I'm the biggest fan of Facebook as a service but what their engineers are doing in terms of pushing the state of the art is fantastic.

Out of curiosity if there's anyone involved in Facebook on HipHop, has there ever been a discussion about just shifting from PHP to a more performant language, or is a case of still reaping the benefits of PHP in terms of dropping a developer in and not worrying about skill sets?

lbrandy• 14 years ago

> ...about just shifting from PHP to a more performant language

We already shift lots of things out of PHP into more performant (c++, java, etc) languages when we are building services and/or extensions and/or other computationally intensive things.

But I think more to your point (and if not, let me indulge in the strawman), since this comes up frequently on reddit/hn: php is not somehow -uniquely- broken. While PHP might be uniquely broken as a language (ha ha), it's not -uniquely- broken as a platform/runtime. Everything you think is performant is broken at a large enough scale. Put another way, it's not just about the switching costs of rewriting in some other language. If we could magically snap our fingers and convert the entire codebase from php to some other language, that's just the beginning of understanding, tweaking, and occasionally rebuilding everything about the runtime/libraries/etc to make it work.

At the end of the day the language itself is given way too much attention in discussions like this.

dkhenry3• 14 years ago

So is your argument then that there is so much vested in facebook in dealing with and optimizing for PHP at scale that the real cost would be acquiring that knowledge a second time on a different platform since you _will_ run into scale problems no matter what language you use ?

From my experience in dealing with PHP performance warts. You can make it somewhat fast, but man the insides are so messed up every time I get into the core and try to do anything I want to rip my hair out.

nicholassmith3• 14 years ago

Thanks for the response.

That's a big part of my point, but Facebook seems to have invested a lot in the HipHop compiler and VM as a way to tackle the rough spots from using php. All languages are in their own way rough in places, but I was more curious about whether anyone has gone "this is great but maybe we're at the symptoms and not the cause". But you're right about the total investment being way, way larger than just a code base rewrite.

Interesting to learn that you do use multi languages for different things, a lot gets made of HipHop and Facebooks use of php that I didn't think to guess that different levels of the service might be getting built with different tools, at least not at scale.

jebblue• 14 years ago

>> Can't say I'm the biggest fan of Facebook as a service but what their engineers are doing in terms of pushing the state of the art is fantastic.

They let people upload comments and pictures. Anything fancy is done mostly in the client or am I overlooking something?

j_col• 14 years ago

> They let hundreds of millions of people upload billions of comments and pictures

Fixed it for you. Scale makes simple things difficult.

nicholassmith3• 14 years ago

Well, for a non-client example they do a lot of analytics, probably more than we know. Plus they created their own highly optimised php compiler from scratch that isn't just production ready, but battle hardened.

HipHop alone should get Facebook's development team praise.

debacle3• 14 years ago

The PHP language is performant just fine, though it has a few warts (that could be solved by making HHVM support a superset of PHP).

The problem is that the interpreter is shite.

ck22• 14 years ago

By the way, there is an alternative to HipHop that is actually easier to implement because it makes php extensions.

It's called PHC http://phpcompiler.org and it's open source.

It was Paul Biggar's PhD thesis http://blog.paulbiggar.com/archive/a-rant-about-php-compiler...

It just needs some community contributions to catch up.

pbiggar25• 14 years ago

While you are very kind to say so, even the original Hiphop blew past what phc could do. Importantly, phc did not support the full PHP language (we had basic object support, and no support for magic methods).

There were two cool things about phc: that it had a really advanced static analyzer, and that it compiled to modules compatible with zend. However, these are somewhat mutually exclusive, and making them work well together is an open problem (though I had some ideas).

I moved on from phc, and there's no-one left in the community really. I now run https://circleci.com (applying my compiler knowledge in a different way). I tried to pass the reigns on, but nobody stepped up. Hard to find people who love PHP but are also able to hack on compilers.

ck22• 14 years ago

Maybe you were just half a decade before your time and the rest of us had to catch up.

I worry the php community is losing or has lost some of the really smart folks.

Suhosin (Stefan Esser) is also fading away without any updates to work with php 5.4 and that's really sad, especially considering the radical performance improvements and memory-use reductions found in 5.4 and 5.5 vs 5.2 - 5.3

runekaagaard• 14 years ago

Slightly OT, sorry. phc was such a nice project, and vastly unappreciated. Your cleanup of PHP's grammar was a piece of art which I spend a long time perusing through and trying to understand! phc could have largely benefited the PHP community as a whole and it's such a shame it couldn't gain (enough) traction!

Never got a chance to say thx, so THX!

rynop2• 14 years ago

First off thanks for all the hard work. Do you have a list of the php extensions you support? I’m wondering if things like cURL, memcache, pdo MySQL etc are supported. I found https://github.com/facebook/hiphop-php/wiki/Extensions-and-m... but its a bit outdated (last mod 2yrs ago).

Also wondering what APC methods you support.

rynop2• 14 years ago

Found ans here: https://github.com/facebook/hiphop-php/tree/master/src/runti...

pbiggar25• 14 years ago

This is curious. Their trace-based approach looks like Mozilla's Tracemonkey - even using the same terminology like side-exits. Mozilla discontinued Tracemonkey because it was really good for deep loops and not much else. They moved to JaegerMonkey, which is a method-compiled VM like v8 (at the time), and are now moving to IonMonkey, which is a best-of-all-worlds version.

So I'd love to hear why using a circa-2009 technology was the right one? Is PHP sufficiently different from Javascript for this to make sense (as someone who has worked on VMs for both, I think there's a good chance of that)? Why not use a method-compiler instead? Very interested in the answers and comparison to other JITs out there, if any HHVM people are here.

pbiggar25• 14 years ago

One more thing - the use of a stack-based bytecode is also curious. PHP itself (the Zend engine) uses a register-based bytecode (well, it uses a horror-show of an in-memory bytecode-like-thing, which could best be described as a register-based bytecode). The PHP engine details leak into the language a lot, so the difficulties they describe aren't unexpected.

Not only that, but bleeding-edge JITs typically use register-based bytecode. Java's JIT and all the modern versions of Javascript VMs use register-based (in many cases actually translating from stack-based bytecode that the parser outputs, to register-based bytecode consumed by the JIT compilers). Since 2003, when my PhD advisor and his co-authors (Google "David Gregg" and "Anton Ertl") showed register VMs could be much more efficient that stack-based ones, the jury has settled on register-based VMs.

So I'm curious as to why a stack-based instruction set was used in the VM design?

eropple2• 14 years ago

Java's HotSpot is a stack machine, AFAIK. Dalvik is a register machine, but I wouldn't want to point to that as a positive. ;)

sb5• 14 years ago

AFAICT, two reasons come to mind:

1) A stack-based ISA is still more space-compact. (AFAIR the Shi et al. paper mentions something like a 40+% increase in space requirement for the instructions.)

2) The performance improvement of a register-based ISA is only visible for interpreters that suffer most from instruction dispatch [1]. PHP is a rather complex programming language that is most certainly not bound by instruction dispatch at all. So I guess it could very well make sense to stick with a simple stack-based ISA, which incidentally is also easier to compile from the AST.

[1] for the sake of completeness: the stack architecture emits many instructions to push operands onto the operand stack. A register-based interpreter does not need those. Hence, the overall number of dispatches is lower, and if dispatch cost is your bottleneck the overall performance increases. OTOH, you need more space, because in addition to the opcodes, you need to specify/encode which registers to take operands from and put results to. (Hint: quadruple code and the likes.)

masklinn2• 14 years ago

> all the modern versions of Javascript VMs use register-based (in many cases actually translating from stack-based bytecode that the parser outputs, to register-based bytecode consumed by the JIT compilers)

Pretty sure V8 doesn't as it has no interpreter at all, it has a fast JIT and a slow JIT, but it only ever runs native code.

dparoski• 14 years ago

As noted elsewhere in this thread, a stack-based design typically produces more compact bytecode. Compactness was a concern for us because of the size of FB's PHP code base. Also, generally speaking a stack-based design tends to be easier to deal with when working to get a prototype VM up and running quickly.

Many of the advantages of register-based designs (ability to optimize by rewriting the program at the bytecode level, ability to write faster interpreters, ability to map bytecode registers to physical registers, etc.) weren't particularly attractive to us because we knew we were going to build an x64 JIT that did its own analysis and optimization to take advantage of type information observed at run time.

Thus, we drafted a stack-based design for HipHop bytecode. It captured PHP's semantics correctly and happened to fit in fairly well with PHP's evaluation order, so we ran with it and here we are.

kmavm7• 14 years ago

It isn't tracing. Don't be fooled by the name "tracelet".

Unlike trace trees, tracelets contain no control flow. It is just a type-specialized basic block. This means that the jit output does not combinatorially explode when we encounter polymorphism. In many ways, our approach is the opposite of tracing; we jit the first time we hit any code at all. And we look terrible on loopy microbench code, but run FB's multi million LOC application very well.

pbiggar25• 14 years ago

Very interesting. Rereading with that insight makes your approach clearer - side exits aren't control flow, they're for the weird shit that PHP can throw at you. Aliased local variables are one of the major differences between PHP and Javascript, and that's a really good way of handling them.

So it avoid trace explosion, and I would describe this as a sort-of local (in the compiler sense of "basic block") version of a method compiler. Can you compare the approaches?

masklinn2• 14 years ago

So it's closer to V8's "basic" JIT?

sb5• 14 years ago

This is somewhat different from my memory. AFAIR, TraceMonkey was a trace-based JIT. JaegerMonkey was a JIT (however not just-in-time like V8 with a template-based JIT, but still using an interpreter initially). IonMonkey is--to the best of my knowledge--JaegerMonkey plus type inference (there was a paper by two Mozilla employees at PLDI'12 about their type inference.) So I guess it's not really using anything from trace compilation.

I think a trace-JIT still gives you a lot of bang for the buck and is (in theory at least) easier to implement. Two known projects using trace-compilation are LuaJIT2 (usually well known) and Dalvik VM's JIT compiler (not so well known, needed to watch the Google I/O 2010 announcement.)

rpearl• 14 years ago

There's a bunch of not-quite-correct information here... Let me try to summarize. Jaegermonkey is a JIT. It takes spidermonkey bytecode, and when functions grow hot, it compiles them to native code in a way quite similar to v8's full AST compiler (wit no optimizations).

IonMonkey is also a JIT. However, instead of just doing translations directly from bytecode to native code, it builds up an SSA-based IR, does some optimizations (last I checked, loop-invariant code motion, range analysis, global value numbering, on-stack replacement to be able to jump to JITted code in the middle of a hot loop, and some other small optimizations). It then does register allocation using LSRA. It takes advantage of type inference to aggressively type-specialize these optimizations.

This is all more expensive than just blatting out native code, but the code performs better.

In IonMonkey-enabled Firefox (Firefox 18+), the compilation strategy is to interpret, then use JM on hot functions. Particularly hot functions will get IM compiled.

This is similar to what "crankshaft", v8's version, does. The compilation strategy is quite similar, but v8 has no interpreter--it has two compilations: a fast, non-optimized one and a slow, optimized one.

Trace-based computation is...I would say... more complicated than a traditional compiler approach. It is certainly differently-complicated than the usual compiler problem. Recording traces requires some relatively complicated state, and significantly muddle up the interpreter. Removing the tracer from the Mozilla code base made things substantially cleaner--the tracer was holding back the method-based JITs.

monopede2• 14 years ago

Note that Dalvik is actually not a very good trace JIT. There's a recent paper comparing Dalvik to an embedded version of Hotspot. The Dalvik interpreter was slightly faster (~5%), but the JIT speedup was only 3-4x whereas it was ~12x for the method JIT. The reason is that Dalvik's traces are way too short and cannot even span method boundaries. That means little scope for optimisation and a lot of code just to load the state from the stack into registers and then write it back. Weirdly enough, Dalvik also generates more code than the method JIT, yet low amount of compilation overhead was their main goal.

Fortunately for Dalvik, most applications actually spend most of their time in native code.

The paper (behind ACM paywall): http://dl.acm.org/citation.cfm?doid=2388936.2388956

mcmire• 14 years ago

Does anyone else think this is a tremendous waste of time for Facebook? I mean, obviously PHP powers a lot of stuff at Facebook, I'm sure there are zillions of lines of code they can't just replace today, and now they're forced to make it scale. I don't mean to be negative -- building faster, better systems is inspiring, and the stuff they are doing with PHP is pretty neat, and there are really smart people trying to figure this stuff out. But, you have to ask, why are they still using PHP?? Why not use some of the new stuff that's out there now or heck, why not go with the JVM instead of reinventing the wheel here?

No I'm genuinely asking. Isn't some of the stuff here already being done by other languages, or is Facebook really breaking new ground here? (yes lbrandy I saw your earlier comment)

mitchellh• 14 years ago

My personal opinion on it is that this is in fact a very efficient way of doing things at Facebook. Facebook probably doesn't hire more than a couple people working on HHVM. Let's say their salaries individually are around $100,000 a year. That's cheap for them versus the value they're getting out of it.

Facebook probably has millions of lines of PHP code, PHP code proven to work. This isn't just Facebook.com, but internal websites, their bug tracker software (that is open source), analytical software for PHP, etc. etc. Rewriting all that would be _expensive_.

Running it on normal Zend PHP, based on the benchmarks they get on HipHop, requires _way_ more hardware at the scale of Facebook, so again, very expensive.

PHP developers are abundant, PHP is a relatively easy language to teach, resources are everywhere. Due to their abundance, PHP developers (even at Facebook quality) are probably less expensive than other languages. Easier hiring and lower cost developers, cheaper.

maratd4• 14 years ago

> But, you have to ask, why are they still using PHP??

If you read the comments, you'll find the answer further down. They do address it. At their scale, all languages/platforms break and they would need to invest the same amount of time and effort getting it to work.

It won't make a difference if you switch to the latest hipster language. You'll still need an effort of this magnitude to get it to scale the way they need it to.

Furthermore, rewriting an entire code-base is a monumental undertaking that is both expensive and complex. Especially for something as large and diverse as Facebook. It also makes zero business sense and Facebook is a business.

Why should they take a risk with a new code-base, when what they're doing is clearly working? PHP just works.

pjmlp2• 14 years ago

Because you also need to consider the human factor.

Changing languages in the scale of Facebook, means that everyone, internal or external, need to know the new language.

This transition is a big process, which usually costs much more money in trainings and loss of productivity, than having a dedicated team improving the performance of existing tools.

lectrick• 14 years ago

Like putting a rocket on a turd.

Speeding up PHP with the HipHop VM

Related Stories