Computing at scale, or, how Google has warped my brain

http://matt-welsh.blogspot.com/2010/10/computing-at-scale-or-how-google-has.html

by adambyrtek • 16 years ago

119 43 16 years ago

matt-welsh.blogspot.com

retube3• 16 years ago

When I was at CERN 10 years ago, the set up was similar. Instead of Macs the desktops were PCs. The code was Fortran, Perl and c++. All work was on remote Linux boxes (typically exporting a desktop over X), although physical not virtual OS instances. You'd ssh to a gateway IP and be logged into one of the hosts. All data and user accounts were stored on a distributed filesystem AFS. There was job scheduling and management middleware that'd batch process your analysis across hundreds of hosts in the datacentre.

"The cloud" has been around for a while, although implementation has changed.

irahul3• 16 years ago

Yahoo uses a similar setup. Instead of virtual hosts, the hosts here are physical machines. Search team had a dedicated lab(not sure what's happening now with search migration) and all developers would ssh to their assigned lab machines, fire up their vim/emacs and crank code.

Search team has a mixture of windows laptops, macs, linux desktops, linux laptops..and it didn't matter because everyone just ssh'd to a RHEL box and worked.

Yahoo used to be a BSD shop, then moved to linux. Search team used both Debian and RHEL in the beginning, then transitioned to RHEL fulltime.

I don't understand the obsession with RHEL. In my experience, Debian/Ubuntu will give the same benefits as RHEL sans the cost.

The rationale for moving from BSD to linux was practical. The gist was most of the vendors either do not do bsd or bsd was second class citizen. And BSD wasn't as quick as linux in catching up with esoteric hardware. But discarding Debian didn't actually make sense to me.

retube3• 16 years ago

CERN used RH too, 6 when I was there. I believe they've now switched to Scientific Linux.

Actually I don't understand why Google would use virtualisation. Why would they split a physical box up? Making many hosts look like one, yes (is such a thing possible?), but not the other way around.

And AFS was cool: it was actually a global filesystem that many institutions used. you could cd /, ls and browse to nasa.gov amongst many thousands of others (much was protected!) (cern was mounted on /cern.ch)

cperciva2• 16 years ago

Yahoo used to be a BSD shop, then moved to linux.

Yahoo still uses BSD. They're not completely BSD, of course; but they're not completely linux either.

regularfry• 16 years ago

I am reminded of a William Gibson quote: "The future is here, it's just not evenly distributed."

swah• 16 years ago

Huh? His "dumb terminal" is a Mac laptop that will run on a single machine most less-than-Google-sized tasks...

mechanical_fish2• 16 years ago

Even if you are avoiding Google-sized tasks, there turn out to be advantages to treating your local machine as a dumb terminal.

One is that remote machines can be rented instead of being bought.

Another is that the configuration of everything becomes easier. You buy new laptops by taking them out of the box, running a handful of app installers, and moving over a SSH key or two and an encrypted password file. Your career as amateur sysadmin might just be over. (If you miss the chance to build your own Linux kernel, of course, there's plenty of work left for you as a cloud sysadmin. But cloud system admin is also usually easier, because machines in data centers come with nifty administration tools, like SANs and dashboards, that benefit from economies of scale.)

Security: When you lose the "dumb terminal" you lose no data. You do lose the keys that are on the machine, but you can revoke those. And it is much, much easier to get consistent automated backups on a datacenter box than on a laptop.

There are definite problems with the "cloud" approach, such as "who owns the data and how do you trust it is being handled properly", but the advantages are strong these days.

adambyrtek3• 16 years ago

It's more about convenience than resources. For example I'm very happy that I don't have to carry my laptop with me anymore as I can always reconnect to my virtual machine from home and continue at the same place where I left.

jedbrown6• 16 years ago

Forget about complex shared-memory or message passing architectures: that stuff doesn't scale, and is so incredibly brittle anyway (think about what happens to an MPI program if one core goes offline).

MPI is not good at fault tolerance, though MPI-3 will help some. MapReduce and Sawzall are well-suited to the problems they were made for: largely independent operations on very large data sets with occasional reductions and weak synchronization. Usually disk is a significant factor and strong scalability is not especially important.

But they really can't compete with MPI for other problem domains. Claiming that MPI doesn't scale is demonstrably false, as evidenced by PDE solvers running on 200k - 300k cores (Jaguar, Jugene, etc). This is a comfortable order of magnitude larger than Google is running, and the communication requirements of those algorithms are far more demanding, with essential all-to-all communication as well as low-latency global reductions (for which Blue Gene has dedicated hardware). The need for these low-latency all-to-all operations is actually very fundamental (formally proven). The algorithms also tend to have much stricter synchronization requirements, for example, nothing can be done until everyone gets the result of a dot product, because the next operation needs to happen in a consistent subspace (though MPI has fine asynchronous support). There has been plenty of work trying to make algorithms less synchronous, but weakly synchronous algorithms almost uniformly have worse algorithmic scalability (number of iterations to solution increases by more than a constant factor). And since problem sizes are usually chosen to fit in memory, the network doesn't get a free pass due to the disk being slow.

If you care about strong scalability, MapReduce and Sawzall are probably not good choices. Similarly, if your problem domain requires low-latency all-to-all, reductions, or "ghost exchange".

Retric5• 16 years ago

200k - 300k cores... a comfortable order of magnitude larger

I would assume Google has well over over 20,000 cores. Or did you mean something else?

jedbrown6• 16 years ago

From talking with someone at Google Zurich who is involved in page-rank computations, those jobs run on ~10k to 20k cores. Of course Google has many more distributed around the world, but unless something has changed in the last few months, they are not running single jobs on more than about 20k cores.

snissn• 16 years ago

http://en.wikipedia.org/wiki/National_Center_for_Computation...

edw519• 16 years ago

So, printf() is your friend. Log everything your program does, and if something seems to go wrong, scour the logs to figure it out. Disk is cheap, so better to just log everything and sort it out later if something seems to be broken. There's little hope of doing real interactive debugging in this kind of environment, and most developers don't get shell access to the machines they are running on anyway.

1985: Interactive debuggers suck. PRINT() is your friend.

1990: Interactive debuggers have matured. No one uses PRINT() any more. Debugger questions are now even part of technical interviews.

2008: Concurrent processing and cloud computing have made interactive debugging difficult.

2010: printf() is your friend again.

Developing software is getting to be like fashion. Keep those old skinny jeans and workarounds in your closet. Sooner or later, they'll be in style again.

jakevoytko• 16 years ago

All projects are different. Use the most useful tool for your job, and pay little attention to other developers. They may be using something better, or they may just have different needs.

tibbon• 16 years ago

This seems to be even more interesting to compare the Ruby community (from what I've seen is massively test-driven and shuns use puts()), to the use of Python at Google. Of course, most Ruby stuff that I mentioned is Rails based, and there isn't that much concurrency happening and its set up well for testing- but just an observation.

adambyrtek3• 16 years ago

Google is extremely test-oriented, every project is covered by a test suite and there is a powerful distributed continuous integration infrastructure in place that executes tests "in the cloud" and doesn't allow to commit code that doesn't pass. Not to forget about the public Google Testing Blog[1] and the famous Testing on the Toilet[2].

People in the Ruby community also use temporary puts/logging statements for exploratory debugging, but of course nobody in right mind commits such code to the repository.

[1] http://googletesting.blogspot.com/

[2] http://googletesting.blogspot.com/2007/01/introducing-testin...

hvs2• 16 years ago

This just tells me that we need to figure out a new way to debug concurrent/cloud processes more easily. The traditional debuggers just don't cut it anymore.

aristidb• 16 years ago

How about trying to find errors as early as possible, by for example using the type system? That will not catch everything, but it will catch quite a lot.

polynomial• 16 years ago

> Developing software is getting to be like fashion. Keep those old skinny jeans and workarounds in your closet. Sooner or later, they'll be in style again.

Or is developing software like business? Sell those skinny jeans and workarounds when they are in demand, then buy some back when they go out of style. You're now ready for their comeback and you have some mad money in your skinny jeans' pocket.

wglb• 16 years ago

Actually around 1985, Mark Williams invented the C Source-Level Debugger, which certainly did not suck, and was quickly imitated by other larger outfits.

However, my use of debuggers has always been quite minimal, perhaps because despite the CSD experience, I am often behind the times. Or perhaps it was due to my early (not quite childhood) experience building real-time interrupt-rich programs in Sigma 5 assembly language.

I also read recently that most of the grownups don't use debuggers but have always stuck by printf or equivalent.

It kind of reminds me of an early attempt on my part, not wanting to do laundry in my early bachelor years, to start a wrinkled shirt fashion trend. I was not successful, at least for a large number of years. It finally did become fashion.

So stick with printf and build useful but lean logging habits would be my advice.

brown9-2• 16 years ago

I do all of my development work on a virtual Linux machine running in a datacenter somewhere -- I am not sure exactly where, not that it matters. I ssh into the virtual machine to do pretty much everything: edit code, fire off builds, run tests, etc. The systems I build are running in various datacenters and I rarely notice or care where they are physically located.

I love moments like this when you realize as a developer that the bits that are the fruit of your labor are off executing in some far off physical location you'll never know about.

global_hero• 16 years ago

This guy is a total lamer!

jessriedel• 16 years ago

>The idea that you need a physical machine close by to get any work done is completely out the window at this point. ... I ssh into the virtual machine to do pretty much everything: edit code, fire off builds, run tests, etc. ...Wide-area network latencies are low enough that this works fine for interactive use, even when I'm at home on my cable modem.

This situation sounds fine if you're a developer and your job requires only low-bandwidth text streams between you and your physical computer. But for anyone who does anything graphical, or for anyone at home who wants to enjoy media, the bandwidth and latency are big problems.

Maybe I should just assume that because this is on HN and written on a developer's blog, context is assumed. But it sure seems like a disconnect between hard programmers and the other 99% of the population.

Computing at scale, or, how Google has warped my brain

Related Stories