Linux dominates supercomputers as never before

http://www.zdnet.com/linux-dominates-supercomputers-as-never-before-7000030890/

by tanglesome • 12 years ago

129 81 12 years ago

Alupis4• 12 years ago

I really amazes me, that Linux is so darn flexible that it runs on some of the world's largest machines, down to the smallest embedded devices.

After years of being a Windows fanboy -- I look back now and am very glad I made the switch. With Linux -- it's never a case of "you can't do that".

TallGuyShort3• 12 years ago

What I think is more amazing is that with the exception of the 2 Windows systems, EVERYTHING is either AIX UNIX or Linux, or a combination. UNIX started as a "microcomputer" OS, but it really has proven to be incredibly flexible. Personally, I think the ideas that have come out of the Solaris and BSD communities are invaluable and I always like to see them included in projects - I'm a little disappointed they have lost favor in the HPC community. They are similar enough to Linux and other UNIX-like OSs that targeting all of them is relatively practical in most cases, and users will find similar interfaces everywhere, but their ideas, communities and approaches are different enough that I think they would prevent it from being a monoculture.

_delirium2• 12 years ago

One of the things that imo has pushed some of the other Unixes out of HPC is the move to using cluster file systems. Any Unix supports a traditional NFS-mounted NAS solution for storage, as used to be typical in HPC clusters (and still is fairly common in smaller installations at universities). But only AIX and Linux really have solid cluster filesystems; there's nothing in the BSD world analogous to IBM GPFS, or to Lustre, and porting is nontrivial.

kjs34• 12 years ago

UNIX started as a "microcomputer" OS

One very minor nit...the PDP-7 where Unix started out cost US$72k+ in 1969 dollars. Nothing compared to the big iron of the day, but very far from a microcomputer (and, really, that market didn't exist until 1972 or so).

hadrianoliveira• 12 years ago

It also really amazes me how vendors make crappy drivers for linux compared to windows

ckaygusu• 12 years ago

Seconded. I wonder if Microsoft aggresively pays people to not to develop drivers for other operating systems.

Alupis4• 12 years ago

This is not really true anymore.

About 99% of all devices "just work" out of the box, the other 1% either are very obscure/specialty hardware, or are new hardware. Linux doesn't need a vendor to support Linux like it used to... most generic drivers are built in and work OK (just like the generic drivers built into Windows now-a-days).

alephnil3• 12 years ago

That is not really the case for supercomputers though. Here Linux drivers obviously have first priority.

If you are thinking of video driver it is a totally different story.

venomsnake• 12 years ago

Except you "can't play your game library". Lets hope for some really good compat layer from valve.

Alupis4• 12 years ago

Well, Valve already has ToGL, a DirectX2OpenGL abstraction layer... so that is already happening. Also, SteamOS (and Steam on Linux) now has low latency Streaming from your Windows pc to your Linux PC, SteamOS/SteamBox on your TV, etc.

So, in large part, just about all games are now playable on Linux, and more are coming as developers re-release games on Linux, or as many big AAA studios have committed to releasing all future games on linux as well (Crytek, Valve, Unreal, Firaxis Game, etc etc... there are a lot and more coming!)

CraigJPerry• 12 years ago

What does Linux not dominate?

I can think of only a few categories: desktops, ATM machines, real time, SCADA.

What are the remaining big sectors Linux doesn't yet dominate?

epochwolf• 12 years ago

Point of Sale machines, machine controllers, etc.

marcosdumay• 12 years ago

Desktop and laptop computers. Also CAD plataforms (that are a subset of the first, but an important subset).

legulere• 12 years ago

Real time computing, routers, sensors

dublinben2• 12 years ago

More routers run linux than you'd think. There's very little reason to reinvent basic networking and routing components and a kernel, when all the common SoCs are supported by a free OS. For the small price of stashing a source code bundle somewhere on their website (if even!) companies can get most of what makes their product tick off the shelf.

ptfoobar• 12 years ago

RT is a matter of time (no pun intended). In fact hard real-time is already done using Linux and RT_PREEMPT is believed to go in the main tree soon.

nardi• 12 years ago

How has no one mentioned mobile? Smartphones/tablets? You know, the largest segment of general computation devices?

gamegoblin• 12 years ago

Android is based on the Linux kernel at least.

azdle• 12 years ago

How has no one mentioned mobile?

Because Linux does dominate mobile? Android is Linux and iOS is a *nux. (iOS is a BSD derivative.)

robotresearcher• 12 years ago

The profitable part of the cellphone business.

kiram92• 12 years ago

A lot of broadcast TV is all windows/mac. Even for things like playout servers, automation, and real time graphics engines mostly run on windows or mac.

pjc502• 12 years ago

Pretty much all end-user facing systems?

(Android barely counts as Linux for reasons discussed in my other comment)

odonnellryan• 12 years ago

Small and medium sized business servers?

lucidguppy• 12 years ago

What alternatives are there really?

onalark2• 12 years ago

It's worth pointing out that this is what most people think of as the traditional "Linux Operating System" but sans the Linux kernel.

Also, there are a few components on these supercomputers that you won't find on a typical workstation or cluster machine.

First, the kernel itself is frequently very small, lightweight, and much closer to what you would find on an embedded system than on a traditional desktop computer. That's because once the program on a supercomputer is loaded, the kernel's job has been mostly to get out of the way. This isn't to say that you don't find the Linux kernel on these supercomputers, it just isn't as common as you would think from reading this piece.

Fear not, many pieces of these operating systems are still open source. Here's IBM's fusedOS prototype:https://github.com/ibm-research/fusedos.

Second, many of these computers primarily only run code in C, C++, Fortran, and Python. These tend to be the only major languages in play on the HPC machines, with acceleration frameworks such as OpenMP, OpenCL, and CUDA playing major roles.

Finally, everything is glued together with MPI, a high-level (at least it was in the 90s) abstraction for scientific programming that maps down to very high-performance networks designed to help scientific codes "scale", that is, run effectively when millions of cores are simultaneously engaged.

These are beautiful machines producing important science, and the GNU/Linux operating system plays an incredibly important role in both their implementation and culture.

gh02t3• 12 years ago

This is not exactly true, most of the ones I know are running straight up Linux. Titan, for example, runs basically SLES on the login nodes and Compute Node Linux on the compute nodes. CNL is a pared down version of Linux, but it definitely is the "real" Linux kernel with the functionality you'd expect to be there.

Most are just slightly spruced up commodity server hardware running Linux. I'm not sure if this is what you're suggesting, but they don't run C/Fortran/whatever on bare metal. They're run by the OS on the compute node just like a normal OS process, except that tasks are dispatched to compute nodes by a central cluster manager. Processes running in a gang communicate via MPI to share data, though coprocessors are also pretty popular as well so you see a lot of communication between the host processor and a coprocessor too. Titan and Tianhe both actually have most of their compute power in the coprocessors (Xeon Phi and Nvidia Tesla, respectively), but they're still arranged in a master-slave arrangement just like if you bought a Phi or Tesla and stuck it in a spare PCI-E slot. They use plain old PCI-E, too. The Cray XT/XE series (a popular model of which Titan is an example) is basically just really nice blades with integrated cooling and a network backbone in a custom cabinet and possibly coprocessors attached to each blade. You could just as easily run Windows XP and play Minesweeper on each blade if you really wanted to, except maybe for some driver issues. The most foreign thing is probably the network backbones, where fabric architectures like Infiniband are popular.

They're also not limited to specific programming languages. In truth, you can run whatever you want if someone has paid the bill for your resource allocation. I watch people run MATLAB on large clusters all the time, which hurts me because it's so damned inefficient. That said, Fortran and C++ comprise the overwhelming majority of large and computationally taxing codes. Just because all that power is there doesn't mean that all of the users take proper advantage of it. One of the larger calculations run on Titan that I know of (Denovo, a nuclear reactor simulation code) didn't even use the GPUs, only the CPUs. Making codes that can take advantage of GPU processing ergo Titan and its predecessor Jaguar has been a major project at the DOE, with libraries like Trilinos being developed to make it easier on scientists, many of whom are only computer programmers as a secondary concern.

The setup you're describing used to be how it was until up to maybe 10 years ago and there are still systems in the top 500 that work like that. Probably some new ones being built, too. But what I've described is what seems to be in fashion these days mostly and the machines I use are all like that. I've heard mumblings about FPGA coprocessors being the Next Big Thing, but we will see.

onalark2• 12 years ago

Yup, thanks for the comment. I think FPGA has interesting potential for sequencing (and we need more sequencing compute), I'm not sure what's ahead for the simulation machines.

nroose• 12 years ago

Um... That graph goes up to 600. Kind of an epic fail when it is an article about the fastest computers in the world.

mnw21cam• 12 years ago

Yes, and also a fairly basic research fail, referring to the ECMWF's old supercomputer at position 60 as the UK's weather predicting system. They have couple of new systems at positions 19 and 20. ECMWF is for Europe, not UK, even if it is based in Reading. The key is in the name.

zurn• 12 years ago

Another way to put it is that specialized supercomputers no longer exist, and the top500 is now populated with super-branded racks of PCs running Linux with souped up network cards.

For the young ones in the audience, the supercomputer term has historically been used to describe machines with largely custom hardware from the CPUs up - or at least from the CPUs onwards. Cray vector machines are the canonical example. These were shared-memory / single-system-image machines, software really saw it as one computer instead of a networked cluster.

alephnil3• 12 years ago

I don't think this description is entirely fair. What is expensive with these computers is the fast interconnection buses between processors and modules, which is needed for many scientific problems that can't easily be divided into independent subproblems. Just having a large cluseter of computers connected with normal Ethernet would not cut it. OK, you say souped out network cards, and if you had said souped up switches as well, I could have given you some points, but this is the major cost of modern supercomputers. You can get a non souped-up cluster with the same number of processors for a one digit percentage of the price of the supercomputer.

walshemj• 12 years ago

Some hadoop clusters are looking at/using infiniband - cluster network design depends on what sort of problems your dealing with.

exDM69• 12 years ago

Another way to put it is that specialized supercomputers no longer exist, and the top500 is now populated with super-branded racks of PCs running Linux with souped up network cards.

While there's a grain of truth in this, in reality, a supercomputer still looks like a room full of refrigerators and not at all like a typical computer or even a server farm.

Some of the computer hardware in there is just commodity hardware but for example the interconnects can't really be described as "souped up network cards", they're in a category of their own.

Yes, the era of the Cray boxes is gone but a supercomputer is still a supercomputer.

weland2• 12 years ago

They're not exactly "super-branded" racks of "PC"s the way we tend to think about them, a bunch of white/beige boxes. They are server-grade computers, with all that comes there (special interconnect features, ECC memory and all that), but there is indeed no question as to their distributed nature.

This is probably a technological response to the rapid pace of development. It's more efficient this way, instead of developing custom chips and boards for two years and deploying them when the high pace of advance makes them half-obsolete.

dekhn• 12 years ago

Nope, supercomputer has and always will refer to capability of the machine, not the details of its implementation.

TallGuyShort3• 12 years ago

In addition to just the networking hardware, the networking stacks and topologies are very different. I would consider something a supercomputer if it was using a high-dimension torus topology, and some MPI variant, and a different internet / transport protocol (I forget the name, but as I understand it's a very different approach where there's much less decision making in the routers). On the other hand, what I would consider to be a cluster is almost going to use off-the-shelf networking hardware (as you mention), TCP/IP, and a simpler hierarchical topology with a switch or two per rack.

apw• 12 years ago

The IBM Blue Gene/Q cannot in any reasonable way be described as "racks of PCs running Linux with souped up network cards".

bch• 12 years ago

What _are_ examples of HPC installations that *BSD is key? Free, Net, Dragon Fly, Open?

piokuc• 12 years ago

In the article it was mentioned that ECMWF's supercomputer was the fastest one of those running not on Linux. It's worth mentioning that ECMWF is in the process of migrating to an XC30 Cray system which runs on Linux.

jason_slack• 12 years ago

I think that I read a long while back that there was a cluster of PlayStation 3's, all running Linux that was pretty fast too.

Edit: Here is one of the articles: http://phys.org/news/2010-12-air-playstation-3s-supercompute...

Linux dominates supercomputers as never before

Related Stories