Summit Supercomputer Up and Running, Claims First Exascale Application

https://www.top500.org/news/summit-up-and-running-at-oak-ridge-claims-first-exascale-application/

by rbanffy • 8 years ago

57 55 8 years ago

top500.org

taliesinb• 8 years ago

27,648 Nvidia Tesla V100 GPU modules

holy shit

dogma11383• 8 years ago

I always wondered what’s the bulk discount on these since it’s north of $200,000,000 for GPUs alone.

tempdeadbeef4• 8 years ago

Bulk discount when currency miners will pay full price?

IncRnd• 8 years ago

I'm not sure you can get a discount for any purchase, since 100% will be sold anyway.

gh02t3• 8 years ago

This article suggests that Summit cost somewhere in the low $100 million range and last I heard from DoE that was accurate, so they aren't exactly paying retail.

https://www.nextplatform.com/2016/11/20/details-emerge-summi...

Price isn't really a concern with these computers, though, because the sort of experimental work they are intended to "replace" (using that loosely, since a lot of the things they simulate are impossible to actually do) is far more expensive. Leadership-grade computing is all about enabling new classes of problems to be solved.

tempdeadbeef4• 8 years ago

What’s not inside the Summit Supercomputer speaks volumes: Intel.

Knights Landing/Hill/Mill is simply not compelling; Omni-Path was created as an infiniband knockoff that doesn’t beat Mellanox. The Cray Gemini/Aries interconnects can be found all over the top of the list (and the Intel acquisition of those interconnects happened in 2012), but you don’t see Omni-Path replacing anything.

Meanwhile, Nvidia comes out with NVLink and begins to build small clusters of GPUs connected by larger networks containing IBM and Mellanox. A vacuum was created, and IBM and Mellanox moved (back) in.

quadruplebond4• 8 years ago

Yes, but rumor has it that Intel will still be the supplier for Argonne’s next computer. Which if everything goes to plan will be the first exascale machine.

rbanffy3• 8 years ago

Even the previous generation Phi's pack about 3.x peak GFLOPS per socket. Tuning HBM size and AVX512 pipelines can certainly help increase that, even if their 10nm is proving harder than expected. It's a matter of time before they can go full 10nm.

davidmr• 8 years ago

The DOE (and DOD, but with whom I’m less familiar) tends to spread out these purchases over multiple vendors to keep multiple US-based providers able to build and support these machines (and I imagine to keep costs competitive).

The last few acquisitions by ORNL and LANL have been Crays while ANL and LLNL were buying IBM Blue Genes. With this generation, it looks like things have switched. As another poster mentioned, it certainly seems like ANL’s next one will be Cray/Intel. It was going to be based on Knight’s Hill, but Intel cancelling that sort of put the architecture up for grabs.

tempdeadbeef4• 8 years ago

Intel knows better than anyone that they need to sell a roadmap, not a chip. Who’s going to put in a large advance order on a bunch of future silicon that may get cancelled as well?

rbanffy3• 8 years ago

I would love to see Intel tweaking the Phi line with asymmetric cores like some ARMs do. Having a couple brawny proper Xeon cores and a bunch of smaller 4-thread cores, all coupled with local HBM (and maybe some dedicated HBM for each core) would make it a very versatile part that, with some tuning in number of cores, cache sizes, HBM size, etc, could cover from low-end server all the way to supercomputing.

I don't think there is much doubt core count will increase on all segments and that asymmetric core tech that's currently used in ARM is pretty cool.

sliken• 8 years ago

Omni-path isn't support on AMD chips, let alone Power 9. I don't agree with the decisions, but there you go. We switched from 3 generations of Pathscale IB adapters to Mellanox because of it.

greglindahl2• 8 years ago

Knockoff? Infinipath used Infiniband at first because we could only build one chip, not a host adapter and a switch. Turns out that Infiniband is flexible enough that we kept on doing that for several generations.

tempdeadbeef4• 8 years ago

Omni-Path Gen1 has a Wolf River NIC and a Prairie River Switch.

jabl• 8 years ago

Sometimes I wonder about the longer-term future of infiniband. Originally it was a multi vendor standard, but nowadays it's a one-shop show. Squeezed from above by high end networks supporting things like adaptive non-minimal routing, and from below by ethernet (roce etc.) And squeezed from the side by Intel with their deep pockets.

And apparently mlnx is pressured by some activist investors to reduce R&D expenses and pay more dividends instead.

gnufx2• 8 years ago

You certainly see OPA replacing IB -- I personally know several examples, though some HPC vendors now seem to be making it difficult to buy. It has a fairly healthy showing at top500, considering it hasn't been widely available for long. I know the Mellanox propaganda, and have some experience of it, along with promises of help to make it work that evaporated.

I can't remember where to look for OPA's features to help MPI implementation, but someone else might be able to comment.

yaschobob• 8 years ago

This is very short sighted. Just wait a month or so and this machine will not look like anything special. ORNL typically doesn't use Intel.

fermienrico• 8 years ago

This is a poorly written article.

Here is a better source: https://www.top500.org/news/summit-up-and-running-at-oak-rid...

Interesting bit about nVidia Tesla V100 GPUs:

Assuming all those nodes are fully equipped, the GPUs alone will provide 215 peak petaflops at double precision. Also, since each V100 also delivers 125 teraflops of mixed precision, Tensor Core operations, the system’s peak rating for deep learning performance is something on the order of 3.3 exaflops.

Those exaflops are not just theoretical either. According to ORNL director Thomas Zacharia, even before the machine was fully built, researchers had run a comparative genomics code at 1.88 exaflops using the Tensor Core capability of the GPUs. The application was rummaging through genomes looking for patterns indicative of certain conditions. “This is the first time anyone has broken the exascale barrier,” noted Zacharia.

distortedlojik• 8 years ago

Agreed, check out the top500 article or the direct ORNL System Overview. https://www.olcf.ornl.gov/for-users/system-user-guides/summi...

dang• 8 years ago

Ok, we've changed the URL to that from mhttps://www.eetimes.com/author.asp?section_id=36&doc_id=1333...

yread• 8 years ago

I wonder what the application was. There is not that much deep learning in genomics.

alfalfasprout• 8 years ago

The tensor cores are simply matrix multiplication + accumulation. They don't need to be used purely for deep learning.

dekhn• 8 years ago

The code is Comet https://www.sanger.ac.uk/science/tools/comet and based on the PR I think they ran it in embarassingly (or lightly) parallel mode (IE, very little if any communication between nodes except for periodic reduces.

agumonkey• 8 years ago

I think I still have my nv3 graphics card. Fun to think of how Nvidia grew.

newnewpdro3• 8 years ago

"Competition remains stuff."

newnewpdro3• 8 years ago

Is there anything preventing them from mining crypto as a burn-in to help subsidize the costs?

qop• 8 years ago

Yeah, common sense.

Also, actual science work that they have to do.

newnewpdro3• 8 years ago

Yeah, common sense

Could you explain

Also, actual science work that they have to do.

Presumably these clusters aren't 100% utilized 100% of the time. They certainly weren't at the national lab I had access to ages ago...

My question was more inquiring if there's any red tape preventing the laboratory from paying the bills with mining, if it could do so profitably. Let's just assume there's idle time where this could hypothetically occur.

I'm not even trying to suggest that they should do that, it's just an interesting relatively new possibility and these things are quite expensive.

vertexFarm• 8 years ago

lol yeah, they have actual productive things to work on.

kristianp• 8 years ago

Why do they need expensive Power 9s when the bulk of the processing is done by GPUs?

bryanlarsen• 8 years ago

Because Power 9's have on-chip NVLink ports.

JohannFlobuster• 8 years ago

This. CPUs are hungry.

crb002• 8 years ago

Because all the Blue Gene/L/P/Q code is POWER.

quadruplebond4• 8 years ago

Sort of, but Blue Gene tech was always an Argonne thing, Oak Ridge has been the one doing GPUs for awhile now. Unless something weird happens Argonne and Oak Ridge will always be running different types of computer since we don’t want to be locked into one specific vendor or hardware configuration.

quadruplebond4• 8 years ago

Partly because we don’t want to be locked into using Intel for both of DOE’s open science machines. In the lead up to the exascale or near exascale machines (2012-now) I don’t remember hearing much about AMD, so presumably Intel and IBM were the only vendors supplying general purpose CPUs.

code_duck• 8 years ago

I drive by a site with large signs declaring it to be an Exascale computing construction project at Los Alamos, but I'm not sure what they're building. I'm sure they've announced it somewhere.

Summit Supercomputer Up and Running, Claims First Exascale Application

Related Stories