US Air Force connects 1,760 Playstation 3's to build supercomputer (2010)

https://phys.org/news/2010-12-air-playstation-3s-supercomputer.html

by jamesdhutton • 4 years ago

126 92 4 years ago

chrisma0• 4 years ago

This is one of those headlines that initially reads like a joke until you think of the economics of it. "cost is about 5-10% of the cost of an equivalent system built with off-the-shelf computer parts."

I wonder if this is due to the fact that the Playstation hardware is (was?) competitively priced to encourage revenue generation through games? Or was Sony simply very good at mass-producing these units?

throwaway0a5e• 4 years ago

I wonder if this is due to the fact that the Playstation hardware is (was?) competitively priced to encourage revenue generation through games? Or was Sony simply very good at mass-producing these units?

By 2010? Likely both.

manishsharan• 4 years ago

I thought PlayStation and XBoxes are subsidized by the Sony and Microsoft respectively as they make most of the money off the games and the network subscription. Nintendo is the only one that does not do that.

vel0city• 4 years ago

When it comes to network subscriptions, at the time of this article Playstation's online services were pretty much all free. It was one of the differentiators compared to Xbox's paid online services at the time.

That said though, the original PS3 hardware was definitely sold at a negative margin and wouldn't be profitable until four years into the console's life.

Article from 2010 talking about how the PS3 hardware had only just become profitable: https://www.pcworld.com/article/512740/article-4244.html

The console released in 2006.

chomp• 4 years ago

This is my understanding as well. A big part of this was the HD-DVD v Bluray war around that time - Sony winning with Bluray would be worth millions of dollars more, so they were willing to take a loss to get a Bluray player in each person's home.

paulmd3• 4 years ago

The PS3 was an extraordinarily expensive console, between the cell processor and the bluray drive (iirc this alone was several hundred dollars) and in early models also including a whole embedded ps2 implementation. So PS3 was sold at a loss, and not just a small one, but a heavy loss, like several hundred dollars per console, and even still the PS3 was derided for being far too expensive. It was a financial disaster for sony really, moves like ripping out the embedded ps2 make complete sense in that context, and they absolutely changed their business model for subsequent consoles.

Since then, consoles moved away from the exotic POWER/cell/etc custom hardware towards commodity x86 hardware based on integrated x86 APUs and haven't really been sold at a loss outside of maybe a small window at launch. PS5 moved into hardware profitability about 9 months after launch, microsoft said that the xbox series is still sold at a loss but I don't believe them because the xbox shouldn't be monumentally more expensive to build than the PS5. This is in the context of them trying to argue during the apple app store lawsuit that their lock-in on xbox store was different from the lock-in on the app store, so they have a financial incentive to make sure they "run a loss". It's either not much of a loss, or it's hollywood accounting and the money is going into their other pocket somewhere, like making the xbox division pay a parent holding company big licensing fees for on every console sold.

https://www.extremetech.com/gaming/325504-sony-finally-turns...

(again, I don't agree with the "finally" spin here, this article was roughly a year after launch and they may have been turning a profit for a while before disclosing it... the consoles themselves become profitable pretty quickly.)

However, this mindset that "consoles are sold at an initial loss" still persists. They're not, Sony has said they're selling the PS5 at a profit. Previous generations also reached profitability pretty quickly after launch as well. It's not 2005 anymore and the ps3 is gone. Slapping some GDDR5/GDDR6 on a semi-custom APU is dirt cheap.

Even during the launch window when they do lose money it's much smaller, nobody is losing a couple hundred dollars on each console anymore like on the PS3, that model is gone.

smoldesu2• 4 years ago

This is largely not true these days. The Switch sold at a loss at-launch, but slowly turned a tiny profit as shipping prices went down. The modern revisions (Switch Mini and OLED) are also priced similarly, so any profits they're making off the hardware itself is incredibly marginal.

dragontamer• 4 years ago

IBM cell processor was incredibly expensive.

The PS3 probably was the cheapest way to play with it in practice. There is a reason why modern GPUs share an architecture with video gamers.

It's not about technical advancement, as much as economics. There are two groups of people who want TFlops of SIMD compute. Supercomputer groups, and video gamers.

Cell / PS3 was one attempt at making one device work with both groups, sharing research and economic investment.

NVidia over the next 15 years would execute these economics better however.

solardev• 4 years ago

There are two groups of people who want TFlops of SIMD compute. Supercomputer groups, and video gamers.

And crypto people, unfortunately :(

riskneutral• 4 years ago

Sony subsidizes the cost of the hardware, in order to sell games. They were not happy with the Airforce doing this and it probably played a role in Sony pushing out an update that disabled Linux on all PS3s.

flutas2• 4 years ago

it probably played a role in Sony pushing out an update that disabled Linux on all PS3s

Pretty sure that was due to it being used to hack the PS3.

Blame for the latest culling has been pinned on computer hacker George Hotz, who was originally infamous for unlocking Apple's iPhone. In January of this year Hotz claimed that he had successfully hacked Sony's PS3 by exploiting Linux, gaining "read/write access to the entire system memory, and HV [hyper-visor] level access to the processor".
Hotz released this to the public on 26 January, boasting, "Sony may have difficulty patching the exploit". He may well have been right, since Sony's latest response has been to completely lock off the required 'Install Other OS' feature. Shame on pirates, shame on Sony.

dntrkv• 4 years ago

You think Sony didn't know the Airforce was doing this? It's not like they walked into Walmart and purchased 1,760 PS3s. It had to have been a direct purchase from Sony.

bee_rider• 4 years ago

It's a shame they couldn't make it work out anyway. If the cost really was 5%-10% of a similar off-the-shelf cluster, even without the subsidy it would be, what, 10%-20%? That still seems like a steal.

And Sony gets to sell their gaming console as "US Airforce proven" or "a supercomputer in a box" or whatever marketing spin they want to put on it.

And it is only a couple thousand PS3s anyway, so it is a drop in the bucket.

I wonder if there was some behind the scenes stuff going on, maybe IBM worried or angry that this would devalue Cell processors somehow.

monocasa• 4 years ago

They were very happy about the air force doing this.

The whole point of OtherOS (and the official Linux port to the PS2 before this https://en.m.wikipedia.org/wiki/Linux_for_PlayStation_2 ) was to get the system classified as a general purpose computer rather than as a game console because that gave them import tax benefits in quite a few jurisdictions.

giobox• 4 years ago

The death of otherOS was entirely to do with the fact it facilitated a lot of the reverse engineering efforts on the PS3, nothing else. As others correctly note, this was largely in response to George Hotz's hacking which required otherOS. Sony actually exposed themselves to legal action in many countries by selling a customer a product with an advertised feature then removing it after the fact, as such behaviour can often fall foul of sale of goods legislation.

Sony absolutely adored that supercomputer project if you lived through this period and followed the company; the idea that the cell processor was a supercomputer for the living room and we'd all be using our PS3s for media editing etc was genuinely a thought Sony had back then. It all fed into much of the (at times ridiculous) marketing for the Cell chip. Sony had plans for more Cell based devices that never materialized too.

https://www.gamespot.com/articles/sony-gives-glimpse-of-ps3-...

"First, the company will manufacture a high-end workstation using the Cell CPU. Planned for release at the end of 2004... the Cell workstations will be marketed directly to the game and special-effects industries. The labor in their creation will be divided between Sony and IBM. SCE will develop middleware and other tools for game development and film effects. The Cell chips themselves will be manufactured by IBM, who will also work on the OS."

Matthias247• 4 years ago

I don't feel Sony should have to worry about it. That amount of consoles would have been 1M$ at the time that happened, and represents a minor amount compared to the total sales that Sony had. It's probably a good marketing/PR budget, and messaging like "the PS3 is so good that even the military uses it instead of other computers" sounds like it could make it more appealing for customers.

The main risk would have been that more companies buy PS3 without buying subsized consoles without buying games. But due to the highly specialized architecture I'm not sure if there was ever a real risk of that happening.

zappo2938• 4 years ago

Navy submarines use XBox controllers to control parascopes.

selectodude• 4 years ago

The USAF uses them for Predator drone flying as well.

I'm not really a pacifist but there is something profoundly unsettling about real-life Call of Duty.

tcptomato• 4 years ago

periscopes ...

29athrowaway• 4 years ago

Also used to control drones and robots in other branches of the armed forces.

masklinn2• 4 years ago

Sony sold the consoles at a loss, and likely had the CPUs for much cheaper owing to having bankrolled the chip itself.

mywittyname• 4 years ago

This used to be fairly common, I think.

I remember in college we had a lab full of PS3s because they were cheap and powerful and there was a course you could take to learn to develop on them. The lab later expanded after a big donation from nvidia of high end gaming machines to teach students CUDA.

The courses sucked though, because everyone would take them just to get access to the labs.

srvmshr2• 4 years ago

The challenge was not rigging this up - but making things usable & efficient. The Cell Broadband Engine was notoriously hard in architecture & needed lots of optimization.

Part of the reason why PS4 & Xbox later pivoted towards x86-64 ISA

bilekas• 4 years ago

I would have thought that the CBE was one of the reasons they chose the ps3 over conventional servers though ?

srvmshr2• 4 years ago

In my naive understanding, instead of modern SIMD type execution they were using some form of token passing between cores to execute instructions async. That made programming such multicore APUs pretty hard for the programmer. They had a SDK for mitigating some of these pain points. I could be somewhat wrong on the specifics (and the fact I last touched any architecture related topic about a decade ago).Please correct me if I am wrong.

masklinn2• 4 years ago

If you were building a supercomputer the CBE’s peculiarities (specifically the SPEs) were what you wanted.

And the 360 didn’t use SPEs, it has a pretty standard 3-core CPU built out of the PPE. The weird-ass structure of the CBE was not why it switched (although PPC might be).

trimbo• 4 years ago

Related: Los Alamos' Roadrunner was based on the CELL as a co-processor, and was #1 in TOP500 in November 2008.

https://en.wikipedia.org/wiki/Roadrunner_(supercomputer)

schaefer• 4 years ago

I used to crash their (Roadrunner Team's) lunch-time Linux kernel study group. So much fun.

INTPenis• 4 years ago

I remember reading about the air force buying a lot of ps3's back in the 2000s, and that just cemented to me that information is power and any government will do all they can to own as much of it as possible. IBM just published some quantum breakthrough[1] and my first thought was that it will be used by the military or intelligence agencies. Hopefully by the time this technology trickles down to more corrupt agencies the people will have caught up.

You might as well be talking about magic when it comes to quantum computing and me, I have no idea if that new openssh standard really does protect against a quantum computer trying to break it.

1. https://www.techradar.com/news/ibm-claims-to-have-mapped-out...

zaptrem3• 4 years ago

Wouldn't seeing the Air Force clamoring for commodity gaming hardware usually destined for a kid's bedroom be an inspiring example of equal access to technology?

smm11• 4 years ago

Just think what they're doing with all our phones now.

hypeatei• 4 years ago

I believe they had to jailbreak these to get Linux installed on them.

Just imagining an Air Force employee (?) browsing community modding forums and such for older firmware and tools, kinda funny.

flutas2• 4 years ago

I believe they had to jailbreak these to get Linux installed on them.

OtherOS was a fully supported feature on the PS3, that enabled you to install Linux on a secondary partition on the built in HDD. At least until George Hotz used it to get hypervisor access and exploit the PS3.

This lead to the feature being removed and him being sued by Sony.

https://en.wikipedia.org/wiki/George_Hotz#Sony_lawsuit

humanistbot2• 4 years ago

Nope. PS3s were initially released with a built-in feature to install other OSes. This Air Force project got a ton of publicity. It directly led to Sony deciding to lock down the hardware and firmware on future releases.

paulmd3• 4 years ago

Sony actually officially supported it at the time... and then realized that since PS3 was sold at a massive loss, selling thousands of units to bulk customers who would never buy any games was costing them millions.

Subsequent consoles dropped the "sell the hardware at a loss, make it up on games" model, they only lose money during a brief window at launch and then economies of scale take over and most of the console life it's profitable. It's likely that other kinds of hardware also share this model but we just don't hear about it - steam deck is running tight margins and day-1 sales are more expensive than the "average" unit sold mid-gen or late-gen, so similarly they are probably selling day-1 units at a loss even if they make a profit on the expected cost.

https://www.extremetech.com/gaming/325504-sony-finally-turns...

Of course... even though sony is selling the hardware at a profit, they still maintain the lock-in, just like apple/etc. The linux feature never came back even though the reason for the removal went away.

AdmiralAsshat• 4 years ago

The PS3s are the older, larger variety, since the newer slim models don't allow for the installation of Linux.

Big caveat there. Sony later removed the ability to install "OtherOS" (a.k.a. Linux or FreeBSD) on PS3s from those models. [0] I wonder if that firmware update basically neutered the supercomputer or if they were able to keep them from updating automatically.

[0] https://en.wikipedia.org/wiki/OtherOS

eslaught• 4 years ago

1. No one updates supercomputers automatically. Instead you firewall the compute nodes and have a team dedicated to maintenance and support to do periodic upgrades.

2. I'm sure they had a support contract, which may have included running an entirely different firmware. I have no specific knowledge, but based on other supercomputers I've worked with, these are often very custom machines, despite the "commodity" hardware. The odds that they were running the consumer version of the firmware are low to nil.

thatguy0900• 4 years ago

I can't imagine Sony really gave them custom firmware for just 2k units, right? Especially since all the money from a console is from games sales. Seems like a lot of work for nothing. Maybe they hacked something in on their own though

thetinguy2• 4 years ago

They did not have a support contract.

https://www.syracuse.com/news/2011/03/rome_labs_supercompute...

Rome Lab asked the Department of Defense for $2.5 million to assemble its supercomputer. By the time money to buy that many was approved in 2009, PlayStation 3s were hard to find. Rome Lab bought as many as they could — 1,700.

system16• 4 years ago

that cost is about 5-10% of the cost of an equivalent system

I bet 1,760 chickens cost roughly 5-10% of the cost of a Clydesdale, but I know which one I'd prefer to pull a cart.

humanistbot2• 4 years ago

Literally every single "supercomputer" is a distributed cluster like this.

Communitivity2• 4 years ago

I was there, but not on that project. It worked well. The reasons it was attractive were the cost savings, but more importantly the unique aspects of the Cell processor in the PS3s.

PartiallyTyped• 4 years ago

Why is that? What was it about the Cell processor at the time that was desirable?

Afaik PS3s were sold at a loss for Sony, so it seems likely that they were very beefy computationally wise, but I am use USAF could have gotten an incredible deal with Intel, IBM or AMD, so what is it?

Communitivity2• 4 years ago

IBM, Sony, and Toshiba worked together to develop STI, the architecture behind the Cell and PS3. What was really unique about the Cell was two things: it was the first dual-core processor machine, and it had 8 specialized streamlined cores as well: the SPEs (Synergistic Processing Elements).

Michael Van de Luur of Gorilla Games said in an interview [1] “Even desktop chips nowadays, the fastest Intel stuff you can buy, is not by far as powerful as the Cell CPU, but it’s very difficult to get power out of the Cell. I think it was ahead of its age, because it was a little bit more like how GPUs work nowadays, but it was maybe not balanced nicely and it was too hard to use. It overshot a little bit in power and undershot in usability, but it was definitely visionary.”

[1] https://www.gtplanet.net/playstation-3-cell-more-powerful-mo...

danso• 4 years ago

What's the best lengthy write-up of the Air Force's PS3 cluster (if any exist)? From the headlines, I had always thought it to be some marketing gimmick.

dc-programmer• 4 years ago

The most surprising aspect of this story to me is that the Air Force was allowed to exercise this level of resourcefulness. I just assumed that the procurement process would stipulate 10 year support contracts or something similar.

fdr• 4 years ago

me too. Large, even medium-large private bureaucracies may have also made it a challenge.

smallmouth• 4 years ago

Most would be quite surprised by how resourceful the military can get when a need arises. It's simultaneously impressive and scary.

tomrod• 4 years ago

There is a lot of innovation in the contracting/procurement space with Government. As an example, check out AFWERX.

bitwize• 4 years ago

If some officer can cobble together a proof of concept from COTS parts and show that it is just as reliable and effective as the boutique stuff from Raytheon, et al., at a way cheaper price, you bet that it will be met with considerable attention if not approval. Most ground-based military drones these days are controlled with Xbox controllers because those controllers are cheap and ubiquitous. In the 90s, Doom (with mods) was pressed into service as a simulator to help teach Marines fireteam tactics.

About the most stiff-necked branch of the U.S. military is the Navy. I'd be more surprised if the Navy approved a PS3 supercomputer, especially if it were to be run aboard ship.

blueboo• 4 years ago

Is there any evidence of the resulting “supercomputer” being used? It’s one thing to buy two thousand PS3s. Another to deploy them into a data center. And yet another to build processes and software that uses it. The challenges involved seem discouraging.

thetinguy2• 4 years ago

It wasn’t just the air force.

https://en.wikipedia.org/wiki/PlayStation_3_cluster

Scuds• 4 years ago

UGH! All this infrastructure and all this work into a software platform and it's something that will be obsoleted in a few years by off the shelf GPUs built on an standard development environment backed by industry

That's the problem of building something super-cutting-edge on a grand scale - you run the risk of making a super evolved version of an evolutionary dead end.

Sure it sounds like a cool idea at the time, the Cell optimized SETI At Home demo around the PS3 launch ripped through work units far more quickly than any Intel system.

I wonder what became of the cluster once it was decommissioned? Military surplus PS3s anyone?

HPsquared2• 4 years ago

Many such cases in high performance computing.

avian• 4 years ago

I wonder what became of the cluster once it was decommissioned? Military surplus PS3s anyone?

If it touched at any time any kind of confidential data or algorithms it was probably shredded into tiny pieces.

latchkey• 4 years ago

In semi-related news...

https://www.techspot.com/news/93980-14800-asrock-mining-rig-...

kitplummer2• 4 years ago

I worked on a DARPA project a few years before this - where we were using CBE as the core for a polymorphic processor (one with an FPGA attached to every IO). We were also gutting PS3s to make mission computers for early unmanned systems - running Ubuntu on top. USAF wasn't the only one - not only were there commercial supply challenges with the PS3, various components were being horded by various nation states. We were pretty sure they didn't even no what to do with the parts, but was a basic attempt to prevent projects like this from getting off the ground.

kitplummer2• 4 years ago

Another interesting aspect of the PS3 is it was one of the first OTS device platforms that integrated hardware DRM.

omershapira• 4 years ago

I'm old enough to remember the rumors about Saddam Hussein fusing clusters of 15 PS2s to make a UAV realtime controller [1].

To be fair, he could have been more upfront about it. Some marketing managers would have shown that at E3.

[1] https://www.reddit.com/r/AskHistorians/comments/l3hp2i/did_s...

jlink• 4 years ago

Make me think of this cluster of 200 PS3 that was used in 2009 by EPFL to solve the elliptic curve discrete logarithm problem: https://www.epfl.ch/labs/lacal/articles/112bit_prime/

marcodiego• 4 years ago

In the PS2 era I considering buying it and using the "linux kit" instead of buying a desktop. I was prepared for the differences in performance but then I knew that sony didn't release all the drivers or specs and graphic acceleration was not available. Just gave up the idea.

The PS3 could be used as a computer, this allowed sony to pay less taxes in the EU. Since it is such a closed platform the "install other OS" feature could be disable remotely by the vendor automatically and, I think, without any user intervention. When it became economically "better" for sony, they disable the feature.

These are good examples of the problems with such closed systems.

textcortex• 4 years ago

Reminds me the scene in Chappy

Barrin92• 4 years ago

Huh I wonder if that's where Jonathan Nolan got the idea from for his show Person of Interest.

etempleton• 4 years ago

Someone must have had a pretty amazing internal pitch deck to get this approved.

elkos• 4 years ago

Which would be a good alternative to build similar cluster supercomputers nowadays?

torginus• 4 years ago

The Cell's SPEs were such an inspired design - if you look at CPU architecture, 99% of the complexity comes from pretending there's a flat memory space - data hazards, cache coherency, latency hiding, prediction etc.

If you make a beefy processor that works like a microcontroller - reading and writing everything from SRAM, and making all main memory DMA explicit - you get all the speed a fraction of the logic budget.

If you look at a modern CPU pipeline, it's 20ish stages, with most of them about handling the aforementioned complexities, with fetch-decode-execute taking up at most like 6 of them. In MCUs (and SPEs) it's basically all there is.

Unfortunately they never did figure out how to write programs for such a curiosity in an accessible manner.

I'm curious if it's worth taking another crack at this architecture, this time with better tooling.

pyinstallwoes• 4 years ago

Have you seen https://www.greenarraychips.com/home/products/index.php

US Air Force connects 1,760 Playstation 3's to build supercomputer (2010)

Related Stories