The slow rise of robots in the data center

https://www.datacenterdynamics.com/en/analysis/caves-of-steel/

by vanburen • 5 years ago

103 72 5 years ago

johnohara4• 5 years ago

Gone are the days of tape operators on roller skates, but this article is correct in its assessment that change in the datacenter moves at a very deliberate pace.

Oddly enough, I found the reading pace of this article similar to the change it describes. Nothing negative mind you, just gently rolling in its delivery.

markus_zhang3• 5 years ago

Hi do you have any book/video talking about skaters in data center? I checked Youtube and only found one for switch board. Thanks!

donalhunt5• 5 years ago

From personal experience, health and safety officers prefer workers to walk around facilities. Accidents while rare can result in hospitalisation (broken bones) when using skates, scooters and bicycles.

I was once in a datacenter that had a golf cart to get from one end to another (yes - it was that long).

johnohara4• 5 years ago

No. Sorry. My comment was from a conversation I had in the mid-80's with a couple of field service reps who worked the site and couldn't believe it themselves.

The magtapes used to hang on row after row of metal racks. The operators would retrieve the tapes, hold them on their forearms, and bring them back to what were usually a centralized set of tape drives.

Here's an offset into a video showing what it typically looked like. [0] The narrator refers to a "search tape" used to retrieve the person's information. The tape had to be retrieved from the racks.

The sheer weight and overall length of the rows/racks meant the rooms were often lower level with vinyl flooring over concrete (raised tile later). Imagine walking up and down for an 8-hour 5pm-1am shift. Not difficult to imagine skates.

Here's another video with a good representation of how it worked. [1]

As I have come to accept, the mid-80's was pre-everything. Deliberate change takes time.

[0] https://youtu.be/Iddrm7mHPrY?t=111 [1]: https://www.youtube.com/watch?v=Cwj6pfhWBps

johnohara4• 5 years ago

Couldn't resist. Here's another video showing a streamlined and "much more compact" computer room from around 1990. That 44GB of disk storage (DASD) probably cost $150,000 at the time. Somebody signed the P.O. and slept well that night. [0]

Best part. You could roller skate to the music.

[0] https://www.youtube.com/watch?v=vlvUz3T4WTA

sithadmin3• 5 years ago

Not sure if I really buy that the Switch security bot is a serious project, given that over-the-top security theater seems to be one of Switch's main marketing points. Their employees wear military tactical gear; their facility walls and gates are designed like they're meant to house a prison; they have racks of what appear to be real guns and riot gear displayed prominently behind the security folks that check you in. Factor in the Bond-villain architectural aesthetic and it's really just too much to take seriously.

Bizarro security roleplaying aside, Switch is probably the most professional and competent data center operator I've worked with.

formerly_proven• 5 years ago

Factor in the Bond-villain architectural aesthetic and it's really just too much to take seriously.

-- Sith admin

donalhunt5• 5 years ago

[deleted]

Firerouge• 5 years ago

I think you're confused, the poster you're replying to is talking about a company named Switch that makes robots for monitoring/patrolling/securing data centers.

Nothing to do with packet switched networks

foobarbazetc• 5 years ago

Presumably the Fed/State government people who hand out the DC contracts like the security cosplay. :)

dmurray• 5 years ago

Doesn't explain the high level of competence, though.

systemvoltage• 5 years ago

Isn't security theater exactly the thing that's needed to deter threats? When you see guards that look like Navy Seals, it is a pretty strong deterrant.

markzzerella2• 5 years ago

Regular datacenters don't seem to be having major problems with technicals full of insurgents stealing hard disks, but maybe I'm just out of the loop.

kjs35• 5 years ago

No? A cursory reconnaissance (e.g. "I'm a potential customer and I'd like a tour") will usually show that sort of theatre for what it is.

ClumsyPilot• 5 years ago

Deterrent of what? Did someone ever hold a datacenter hostage?

SilverRed2• 5 years ago

Is physical attack a common method to breach a datacenter? Sure you need some physical security and auditing but I would imagine the main threat is online.

eru• 5 years ago

Depends on your threat model.

(Ie any serious attacker will look past the theatre.)

markzzerella2• 5 years ago

I know half a dozen folks that have worked there who all independently described the security as a joke to impress clients. When I took a tour once they claimed that because of their government contracts they can legally commandeer fuel at gunpoint to run their generators if they need to, which is a cute fantasy.

worewood• 5 years ago

They can, then they will be charged with an armed robbery afterwards

OldHand2018• 5 years ago

commandeer fuel at gunpoint to run their generators if they need to, which is a cute fantasy.

Indeed it is. But sometimes real life has odd moments, and some data can be extraordinarily important:

The dramatic rescue of title books from doomed abstract businesses proved a greater public good when all official land records were lost [in the Great Chicago Fire of 1871]. John G. Shortall, who forced a passing wagon driver at gunpoint to load his records, would thereafter be remembered for more than the arrangement of legal conveyances.
Shortall and Hoard, Jones and Sellers and Chase Brothers and Co. - who have saved nearly all of their papers, including the indices to every piece of property in Cook County, and actual abstracts to a large proportion of this property. We have one firm - J.H. Rees and Co. - who has saved copies of all the maps and all the plats ever made of Cook County property.
In April 1872, the Burnt Records Act was passed by the Illinois Legislature, and the existing abstract records of the three companies were made admissible as evidence in all courts of record.

https://www.ctic.com/history2.aspx

pram• 5 years ago

When I worked at Oracle our DC had similar, but less stylish armed guards. It was because we had equipment for Federal Government contracts. Another smaller (former Siebel!) DC with no government stuff I visited basically had zero security in comparison.

So it might be a requirement?

cobookman• 5 years ago

it is.

"Prior to FedRAMP, the Security Control Assessor (SCA) had to visit the data center to check the “gates, guards and guns” every single time, even if that specific assessor had previously visited that data center. That is no longer necessary. The FedRAMP ATO takes care of all of that"

https://www.linkedin.com/pulse/security-control-spotlight-in...

walrus01• 5 years ago

Not sure if I really buy that the Switch security bot is a serious project

It isn't. Switch is famous in the telecom/datacenter industry for weird security theatre. Notably they started having conspiciously armed-with-AR15 guards at their Las Vegas area facility 15+ years ago. Compare them to any major carrier hotel/critical IX point in the USA, they stand out as weird. And they're not even as critical as they claim to be.

I can't share the details but the real, hidden, non-theater security behind some of the US's major traffic exchange points very much exists but is not on public display for touring the rube customers.

aperrien• 5 years ago

I'd think that a robot that could neatly wire up racks with cabling (ethernet, power, & fiber) would be quite doable and useful. Does anyone here have experience with such a project?

sithadmin3• 5 years ago

Having wasted way more hours of my life than I would like on rack-and-stack...I don't think it's possible to fully automate this in a scalable manner without a radical re-work of rack design. Which isn't going to happen any time soon, given that the current standards have so much momentum behind them.

As things are: even in well-managed data centers, the racks themselves are always somewhat finnicky, with varying levels of precision in assembly from rack to rack that require odd workarounds for equipment installation more often than one would expect. And that's not to mention how incredibly variable the rack enclosures themselves can be, which has big implications for cable routing. And nevermind the fact that there's basically no standard for port placement on rackmount systems.

Rack-and-stack labor is dirt cheap, or easily foisted off on your sysadmins for small deployments. I don't see a robot for this being competitive from a cost standpoint unless that robot is extremely general purpose and able to fulfill other roles.

bluedino• 5 years ago

> I don't think it's possible to fully automate this in a scalable manner without a radical re-work of rack design.

I think the same thing when people talking about sending a 'plumbing robot' to your house.

donalhunt5• 5 years ago

This.

Contingent workforces are cheap (minimum wage essentially).

It's also cheaper to use connecters that mate on insertion than to use connectors that humans are accustomed to.

xyzzy_plugh• 5 years ago

Indeed. Even the largest DCs are putting so few racks up per day (still many, many racks) that throughput is rarely the problem.

Actually getting the parts on time has been more of an issue in my experience, or finding out that your entire rack is forfeit because it fell over at the docking bay.

I can count on one hand the number of times I've seen rack-and-stack go too slow for the consumers to actually use the hardware. Usually the hardware is sitting waiting to be provisioned for eons.

The real labor is pulling bad drives/boards and rejiggering the network after the racks are in. So many outages are caused by magic hands making an error and pulling the wrong cord.

Robotbeat• 5 years ago

Basically, a giant blade server is what a server rack optimized for automated server install and replacement would look like. Something like a cross between a vertical warehouse robot and a tape jukebox is what would service it.

Of course, at that point it you’re adding a bunch of cost and may be better off just designing the servers to last for like 10 years with enough redundancy (and hot/warm/cold spares) to not need any physical swapping, then replace the whole thing with a forklift at end of life.

jjeaff• 5 years ago

I think it would be a lot more likely that the next step would be a different "smart rack" where connections, or a single, huge bandwidth network cable would come in to the rack in a single place. And then have a single connection per device in the rack. Then software would handle the routing.

panic_on_oops• 5 years ago

Maybe not rack and stack, this will definitely not fully replace traditional datacenter operations but complement it and make it more streamline. Thinking about how we currently do disk and cable swaps,memory or even system boards misses the point. Of course it can't be automated because there's no unified standard. There needs to be one for such tasks to be fully automated with robots, the rack, server, network switch and even hot/cold aisles need to be redesigned to work with a standard that supports robotics arms instead of human ones. No more wires, maybe conductive rails.

This will take infinity to accomplish (barring disruption in compute/storage) because -

* Remote human hands are ridiculously cheap, and they can perform functions robots can't without extra pay or a paid for software upgrade (think shipment handling, etc). Only real use case that comes to mind is one without economic justification, but of necessity. Think datacenters in space or deep underwater, or hostile environments. So maybe innovation in this field will come from the government this time, and trickle to the private sector. Maybe.

* The cost of such investment far outweighs the financial gains. In the very competitive cloud business, it's the services you offer that matters most, and the reliability of your systems. Sending a person with a code scanner to verify and do a quick disk or cable swap is not a risky endeavour. Unplugging the wrong cable and causing an outage never happens on properly designed systems, with properly set maintenances, so this is irrelevant (when was the last time you've heard that in an outage retrospect?)

pm902• 5 years ago

It would be fascinating if we had infinitely reconfigurable data center network topologies. Among other things, you could have eg Just In Time network rebalancers that could add more capacity between nodes that have a lot of traffic between them. The reconfiguration would have to be instant, or at least take less time than the actual data transfer to have any benefits.

sithadmin3• 5 years ago

Things are already moving this direction with adoption of network virtualization solutions and overlays becoming the norm in corporate data centers.

wmf2• 5 years ago

There has been a bunch of research on reconfigurable optical datacenter networks to do this.

kjs35• 5 years ago

You do that virtually, not physically.

kijiki• 5 years ago

Plexxi tried something along those lines. It was cool, but very expensive, so it didn't really go anywhere.

donalhunt5• 5 years ago

My sense is that robots are in use more than the article makes out. Anywhere there is a risk of human error or repetitive work that can be simplified, you'll find robots are a candidate.

The main issues with robots in my experience is whether you can get the desired throughput while including planned and unplanned maintenance work, reconfigurations, etc. The upfront capital cost generally means you don't have the ability to have a drop-in replacement ready to go when a change is required. That results in downtime. Folk familiar with LEAN six sigma will know that hidden bottlenecks should be avoided or you build up large backlogs during downtime.

Good news for datacenter engineers is that fixing robots may become part of your remit in the future (interesting niche given the mix of software and mechanical). More interesting work will always exist!

dv_dt• 5 years ago

Are there any interesting mathematical or other tools that you recommend to evaluate hidden bottlenecks systematically? I wonder for example, about Toyota making a decision to add inventory buffer for chips vs other auto makers. Great decision in hindsight, but how did they evaluate that in context with other possible actions.

pm902• 5 years ago

The only way to know for sure is by experiments/simulations.

The heart of any competent operations team, whether its in computer systems or plain old logistics, is drills. You can't simulate every possible scenario, but you try to analyze and get an initial set, and the build that up as you run into actual production issues. What you want to avoid is running scripts for the first time in the middle of a disaster.

wiml• 5 years ago

From what I remember this (at least used to) fall under the heading "operations research", which spawned a lot of mathematical tools used elsewhere like queueing theory. Related terms include supply chain engineering, logistics, industrial engineering, and the general ideas of scientific management, Taylorism, etc.

MR4D• 5 years ago

The slow rise of robots in the data center

Related Stories