Moved a server from one building to another with zero downtime

https://www.reddit.com/r/sysadmin/comments/i3xbjb/rant_sorta_physically_moved_a_server_today/

by huhtenberg • 6 years ago

1043 400 6 years ago

panpanna2• 6 years ago

Really disappointed they didn't use a wireless network of some kind.

lordnacho• 6 years ago

My first thought as well. Set up WiFi along the path, basically turn the machine into a laptop. But I think there might be a disconnect when you change base stations? At least when I move my laptop between rooms in my house there's often a momentary problem while on video call.

The other way I'd do it is more similar to described. Create redundant network paths to the server, then cut one.

dbalatero• 6 years ago

I wouldn't, the risk of disconnects is high.

panpanna2• 6 years ago

But the risk of sometime tripping over your looong cat6 and breaking the network is not negligible either...

D895n9o33436N42• 6 years ago

This reminds me of a famously obtuse and obdurate boss who asked for things that were utterly impossible. He had delusions of grandeur which left him convinced that he and only he was qualified to challenge the “cheap, fast, good - pick any two” triangle.

Naturally, I did my best to explain the laws of physics to him, but he wouldn’t hear it. In a spectacular display of Stockholm syndrome I did my best to appease him for four years, but, as many of you can surely predict by this point in the story, I failed in every possible way and eventually gave up. Just wish I could have my four years back.

I was glad to read that OP at least got paid well for his efforts.

Tade0• 6 years ago

I applaud you for being able to stand four years of this.

I usually get fired from such positions in less than two.

sleepybrett2• 6 years ago

I usually walk out about six months in if not sooner. Maybe it's just because I spent so much time freelancing that I had enough experience to recognize a no-win situation.

jtbayly2• 6 years ago

Seems very risky. Not something I’d want to do if minimum downtime was the goal. One wrong piece of gravel ends up with catastrophic failure instead of 5 minutes of downtime.

dmurray• 6 years ago

But the goal was zero downtime, not minimum downtime. The client made it clear that 5 minutes of downtime was equivalent to catastrophic failure. So they correctly found a solution that reduced the chance of "5 minutes of downtime", at the expense of an increased risk of catastrophic failure.

jtbayly2• 6 years ago

I understand that. I just doubt that the risk was worth it, if downtime is such a big deal.

fooblat• 6 years ago

Stupidest thing I've ever had to do.

I don't really understand the "ranty" tone. The client had very specific requirements and the author came up with an effective solution and was fully paid to deliver it. Sounds like a win for everyone.

ianhawes• 6 years ago

I believe the expectations of having 0 downtime was not expressed until the day of the transfer.

dna_polymerase• 6 years ago

In addition to that the customer runs a single server but expects the guys to maintain a property not even feasible at Google scale: Zero downtime. Overall the whole thing was just ridiculous, but luckily the customer got a nice bill in the end.

intpx• 6 years ago

probably because proper architecture (clustering, HA etc) and planning would have never made this an issue. This is still an extremely risky operation, hot swapping power and switching interfaces on the fly all while sitting on a cart in a corridor. In any disruptive work there is never a guarantee of no downtime for affected assets. I know the OP came in as a consultant, but If I was the MSP tech, I would have demanded a paper trail a mile long to cover my ass if this went sideways and If i was the account manager for the client, I would have refused the work. Its not good business to agree to do work where you know there is a better than good chance there will be an outage and your client is saying they wont pay if there is an outage. Even agreeing to it puts you in a bad spot for future work. I guess as an outside consultant, bewilderment is a better reaction than ranting, but this is the kind of shit that drives ops folks crazy

dspillett2• 6 years ago

planning would have never made this an issue

Hard work is wonderful stuff. Days and weeks of it can save you whole hours of planning.

jabroni_salad• 6 years ago

My SOWs leave zero room for 'and it will go flawlessly or you wont get paid at all'. If you are occupying my time in a way that makes me unable to serve other clients you will definitely pay for it.

jedimastert• 6 years ago

Me: You didn't notify them of scheduled maintenance like we discussed on Friday?

It appears that the client didn't have the specific requirements on initial consult.

throwaway0a5e• 6 years ago

Reddit (for reasons related to user demographics and feedback loops) rewards certain types of writing and implied viewpoints. Following best practices and rules is one of those things. This server migration clearly runs counter to established wisdom so OP using a writing style of "look how terrible and asinine this was" will be rewarded and gain traction much more than a "look how interesting this was" writing style.

user59944614• 6 years ago

It's reddit /sysadmin, the channel is dedicated to rants and horrible experiences from sysadmin and helpdesk folks.

It's quite sad IMO, don't recommend to go there unless you want to have a bad day reading about the most horrific work environments and bad practices in the world.

me_me_me2• 6 years ago

<any group of 2 or more people> (for reasons related to user demographics and feedback loops) rewards certain types of writing and implied viewpoints.

This is literally the basis of human interactions, thats how we humans work at every scale to form friendships/families/societies/nations.

DoreenMichele2• 6 years ago

The flair for the piece is "Rant." That's an official category for the sub. There are going to be expectations surrounding how you write when using a tag like that.

aphrax• 6 years ago

Its funny you mention the rewards on certain types of writing within Reddit. I was thinking about it the other day & couldnt quite put my finger on why I dislike a lot of the stuff on there - even across Reddits. I think this is probably the cause...

fortran77• 6 years ago

Reddit (for reasons related to user demographics and feedback loops) rewards certain types of writing and implied viewpoints.

Just like Hacker News! Here's a clue--it's in a subreddit where these types of stories are welcome.

thehappypm• 6 years ago

So true. Reddit loves these "pro revenge" type stories, all one-sided and unverifiable, where the author is a lone hero, toiling against an uncaring world.

swarnie_• 6 years ago

I see reddit so i assume this is the sysadmin subreddit?

They're famous for not being a cheery bunch. Because reddit's demographic does swing younger the sub used to be filled with endless posts about being socially incompetent or possessing 0 business craft.

Does anyone know if it improved?

some_random• 6 years ago

The client was an asshole who demanded 100% uptime and stated that they wouldn't pay if there was any downtime at all. The rant is entirely justified.

macintux• 6 years ago

At my first job we were starting up the company and didn’t really know what we were doing; one early server was sitting on a folding table and its power cord was wrapped around a leg, so just to replace the table with something more robust involved downtime.

elliotpage• 6 years ago

I've been there, solidarity for cheap furniture based maintainence windows.

em-bee2• 6 years ago

the careful application of a saw or an angle grinder would have made it possible to remove the folding table without unplugging the power cord. :-)

crumpled• 6 years ago

I heard an anecdote about a company splicing some fiber cable in the middle of a utility van and having to cut the van apart at the end.

Humphrey• 6 years ago

I haven't read the article, but I'm reminded of that episode of Seinfeld and the frogger arcade game

dfsegoat• 6 years ago

I have pondered this exact scenario (server move w/0 downtime) - because of watching that episode - wouldn't have thought about it otherwise.

..It's interesting how pop-culture and your chosen profession intersect, at times.

g051051• 6 years ago

I guess servers have gotten a lot more robust in the last decade...there's no way any server I ever managed would survive something like that.

maeln• 6 years ago

A lot of server are SSD-only these days which make them less fragile. Still, I really wouldn't see myself pushing a running server in a cart.

tyingq3• 6 years ago

Yeah, there's certainly still things like riser cards and connectors that could come unseated due to vibration.

throwaway7446782• 6 years ago

On a parking lot, no less! Let's hope it will not rain on the way!

anfractuosity• 6 years ago

Reminds me of this - https://www.youtube.com/watch?v=vQ5MA685ApE

'Moving online webserver using public transport'

tyingq3• 6 years ago

The Indiana Bell building move is pretty impressive. http://www.paul-f.com/ibmove.html

sschueller2• 6 years ago

https://www.youtube.com/watch?v=CNqul9TfJwI

joncrane• 6 years ago

Wow this is literally the exact same thing as the OP but for an entire building. Insane.

zomglings• 6 years ago

In the rain!

salzig• 6 years ago

had the same thought :)

umarniz• 6 years ago

Interesting read, makes me wonder as a thought experiment if it counts as downtime if the latency of commands on the machines rises to 5 minutes?

You could clone the VM to another instance and record commands going to VM1 and replay them to VM2 after 5 minutes.

This whole brain fart of mine doesn't make much sense but if you play along with it, does it still count as a downtime or just very high latency?

pc862• 6 years ago

Wouldn't requests time out on the client side long before five minutes?

heavenlyblue• 6 years ago

I don’t know whether it’s the software in general, but ever since I’ve started using Three 4G broadband in the UK; all of the software started behaving really weirdly (lots of lockups, hangs, etc). Apps often need to be restarted.

If you do a ping during “bad weather”, you can see that they buffer up to 5 minutes of packets (i.e. there will be no communication for some time, then you’ll receive a bunch of them with a huge latency with sequences intact).

So I would assume a lot of software could even work that way. I think a lot of software don’t set any (TCP) timeouts at all.

tyingq3• 6 years ago

That works where you have control over all of the timeouts and failure detection at every level and layer. TCP keepalives, for example, could thwart you. Or client side timeouts, or firewall connection state tables, etc.

5 minutes of unplanned downtime in a pub/sub setup could easily go unnoticed, since that setup is typically tuned for long timeouts and/or repeated retries.

NateEag• 6 years ago

It depends on how downtime is defined in the contract.

That sounds like I'm being snarky but I mean it - whether an actual legal contract or just the documentation given to users, any system where downtime matters should have some discussion of what impacts downtime can have and how it's measured and managed.

That documentation is what defines "downtime".

I'll add that what you've described is a sort of low-fi manual version of DB replication (https://en.m.wikipedia.org/wiki/Replication_(computing)).

user59944614• 6 years ago

I am so scared to imagine what would happen if there was any issue during the move (very likely when dragging live cables and powers over hundreds of meters).

The client would immediately refuse to pay anything because he was very clear he wouldn't pay a thing if there is downtime.

Then, the next contractor would be super quick to judge you and the situation, reinforcing that you were an incompetent idiot and the client was right to kick you away on the spot and not pay a dime.

Glad it went well in the end. There is so much to lose for the person trying to help.

willcipriano• 6 years ago

This is a junior sysadmin I suspect. With a bit more experience you'd learn to say something along the lines of "no downtime, sure, that will be 30 grand" and the ability for downtime will suddenly materialize. Him and his friend did this big song and dance, took a huge risk and only got paid for ten hours worth of work in the end.

zrail• 6 years ago

$30k, 80% up front, strict liability waiver that says I’m not responsible for loss of business or anything else if there is downtime.

cellularmitosis2• 6 years ago

sure, that will be 30 grand

I am having trouble finding a reference to it now, but I've heard patio11 refer to this as "the Japanese no". Don't ever say "no" directly, just quote an astronomical price.

ThePowerOfFuet• 6 years ago

Also, moving a server with spinning disks? What could possibly go wrong.

daemin• 6 years ago

Wasn't here a story about Sun (or HP or someone like that) where they moved a bunch of disk servers across a parking lot to another building and found that many of them had died from the vibrations on the trolley cart used to transport them.

sschueller2• 6 years ago

I had a spinning disk in my car back before we had all these cool embedded PCs. The disk was never an issue, these things can take a lot of abuse (Even New England roads). I had it mounted sideways so a large pothole wouldn't push the heads into the platter.

driverdan• 6 years ago

Spinning disks can take a surprising amount of shock and vibration before they fail.

nwallin• 6 years ago

Disks aren't that sensitive to motion.

At my last job, we had 2 airplanes with 5 computers each with 6 disks each mounted in an aircraft. These were regular servers from Dell, not special hardened or resilient hardware or anything. So 60 or so hard disks flying around. Takeoffs, landings, turbulence. Two flights per day, 3 hours each flight, 6 days per week. So 626 landings per year.

Disk failures were not particularly common.

nilssonanders• 6 years ago

Reminds me of the video where they yell at hard drives and measure disk latency. https://youtu.be/tDacjrSCeq4

sleepybrett2• 6 years ago

If there was downtime during the move and the client was there and declaring that they would not pay, you just walk away. You'd be surprised at how fast they can cut a check in that situation.

gnopgnip• 6 years ago

4.5 hours of a consultants billing rate can be much more than 10 hours of your regular hourly rate working a similar job. A good consultant will have a contract. The client saying I won't pay if XX happens doesn't mean anything unless it was in the contract.

Networking/spanning tree loops, arp table mismatch/corruption, the switches at the destination being misconfigured are all realistic problems that would result in downtime here. The normal way you do this is with live migration from hyper-v or vmotion from ESXI. If the initial migration is not successful, you just leave the server powered on while you address the issues. Once the VM has been migrated you can do whatever you want with the original server without worrying about downtime.

maire• 6 years ago

This reminds me so much of when I joined vmware in 2006. vmotion had already been around for a few years - but I believe this was the first release of vCenter with DRS.

A couple of months I joined, a room full of customers chewed us out for not publishing our vmotion compatibility tables. After 4 hours of chewing out - they then told us they reverse engineered the compatibility tables and reorganized their entire data center to conform to vmware vmotion. Then (of course) we worked with intel to make sure the compatibility matrix worked in the future.

I realized at that point that I joined the right company.

neilv• 6 years ago

Good thing the server had two power supplies. There was a YouTube video (which I can't immediately find) of people moving a server across town, on the train, without powering it off, and, IIRC, they had to splice the UPS into the power cable.

When it's done for pay rather than for fun, and payment is conditioned on zero downtime, I hope they charged a premium to make up for the risk of no pay. Offhand, I don't know what's a good way to do that -- I've never had a consulting client demand terms like that for billed-by-the-hour work.

kijin2• 6 years ago

Effective hourly rate = base hourly rate * risk.

Risk = client risk * task risk.

Client risk is based on your past experience with the same client. If they're prone to demand last-minute changes or stupid stuff, they get charged a higher rate on every project afterward. Jacking up the client risk factor is also a nice way to fire a client you don't want.

discordance2• 6 years ago

https://www.youtube.com/watch?v=vQ5MA685ApE

Milank4• 6 years ago

No downtime is acceptable, but they have only one server?

What if a technical failure happen? What if there's a fire in the server room? What if there is an earthquake and the building collapses? What if... many things can happen that can result in a long, long downtime with this tactics.

If uptime is so crucial, the system should be setup in such way that moving one server should be a peace of cake, not a spec-ops mission.

redwood• 6 years ago

Remind me of how IBM positions mainframes: they are so highly available that you simply never let them shut down.

lasereyes136• 6 years ago

IBM Mainframes are designed to be serviced while running so if you have multiple CPUs you can offline one at a time for upgrade it without the whole mainframe going down. Big Sun Solaris boxes where built like at as well.

If your mainframe had only one CPU, you did have to turn it off in order to service it. But you could upgrade the OS without turning it off. While they aren't cool tech now, mainframes are a marvel of hardware engineering.

chasd003• 6 years ago

plus, i would imagine turning them on and bringing them online isn't just a press of a button.

walrus018• 6 years ago

From an ISP perspective this seems like the sort of company that orders one $250 a month business DIA circuit (at a price point where there is no ISP ROI for building a true ring topology to feed a stub customer) and has no backup circuit. Then the inevitable happens like a dump truck 2km away with a raised dump driving through aerial fiber and causing an 18 hour outage.

Some circuits might average 5 to 7 nines of uptime over a year, but the next year is dump truck time... You can never truly be certain.

momokoko• 6 years ago

You’d be shocked how rare downtime is with modern hardware. A redundant power supply and SSDs in the right RAID configuration typically will not have any issues for years until it can be replaced by a newer model. Also, hardware monitoring is significantly improved to the point where you’ll typically know if something will fail and can schedule the maintenance.

In the past power supplies and spinning disc hard drives would fail much more often.

It’s basically a solved problem, outside of extremely mission critical, 5 nines kind of stuff, that we all forgot because of AWS.

HN ran, and may still run, on a single bare metal server.

paulie_a• 6 years ago

Quality hardware has existed for years. At a ford motor plant they were doing an inventory and couldn't locate a 10 ton mainframe. It was working so well for 15 or so years the tribal knowledge of where it was physically located was lost.

walrus018• 6 years ago

Unfortunately complacency about how reliable modern hardware is can lead to neglecting things like off site backups. And other issues. Yeah your one big critical on premises server may be super reliable. But what happens when the building is flooded with 6 ft of water, catches on fire, is leveled in an earthquake, or anything else?

If a function is super critical to business, it also deserves to have some thought put into the blast radius of its failure.

The sort of places that would insist on rolling a live server 700 ft across a parking lot probably don't have any real disaster recovery plan.

mwcampbell• 6 years ago

Still, sooner or later, the data center will be hit by a natural disaster, a DoS attack, a network problem, or the like, and you'll have to be ready to move to a different one to get your service back online. Or you'll have to reboot your server to apply a critical kernel security update, in which case you need to be ready to fail over to a hot standby. So, since relying on a single server with high-uptime hardware is penny-smart and pound-foolish, might as go with a cloud-style architecture with commodity hardware.

user59944614• 6 years ago

AWS and older hardware is no different. Set it once and it keeps running for many years.

I've came across old AWS account (startup have been using AWS for the longest). All the network traffic or VPN goes through a single instance with 3 years of uptime.

marcosdumay2• 6 years ago

HN ran, and may still run, on a single bare metal server.

I bet HN wouldn't do a 10 hours high-risk operation for moving their servers because they can't afford an outage. (But well, running stuff on a single bare-metal server is expensive enough that even if they could, I expect they don't.)

What would that company do if a pipe broke inside the datacenter? Besides, if you never restart your servers, you are guaranteeing that the one time when the power goes off on the entire city, they won't come back online.

Thaxll• 6 years ago

Yeah that's how you end up with 3years uptime on some forgoten servers... :)

packet_nerd• 6 years ago

Human error is a bigger cause of downtime than technical failure or natural disasters. And in practice, a single server like this tends to be a hand managed one-off which only exasperates the human error component.

Johnny5552• 6 years ago

Even in modern hardware there are plenty of single points of failure.

Single server and "can't tolerate any downtime" are mutually exclusive.

closeparen2• 6 years ago

hardware monitoring is significantly improved to the point where you’ll typically know if something will fail and can schedule the maintenance.

There's SMART for disks... what else?

jayd16• 6 years ago

HN ran, and may still run, on a single bare metal server.

HN also has downtime fairly often.

sixothree• 6 years ago

What's the current advice on SSD RAIDs?

galoisgirl• 6 years ago

Should have been a 5 minute job if done correctly. Owner ended up paying for over 10 hours of work. Stupidest thing I've ever had to do.

You can see the common sense ship has sailed.

coldcode2• 6 years ago

I worked at my last job for a place with a single rack mounted set of Windows servers at a data center - with no backup power supply, no backups of any kind for that matter, no UPS and no redundancy of any system, plus they didn't even have an admin for 6 months. The CEO refused to spend money on a 2nd anything. The company has 2000 employees. One server held all of the companies photos (which is basically the core of the business) and of course was not backed up.

Milank4• 6 years ago

Of course it can work, you can get far with one server and no spending on anything like backups, UPS, etc.

Whether it's smart and good for your business/reputation is a different question.

elliekelly• 6 years ago

This is the kind of company that could benefit immensely from a ransomware attack.

closetohome3• 6 years ago

My boss refused to use UPSs for years because he bought one once and couldn't get it to stop beeping.

gear54rus• 6 years ago

He should have taken it offline without notifying this brain-dead manager. Probably wouldn't have noticed lol.

And then charge for those 5 hours for good measure.

In general, this stupid trend of wanting 0 downtime makes no sense to me. If you're not NASA, police or other emergency service you 100% can afford a few hours of downtime with scheduling it be forehead.

icedchai5• 6 years ago

Never mind these less common scenarios... What do they do about Windows updates?

YetAnotherNick2• 6 years ago

You wrote one server but describe the failure modes of having one data center. I think it is very very uncommon and hard to allow for data center level issue. After all Instagram and 100 other site failed when one AWS data center went down. I would interested to know how/whether anyone's backend will work if any data center and its databases completely fails due to fire/earthquake/networking etc.

Second thing is having multiple machines for server. In theory it might help in increasing the availability but in practice I haven't seen any random issue due to machine which occurs just based on probability. I think almost all failure modes that exist, they are correlated between machines. eg suppose you have data loss on one machine, you could more likely than not, blame it on code and it would be similar across machines.

toast02• 6 years ago

Re: single datacenter. At the basic level, you need a second datacenter with enough machines to provide your service (or a emergency version at least), replication of data, and a way to switch traffic. It's doable, but expensive in capital and development. If you're dependant on outsourced services, they also need to be available from both datacenters and not served from only one. In an ideal world, your two datacenters would be managed by different companies, so you would avoid any one company's global routing failure (IBM had one recently).

Re: multiple servers. Power supplies fail, memory modules fail, cpus fail, fans fail, storage drives fail. Sometimes those are correlated --- the HP SSDs that failed when the power on hours hit a limit (two separate models) are going to be pretty correlated if they were purchased new and stuck into servers at a similar time and then on 24/7. Most of those failures aren't that correlated though. Software failures would be more likely to be correlated though, of course.

The key thing is to really think about what the cost for being down is, how long is acceptable/desirable to be down, and how much you're willing to spend to hit those goals.

misiti3780• 6 years ago

or even better, how do they apply OS patches?

wastedhours• 6 years ago

We used to have one server for a website I was a content guy on - it was in a standard PC case, plugged into a switch in the IT team's office (this was not a tech-centered org).

The main IT guy went on holiday and one of the cover guys from another office decided to tidy up. He unplugged the server and thought (and told me after his thought process) "if anyone was using it, they'll let us know".

This was the one, single box for the whole website - no one else was monitoring (even though the central office had a proper, dedicated web team) and the assumption was I was sysadmin.

An hour later I'm sprinting down the corridor to find out what the hell happened and why I can't even SSH into the box.

We put a sticker on the case saying not to unplug it after that...

PinguTS• 6 years ago

It's been done 7 years ago even using public transport.

https://www.reddit.com/r/uptimeporn/comments/1kf26r/moving_a...

jagermo• 6 years ago

The most dangerous part is them expecting the 3G to be available during the subway ride.

deftnerd• 6 years ago

I'm surprised they weren't stopped by police to investigate a very suspicious heavily loaded cart on the Subway. It easily could have been 300lbs of explosives on that cart.

mercora4• 6 years ago

in Germany mobile networks work just fine in the subway as ISPs have deployed hardware there. I actually have more issues with the network when using classical railroad transport...

csours• 6 years ago

I really thought this post on HN was going to be that story. Thanks for digging it up.

walrus018• 6 years ago

Sort of on the subject, i've seen a brochure for a specialty product marketed to law enforcement. It's meant for use with the seizure of live, powered on desktop PCs and similar that have a high likelihood of full disk encryption.

Essentially it's a medium sized double conversion ups, with a really high quality sine wave inverter, and some electronics that can match phase with a live 120vac 60Hz circuit. And a tool kit which consists of the insulated electrical hand tools needed to do a midspan removal of the cable jacket and splice into the wires in an ordinary PC power cable. The person using it is of course supposed to be trained in advance, and competent at the process of attaching the UPS to the live circuit.

throwaway7446782• 6 years ago

Wouldn't it be safer to open the case and connect some kind of battery + adapter after the power supply?

walrus018• 6 years ago

Splicing into the many wires that is an atx+12v power connector, between the output of the power supply and the motherboard is way more fiddly than just dealing with the hot and neutral on an ordinary $5 PC power cord. You could also never be certain what weird ziptie and cable management system (or lack thereof) might exist in a home built x86 PC case, or if there's any room for hands to work at all...

I think the thing I saw is also meant to deal equally well with a commodity x86 PC built from parts, or an Intel NUC size thing, or a corporate desktop machine with proprietary internal wiring like a slimline Dell, Lenovo, HP, etc.

ajross• 6 years ago

Safer for the operator? Sure. But certainly not for the device, if you're trying to keep it operating. An ATX power supply has 24 pins at 5 different voltage levels (plus any auxilliary power connectors for the GPU and drives, etc...), and motherboards are a lot less tolerant of spikes and transients than the PS on the other side.

Dealing with AC power isn't really that dangerous if you're careful.

regularfry• 6 years ago

Cases can have "case open" switches that tell the machine to switch off. You can't necessarily tell beforehand.

jrootabega• 6 years ago

Case intrusion alarms (built in or Homebrew)

mercora4• 6 years ago

linked below is an old advertisement/demo video of a similar device or maybe even the one you mentioned :)

https://www.youtube.com/watch?v=-G8sEYCOv-o

walrus018• 6 years ago

Very similar, yes

huhtenberg2• 6 years ago

In a similar vein, there are USB gadgets that emulate a mouse that keeps on jiggling, to prevent the machine from locking out on user inactivity.

However, there are anti-jigglers too that lock the machine when any new human input device is plugged in.

http://codefromthe70s.org/antijiggler.aspx

kawsper2• 6 years ago

That's interesting.

You could have a list of known USB device IDs you trust, and if a newly plugged in USB device wasn't on that list you could lock or power down.

ace32229• 6 years ago

You will be seen as active (including on comms software (at least the ones I've tried)) if you have any sort of video playing e.g. Youtube in an active tab. Quite handy.

warrenm2• 6 years ago

They should call it the Jiggle-No

vangelis2• 6 years ago

That's where the analog mouse jiggler comes in. Apparently watche faces work quite well for for optical mice.

reaperducer4• 6 years ago

If you ever come across a jiggle-and-click gadget, let me know. Some of the computer activity trackers I've seen lately require the user to click every so often, so plain jigglers are no longer effective.

donpott• 6 years ago

For the purposes of preventing locking out, I've had some success installing Autohotkeys and sending the MouseMove event every minute or so in a loop. No need for plugging additional USB devices.

jlgaddis• 6 years ago

Search for "HotPlug" on YouTube.

dghughes• 6 years ago

I thought about HotPlug too. And the obligatory Seinfeld Frogger scene (become much less familiar to younger folks).

HotPlug must only work in countries with terribly designed plug outlets like the US and Canada. Our NEMA 5-15 plugs are live when the plug's hot (electrons be here) and neutral (return to sender) blades are still visible. I don't think this device could work in the UK I'm not from there but I think their plugs can't be live with exposed plug blades.

https://www.cru-inc.com/products/wiebetech/hotplug_field_kit...

bob1029• 6 years ago

How do they deal with the loss of network connectivity?

I could pretty easily write a script that forces my machine to reboot and do all manner of other things if some sort of network change is detected.

walrus018• 6 years ago

I don't believe that specific product addresses it at all. Undoubtedly the persons operating the kits have put some thought into it, but given the myriad of possible LAN configurations and types of software deadmans switches, it must be a difficult problem to solve.

numpad0• 6 years ago

Or motion, inactivity, vibrations in the room, etc. But that’s for another product/specialist I guess?

CydeWeys2• 6 years ago

You could, but what % of running servers actually have such safeguards in place? I'd say almost none of them.

jimmaswell• 6 years ago

I'd have thought plugging something into the outlet and unscrewing the outlet to take with you would be more convenient than carefully splicing wires just enough not to disconnect them. All the easier if it's on a power strip.

Scoundreller8• 6 years ago

Sometimes they are on different circuits.

jaclaz• 6 years ago

Technically you don't need to touch the naked wire, you only need to remove (carefully) the outer insulation and have the two (still insulated) wires separated for a few centimeters.

Then there are splitter clamps.

The most common ones are used (low voltage) on cars and motorcycles, they look like these:

https://www.mesconnettori.it/index.cfm/it/ricerca/?flbr=ruba...

But there are professional ones, suitable to 110 or 240 V example:

https://www.techno.it/en/products/all/thb-370-a2a/

miles• 6 years ago

HotPlug Field Kit https://www.cru-inc.com/products/wiebetech/hotplug_field_kit...

"With the CRU WiebeTech HotPlug you can transport a computer without shutting it down.

"The HotPlug allows hot seizure and removal of computers from the field to anywhere else. The HotPlug's patented technology keeps power flowing to the computer while transferring the computer's power input from one A/C source (such as a wall outlet or power strip) to another (a portable UPS) and back again.

"We created this product for our Government/Forensic customers, but it has IT uses as well. Need to move a server without powering it down? The HotPlug can do it.

"It's great for digital forensic investigators and techs who can't risk losing access to data on a running computer. With many computers now employing full-disk encryption, shutting them down poses the risk of having to crack a password after moving the computer to a lab for analysis, which can greatly increase the time and expense of an investigation. When combined with a WiebeTech Mouse Jiggler, you also won't have to worry about the computer entering password-protected screensaver or sleep modes."

Scoundreller8• 6 years ago

Time to geo-fence the servers with an external GPS antenna (often useful for time-sync). Or maybe FM signal strength locks?

nobrains2• 6 years ago

I don't know. If the "boss" was charged "4.5 hours of work, 2 hours of consultancy, and 4.5 hours of consultant", and assuming he would have been charged half of that with downtime, maybe the boss did get a good deal. We don't know the cost of downtime for him.

I mean if he had access to technical resources who were willing and capable to do this for him, he chose to do it.

notwhereyouare• 6 years ago

I personally find it hard to believe that a rough estimate of $450 for the job (spitballing $45/hr for 10 hours) is less than 5 minutes of downtime and they only have 1 server.

Then again, could easily be wrong

nobrains2• 6 years ago

You cannot compare it to zero. You have to compare it to the cost of doing it with the downtime. There would be cost to that as well. It will not be free.

moduspol• 6 years ago

It's also possible that "downtime" has different meanings to different people. The client may be seeing "downtime" as the net result of what happened the last few times the server was "down," which could have been for any number of reasons (potentially even unrelated to the server itself).

When you get clients describing things like this, it's possible they've been promised things about this server before by other consultants that didn't pan out. They don't want to give you the full details because then you'll recommend a different route that they don't want to take (justifiably or not).

It's easier for them to frame the problem to a consultant in a way that allows for only one potential solution, even if perhaps better ones exist, because the guy in charge of making the decision isn't technically skilled enough to assess whether others proposed by consultants are as viable.

And, of course, one might read a little into why there exists a "boss" with such a highly-critical IT need that is hiring a consultant to do work like this, and thinks that threatening to not pay at all if there is any downtime is the best way to do it.

I mean, what if they opened the door to this closet and it grazed a power cable on the floor and the machine just shut off? Why even bother staying around to bring things back up? It wasn't your fault and there's already downtime: you're not getting paid.

closetohome3• 6 years ago

Someone upthread was talking about how, as a Salesman, you have to read the room and know how to talk to clients. I did that for awhile, and always got a lot of mileage out of asking the customer what they ultimately wanted to accomplish, which usually revealed that what they were asking for was a solution to a self-made problem, and there was a better alternative altogether.

growt• 6 years ago

Setting up a new server at the new location and moving the VMs one by one to the new server as they become idle should be possible without downtime. But maybe there were other requirements (like no new/additional hardware) that weren't mentioned in the article.

xyst• 6 years ago

Why wouldn’t cloning the VMs to a second server, then split the traffic between the primary and secondary server work? Once traffic to the second server is confirmed, you could shut off the second server and haul it off to the new location.

I would probably still charge a much higher rate since the owner was an arse, but at least you would get back your 7-8 hours.

blcArmadillo• 6 years ago

It sounds like there was no second server.

viraptor• 6 years ago

You're making assumptions about what's running on the servers. Let's say it's a VoIP conference server with a shared dedicated room - effectively you have an ongoing session shared between multiple connection and you cannot stop it. Or you have stateful local processing so you can't "split the traffic". Or a number of other limitations...

AnIdiotOnTheNet2• 6 years ago

Not all services can be load balanced in this way

Live migration of VMs would have been a better option, which was brought up in the reddit comments and dismissed because HyperV live migration is spotty. While I'd have to agree with that assessment, it isn't so spotty that what they actually did was less risky.

morphogenesis• 6 years ago

Database inconsistency for one thing. This works for frontend web services but how do you reconcile the writes between the two servers?

exabrial• 6 years ago

Reminds me of the "hot slide" technique used for old telephone switches

ansible2• 6 years ago

They did some crazy stuff in the old days. Like when they moved a telephone exchange live... the whole building.

social_quotient• 6 years ago

Slight topic drift - Any thoughts on how the pandemic might materially change assumptions about an onsite/onprem being better than cloud or manage data center when the code people are now actually remote to the “Local” infrastructure. Something specific to the reality of the pandemic strikes me as something that would make the die hard local only folks have to start rethinking the position.

(Not to suggest it’s bad, just different now that a primary assumption about people work in the office is less true)

ocdtrekkie• 6 years ago

I feel a lot safer knowing I'm controlling all the variables during a global crisis, actually.

This article provides an example of how when you operate on prem, literally any crazy option remains on the table for you. If you asked your cloud provider to do this, it'd be a no.

AnIdiotOnTheNet2• 6 years ago

As someone who works in a very anti-cloud company culture (which I happen to agree with), this incident has had no effect whatsoever on that mindset. We don't dislike cloud because it is accessed remotely, we dislike cloud because of the lack of control we have over everything running there. If something happens and our local systems have a problem, there are people here, like myself, who's highest priority will be fixing it and second highest priority will be communicating the status of that. Your problems are never a priority to a cloud vendor and communicating with you is even less of a priority. That's before we even get into the absurd expenses and reliance on big fat pipes.

chisleu• 6 years ago

I didn't want to lose my many months of uptime for a lan party back in 1999/2000 and we used the UPS to migrate my linux box across town for some Quake 3 Arena action.

Things were so much simpler back then.

mercora4• 6 years ago

i wonder if its really possible to do the initial setup of the ethernet failover without interruption. i have never done this, but i would expect the interfaces themselves will become unavailable for direct use and you get a completely fresh virtual ethernet interface which represents whatever physical interface is currently active... at least this is what happens when you add an ethernet interface to a bridge in linux...

webscalist• 6 years ago

Could've been cheaper to buy/rent another server, put it on the new location, set up redundancy/replication, power off the old server, move it to the new location, return the new server. Or just keep it for sanity.

imglorp• 6 years ago

The moving server on cart part made me nervous. If there was any rotating rust in there, bouncing across the parking lot would make things difficult for flying heads. I'd have hand carried it from stage to stage, setting it on a padded cart each stop, treating it like sweating TNT.

rafaelturk2• 6 years ago

Fun reading. But my advice is never accept a job like this. This could easily become 2 weeks down time

codingdave• 6 years ago

I'm surprised that part of the story wasn't to drill down into the requirements. No downtime ever? Not even at 3 AM on a Saturday?

I've found that when people are being unreasonable it is because they haven't split out their true needs from their first idea of how to meet those needs. In this case the true need is zero impact to users. The owner translated that to "zero downtime", and then didn't accept alternative solutions that still would have met his true business need.

robin_reala• 6 years ago

I always remember this post by the Amsterdam Police who managed to maintain their uptime on a VMS cluster despite moving data centres in the middle: http://web.archive.org/web/20120229042903/http://www.openvms...

digitalsushi• 6 years ago

I had to search the reddit commits for 'vmotion'. They have it covered.

This anecdote is an amazingly good story for telling at the pub over a few beers. It's a terrible story for a strategy.

If this is a mountain, my molehill is that one night in the late 90s, I got paged cause the SMTP outbound server was overheating. At midnight I drive across sleepy NH backroads, and stopped at a Wendys to get a chicken sandwich and iced tea, for the caffeine.

When I got to the server room, I pulled the 2U Dell server out of the rack and discovered the CPU cooling fan had seized up. Mind you, this is a New Hampshire data center in 1999, and it has a filing cabinet with manilla folders, and carpeted floors. This thing was never prepared for any disasters.

A half hour later, the SMTP server was up and running cool again.

I greased the fan with the mayonnaise from my sandwich.

dsr_2• 6 years ago

The (probably soybean) oil is a fine lubricant, but the constant motion should cause the egg proteins to coagulate. How long did it operate before you replaced the fan properly?

emeraldd• 6 years ago

I can't help smiling at this analysis. It feels like something you'd hear from a sci-fi story engineer working on an rundown ship that just keeps going no matter what ...

kijin2• 6 years ago

The egg proteins are already quite coagulated. I'd be more worried about the vinegar component. You need to neutralize that acid with something.

Corrado• 6 years ago

It probably lasted long enough to get a replacement fan installed the next morning.

coding123• 6 years ago

Maybe it was vegan mayo? Oh wait, 1999.

tempestn• 6 years ago

Reminds me of the foosball table in our old engineering students' society room. You can buy special lube for foosball bearings. OR you can just rub popcorn butter on the bars. Guess which one we had in ample supply.

Nextgrid2• 6 years ago

I’ve once used cooking oil as a thermal paste substitute. Worked well enough and nothing went wrong.

unlaxedneurotic• 6 years ago

That sounds too close to a fire hazard

Spooky23• 6 years ago

The real lesson is that the teller of the tale sort of did initally -- fire the customer.

If the story is true, the client is a stereotypical know-it-all small business owner who gets by on bullying. You see them frequently in businesses that pay low-skill workers a small premium that is hard to replace. (ex: cleaning services, pool guys, mechanical contractors that do low-end maintenance work, etc)

As a contracted SME, taking a job like this is dumb. The chances of failure, where "failure" == the server going down is high, and the customer will just stiff you.

donmcronald• 6 years ago

I deal with a fair bit of customers like this (not via my business), enough that firing them all isn’t an option because it’s a large portion of the market where I am. It’s something. They’ll have low end servers with no redundancy, terrible or no backups, and no contingency plans for anything. They won’t spend a nickel and are the most likely to lose their minds if anything goes wrong.

It’s so frustrating and stressful.

paulsutter• 6 years ago

Key is to set the price. If the job is a hassle, you're not charging enough. That will also filter out bozos. Sometimes people really do have extreme requirements. And when they do, they're willing to pay 10x for it.

Agreed though, this particular customer disqualified himself as soon as he said he wont pay if the server goes down. He should have offered a big bonus if the move succeeds without downtime.

dialamac2• 6 years ago

There’s a market for everything. The “shit customer” would have hired someone. Sure things could have gone to shit and they wouldn’t be paid, but they didn’t, and there is a living to be had serving this market segment. Every contracting business has difficulty with AR, stereotyping often isn’t even particularly acccurate. Suffice it to say there is a business to be had in catering primarily to difficult customers.

renewiltord• 6 years ago

Presumably at least part of the value of the job from his POV was the excitement of it. I've definitely done suboptimal jobs where I just enjoyed them a little and they were a break from the replicable tasks.

Since the customer is never going to know the awesomeness of it, it's really just for yourself.

jjice2• 6 years ago

I'm a young developer, so I've never had the chance to work with on prem servers (and the chances that I will are looking slim), but I've always loved these "war" stories.

laurent92• 6 years ago

In my former service company, the story of the server room which has become much much more important and reliable with massive investment like a diesel generator, but the teams haven’t grown enough in maturity. One day they have a problem with a server. A system admin is granted permission to go to the bay, since remote desktop didn’t work. They discuss the problem in front of the bay. One leans on another bay that was just between two locations. Wheels weren’t locked. It just flew across the room.

It was ok, just the power cord and a fee RJ45 torn. No serious damage besides downtime.

jacobsenscott• 6 years ago

I miss the days when the servers came with seating (Cray).

mike_d3• 6 years ago

You can buy yourself a pair of old Dell servers from craigslist or eBay for a few hundred bucks. With a $200 membership to VMUG Advantage you'll get all the licences you need to build an enterprise grade cluster.

Build yourself a home lab and learn how systems work. Figure out what is really running your code. Learn how to resource optimize.

Don't end up only being able to work on webapps and small datasets that fit comfortably in the cloud.

y_tho• 6 years ago

It´s 2008. A manager that just doesn´t care anymore tells the new IT person to replenish the fan mayo.

-"Why? I don´t know why. It just works."

-"Is Hellmann's ok?"

IT person documents that Hellmann's is preferred.

bentcorner• 6 years ago

And then later on Hellmann's is discontinued, so the company solicits quotes for a mayo supplier.

m3kw9• 6 years ago

Mayo is around 1 part water to 1 part oil ratio..

arexxbifs• 6 years ago

Mayo usually has around 80% fat content.

geocrasher• 6 years ago

I once shut down a PC, moved it to another desk, and it wouldn't power back on. Another time I moved a server to another rack. It had 2 years uptime. Had to power it down, and it wouldn't power back on. Both required PSU replacements. Had I moved them while powered on I can only imagine the fun times.

Perhaps they should have just told the customer they couldn't find it: https://www.theregister.com/2001/04/12/missing_novell_server...

ericyan• 6 years ago

The consultants really should told the client that if all you have is a single server then there is no such thing as "zero downtime".

neya• 6 years ago

Dude has ONE server and talks about having 0 downtime for his clients? What the hell?!

In a way, this is Darwinism for the IT industry and I'm happy the people involved got paid well. Due probably paid as much as it woulda costed him a new server. I bet he'll never forget this lesson.

Zenst• 6 years ago

Interesting story and one that has played out a few times, I'm aware of a couple verbatim to that. Another - used power extension leads to cover power. Key being systems with dual power units (most servers do) and networking so you can switch from one run to another.

But have known some large companies who have in their history, done things like this and other creative solutions to impossible problems.

cpuguy83• 6 years ago

I'm picturing that Seinfeld episode where George tries to move the Frogger arcade from a restaurant that is shutting down but doesn't want to lose his high score.

will_pseudonym• 6 years ago

HOLES! I need HOLES! :)

kuon• 6 years ago

When I was younger (read 20 years ago), I did crazy things like that, not over that long distance, but moving live servers in different racks.

Now that I am older, I don't think I would do it anymore, too much stress for a small reward. Also today, most of the time, I am able to "talk out" customers of crazy requirements, while I would just have said "OK let's do it" in my younger years.

devchix2• 6 years ago

I recall sometime in the mid 2000s there was a fever for achieving five-9s (99.999% uptime, I think -- it became fodder for a few episodes of Mr. Robot). Not that the metric ever went away, but back then a lot of BigIron(TM) vendors advertised achieving five-9s by replacing hardware while the OS remained running and continuing service. Sun 15K and 25K series (Gilfoyle had a used one in the garage running his network) were behemoths whose mem/cpu boards you could swap out wholesale while the entire frame and backplane was powered on, and while the OS the board came out of remains functioning. There were many caveats around the procedure but it worked. Execs and sales guys loved those demos. These monsters were expensive and banks and energy conglomerates were buying them by the dozens. There was also a big todo about hot swappable drives. The idea that you could be doing hardware maintenance while the machine was still running was a novelty, something like brain surgery while the patient was not only awake, but awake and eating, driving his car, talking on the phone, etc.

A decade later I look back with deep surprise that we didn't think to abstract out the service instead of the hardware. I don't know how many of those behemoths are still being bought, now I work almost exclusively with small server instances that can come and go on the fly. Micro services and AWS have taken five-9s in a different direction. I frequently think of Sun as a failed Hephaestus, in a Christopher Nolan film he would be brilliant but could only turn out clumsy tools because of his deformity, he hates the things he makes so he throws them away before completion. Men find these cast-offs and temper and refine them.

chasd003• 6 years ago

i remember those. A friend of mine was a network engineer at a local datacenter ( UUNET then MCI pre-scandal ) and said companies were buying Suns for everything no matter how trivial.

He worked a night shift and i use to go hang out with him in the noc and download movies (residential bandwidth was not what it is today). Odd he nor i ever get in any trouble for that heh.

tlack• 6 years ago

Well as I recall there were a few reasons that people focused on reliability in hardware in the late 90s:

1. Shared state storage systems that supported replication were rare (I think Oracle and Informix maybe?)

2. Virtualization software was in its infancy (did SunOS have something before Solaris?)

3. RAM and hardware were waaaaay more expensive, meaning you often had to buy more pure metal just to answer questions fast enough

At least that's my take on it based on my dim faded memories

theevilsharpie• 6 years ago

[In the mid-2000's], a lot of BigIron(TM) vendors advertised achieving five-9s by replacing hardware while the OS remained running and continuing service... A decade later I look back with deep surprise that we didn't think to abstract out the service instead of the hardware.... Micro services and AWS have taken five-9s in a different direction.

In the mid-2000s, enterprises were (and in many cases, still are) running proprietary software with proprietary RPC protocols that had no available source code or other means of modification, and most had no support for application-level high availability, access control, or any other operational quality-of-life feature that people take for granted today. Rather, that functionality was handled at the infrastructure level, through things like the aforementioned Big Iron.

The world looks different today, but those machines made sense for the environment at the time.

ClumsyPilot• 6 years ago

I think it kind of makes sence in general, and the obly questuon is whether it could be achieved at lower cost. Complexity of Todays commodity machines is conparable to big iron kf yesteryear

larrik• 6 years ago

a lot of BigIron(TM) vendors advertised achieving five-9s by replacing hardware while the OS remained running and continuing service.

AS/400's were capable of that in the 90's (possibly the 80's as well). Heck, they'd call IBM for replacement parts on their own. You'd show up for work and there'd be an IBM guy waiting to be let in. He'd swap out a part with no downtime, and be gone. I've seen machines with uptimes of over a decade with zero on-site IT.

blhack• 6 years ago

We had one of these at an old office of mine. I actually think it's really cool.

pmiller23• 6 years ago

I recall working on some Sun machines with hot-swappable CPUs (and, I assume, disks and other peripherals). If they somehow made memory hot swappable (I'm sure it's possible, just uncommon and/or verrry expensive), with hot swap CPUs and disks, and redundant power supplies, you could tear the machine half apart and it would still keep running. Of course, at that point, once everything is hot swappable, there are generally multiples of everything, so your one machine is really more like multiple machines inside one box than a single discrete machine.

bcrosby95• 6 years ago

AWS single region SLA isn't 5 9s though. If you want 5 9s in the cloud, not even multiple AZ is enough - you need to go multi region or even multi cloud.

closetohome3• 6 years ago

Multicloud

This sounds like something they'd make up on NCIS.

techslave2• 6 years ago

a novelty

not a novelty. the true bigiron vendors (not sun) had been doing this for decades.

mainframe reliability puts the upstart unix systems to shame.

devchix2• 6 years ago

Up to that moment, hardware maintenance meant having to power cycle the server. True HPC systems like those that ran at the US National Labs didn't find its way to the general market, and still haven't as far as I know.

tobyhinloopen• 6 years ago

10 hours investment for no downtime seems like a good deal for the owner

topkai22• 6 years ago

Depends on if he really has customers accessing the system "all the time."

Besides, as pretty much everyone has noted, running a zero-downtime system on a single physical machine in what sounds like is just a normal cable room is kind of nuts. Those 10h would have been much better spent to move that puppy to someone else's data center and get some redundancy.

Although reading between the lines, maybe the lease was up and they were waiting to the last minute to move it.

linsomniac• 6 years ago

Lower stakes, but ~15 years ago a friend had a Linux box in the corner that had huge uptime. I want to say the uptime started shortly after the kernel patch that fixed the 400-ish day overflow of the uptime counter. He moved to a new home and very carefully moved the running server using it's UPS. He didn't have to worry about keeping networking up though.

I used to be all about long uptimes. I eventually started seeing long uptimes as a negative though. A long uptime probably means patches have not been applied.

jccooper• 6 years ago

I also did that once, about the same timeframe, specifically to preserve an uptime.

I think the cult of runtime came about simply because it was impressive that a personal computer could stay running for more than a few days when most of the world ran Win95. And because development cycles were longer and there weren't a lot of network threats.

JeroenKnoops12• 6 years ago

Reminds me of the OpenVMS clusters.. Police in Amsterdam celebrated in 2007 an uptime of 10 years of their cluster. In this period, all hardware was replace, and half of it was moved to another location 7 km away. All data moved from DAS disks to SAN without one application needed to be stopped. Also VMS was upgraded from 6.2 to 7.3-2. The VMS cluster did not go down during all of these changes. I <3 OpenVMS

JeroenKnoops12• 6 years ago

During Y2K I've also had to shutdown various OpenVMS servers with uptime over 10 years... Only because of company policies, not because OpenVMS required the reboot.

tandr• 6 years ago

Would be interesting to know if it is still up and running?

tzs2• 6 years ago

I needed to restart a server where I worked. My boss was complaining about the revenue loss during the down time. I knew the revenue loss (if there even was any, as opposed to a couple of minutes of revenue simply shifting to a few minutes later...) would be well under a dollar.

So I listened to him whine for a couple minutes, then tossed a dollar on his desk, told him that would cover it so he could shut up now, and rebooted the server.

Warning: you should probably only try this if you are good friends with your boss. That boss had been my best friend for years before I came to work for his company.

jfcorbett• 6 years ago

Reminds me of the time where IT at a previous employer told us that due to a "new IT strategy", our production cluster that had been sitting comfortably in the basement for years had to be moved to an "approved IT hub facility"... in another office 500 km away and across the North Sea.

There was downtime.

Promptly after our cluster settled into this wonderful new facility, a cooling pipe in the ceiling leaked on it, frying 1/3 of our nodes.

yjftsjthsd-h3• 6 years ago

On a personal selfish level I was quite happy to see our workloads moving to datacenters that we couldn't (reasonably) physically access, because it replaced "can you go drive to the DC and replace a failing disk" with "we put in the request for smart hands to replace the failing disk". Of course, there's some notable tradeoffs, but it makes me feel better when the business decides to do such things...

arethuza• 6 years ago

This reminds me of a small company I joined many years ago that did deployments by RAID - find a working server (possibly at a customer site) swap in a blank HD, wait for it to rebuild then take it and put in a new server and repeat the process.

Like finding people who argue against revision control systems, it's really quite a challenge convincing people why things like this are a bad idea - after all "it works!".

yjftsjthsd-h3• 6 years ago

That's... actually fascinating, if in a slightly insane way. There's pets, there's cattle... and apparently there's a herd of cloned pets, which I'd somehow never considered before:)

rafaelturk2• 6 years ago

Pictures please!

mercora4• 6 years ago

when i was younger i was super proud that i could replace my disk while i kept working on the device. i would put the new disk into my LVM volume group moved all extends to the new disk and dropped the old disk out of the VG afterwards, when done i could just unplug it and be done without halting work except for kicking off the process.

Johnny5552• 6 years ago

Decades ago an ISP I was colocated at did the same thing. I don't remember the exact details, but it was a DNS server and they either couldn't log in or were relying on the zone files cached in memory or something but for some reason they couldn't power it off.

It was already plugged into a UPS, but they had to cut one of the posts off the rack to get the server out without unplugging it, then they plugged that UPS into a bigger UPS on a cart and wheeled it to the new data center they built out in the building next door.

The world was much different at the time -- this coloc provider had a good reputation, yet.... they had a keg of beer in the corner of the server room and a stack of adult magazines in the men's room.

Reedx• 6 years ago

That reminds me of the Pixar incident where Toy Story 2 was accidentally deleted while in production and had no working backups.

Luckily one employee was working from home (rare at the time!) and had a copy of the entire movie on her desktop computer. Which they very carefully moved back to the office and were able to restore from that.

https://www.youtube.com/watch?v=7MAedEXri7c

tzury• 6 years ago

I once was called in to export data from a DOS program that had no export option. Single Author died of heart issues and the company needed the data for the migration.

After several attempts to understand the binary format I gave up and ended up printing tabular reports to LPT1 which I connected my laptop to, extracting it and rebuilding CSV files.

Lucky enough, printing those days were the most important feature of a business app.

fredley• 6 years ago

This is the kind of content I've only ever seen previously in TDWTF (which is entirely this sort of content...)

https://thedailywtf.com/

merb• 6 years ago

meanwhile in germany, german telekom have their connect ip lines (leased lines..., company internet..) shutdown since tuesday morning. so a downtime of over 48 hours, besides a sla that no downtime will be longer than 8 hours and a availability of 99,9%.

what a crazy world.

kyuudou• 6 years ago

It's called vMotion

akssri• 6 years ago

Was it George Costanza ?

https://m.youtube.com/watch?v=a-FbktgqCqY

johnklos• 6 years ago

I've done something like this - server running off of UPS moved from one building in Manhattan to another about 1/4 mile away, in snow... Not for someone with weak arms.

BrianB• 6 years ago

It's been done. https://i.cdn.turner.com/v5cache/TBS/Images/Dynamic/i439/sei...

neycoda• 6 years ago

I wonder if there was any legit reason to require no downtime. Otherwise the owner doesn't understand what downtime means for his business.

jcrawfordor2• 6 years ago

I once basically spent a summer doing this, not over a parking lot but to consolidate the remaining equipment in a large number of racks into a few new ones - this was a former sales office of a megacorporation that had been built to have its 1970s-era computer room proudly displayed through windows into the main conference room, a very weird setup without the context that in said '70s that conference room was used to pitch prospective customers on business automation.

Anyway, by the time I was there it was still a '70s-vintage large computer room but now massively overprovisioned on space, cooling, etc, particularly with most IT functions having moved to corporate. A decision was made to repurpose part of it as a test lab and move all the actual remaining equipment to three racks in the corner.

I'd do about two servers a day in between other things, taking advantage of redundant power supplies to transfer the PSUs one at a time to extension cords, swap to a long network cable fast enough that TCP sessions probably didn't time out, and then unrack onto a hydraulic lift card and do the same procedure the other way.

I presented this at the start as far from a guaranteed strategy - that it would minimize downtime but there would inevitably be some due to mistakes. None of this was really that critical. There were a few devices that were pretty old and poorly maintained, we agreed up front that if these lost power for some reason and then failed to boot, we would just say they'd lived long lives and purchase replacements.

I guess the point is that this whole situation was kind of unusual and I would generally not recommend doing this, we were lucky that all the equipment left had stakeholders that acknowledged it was legacy stuff and they could tolerate losing it.

The irony is, of course, that it went perfectly. So far as I know there was not a single problem experienced through the whole thing. I even managed to swap the phone lines to the (surprisingly busy!) legacy fax server when each was out of use.

groby_b• 6 years ago

There were a few devices that were pretty old and poorly maintained, if these lost power for some reason and then failed[... we'd] purchase replacements.

And the sysadmin let that opportunity pass??

jcrawfordor2• 6 years ago

Heh, I know of a couple of people who would have been far happier if the fax server specifically was somehow accidentally tipped down a stairwell... by this time there were enough dirt cheap "cloud" fax services that it really didn't make sense to keep the on-prem system, particularly since the office had been almost entirely migrated to VoIP except for a few oddball devices like that fax setup. But that whole thing just becomes a story of the internal woes of that particular megacorporation, from the computer room to the front office there was a whole lot of stuff just being kept over from the '90s.

michaelmior• 6 years ago

swap to a long network cable fast enough that TCP sessions probably didn't time out

I'm just picturing the networking equivalent Indiana Jones swapping the idol with sand in Raiders Of The Lost Ark.

noobermin2• 6 years ago

Sorry but this is ridiculous. It's a great story of a feat of sysadminery, but the client should have just accepted some down time, even a few hours. The level of entitlement from some clients people get is just infuriating. Even down to calling him back for not agreeing to help, what an infuriating person.

That was my main take away from this. Endeavor to be the sort of person who can refuse clients, the entire idea that "the customer is always right" enables so much ridiculous behavior.

pengaru• 6 years ago

Decades ago working in a sysadmin role at a hosting company I had a similar situation.

The solution I came up with was to fashion a custom male<->male power cord, like a gender changer, from some broken ATX PSU scraps we had laying around. By rearranging the power sockets from multiple donors, two male power cords could be connected on a single enclosure. Internally the sockets were simply bridged, otherwise the PSU was basically gutted.

With this goofy metal box having two male power cords dangling from it in hand, I just used a very long extension cord plugged into an outlet on the same AC phase as the existing server's power source. The extension cord powered one of the bridge cords. The other bridge cord plugged into the server's existing - and hot - power strip, forming a redundant power source. Now the power strip could be unplugged from the primary power source without losing power, and we just moved the server to the new location with the bridge box and power strip in tow.

If memory serves the only tricky part was determining which outlet at the new home was on a compatible circuit. We didn't have much in the way of electronics tools, no oscilloscopes or anything. Even the soldering involved to make the bridge box was done using my personal soldering iron, which just happened to be in the office because some of us raced RC cars there after hours.

I think I just used an incandescent desk lamp to verify a normal brightness on the bridged circuit before proceeding with the server, but it's been a while.

I wonder how many people have fashioned AC power cord gender changers throughout history... :)

shireboy• 6 years ago

You realize the client is condescendingly mocking the guy for saying it can’t be done now, and will expect this next time they run updates on the server, which is to say never

partiallypro• 6 years ago

This reminds me of the Seinfeld episode with Frogger

aleem• 6 years ago

Sometimes it's better to seek forgiveness than to seek permission.

Saturday 3AM shift with a 5 minute downtime would work just as wwll. Unless this server has had historically 100% uptime this would go unnoticed.

aeno• 6 years ago

A real classic comes to mind: https://www.youtube.com/watch?v=vQ5MA685ApE

Moving a running server about 7km through public transport without downtime.

snvzz• 6 years ago

Incredible. Five minutes of VM downtime were not acceptable. And yet, they had a single VM host. Should it catch fire (hardware FAILS!!!), what then?

lightlyused• 6 years ago

I seem to remember a similar story from another site. I'm thinking it was thedailywtf.com, but I can't find it.

Moved a server from one building to another with zero downtime

Related Stories