FedEx to close data centers, retire mainframes

https://www.datacenterdynamics.com/en/news/fedex-to-close-data-centers-retire-all-mainframes-by-2024-saving-400m/

by taubek • 4 years ago

492 559 4 years ago

sschueller• 4 years ago

So are the moving to AWS? Are we going to have a complete breakdown of packages delivery next time there is an outage at AWS?

joekrill• 4 years ago

The company is a known Oracle Cloud and Microsoft Azure customer.

ehnto• 4 years ago

Hard to say for certain. Companies that size often have many very independent operations run by separate internal departments. I have worked places with internal, AWS and Azure operations underway.

andyjohnson0• 4 years ago

I read the parent comment as questioning the wisdom of relying on a cloud provider (but not specifically AWS, as you indicated) for a global operation life FedEx. Even with service zoning, in extreme cases that provider constitutes a potential single point of failure. Unless FedEx are able to run their core systems on Azure and Oracle‡ then I think the point stands.

It would be interesting to know how Fedex sees the risk/reward equation, and how Azure (for example) compares to a (IBM?) mainframe data centre in terms of reliability and uptime

‡ I'm not sure if there are any cloud neutrality solutions that permit this, but it is possible that FedEx have rolled their own.

sokoloff3• 4 years ago

The article says: The company is a known Oracle Cloud and Microsoft Azure customer.

With Amazon being a formidable logistics competitor, I can see why Fedex might be loath to fund a major competitor.

rbanffy4• 4 years ago

The company is a known Oracle Cloud and Microsoft Azure customer.

I guess I'll prepare for the inevitable apocalypse then...

vasco• 4 years ago

Do you trust FedEx to operate datacenters better than AWS or Microsoft?

voxadam• 4 years ago

I don't even trust FedEx to deliver my packages and that's supposedly their core competence.

kragen25• 4 years ago

Not especially, but that hardly matters; what matters is that FedEx's CIO doesn't trust FedEx to operate data centers in a way that is better for FedEx than AWS or Microsoft.

anonymoushn2• 4 years ago

If AWS is charging a 1000% markup on a bunch of stuff, you don't have to be as good as them to save money doing it yourself

badgers• 4 years ago

This was from 2019: https://www.switch.com/fedex-signs-10-year-data-center-infra...

Thorentis• 4 years ago

Is there any gain here aside from cost saving? Will this saving be passed on to the customer? (laughs). What happens when the inevitable AWS outage then causes a global shipping bottleneck? Have we still not learnt that having sovereignty is better than saving money? Has COVID taught us nothing? sigh

jupp0r• 4 years ago

What makes you think that a cloud provider outage is more likely than an existing-Fedex-datacenter outage?

rbanffy4• 4 years ago

Mainframes themselves are extremely reliable. When configured as a cluster they get five nines. I'd say a power or communications failure is much more likely than mainframe downtime.

With cloud you can rely on more datacenters and, if your application is built right, it may end up being more resilient than what a mainframe setup would give you.

Downtime is a fuzzy thing. Whenever possible, I design my apps for graceful degradation - if the database becomes read-only, we can still operate in degraded mode. If we lose queues, some things will not work, but others will continue normally and many users will be completely oblivious to the alarm bells at the NOC (just kidding, there's no such thing). My SLA for full operation is much more relaxed than for degraded operation.

dchftcs• 4 years ago

I'm not sure having mainframes counts as sovereignty. Yearly savings suggests it doesn't.

sokoloff3• 4 years ago

What’s to gain other than $400M in savings every year indeed?

If it drops all the way to the bottom line, that would boost their 2022 profits by 35% and 2021 profits by about 45%. Seems like a decision you have to consider, even with possible failure modes.

oblio5• 4 years ago

https://en.wikipedia.org/wiki/FedEx

Wikipedia says their net profit for 2021 was $75 billion, so your math seems off.

ClumsyPilot5• 4 years ago

In a security business, if you drop all precautions your profit margin becomes 100%! Great business decision!

iso16313• 4 years ago

Fedex will continue to charge as much as they can to boost their profits. This move will increase their profits, making $1.50 on each of 100 million deliveries instead of $1, boosting profits from $100m to $150m

However if they were to undercut UPS and others they might be able snap up UPS business, dropping the profit to $1, but on 200 million deliveries, and thus they'll make $200m instead, you the customer will have a cheaper delivery, and everybody wins

nly• 4 years ago

Until it becomes a race to the bottom and UPS drop their prices as well. Then both companies suffer

jon-wood2• 4 years ago

The immediate gains I can see are:

* In-house development teams have more agility in standing up new services, since they can just use EC2/ECS/one of the other 1500 container runtimes AWS has, rather than having to wait for someone to provision new physical servers for them.

* An extensive suite of complementary products are available from cloud providers, it's not just about being able to start a VM, in many cases you don't even need to because you've got Lambda, and the various managed services.

* Less training/hiring overhead, since people who can use AWS are pretty much a commodity at this point, whereas people who can manage a mainframe are an increasingly rare breed.

* Redundancy becomes easier. As does data locality, let's say Singapore declare all services delivered in Singapore must be hosted there. That's a lot easier to handle if you don't have to go and build/acquire a data centre first.

* The aforementioned 400 million dollars per year.

throw0101a• 4 years ago

In-house development teams have more agility in standing up new services, since they can just use EC2/ECS/one of the other 1500 container runtimes AWS has, rather than having to wait for someone to provision new physical servers for them.

How many people run physical servers 'raw' anymore? I would hazard their current stack involves VMware, Hyper-V, Open Stack, or a combination of the three.

You can have an API system spin-up just as effectively in a private cloud as a public one.

knorker2• 4 years ago

You can take part of the savings by buying NYSE:FDX stock.

I'm not being facetious.

aliswe• 4 years ago

These hyper-ideological takes on finance are so tiring. If you think they are bound to make more profits, I suggest you invest in them, they are called FDX on NY stock exchange.

EDIT: I used to be like that, too.

baud147258• 4 years ago

Are they really going to save 400 m$, or was it the figure "promised" by the shop that will handle the migration?

makeitdouble• 4 years ago

Is there any reasonable answer to this question ?

arghwhat3• 4 years ago

It certainly seems unlikely. Maybe it's the current operational cost, not subtracted the new operational cost. Maybe it's due to already adopting cloud and thus having paying double. Maybe it's due to the use of mainframes, rather than conventional servers.

Either way, it's definitely not the cost difference between on-premise and cloud. Cloud providers are not charities, and their buildings and staff are not cheaper than yours.

Aeolun2• 4 years ago

Either way, it's definitely not the cost difference between on-premise and cloud. Cloud providers are not charities, and their buildings and staff are not cheaper than yours.

Exactly, which is why it’s quite a bit cheaper to not have a building and staff right? The premium you pay for the cloud provider having them must be less than having them yourself, or nobody would ever move to a cloud provider.

oceanswave• 4 years ago

So do you feel FedEx should make their own trucks, ships, and planes?

afavour• 4 years ago

Either that or it’s an accurate figure for the first year or so while the cloud provider gives a ton of credits. By the time that all runs out the senior exec behind the migration will have moved on and someone else can do the maths.

kurupt2132• 4 years ago

$400M sounds huge, but why do I think it’s trivial next to FedEx’s operating budget? A couple years ago they were making 18 Billion* a quarter.

* edited from million (meant to type billion)

oblio5• 4 years ago

If think you mean $90 *b*illion a quarter.

pclmulqdq2• 4 years ago

They were probably promised a very low price for their first few years, before their cloud costs get re-negotiated. Once they are locked in, the cloud cartel can jack up their price to make sure they get as much money as possible.

But also, mainframes are extremely expensive, so they can probably do better on commodity hardware. The cloud is a way to rent a lot of commodity hardware.

The big-brained move that they are almost certainly not doing is to use cloud services to bridge the retirement of their mainframes and move everything back on-prem with Linux boxes in 5 years.

sofixa3• 4 years ago

They were probably promised a very low price for their first few years, before their cloud costs get re-negotiated. Once they are locked in, the cloud cartel can jack up their price to make sure they get as much money as possible

People often say that, but that literally has happened once, with GCP, and has no relation to how AWS and Azure do things. Please go and find an example in the 10+ years that AWS have been a big serious contender for the world's IT workloads of them jacking up prices.

throwaway_2341• 4 years ago

It's likely the 400 m$ is inflated, "by 2024" will be many years late and there will be significant increased costs during the interim period, also likely kept artificially low.

tinco3• 4 years ago

There's not a company in the world that spends more than $1m annually on cloud costs that has saved money by doing so. You don't go to the cloud to save money, you go to the cloud to reduce technological risk.

If you go to the cloud you don't need to fire anyone for choosing IBM, you're not getting strangled by any Oracle contracts, you're not gonna lose all your data because of security holes in your use of Microsoft products.

You're also not going to be dealing with downtime because you couldn't find staff talented off to properly configure your Cisco networking equipment.

Your system administrators department is not gonna block any innovative employee initiatives because of the strain maintaining more projects puts on a deployment stack carrying 150 different projects but which was architected to solve a single business goal 15 years ago.

Imagine being a 67 yr old manager who just wants to be in the business of getting letters printed on tree pulp from Bumfuck, Idaho delivered to Louisville, Alabama reasonably effectively. Knowing nothing about computers or information technology, imagine the insane stack of perfect decisions that have to be made to get the IT infrastructure of a company like FedEX running.

Not saying going to the cloud is the best decision. Not even saying it's the decision I would make. But it does sound very enticing to just have all of those problems go away by throwing a couple hundred million dollars a year at Google, Microsoft or Amazon (btw lol if they go with Oracle or IBM instead).

hotpotamus2• 4 years ago

Article says they're going Azure and Oracle. I actually use Oracle myself, but only their free tier because it's very generous. It probably works as marketing because I'd be inclined to throw them in the mix if I was looking for a real provider.

dopylitty• 4 years ago

You're also not going to be dealing with downtime because you couldn't find staff talented off to properly configure your Cisco networking equipment.
Your system administrators department is not gonna block any innovative employee initiatives because of the strain maintaining more projects puts on a deployment stack carrying 150 different projects but which was architected to solve a single business goal 15 years ago.

As someone in the “cloud” team at a legacy enterprise I strongly disagree with both of these points.

Cloud networking is as complicated as anything the Cisco people ever did but instead of CCNA you have certifications that barely scratch the surface of the complexity. So you get cloud people who barely understand the platform they’re administering flying by the seats of their pants. And instead of having a networking team to focus on networking the same people trying to figure out static routing across regions in AWS are also the people responsible for migrating EC2 instances from GP2 to GP3 but they were deployed with cloudformation which will replace the instance if your change of the disk type.

So getting to the second point this team will be totally overwhelmed and likely inexperienced so good luck getting them to do anything to help your “innovative” project because right hope they’re too busy trying to figure out how EKS is using up all the IP addresses in us-west-2.

Executives who think they’ll save on money by moving to the cloud are delusional. They’re also delusional if they think it’ll increase stability or resilience. And that’s not even getting into the EMR clusters the “data science” team spun up and left running at $30k/day.

mbesto12• 4 years ago

There's not a company in the world that spends more than $1m annually on cloud costs that has saved money by doing so. You don't go to the cloud to save money, you go to the cloud to reduce technological risk.

I don't know why I have to explain this every time "data center vs cloud" discussions come up, but if you reduce risks, then you are in effect saving money.

Rochus• 4 years ago

you go to the cloud to reduce technological risk

And exchange it with dependability risks

you're not getting strangled by any Oracle contracts

Unless you go to the Oracle cloud

Your system administrators department is not gonna block any innovative employee initiatives

They're still there, aren't they?

ryanmercer2• 4 years ago

Are they really going to save 400 m$

I'm sure they'll save many millions.

Earlier this year I left FedEx after 15 1/2 years of service. Every single day I used an IBM AS/400 terminal to interact with several systems to do my job, the bulk of my job was done via that terminal. Yeah, that's how old most of FedEx's tech is. A few months before I left they had just migrated one of our in-house systems over to Oracle likely as part of this. That said, the mission-critical system was having almost daily errors/downtime once migrated to Oracle soooo...

I imagine some of this is the simple fact that they need to replace these severely aging, no longer supported, IBM AS/400 servers throughout the company. They lost hardware support around 2020 and, if I'm not mistaken, haven't been made since like 2008 or something. That alone is going to save a pretty penny in new hardware and energy costs as well as free up physical space at often already crowded areas.

It'll also save time-lost costs. Any time power would go out at our building, the servers would usually be done for tens of minutes even with the generator kicking on. We'd lose a couple of hours a year usually to the servers that were in our building coming back online. A couple of hours, times 100~ employees at one site, equals a LOT of backlog being created which ripples through the company. While that few thousand dollars they're paying employees during that downtime isn't much, it would cascade and disrupt the freight handling. If a single package didn't clear customs in time, then it might end up as an overage and have to go to a bonded cage, that's now 2 extra movements added, that's freight planning for possibly multiple trucks that will be part of the delivery once the package landed in the United States, you might see a thousand or more shipments (that may or may not be single package shipments) now needing to be handled extra at a half dozen ports, even more sort facilities, and even more local facilities. That's just from 1 office losing power for say a half hour.

Moving those servers to a cloud provider should provide a much better uptime which should translate to a notable savings in the above situations.

andy81• 4 years ago

IBM i on modern Power machines is fine as far as performance goes.

The hideous frontend of greenscreen RPG programs is optional, you could replace them with Java if anyone cared enough about UX for internal tools (they don't).

The hard part with that stack is getting RPG devs - most of them are 50+ and expensive.

hindsightbias• 4 years ago

Save money by replacing 14 year old machines? More Oracle in the cloud? IBMi is available in the cloud now.

Now they'll need how many engineers to move the code stack (that probably no one knows) to some groovy stack with 12 layers.

benj1112• 4 years ago

What's the 8MW in this context?

Is that power. Or something else?

bcraven• 4 years ago

"We presently own and operate four primary campus locations (“Primes”), encompassing 12 active multi-tenant data center facilities with an aggregate of up to 14 million gross square feet (GSF) of space. Our existing facilities are equipped to provide up to 490 megawatts (MW) of power, with the potential to scale to more than 1,300 MW upon full build out of our existing footprint."[0]

It sounds like Switch Inc. advertise their datacentres by Wattage. I am not sure how common that is.

[0]https://investors.switch.com/company-profile/default.aspx

tgsovlerkhgsel2• 4 years ago

Very common.

rbanffy4• 4 years ago

Reverend Mother Mohiam: Many men have tried

Paul: They tried and failed?

Reverend Mother Mohiam: They tried and died.

bryanrasmussen• 4 years ago

context is everything, reading that I thought it would be hilariously funny as an exchange in Father Ted.

sackerhews• 4 years ago

Wow, this both a fantastic joke and could be used as a school book example of the importance of context.

Given of course that everyone would have watched Father Ted. But the world would be a better place if everyone did.

bradfa2• 4 years ago

It would be interesting to see the actual numbers on how much such a transition will cost.

Can their existing mainframe software and databases simply migrate into a cloud offering? If not, then if their planned transition has delays or technical hurdles, how much of that cost savings is affected?

If their cloud providers change pricing structures in future contract negotiations or if the amount of data they transfer in/out of each cloud provider location changes dramatically in order to better serve their customers, how much of that cost savings is affected?

It's clear that you can run a business with minimal amount of physical plant for servers. It's still not entirely clear to me that large established businesses can actually save money this way.

kragen25• 4 years ago

Ultimately companies that abdicate their informatics operations like this will give their profits to their data-center operators, who will be empowered to charge them whatever price they want. Because what's their BATNA? Migrating from Azure to AWS when Microsoft doesn't want to let them?

Renting your information infrastructure is a great way to reduce startup costs, but down the road, that information infrastructure runs your company. Trying to outsource it is like trying to outsource upper management.

To be clear, I'm not saying that the optimal amount of cloud services for an established company like FedEx to buy is 0. They bring in management consultants, too. But it sure isn't 100%.

cs7024• 4 years ago

Without a doubt, a future generation of executives will revisit and reverse the decision to rent all information infrastructure, but that will likely be many, many years down the road. In the meantime, the current generation of executives who made this decision will look very smart for saving the company lots of money for a good number of years. And they stand to benefit personally from it. They're doing the rational thing!

noobermin3• 4 years ago

It's amazing there are no other voices in the room pushing back against this. It really makes you wonder how companies even function at all, much less make money. Being so short sided never ends well.

ithkuil• 4 years ago

Tragedy of the executives

tunap• 4 years ago

Sounds a lot like farming out NOC-Ops to Wipro. Somebody had to think it was a good idea & managed to sell it to the policy makers for a nice promotion. So, when reality hits, does the same person get demoted further down than their original ladder rung?

kragen25• 4 years ago

I don't think it will be a future generation of executives; I think it's the current generation of executives at companies and agencies like Google, Amazon, Microsoft, and the NSA who will not rent all their information infrastructure. I agree that the executives who thus turn their companies into sharecroppers on land owned by Microsoft et al. will be richly rewarded despite the misfortune that befalls their stockholders; like Stephen Elop, if things go badly, they have an impeccable excuse.

goindeep• 4 years ago

I used to work with a very smart man that I'm sure was some kind of secret genius. He's was that sort of tech gofer. Hardware, software, didn't matter, if there was a problem he'd solve it. Sort of guy you'd see carrying a thick ass SQL book around because he 'needed to learn it' to solve just one little problem. He built whole entire solutions for the company I worked at in his spare time that the company once tried to sell for 500k and at a previous company I heard he figured out a way for the pain mixing machines to save on paint or recycle it or something saving them 1.3 Mil a year. When Raspberry Pis first came out he was one of the first people I saw tinkering with them and he was in his 50's doing it just for fun, I think he ended up using it to open and close his garage door from work or something just to scare his wife.

That sort of guy. Well he once told me something about executives and upper managers working for corporations that I have never forgotten. He said to me, and of course I am paraphrasing:

"Change gives the illusion of progress". I asked him what he meant and he responded with something to the effect of "They have the habit of changing big things every 5-10 years on purpose to make it look like they are productive, and to justify their own roles, one guy will come in and 'cut costs', the new guy after him will 'invest'".

VBprogrammer2• 4 years ago

I think you are absolutely right, however when they do reverse the decision they will publish a great article about how bringing their datacentres in-house is going to save them $800m a year and a bunch of execs will grant themselves a bigger bonus for saving the company so much money. It's a win-win!

pid-1• 4 years ago

That seems to be a common take on HN - cloud is too expensive.

I'm curious whether the folks claiming that have any data center ops experience.

Because, personally, I'd rather retire than deal with Dell, HP, Cisco, fibers, cooling issues, physical security, hardware failling... And that's just the hardware. Then you still need to pay VMWare for a decent virtualization platform, monitoring tools, etc.... Seriously, no amount of money would make me work in a DC again.

I believe companies selling bare metal as a service are a happy compromise of cost and convenience, though.

jes• 4 years ago

You are right about this.

One of the visible signs of an unrecognized (and therefore, unresolved) dilemma is oscillation between two poles.

I got this from the late Eli Goldratt and the problem-solving tools he created. One of those tools is the "Evaporating Cloud," which is a way to visualize a dilemma of some kind.

I'd suggest that there is an unresolved dilemma here: Should we rent or own data centers?

Over a period of years or decades, you can watch these things flip-flop between two extremes.

It's interesting to me that this dilemma is kind of like a specialization (in OO terminology) of a more generic dilemma, which might be something like: "In general, do we want to own or rent the things we need to run our business?"

If your turn your head a bit, close your eyes a bit and squint, you can see how a dilemma like this can apply to things like "What should be our policy regarding employees vs. contractors? Should we try to hire and retain over a long period of time, or should we rent the people we need to get the job done and release them as soon as we are done with them?"

The overall point is that whenever you see flip-flopping between two poles, think "Unrecognized dilemma."

Sorry if this is vague.

mbesto12• 4 years ago

rent all information infrastructure

Most of them were renting most of the infra anyway:

- Co-location (physical space rented)

- Managed service to keep the lights on (power, back-up generators)

- Leased hardware that gets replaced every 3 years

- Managed service for switch, firewall, etc. monitoring

- ISP, back-up generators, etc.

but that will likely be many, many years down the road

They're going to explain to their future bosses that buying land, building a huge building, buying servers, figuring out cooling, setting up redundant ISPs, etc. etc. is somehow going to be smarter? Explain that one to me?

synergy20• 4 years ago

Some might return to on-premise data center, but most will not.

It's really just like the utilities these days, 98% people will not dig their own well and install some sewage system and buy a generator, but big companies can still opt to have them on-premise installed, even though some of those are just backup systems.

there is no return for the cloud for 98% of us.

WFHRenaissance• 4 years ago

Great concrete example of the principal-agent problem here.

gravypod• 4 years ago

that will likely be many, many years down the road

And this is where people in my age range (20-30) will get to swoop in if we invest the next 10-20 years learning about and tinkering with server hardware.

I highly recommend it for everyone.

jollybean3• 4 years ago

They are doing the rational thing and there is no going back.

HNers need to get their heads around this: the economies of the could represent a fundamental, secular shift.

Imagine if Fedex designed and made their own delivery vehicles. Then they 'outsourced' that to Ford/GM. You might say 'look at Ford's amazing margins, look at the money being left on the table' - but in reality, it's still more cost effective to 'buy vehicles' the to 'DIY' them.

In the 'long run' those Azure/AWS margins will erode - and/or - the long tail will creep up on them. Basically data centers offering less, for less, and companies realizing they don't need the complexity of AWS do do basic things.

What that will look like is hard to say.

With hardware it meant 'pivot to Asia'.

Will this happen again? Chinese companies creating large data centres in the US with minimal staffing, monitored and operated out of China?

If we were not in a giant geostrategic kerfluffle with them, then yes, I would say that is the future. But given security issues etc. it's probably not.

Can India do that? Maybe.

adamc• 4 years ago

This is one of the things bad about Wall Street: Short-term decision-making. Everybody cares what happens to stock prices in the near-term, rather than 10 years out.

dehrmann4• 4 years ago

I disagree. This isn't FedEx's core competency or a differentiator for them. They're also not at a scale where it could make sense to colocate or build their own DCs. Fiddling with low-level tech infra would just be a distraction.

Now, they might find that software is a differentiator for them, but that's different.

systemvoltage• 4 years ago

Some are starting to question the almighty Cloud™: https://www.economist.com/business/2021/07/03/do-the-costs-o...

One of the things that bother me about AWS is that I still need to manage their risk of a data center going down by using multiple availability zones and managing the complexity of extra resilience. There is a conflict of interest: AWS has a perverse incentive in keeping a single AZ robust and unless there is a natural calamity, it should not go down.

I thought one of the major reasons of going to cloud is that you need not manage risk and offloading it from on-prem.

bhewes2• 4 years ago

At least in Oil and Gas. Servers moved back into the personal towers for the first time in a decade. Latency issues.

walrus01• 4 years ago

reminds me a bit of:

https://www.reddit.com/r/AskHistorians/comments/vqr30e/jack_...

"Jack Welch extracted record profits from GE for 20 years, but left it a hollowed-out "pile of shit," according to his successor. What exactly did Welch do that was so damaging, and how did he get away with it for so long?"

from the post reply, in case it's not available from the URL:

alecsliu · 2 days ago · edited 2 days ago Gold2Eureka!Bravo!Today I Learned

Welch took over a GE that was at the time, a major company. At the time, he viewed GE as bloated and needing to change. While he might've been right about that, the approach he took was perhaps less ideal.

One of the worst things he helped make commonplace among American companies is the concept of stack ranking. The way it worked was like this: people were divided into three groups: A, B and C. A's were the top performers who needed to be rewarded generously, B were adequate performers who should be allowed to stay, and then C were those who needed to be fired.

So far so good right? Well, not exactly. For Welch, these three buckets could be separated into the top 20%, the middle 70%, and the bottom 10% (20/70/10). Based on the above points, the bottom 10% would thus need to be fired annually.

In the short term, this helped in making the company lean and look more productive, increasing the bottom-line and portraying an image of success. However, the long term consequences of such a change was cultural degradation and the introduction of new bloat and waste (running all of those performance reviews and firing and rehiring so many people takes a lot of money.) Consider the case of a perfectly adequate team: the entire team has achieved their targets and has contributed to the company as per their job description. The issue? In stack ranking, 10% of this team would still have to be fired even though everyone did their job. Unsurprisingly, the introduction of this competitive atmosphere where it isn't enough to succeed, one must be better than their peers, results in backstabbing, competition, and a host of issues which eventually weaken a company's competitive edge.

This was only a part of Welch's general treatment of workers as numbers rather than humans. Aggressive cost-cutting, offshoring, etc. were the norm under Welch's regime and he would destroy entire divisions. This again, was great in the short-term but bad in the long term. Welch clearly had a dim view of company culture and believed it to be unimportant.

The other major issue of Welch's was the acquisition of hundreds and hundreds of businesses, as part of Welch's goal of acquiring his way to the top. On the surface, this is what Welch did; on paper, he didn't destroy the main profit makers of GE so much as create new ones, primarily in the form of its financial arm.

The result was that this allowed GE to play with the numbers in a way that allowed him to make sure that GE was always meeting targets set by Wall Street; in order to generate the right earnings, simply buy or sell certain assets, write-off others, etc. and when your company is an acquisition machine, it's not too hard to find the right numbers. This process helped expand GE into a mega-conglomerate but again, ultimately left the company in a weaker than expected state. In particular, on paper the core business of GE became its financial arm, as that was where all the funny business with the numbers was happening (nothing explicitly illegal though). Ignoring the damage the Welch did to GE's profit centers through his horrible business practices, this was a huge part of why GE declined so rapidly in the years following. GE's valuation was based on an inaccurate picture of its profits and value, so when the truth started coming out (especially with the Great Recession collapsing financial services profits), GE was quick to follow.

As for why Welch was able to get away with it all? Well, the answer was because he was delivering. He hit the earnings targets, he made the board and shareholders very happy, by all accounts GE was the paragon of success and everything was going right. What Welch did was unprecedented, and it's hard to really understate that. For all of his faults, he had America tricked into believing that what he could do was unique and that he could avoid the realities of the economy and cyclical markets, that no matter what was going on, GE was special. Welch died a rich, rich man and many, many people profited greatly off of GE during its 20 year bull-run.

Edit: My off-the-cuff writing is always horrible wrt to grammar and structure so I'll probably edit it later haha

dhzhzjsbevs• 4 years ago

We have our own datacentres. Lucky enough to have got them up and running pre cloud era. If we were any later we would be full cloud but the reality is we outcompete everyone on price in our space because of it.

pauljurczak• 4 years ago

And they will gently depart under their beautiful golden parachutes. What a racket.

chrisseaton3• 4 years ago

You can say the same thing for hundreds of parts of a company like FedEx.

For example FedEx flies Boeing jets. It's completely reliant on Boeing for parts for those jets. Presumably Boeing can charge whatever it wants.

Your company is always going to be reliant on other suppliers and contracts. Unless you're planning on building your own independent country in some location that has all the natural resources you need.

Non-argument.

tpetry• 4 years ago

For example FedEx flies Boeing jets. It's completely reliant on Boeing for parts for those jets. Presumably Boeing can charge whatever it wants.

Nobody said that FedEx should build their own server hardware. And that's what you are comparing it too.

FedEx could just buy server appliances from e.g. Dell (buying the Boeing jet) and operating it. Because paying some other air cargo company will eat a lot of their margin, the same with Cloud infrastructure. They are not a startup which could better invest their time in development, they can without any problems hire some people to administer their server fleet. When they switch to a cloud they will likewise hire some engineers only managing e.g. AWS to administer it.

beowulfey4• 4 years ago

The better analogy would be that Fedex, since they need to transport things from city to city, could choose to buy jets to transport with instead of renting them or contracting a company to fly for them, which they already do. To remove datacenters is more like removing the jets.

Edit: removed rude phrasing.

kragen25• 4 years ago

It's true, of course, that every company is deeply reliant on its web of suppliers and customers, and it is common for a company to have a single supplier or customer that has enormous power over it. But I think you may not appreciate the impact of that situation.

FedEx might have a viable alternative to Boeing for repair parts, but not for planes, and more broadly I think you could make that argument about US airlines in general in the late 20th century: despite things like the MD-80 (before the merger) they really had no alternative to Boeing. It is perhaps not a coincidence that the net profits of US airlines in the late 20th century were almost exactly zero, while Boeing was and is one of the most profitable companies in the world. So far Boeing isn't squeezing FedEx that way, but not because it can't.

(I do note, though, that FedEx owns its own jets rather than renting them.)

But I think there's a different sense in which informatics are more core to FedEx than even flying. FedEx is a corporation, which in its essence is a set of business processes: relationships, practices, and information; unlike a 19th-century railroad, its physical assets like airplanes are relatively expendable by comparison. Historically, those processes happened mostly in people's heads, especially the heads of its managers, but today the vast majority of FedEx's business processes are automated, which means they happen in FedEx's data centers.

And whether those automated business processes happen at all, and how well they happen, is a matter of competence in informatics. Dan Luu argues convincingly that Twitter's kernel team has been key to their ability to execute in https://danluu.com/in-house/; FedEx needs not only data centers but a kernel team and a convex optimization algorithms team.

It's extremely common for companies that are deeply dependent on a single supplier or customer to end up either merged into that other company or bankrupted by the situation.

As for the country, when I set it up I'll be needing yeomanry regiments. You in?

yobbo3• 4 years ago

For example FedEx flies Boeing jets

This, in particular, is a bad argument since jets are replaceable and interchangeable.

Possibly, the only thing that is not replaceable in FedEx is their information system.

spr93• 4 years ago

Your airplane analogy is apt, but it cuts the other way.

In fact, FedEx likes to buy out Boeing facilities so that they're not "completely reliant" on Boeing for anything.[1]

As for the more general cloud v. on-prem debate, I usually don't like analogies, but the plane analogy is good because jets and logistics-management systems are two things that FedEx needs to handle extremely well to compete.

So, what does FedEx actually do wrt jets?

FedEx both owns and leases them. It mostly owns them outright. Its owned jets are a mix of new-ish to very old (think DC-9s). It also leases and some financing magic (such as some big lease agreements for new 777s a couple years ago).

The reason is that FedEx generally gets excess value from owning rather than leasing, but there are some circumstances where leasing makes sense.

Same goes for "cloud" deployments. The correct answers to cloud "versus" on prem for large organizations like FedEx - "it depends," "that's a false dichotomy," and "it's almost certainly a mix of both" - are neither interesting nor simple, and so those answers don't get execs' attention...or headlines.

For FedEx to brag about taking an extreme position on cloud deployment is breathtakingly foolish. If I were an investor, I'd want to hear something like, "Based on a careful analysis, we've decided to shift certain specific operations to cloud-based systems. We plan to maintain control over the infrastructure and operations of mission-critical systems that have demonstrated resilience." (I'm assuming the latter are systems like the financial institutions rely on, where purpose-built stuff like mainframes using IMS have demonstrated near-zero downtime and darn-near-bug-free software for decades, because FedEx, like banks, needs to handle lots of simultaneous transactions in real time without error or deadlock.)

[1] Example: https://www.ch-aviation.com/portal/news/102874-fedex-to-take...

usrusr2• 4 years ago

But that's only really a change when you come from commodity hardware. For those still on mainframes, what's the difference? Technically the hardware might be charged per replacement cycle instead of monthly, but vendors are just as empowered to demand whatever they want. It's certainly hard to avoid lock-in running stuff on a cloud provider, but I suspect that it is even harder when depending on ye olde mainframe.

kragen25• 4 years ago

I agree, mainframes are almost as bad.

yarky• 4 years ago

I agree, isn't it way easier to switch vendors when running on the cloud? I'd say that encourages competition, since we can literally move the infrastructure using keystrokes instead of muscles.

This makes me think computer infra could potentially become a commodity.

Hnrobert42• 4 years ago

that information infrastructure runs your company

This is silly. There is a lot more to FedEx than hosting physical compute infrastructure. FedEx doesn’t own all its own airports. A CSP is just another vendor.

dsr_• 4 years ago

At almost every airport that FedEx has an operation at, there is another airport within 100 miles that could be used instead, managed by a different company. The barrier to changing is not controlled by the airports.

That's a competitive environment.

How many major cloud providers exist, and what could they do to make it difficult for FedEx to leaving for another one?

tgsovlerkhgsel2• 4 years ago

Theoretically cloud providers should be able to provide both the infrastructure and managed services much cheaper (due to economies of scale) than a single company running their own.

The question is how much of the savings end up in whose pockets, or whether the price at which they actually sell exceeds the TCO of running it yourself.

Outsourcing at least the commodity part (dumb hardware) seems reasonable, and migrating to a competitor doesn't seem like too bad of a BANTA if you aren't locked in.

transpute• 4 years ago

https://twitter.com/cynicalsecurity/status/15407428422381281...

Clown Computing is an elegant, distributed, breach of trust.
Take customers, as many as you like, convince them to hand you all their data, the management of said data, the authentication and, while you are at it, the DNS. Oh, and they pay for it too!
If this was done in real life it would be called a robbery, possibly at gunpoint, definitely in broad daylight. It used to be the case that Oracle licensing was deemed the pinnacle of IT robbery ('90s) but it looks positively quaint now.

lotsofpulp5• 4 years ago

The transfer of various liabilities is omitted from the above tweets, which is clearly worth something to some people.

dalbasal• 4 years ago

I think this is one side of an argument. It's valid, and it needs to be heeded, but still incomplete.

Cloud setups are useful, as are many ways of their associated architecture/infrastructure elements. Saying no to these is problematic too.

It is true that cloud computing has become a monopolistic, categorically non-neutral sector. They'll have you over a barrell.

So, dilemmas. Difficult strategic choices.

lr4444lr• 4 years ago

No, they could use several proprietary solutions that aren't "pay by the hour", that come with their own consultants as well as the physical infrastructure. Also, FedEx is not a customer that a big profit sucking outlet wants to lose by gouging rent. They have more leverage than you're giving them credit for.

JumpCrisscross3• 4 years ago

companies that abdicate their informatics operations like this will give their profits to their data-center operators

Is the expectation data centre operators will see margin expansion? (Genuine question.) I had thought data centres were increasingly becoming commoditised.

kragen25• 4 years ago

Yes, a cloud vendor is a company that has literally all of your company's data and a plethora of cloud services that are subtly incompatible with their competitor's offerings, and their machines make the millisecond-by-millisecond decisions about what your company does. They are in a position to calculate precisely how much you can afford to pay them without going bankrupt, and charge you $1 less. Oracle has aspired to this business model for decades, but as you can see from the fact that many of the other Fortune 500 companies still have profits, hasn't been entirely successful.

etempleton• 4 years ago

I think it depends. There is certainly a long-term cost risk, but future costs are unpredictable and it is not likely a core competency of FedEx anyway. FedEx likely will never invest enough to be great at managing their own data centers so why not remove the complication.

lallysingh2• 4 years ago

On mainframes, they were already giving up profits and flexibility to IBM and that small ecosystem of related vendors.

michaelcampbell• 4 years ago

Trying to outsource it is like trying to outsource upper management.

You say that like it's bad.

tyingq• 4 years ago

I do get your point, but if they have a fair amount of IBM mainframes, they are already renting their infrastructure. The mainframes have a culture of making you pay for operating system and other pieces on a basis of how much compute power you have "varied on". You don't really "own" a z-series box.

Spooky23• 4 years ago

Most public companies already outsource their executive management to Wall St analysts.

I think the reality in this case is that the cloud providers are little different than the incumbent (ie IBM or some other dead platform). Mainframe is a sole source solution, and IBM is always trying to extract their own pound of flesh as they manage that business down to zero.

At scale, you’re better off managing a handful of suppliers (probably Microsoft, Oracle, Azure, IBM) than dealing with one, and figuring out how to sustain the business as the workforce literally dies off. Nobody with half a brain is getting mainframe skills.

I’m not a fan of FedEx, but they seem to be a company well aware of its limitations and able to take action to correct. For example, they went to “outsourcing” the last mile delivery to independent contractors first when they figured out (almost too late) that e-commerce needed ground shipping. They sort of did Uber, except they didn’t have the ability to just break the law.

ArtWomb• 4 years ago

The "real world" counterpoint to this somewhat abstract notion is that even if FedEx wanted to build on-prem, there is not enough cloud ai talent in the universe to meet future demand. It's not just their pref for academic sanctuary, it's that there simply aren't enough cloud architects & AI PhD's with experience at AWS scale. It's hard. Choice is an illusion. GCP AutoML, Tableau, Bubble. Way simpler to train fresh grads via cloud native tooling, than from the cli (the way we did things) ;)

I think the digital transformation to watch is the US Government's IRS Tax Cloud. Never been into taxation law & policy wonkery myself, but Tax Cloud is purported to Save Trillions in TCO!

_the_inflator• 4 years ago

Even in CS we call it "loose coupling". There is no no-coupling.

I agree with you, that the hype train "cloud = total freedom" (whatever that means) is rubbish. More and more it seems, that a cloud model is just yet another option between Mainframe, AWS, Google Cloud etc.

dylan604• 4 years ago

As a startup, the greatest benifit of cloud compute isn't the fact that I can scale up at the push of a button, but that I can scale DOWN at the push of a button. I want to test things on beefy systems, but I don't want to have to pay for those systems any longer than I need to use them. To me, this is the single best thing about cloud compute.

Once your start up has become an established company and are so busy that you are no longer able to scale down regularly, it seems that's the time to start having your own compute vs continuing to line the pockets of the cloud providers.

kragen25• 4 years ago

Yeah, that's my thinking. Also as a small startup they don't benefit from confiscating your profits through price discrimination; if anything, they benefit from "investing" in you as a customer, to help you get to profitability, while remaining as addicted as possible to their services.

bushbaba• 4 years ago

And what do you think ibm could charge them for mainframe repairs?

jeffbee6• 4 years ago

That’s an incredibly silly take. Our whole economy is based on specialization. FedEx will never be as good at running datacenter as Amazon or Google. There’s room in the system for the cloud operator to earn a profit while still offering the product at a lower cost than anyone else can achieve in-house.

Apple fairly notoriously hosts iCloud in other company’s clouds, and they’re a computer hardware and software company! If you’re less sophisticated than Apple, it’s a no-brainer to go with clouds.

jotterbrain3• 4 years ago

The part about Apple is not accurate. Apple operates several dozen datacenters, has spent north of $30 billion building them and expanding them in recent years, operates their own Kubernetes (was Mesos) cloud platform that looks a lot like Heroku internally, and leverages public cloud for a couple parts of iCloud as a redundancy play, and no more. Maps helped prompt the expansion from single-homed iTunes legacy datacenter strategy, but even that iTunes legacy has been online since the early 2000s. Note that my context ends almost a decade ago, so, there’s that. The fleet dedicated to Maps alone when I worked there was large enough to be its own FAANG/MAMAA.

They’re in the datacenter game long term. In no universe does Apple “host iCloud in other company’s clouds”. Apple is notoriously a control freak and would never place their strategic services at the whim of cloud AZs. That idea was proposed and quickly eliminated. (Believe it or not, it is possible to spend that much on Azure/GCP and still only use it for basically blobs. How do you think they moved so easily in 2018ish?)

kragen25• 4 years ago

While my take may be incredibly silly, jwb, it is a different take than the one you're arguing against, and in some sense diametrically opposed.

My argument is not that specialization in informatics operations is unprofitable; rather, it's that specialization in informatics operations is indispensable for maintaining the negotiating position required to earn a profit. I'm not arguing that there isn't room in the system for the cloud operator to earn a profit; I'm arguing that after committing themselves completely and irrevocably to cloud operators, FedEx will no longer be allowed to earn a profit, or rather, just enough profit to keep their stock price from collapsing.

Now, it may be that the advantage of specializing in informatics is so powerful that this situation is unavoidable, and the only viable option for companies like FedEx is to become, in a sense, franchises of Amazon, Microsoft, Google, or Baidu. You remember when this happened to companies like Intergraph in the 01990s: they were reduced to undifferentiated Microsoft resellers. And it's not clear they really had another option. I'm not confident that the situation is currently so extreme, but maybe it is.

As other commenters have noted, you were pretty comprehensively wrong about iCloud, but that's a minor supporting point.

mguerville• 4 years ago

Absolutely, companies outsourcing their data infrastructure to the big 3 will end up having "supply chain" and Intellectual Property issues with their data in the same way companies that outsourced too much of their supply chain top SE Asia / China did.

But by the time this becomes a problem the executive team will have cash out nice bonuses from the short term gains from cost savings...

amelius• 4 years ago

Yes, it sounds like they are about to head down the same road as many large firms who sold all their real estate because short-term investors said it was a good idea.

WJW• 4 years ago

Fedex does not create their own electricity or fuel either, yet without those they have no going concern. Yet they choose to acquire those services on the open market because it is cheaper, even though they do make themselves more dependent on the supplier. In the end, full self-sufficiency is just not viable for large companies in a highly specialized economy.

In this case, if a cloud provider gets too monopolistic then the BATNA for fedex is to re-hire (or train) enough sysadmins to run their own systems again. That puts an upper limit to how much a cloud provider can charge for their services, in addition the competition between Azure/AWS/GCP will also keep prices somewhat down. The cloud providers know this and will price accordingly.

kragen25• 4 years ago

I'm not arguing for autarky, I'm arguing for maintaining a strong negotiating position to keep from being taken advantage of. Fuel is fungible, and electricity is provided by regulated monopolies who are forbidden to price-discriminate, so the same issues don't arise as when you're outsourcing your core business processes to Microsoft.

I don't think FedEx re-hiring a competent sysadmin team and rebuilding data centers once they've lost that knowledge is merely a question of spending enough money on the problem; it's the kind of high-risk IT project that often sinks companies.

renonce• 4 years ago

The network traffic price is absolutely vital. This is essentially the ransom that the cloud provider holds on you for leaving.

thdxr• 4 years ago

it was exactly this kind of thinking that lead to Japanese automakers crushing American ones

vendor relationships do not have to be hostile. AWS has been around for a while and has never raised prices, some services have gotten 99% cheaper.

yes there's some hypothetical day where they say "tricked you, all that stuff about margin being opportunity was a lie" and they try to extract value short term while burning their reputation

but every day that goes by where this doesn't happen is $$$ wasted on trying to avoid it

marcosdumay8• 4 years ago

Hum... The US business all already converted from the GP's mindset, and the US automakers still need to be rescued from time to time.

I'm betting it is not this kind of thinking exactly that causes the problem.

(Oh, and do take a look at the vertical consolidation of the Japanese industry.)

adam_arthur• 4 years ago

In a competitive market, margins are dictated by marginal cost of production of you and competitors.

Data center/SaaS is somewhat naturally monopolistic due to cost of switching though. Legislation should target "high cost of switching" areas to make them more competitive. Translates to lower prices, better for society in the end

MattGaiser• 4 years ago

Was Fedex really running its own infomatics operation or did they mostly have layers upon layers of IBM consultants? Yes, the "owned" the hardware in that case, but it would likely have been no easier to switch.

api• 4 years ago

Yes but the round trip to/from the cloud can be a forcing function in a large bureaucratic company to dump a lot of ossified infrastructure and IT organizational baggage. When you re-localize you'll be building a new IT organization to do it and using new hardware.

Big organizations often do things that seem wasteful as a way of dumping organizational cruft.

knorker2• 4 years ago

You could make the same argument for companies that don't own their own buildings, their own land, or have vendors of any shape or form.

That is, all companies ever.

The opposite of what you're proposing is "do your core business".

Why would fedex be better at running computers than Azure is?

Why does FedEx use vans built by a car company? They should build their own vans. The tires should clearly be made from rubber that FedEx vulcanized themselves.

I've seen huge insurance companies outsource not just their hardware, but their ops too.

McDonalds may be in the real estate business (and not the burger business), but FedEx is not.

menzoic• 4 years ago

Renting your information infrastructure is a great way to reduce startup costs, but down the road, that information infrastructure runs your company. Trying to outsource it is like trying to outsource upper management.

This isn't a one size fits all thing. For a company like FedEx I don't see the advantage of owning their own data centers. They just don't have the scaling challenges that would require that. Data centers or information infrastructure as you put it does't fall within the core competency of FedEx, so I don't think it's a fair comparison to say its like outsourcing upper management. I think its more fair to so that FedEx building their own datacenters is like building their own roads to deliver packages on.

For a company like Facebook or Google, the difference is clear. They need to handle such a high scale of traffic and volume of data that they need custom infrastructure to be able to scale efficiently at significantly reduced costs. The same reason it made sense for them to invest in building their own databases, because the existing options didn't meet the scaling requirements. FedEx won't realistically be needing their own database or infrastructure any time soon.

alexvoda• 4 years ago

Difference being that FedEx pays 0 $/km for the roads, but the $/GB and $/GHz will be at the mercy of their cloud provider.

pfisherman• 4 years ago

They just don't have the scaling challenges that would require that.

I don’t know about that. They are a global logistics organization that rivals Amazon in scale. They are not parsing web scale data like Google, but as far as meatspace data goes they are they seem to be about as big as it gets?

phpisthebest• 4 years ago

>They just don't have the scaling challenges that would require that

Scaling is one of the big cloud advantages. Static Known workloads will almost always be cheaper on Prem than dynamic workloads that need automated scaling

One of the big ways an organization can save money is if they can scale up and down their loads on demand

But if you have a static 24/7 workload almost universally I can build a onprem solution that is be 50% or more cheaper than cloud

shuntress• 4 years ago

I think its more fair to so that FedEx building their own datacenters is like building their own roads to deliver packages on.

Your metaphor is a little bit off. This is more comparable to FedEx renting vs owning their fleet vehicles.

Obviously they aren't going to be setting up foundries to to scratch-build engines and network switches. But it probably makes as much sense for them to own the buildings & computers running their logistics software as it does for them to own the buildings and vehicles running their logistics hardware.

As an aside, I'm sure FedEx would LOVE to get customers locked in to private FedEx roads with private FedEx addresses that no one else is allowed to deliver too. Thankfully, our system of public infrastructure is robust enough to make this infeasible.

FooHentai2• 4 years ago

For a company like FedEx I don't see the advantage of owning their own data centers. They just don't have the scaling challenges that would require that.

Depends how seriously they're locked onto the postman/route inspection problem. Which is a factor for delivery companies and, depending on how hard you chase it, a driver for almost infinite compute. Once you're playing that game, the economics look very different to most 'not an IT company' IT needs.

jollybean3• 4 years ago

Fedex is not in the business of making their own cars, they're not going to be in the business of fairly complex cloud ops either.

This is permanent secular shift.

The advantages of the cloud are just starting to be realized.

That said, for some kinds of operations, they may want a lower cost cloud provider with a more basic stack.

Secondary operators of the world need to get together and start offering a standard stack for basic things so that people can wean off of Azure and AWS.

iso16313• 4 years ago

They don't run their own electric companies, and often don't even own their own buildings, both of those are essential. I suspect they lease their trucks and planes too.

If they're building it on AWS only kit and locking in, then they open themselves to risk. If they build it on standard VMs running on AWS, Azure, Digital Ocean, then they can shop around at contract renewal time, just like they can with Mercedes vs Ford vs VW trucks

dredmorbius• 4 years ago

https://news.ycombinator.com/item?id=31995276

earthboundkid2• 4 years ago

Because what's their BATNA? Migrating from Azure to AWS when Microsoft doesn't want to let them?

Uh yes, exactly?

Clouds give small customers the first hit free as a loss leader. With very large customers, clouds have to be price competitive because if you're at a large scale, it is totally worth spending millions of dollars for a 5 year plan to change clouds. The people who get screwed are the medium to large customers for whom the cost of changing clouds is too high to recoup in a reasonable timeframe. The moral of the story is be very small or very large, but avoid being medium sized.

bpodgursky• 4 years ago

And if all you use is object storage, k8s, and some kind of SQL data lake, migrating between clouds is... actually not that terrible?

It's a huge project but there's not a lot of actual redesigning.

BrainVirus• 4 years ago

With very large customers, clouds have to be price competitive because if you're at a large scale, it is totally worth spending millions of dollars for a 5 year plan to change clouds.

You've literally just explained why a large company would be perfectly willing to be overcharged by a cloud provider in order of millions of dollars. They would have to pay that for migration anyway and there is always a risk of creating disruptions in the process.

themerone• 4 years ago

My company built a data center that was supposed to lead us into the future, and it started running out of capacity within a few years.

The cloud is the only reasonable way to keep up with exploding compute demands.

abirch3• 4 years ago

Exactly, they literally design their own hardware. To me it's like trying to roll your own word processor. You can but unless it's core to your business, it's cheaper to buy.

rendang• 4 years ago

This seems to imply that if processing loads for a given line of business start to level off, it may lead some to move away from the cloud.

njarboe• 4 years ago

Similar to the decision to own a building or rent. Moving is a real pain and can kill companies. Better get a long lease with lots of extension options because at the end of the lease your rent is going to go way up.

marcosdumay8• 4 years ago

Because what's their BATNA? Migrating from Azure to AWS when Microsoft doesn't want to let them?

As long as they have backups, yep, that's a quite possible alternative. And backups cost very little, and don't need to run flawlessly 24x7.

They will probably pay through their nose either way (backups or not, migrating or not), because that's how the cloud works. But well, that's for their bean-counters to count. As long as they have backups, it's not strategy defining or an existential risk.

(Anyway, nobody goes around talking about BATNA of the proprietary, rented mainframes. I wonder why.)

draw_down• 4 years ago

Probably for the same reason nobody goes around talking about the BATNA of having employees. This doesn’t make any sense.

justsomeuser• 4 years ago

I think it's fine to be honest.

Over time competition between the 3 major clouds combined with hardware getting cheaper will mean there is not going to be a massive jump in prices, if any at all.

It is not impossible to design relatively cloud-portable server software.

Also I think FedEx, and companies in general, would prefer to increase revenue rather than reduce costs (by running their own data center) in order to increase their net profit.

ryanmarsh• 4 years ago

empowered to charge them whatever price they want

That's correct assuming monopolistic behavior. I believe pricing in information infra will be a race to the bottom. We've not seen a lack of competition in this space.

avereveard• 4 years ago

Eh as long as one design for portability migrating isn't that hard, which enables them to leverage their own size for volume discounts

As long as they stick to services that have overlap been providers (postgres, mongodb, nfs/cifs, etc) or transformers (s3, azure blob storage) and package their code in somewhat agnostic format (docker images, wars, what have you) they'll be fine.

riku_iki• 4 years ago

Renting your information infrastructure is a great way to reduce startup costs

they can rent hardware only, and still run their own infra (kubernets and all stuff on top on it), then they can negotiate reasonable prices and migrate away if they want to.

wnevets• 4 years ago

You're forgetting the most important part of this entire thing, the massive bonuses the current executives are going to get for saving the company over $400m. Whatever happens to the company afterwards is not their problem, they're already moved on to the next company to spread the legend of them saving FedEx over $400m.

bradly• 4 years ago

I helped with the closure of Intuit’s data centers and it was absolutely the right move for them. The amount of resources needed to maintain data centers world wide is significant.

astrostl• 4 years ago

Ultimately companies that abdicate their informatics operations like this will give their profits to their data-center operators, who will be empowered to charge them whatever price they want.

This "ultimate" endgame has been predicted since at least 2006 and we've yet to see anything but price decreases on cloud services. Tons of labor has been invested into deliberate cloud agnosticism with no apparent results. I am fully on board with economic arguments that favor data centers over cloud for some organizations. I don't think the capture-and-increase argument holds evidential water or ever will in a competitive environment, and the environment is more competitive than ever.

thethimble• 4 years ago

To add to your point I think the competitiveness will increase over time with further technology innovation. Cloud providers are constantly adding features and trying to catch up with each other. New entrants are adding novel competitive dynamics as well (e.g. Cloudflare or Deno Deploy).

The cloud market is so large ($ TAM; also growing rapidly) I think there will always be a tailwind of investment and innovation that prevents monopolistic stagnation.

madrox• 4 years ago

This line of reasoning has been around as long as the cloud has been a thing, and yet companies are still doing it and seeing better results than running their own data centers. At what point do we put this argument to bed? You want to talk about fear of switching costs? Fedex was still on mainframes, for crying out loud.

iancmceachern• 4 years ago

It's like oracle

xnx• 4 years ago

Outsourcing hosting doesn't seem that different from outsourcing electricity. If it's not a core competency, or competitive advantage, let somebody else deal with it.

xwdv• 4 years ago

It is 100%.

Cloud services are about to evolve in ways that non tech-specialized companies can replicate on their own. Cybersecurity requirements will grow and bandwidths will be pushed to the limits. Just leave it to the professionals.

eikenberry2• 4 years ago

Why be an information infrastructure company on top of a delivery company? Why would they want to do both if they didn't need to. And pretty much all companies are going multi-vendor to avoid lockin and provide leverage for negotiations. Their information is the part that runs the company, not the stack.

kragen25• 4 years ago

Why be a management company on top of a delivery company? Or a finance company on top of a delivery company? Or an employer on top of a delivery company? Because if you outsource any of those functions 100%, your vendor owns the actual company.

stanfordkid• 4 years ago

Yeah but cloud infra is a competitive market with multiple players, it's not like they're going to just buy a monthly AWS contract. They will probably have a cloud migration strategy, multiple year fixed contract, and multi-cloud support.

mark_l_watson• 4 years ago

I used to agree with your opinion. Then I worked as an engineering manager at Capital One (deep learning, not infrastructure) and I saw all of the good reasons they had to move to AWS.

kragen25• 4 years ago

That sounds interesting! Can you elaborate?

shaman1• 4 years ago

Disagree, companies cannot focus on too many things at once and get them right. Better for them to focus on logistics, fleet, cargo, etc than investing in data centers. Staying in your area of competence usually yields the best results. Btw, as I was using FedEx for an international shipment, their IT services are abysmal and could use a shake-up.

novok• 4 years ago

Yes, just like leasing office buildings instead of buying them causes companies to give all of their profits to real estate companies. Or using dell windows laptops. /s

If AWS et all get too greedy, competitors will pop up saying 'your margin is my opportunity' and market competition happens for something that is a commodity, and becoming more so as time goes on.

FooHentai2• 4 years ago

I agree that long-term, outsourcing of compute capability is probably going to be a negative for everyone. That said, when your organizations point of difference isn't rooted in computing, and you're not a producer of compute platforms yourself, why wouldn't you source that from a third party? The arguments don't look that different to buying pizza boxes from HP, Sun etc - If they could have gouged your profits away I'm sure they would, but as long as the space is commodified competition should be assured, and that sets pricing down at competitive levels, not monopolistic levels.

Far as I can see the only arguments here are that either cloud compute is naturally monopolistic, which seems a stretch, or there's another factor that will lead to ruin here (e.g. long-term connectivity loss between your and your compute in a particularly nasty disaster scenario). Like, I don't care* if my baker's goals are orthogonal to my own. They just provide crullers for the morning stand up and I can get them from other places despite the insurrection that would occur if staff ever went without. I can likewise not care if my cloud provider's goals are not aligned, because if they screw me I can just shop at another bakery, so speak.

All of this said, outsource to cloud isn't free. You have to have an exit, or at least transition, plan. You have to architect into the configuration, not just transpose what you have. You have to maintain platform-agnostic restorable backups of the information that's critical to your organization, and have an operable proven plan to bring in online. Failure to do these things, which I think is the case for a scary number of organizations that have 'gone cloud', could indeed lead to ruin.

I know this comment is weird. I'm in an odd space here of agreeing with your conclusion but mostly disagreeing with the premises on which you reached it.

*until we see a cloud provider that also owns major fab i.e. if Intel or AMD went into direct cloud provision and took it seriously for about a decade I'm pretty sure it's game and over the world itself would implode from the sheer monopoly that would ensue. Until then, however...

Jistern• 4 years ago

Your reasoning is faulty and very probably self-serving. More and more companies can and will outsource their infrastructure. IT guys hate this because it threatens their "paychecks."

The optimal amount of cloud services for an established company like FedEx is 100%, not merely with a "disaster recovery plan" but with live, 99.99% redundancy by which I mean two almost exact systems running nearly simultaneously (within a second or two of one another) on two completely different networks.

FedEx enjoys almost all of its competitive advantage from its physical network. IT is not core to its business.

Here's the problem... almost no company actually does disaster recovery and "parallel redundancy" properly because most C-level executives only pay lip service to it.

Therefore, the whole notion that disaster recovery and "parallel redundancy" don't work is predicated on the false notion that companies actually have proper disaster recovery and "parallel redundancy" in the first place.

kragen25• 4 years ago

It is surely true that I have a cognitive bias in the direction you say, and that many companies operate at lower levels of informatics infrastructure reliability than Azure.

prmoustache• 4 years ago

I am pretty sure part of it is also some kind of greenwashing move to abstract away your carbon footprint. Like exporting pollution to far east by producing there. Sure there are tools to calculate your energy/carbon footprint from cloud workloads but if you don't operate a datacenter yourself anymore you can't be as much accountable as the cloud vendor is.

But there will surely be a backlash and an - at least partial - return to operating your own datacenter for some companies in the future.

hnarn• 4 years ago

Making accurate calculations of data center cost is of course complicated and rarely manages to take everything into account, but the common knowledge I’ve heard is that there comes a point when a company is so large that going on-prem is what actually saves them a lot of money.

If FedEx is not large enough to be one of those companies, how large do you actually have to be? Or has cloud pricing changed to the point where this is no longer true, and even huge corporations can save money in moving away from their own premises?

Out_of_Characte• 4 years ago

Technology scales faster than fedex's data needs. Maybe somewhere in the future the reverse trend happens as their needs can be met with very small servers. Though I've always been wary of putting your data somewhere because getting your data to a competitor costs alot of money.

ClumsyPilot5• 4 years ago

Technology scales faster than fedex's data needs.

I don't think this sentence has meaning

throwawaymaths• 4 years ago

I don't think it's the server costs they care about it's stuff like db maintenance, security, security, and security. Security professionals are expensive, and it's very hard to get right in house on metal.

lou1306• 4 years ago

Or maybe they are large enough, they just don't realize it right now.

nuthje• 4 years ago

It's not just money, it is also how you spend a limited quantity of organisational focus. You cannot do it all, this is actually even more important when you are that big of a company.

sofixa3• 4 years ago

It's not only about size, there's also workloads to take into consideration.

Take FedEx, their operations are highly dynamic - per day but also per period. The number of packages sent, and being transported, vary greatly between a random Tuesday 15:00 and the weeks before holidays.

In such a scenario, with your own infrastructure, you need to overprovision, by a lot, to be able to handle the heaviest load possible, and then some margin on top. And that capacity is wasted what, 90% of the year?

Meanwhile if you subcontract that part on AWS, it's their problem, and they can afford to handle it, and you only pay for what you actually use when you use it.

whatever16• 4 years ago

The cost does not scale linearly with the #of packages. A table with 1 entry is not cheaper than a table with 1 million entries.

The same routing algorithms have to run either they have one package or 1 million packages.

Plus you need to keep the history of the operations for months if not years (meaning that you will store both holiday and random Tuesday data anyway).

So where is this flexible cost exactly?

tbrownaw• 4 years ago

Someone, somewhere, has to actually buy the computers. "Only pay for what you use" isn't magic, whatever fraction actually gets used (across all customers) has to end up paying for the whole computer.

Taxis are "only pay for what you use", but owning your own car is often cheaper (even if you only use it, what, 1/12 of the day?) and more convenient.

somethoughts• 4 years ago

The flip side is the approach that Amazon took with AWS. Maintain the server capacity but invent some sort of mechanism to sell any extra server capacity during the down time.

Perhaps there isn't the mindset/ability to actually execute this though.

mberning• 4 years ago

The transition to “cloud” is often more about the culture and capabilities of a company than the technology. If you have been on some shitty mainframe and a bunch of other technology from the 60s you cannot bank on that carrying you for the next 20-30 years. The people that know that stuff are ancient, expensive, and retiring. “Cloud” is a very good opportunity to jettison the legacy burden and reinvigorate the technical competence of the org.

chmod6002• 4 years ago

Running your own data centers is a form of vertical consolidation, which is a strategy. Specialization is the opposite strategy. A single company may strategically decide to specialize some things and vertically consolidate others.

The dollars-and-cents are details. Those matter, of course, but don't base your entire analysis there. Staying vertically integrated may help fedex be 4.713% more profitable today, but might miss out on important growth areas and become a dinosaur. Or maybe the important growth areas involve data center innovations, and staying vertically integrated allows them to exploit those opportunities. So it's not always an easy decision, but it involves more than the present-day costs.

cavisne• 4 years ago

Maybe this is true for public cloud pricing but it hasnt been true for some time when it comes to enterprises on cloud. Large enterprises come in and ask for a quote for X cpus/ram/storage over X years. They sign a multi year contract with a guaranteed minimum spend, and a discount on every line item. Then they "lift and shift" their physical data center into a cloud region, in a very static way (no autoscaling etc).

This works great for everyone, the hyperscalars have much lower costs than even the biggest enterprise customers so the company get a good deal. And the cloud company makes some revenue off the minimum spend (making capacity planning a lot easier) and they know once the compute and storage there it will be very likely the company starts using extra managed services.

Aeolun2• 4 years ago

Cue 10 years down the line, when everyone realizes cloud comes with tradeoffs in flexibility and reliability and they all move back to on-premise for business critical stuff.

cs7024• 4 years ago

More like 20+ years -- after a new generation of executives and managers has taken over. Their egos won't be tied up in old debates and they will be more willing to sacrifice old decisions.

Ekaros2• 4 years ago

Where are they using all of these data centers and mainframes? That is they are spending at least 400 million to do what exactly? To me that seems like rather high figure in any case for their sites and logistics...

ajsnigrutin• 4 years ago

Yep... A couple of million of packages daily + tracking a fleet of vehicles, client facing site, apis for their tracking and courrier services/devices plus some internal services (billing etc...).. this seems like it would fit on a few racks, not nearly 400million, or whatever the full number is (since 400m are just the savings).

But yeah... old company, they might still have stuff written in cobol virtualized somewhere.

jeltz• 4 years ago

But even virtualized Cobol should fit in a few racks.

hoofhearted2• 4 years ago

Their Memphis air hub processes 2 million packages by itself during the season. You also forgot to include FedEx Freight in your estimate.

dzhiurgis• 4 years ago

With 300k employees you can expect them to have thousands of disparate services that will likely get rewritten part of such migration

oblio5• 4 years ago

With (most likely) 0 knowledge about the internal details of their business, but with the confidence of the average programmer: "they spend too much".

I imagine Twitter could also be built over a weekend, right? :-)

Ekaros2• 4 years ago

Then again, maybe these savings come from whole same clean-up of services and other stuff they are running... After all there is likely decades of cruft around...

kjs32• 4 years ago

The more ignorant you are about the reality on the ground, apparently the easier it is for some people to tell others "obviously, you're doing it wrong".

83• 4 years ago

Literally every one of their vehicles is a knapsack problem and a traveling salesman problem needing to be solved every day. I'm sure they go through plenty of compute power.

dx03416• 4 years ago

With airplanes especially, optimal utilization is key. Routing millions of packages a day through such a complex network around weather and other incidents while keeping SLAs is simply astounding.

kurupt2132• 4 years ago

What price is acceptable for ownership of your data?

fanf2• 4 years ago

”FedEx first opened an on-premise data center at 250 Spectrum Loop in Colorado Springs, Colorado, in 2008”

That seems implausible: where did they keep their mainframes before 2008?

perlgeek• 4 years ago

Colocated in another company's data center.

dx03416• 4 years ago

They could get back to colocation. I think even AWS & Co don't own all physical buildings, they also rent parts of data centers in some locations. You don't need to own real estate or a physical building to run servers.

dna_polymerase• 4 years ago

Maybe, just maybe FedEx had a bunch of employees making a lot of calculations before going this route. Just maybe they made an educated decision, based on their inside knowledge of the company and it’s needs. Or maybe a bunch of people on HN just know better.

langsoul-com• 4 years ago

How did Dropbox end up saving tens of millions bt making their own cloud infrastructure but FedEx loses money?

Surely FedEx would have a metric shit ton of data?

jeffbee6• 4 years ago

Ask yourself what probably happened a few years later when they had to negotiate new leases from their datacenter landlords. Funny how they never put out a follow up press release about the enduring value of on-prem. Also a company that basically can’t grow and hardly turns a profit.

Really, not the first example I’d reach for.

humanistbot• 4 years ago

I'd bet Dropbox has way more raw bytes to store than FedEx. Plus, the Dropbox business model is based on data egress to clients, which is really expensive on cloud servers. FedEx needs data centers more for internal logistics, which is probably why they're going with Oracle.

fithisux• 4 years ago

Everything will merge to one BIG company because of cost-effectiveness.

thedougd• 4 years ago

From an executive viewpoint, this would be a reasonable forcing function to overcome long, deep rooted stasis in your IT department.

Let's assume for a moment that modernization is good. It probably is wrt mainframes. For every team that refused to modernize, you can now drive them to the 2024 date, absolutely no exceptions.

bythckr• 4 years ago

I never understood the logic behind companies abandoning their own data centers and moving their gold (Data is the new gold) into somebody else hand. I see that many here are also thinking in the same line.

But when I spoke to a CTO of a F500 company. This was the take he had. He feels one of the major driving force is to reduce the carbon foot print of the company. His company is designing the system to have the crucial data on their own infrastructure but keep they other data & do the processing of cloud service. Instead of relying on 1 provider, they are using multiple. Also supporting some small local cloud providers.

This seems very relevant for a logistics company with a huge of fleet of air craft. Especially considering the upcoming carbon tax.

hotpotamus2• 4 years ago

I'm skeptical of the carbon footprint thing for a few reasons, but the other point is interesting and something I think about once in awhile. As we build out more and more bandwidth, compute, storage, and APIs to tie them all together, it seems that the world more resembles some sort of large computer and most of the work is just getting the data and compute closer together when needed and minimizing costs when it's not. I know I'm far from the first to have this realization.

TheRealDunkirk2• 4 years ago

I've seen two mainframe "replacements" fail to either retire the mainframe, OR produce an alternative system which people like. Has any Fortune 500 company successfully "retired" their mainframe?

Someone1234• 4 years ago

I've seen it. It was essentially four steps:

- Build middleware (REST API endpoint) in front of the mainframe that was 1:1 mainframe functions. It initially does nothing except auth and routing.

- Re-write/migrate all other software from using the mainframe directly to using the new middleware.

- Work to further restrict anything directly connecting to the mainframe aside from the middleware, including staff's terminal access by building management solutions (e.g. new UIs, new management middleware).

- Once the mainframe is completely isolated, start building drop-in replacements for business segments and then switch the middleware(s) to point at the new solution from the mainframe. This should be invisible to all external consumers. You can "hot test" it by having the API execute on the mainframe + new solution, and checking for 1:1 responses.

The hardest part about the above isn't the tech, it is the internal norms that you're fighting against (e.g. staff have had terminal access for tens of years, and know the commands to get their work done by memory). So you have to create very compelling alternatives using the new API middleware/web/apps, "export to Microsoft Excel," graphing, mobile support, and similar that a terminal cannot directly offer is clutch here.

malfist• 4 years ago

This is exactly what Hilton did in the 2010s. We stood up an ESB (wso2) between the modern front ends and the legacy mainframe backend, and then broke down the mainframe's functions into microservice domains and wrote them. Had the ESB route traffic to the microservices until the mainframe wasn't doing any work anymore and was able to discontinue it.

There was a lot more complexity that that, but that's the gist of it. By the time I left, my team had carved out the bulk of the mainframe's services and had stood up over 50 microservices.

kjs32• 4 years ago

This is (essentially) how I've seen it done as well, the very few times I've seen it done successfully. But did you guys do it in 2 years, like these folks think they are?

The hardest part about the above isn't the tech, it is the internal norms that you're fighting against

This is so true it's almost painful.

lulzury• 4 years ago

There's so much wisdom in this comment. This is also how you tackle and replace very large legacy systems correctly.

mpyne• 4 years ago

Off topic but I can't tell you how reassuring this is. This approach was essentially what we had come up with as a proposal for a Navy HR migration away from mainframes, though we never even got close to being able to actually attempt it. But at least it sounds like it could have worked had we tried it.

dx03416• 4 years ago

The hardest part about the above isn't the tech, it is the internal norms that you're fighting against (e.g. staff have had terminal access for tens of years, and know the commands to get their work done by memory).

I have sympathy for those people. Not a fan of AS/400, but working in those terminals is incredibly fast once you know all shortcuts. Modern GUIs, especially using all those fancy frameworks, are much slower. If the alternative to the terminal is a new web application, it's extremely likely that it'll be harder to work with.

kamaal3• 4 years ago

Not sure about 'mainframes' specifically.

But one of the easiest way to retire anything, is to stop doing new work on the older systems. Any new project should go on the new tech.

Soon enough you will see the older systems gone. It won't happen in weeks or months, but 1 - 2 years is enough.

At one point the whole banking industry used to de-facto run on Java. Eventually people just started doing new work on Python. These days Java is just another tech there.

dx03416• 4 years ago

I've seen that with a core banking system. A decade later it's still running, with many patches in other systems to avoid its shortcomings. there's still no viable plan for migration.

Just because you don't invest in a system doesn't mean it becomes obsolete.

ryanmercer2• 4 years ago

I left FedEx (Trade Networks) after 15 and a half years earlier this year. The entirety of that 15 1/2 years was spent using IBM AS/400 terminal to interact with various systems and was also used to submit paperwork to Customs and any other applicable government agencies to clear international freight.

It... was an experience.

I imagine that is going to take quite a bit of work to migrate over to someone else's servers, as I doubt they're going to be like "sure, we can accommodate your decade and a half old, mission-critical, systems"

mensetmanusman• 4 years ago

And then AWS will charge $399M…

dontbenebby• 4 years ago

There's not really a market for three big airline shipping things. Amazon air + UPS are tag teaming them in a bad way. This isn't about IT, it's about that FedEx is not in great shape.

lettergram• 4 years ago

Lol “saving $400m” really equates to “moving onto AWS or Azure and paying hundreds of millions to them”

say_it_as_it_is• 4 years ago

This is an ideal project for managers whose end of year bonus is linked to immediate cost savings. Who cares if it costs FedEx more in the long run?

chevman• 4 years ago

I'm in corp land and this to me reads as the CIO using external PR to push internal agendas.

Are they really going to close all their on-prem datacenters in the next 2 years? Maybe.

More likely, this is a way to build momentum around the transition to cloud. Not a bad strategy, and not uncommon from what I've seen in the enterprise.

shrubble• 4 years ago

'Within the next two years we’ll close the last few remaining data centers that we have, we’ll eliminate the final 20 percent of the mainframe footprint'

Does that truly sound achievable, when they have the most difficult to migrate part of their mainframe applications still on their mainframe? The easy stuff they have already moved off.

treesknees2• 4 years ago

I wonder how much of this is motivated by the lack of staff. The market of available mainframe operators has been drying up for decades. When your entire business depends on a technology nobody is willing to learn or go to school to understand, that creates a problem.

Also, from my understanding, it can take a tremendous number of distributed x86 systems to supplement the scale, speed and reliability of a mainframe. It's possible that unless FedEx gets away from managing storage arrays and hypervisor farms, they wouldn't have enough staff to operate the datacenters once they're full of enough servers to replace the mainframes.

thr0wawayf00• 4 years ago

The market of available mainframe operators has been drying up for decades.

Just like many other industries, the market for mainframe operators that want to earn less than technologists in modern stacks has been drying up. A lot of mainframe operators refuse to bring their pay scales in line with what developers in modern stacks are earning. Why take a pay cut to go work on a 40 year-old system when you can work on modern stuff with more money?

They keep the pay low and complain about a labor shortage to deflect from the fact that they're simply choosing to ignore market forces.

kamaal3• 4 years ago

Would be curious to learn the specs of current day mainframes, and cost.

What sort of work is typically done on such machines these days.

treesknees2• 4 years ago

Disclaimer: I have not worked directly on mainframes themselves.

They've been doing the same type of work for decades. Batch processing of records and transactions (payroll, insurance, banking, airline booking, etc.) IBM releases new mainframe models all the time, and typically companies will lease them and upgrade every few years.

When I worked at an insurance company, the mainframes were the core of their financials. They had around 8 of them in an active/standby DR site. The machines could do instruction-level error checking, meaning even a weird processor failure/error wouldn't impact the end transaction. Basically if you want to guarantee accurate handling of your money, you use a mainframe.

One common thing to see nowadays is the mainframe becomes abstracted by some other layer of API, such as a rest API. This has reduced the need somewhat for mainframe operators, but in the end you're still interacting with some COBOL that was written 30 years ago and somebody needs to understand it and the hardware. And the hardware is so different from traditional servers, you can't just pick it up and wing it. You really need someone who knows what they're doing.

filereaper• 4 years ago

Its weird to have companies say they're moving from Mainframes to the Cloud.

The Mainframe is a sort of cloud, each one has usually double the spare capacity and you call IBM to unlock it.

The cost of running a Mainframe is measured in terms of MIPS based on amount of compute used and there's compute reserved for offloading things like JITs (zAAPS). Essentially a version of cloud compute costs.

I can understand if you're locked into COBOL with EBCDIC and can't find talent, but Z runs freaking Ubuntu now! I know IBM works furiously on modern ecosystem support.

Not shill for IBM, but what exactly is it that they're expecting to be so different from their cloud migrations? Or is everyone here caught up in the "Mainframe old" falsehood.

mhh__• 4 years ago

Mainframe = locked in to IBM, is probably the logic.

JamesAdir• 4 years ago

How is it any different than being locked to AWS/Azure/Oracle/Google?

unregistereddev• 4 years ago

I doubt they have too much COBOL lock-in, since the article notes their datacenter opened in 2008.

Here's the thing: A mainframe may have spare capacity, but you still have to call IBM to unlock it. In order to make a Z run in a cost-effective manner, you need to run at 90%+ utilization at all times - which is excellent for batch jobs that can be scheduled, but is difficult to achieve with on-demand loads.

You're paying based on compute available, not just compute used (unless I'm misremembering our contracts back in the day). Sure, the amount of available capacity can be changed, but a phone call to big blue is not automated. Cloud autoscaling is.

You're right that IBM mainframes are far more modern and cost effective than we assume. Sadly, they are still behind the curve of cloud hosting.

stonogo2• 4 years ago

a phone call to big blue is not automated. Cloud autoscaling is.

Respectfully, the hell you say. Cloud 'autoscaling' is something that has to be tenderly maintained by software engineers, who expect to be paid salaries and benefits and so on. It's not like FedEx can just rsync their data into the cloud and have all their software run forever. Instead, they need the engineering team that manages their existing data workflows, and now they also need cloud engineers to translate that into something that won't bankrupt the company, since they're moving from the mainframe world (where you keep your system loaded to an efficient price point) to the cloud world (where you are billed by the second).

Moving your compute spend from capex to opex is a perfectly valid move, but pretending the reason is that someone has to make a phone call once in a while is kind of bizarre, when the alternative is having to hire a whole new cadre of techincal talent.

dx03416• 4 years ago

I'm confused about the first data center in 2008. Surely they had data centers before that, or were they outsourced before? Automatic routing of deliveries has been the standard for many decades, I'm sure the code on their mainframes is much older than just 14 years.

bob10292• 4 years ago

Or is everyone here caught up in the "Mainframe old" falsehood.

It's probably this. I've spent time trying to advocate for the benefits of these systems, but its like shooting a super soaker into the sun.

Spivak2• 4 years ago

I think it’s also both the people making the tech decisions and the people implementing them simply don’t want to work on mainframes. I would 100% of the time prefer to work with an on-prem dc or colo using commodity hardware and OSS to mange it.

I get my pick of a billion different hardware vendors, they’re all interchangeable and interoperable, every problem I can possibly dream up has been solved a hundred times before. The skills are commonplace and transferable so hiring doesn’t mean convincing some poor desperate grad to get trained in the company script, hires will actually want the skills, and you can find senior people.

chmod6002• 4 years ago

Can you elaborate a little? I'll admit I know close to nothing about mainframes.

hinkley4• 4 years ago

A lot of the cool tech that Linux and parts of cloud tech are crowing about are just new versions of stuff IBM was doing in the 1990’s. Sometimes I wish I knew some of those people, because I’m sure it would be fascinating to wind them up over a few beers and see what comes out of their mouths.

dan_quixote• 4 years ago

Its weird to have companies say they're moving from Mainframes to the Cloud.

Ever work for a company with their own datacenters that were NOT cutting edge? The work to provision resources can be so painful that it severely inhibits prototyping. Being able to spin up resources in seconds with some API calls is very valuable.

dx03416• 4 years ago

No one says you can't have both. An on-prem data center for reliable workloads and cloud to try out new stuff and products or to get short-term capacity. But in the end, 95%+ of servers in your data center will run 24/7 and for years on the same software. No need to outsource that.

alfalfasprout• 4 years ago

The fundamental issue with mainframes is that IBM made ramping up talent on mainframes extremely painful. Sure they run Ubuntu but how does one actually get a mainframe environment to learn on?

Mainframes are still king when it comes to transaction processing but having such a closed off ecosystem has screwed them.

Even FPGA's are more accessible to learn. On AWS you can spin up eg; an F1 instance and get a dev environment.

Aperocky2• 4 years ago

you call IBM

you don't need to call AWS. Negotiate a blanket deal and do whatever you need. Run whatever you want. Unsure? Check billing page.

bosveld• 4 years ago

Another way of looking at at: Where own data-center cost gets to high it might be due to lack of responsibility to act or inability to manage.

A company then have two options: (a) replace/pressure top/middle management to get costs down and processes streamlined or, (b) move to the cloud. Its easier (and possibly good for your resume) to move to the cloud and then a few yours later -- decide again.

So moving to the cloud is sometimes an indicator of an inflexible/stale/mature company.

jchook• 4 years ago

Interesting story as I recall Bank of America saving 5x that amount per year by doing the exact opposite move.

https://www.businessinsider.com/bank-of-americas-350-million...

Spivak2• 4 years ago

Doesn’t matter what you actually do, you get the benefits of shaking off all the barnacles doing any rewrite or rearchitectire.

jeroenhd• 4 years ago

The difference may be that they're sticking with their own data center but they don't have a mainframe setup. Mainframes come with expensive, locked out hardware, expensive maintenance, expensive experts, you name it.

With modern open source tooling, ridiculously powerful graphics cards, easily accessible FPGAs and free data center hosting software like OpenStack, Kubernetes, and glusterfs you can replicate many of the features that made mainframes enticing many years ago.

It'll require a heck of a lot of work to get the same performance out of a custom built solution, but long term it's probably cheaper than relying on IBM.

I doubt that they'll be saving any money by moving to the cloud unless they're horribly inefficient with their contracts right now. Just moving away from mainframes alone should save a significant buck, but if they were planning on restructuring their digital infrastructure anyway then now is probably the best time.

mbesto12• 4 years ago

This is 100% reasonable and plausible. It's perfectly acceptable that one F500 company can see savings by using cloud and others see increased costs.

dx03416• 4 years ago

That's how you do it. You don't stick with on-prem but just build your own cloud. At that size, AWS won't really buy hardware much cheaper than you. And while you can't build every cloud product that AWS has, the most popular ones are open source anyway.

fsociety• 4 years ago

The thing I see missing in the comments is that sourcing hardware over the past few years has been incredibly difficult. It will get better, yes, but that may have been the last straw to convince them to switch.

bradfa2• 4 years ago

Past few years? Maybe the past 18 months it's been slightly more difficult but even at the beginning of COVID it wasn't hard to source most anything. And even during these more recent supply-constrained times, you can still source hardware it just might take a little longer for delivery or you might pay a bit more than you did before.

tgflynn2• 4 years ago

I wonder how much of that savings comes from closing data centers and how much from getting off IBM mainframes. It's hard for me to see why a company like FedEx would need to be spending anything like $400m a year on data centers unless a big chunk of that is paying IBM to rent their mainframes.

tgflynn2• 4 years ago

It would be interesting if those down-voting my comment would explain where my intuition is wrong. FedEx delivers 12 million packages per day. If each package requires 10 transactions that works out to about 40k transactions per second. You should be able to handle that with 1 IBM mainframe or much less than 1000 servers and I don't believe either of those would approach $400 million per year to operate.

sbf501• 4 years ago

I'm surprised at the amount of vitriol being spewed at FedEx in this post for their decision, mostly about the doom-and-gloom that awaits FedEx.

Either there are a lot of Fortune-500 CTO's on Hacker News that know better than FedEx, or there is a deep-seated fear of the cloud, specifically PaaS.

simonw• 4 years ago

"FedEx first opened an on-premise data center at 250 Spectrum Loop in Colorado Springs, Colorado, in 2008" - what were they doing before that?

epberry• 4 years ago

Surprising to see a company claim massive cost savings for moving _off_ the cloud.

But perhaps they're talking about engineering velocity, reliability, etc. Of course everyone knows Dropbox's story with this: https://www.datacenterknowledge.com/manage/dropbox-s-reverse...

dend2• 4 years ago

This makes a lot of sense if you think through the angle that their core value prop does not come from the network/data latency optimization. It's not Netflix and Dropbox - this is FedEx. Their core business value proposition comes from logistics and being able to react to the information quickly, but extra millisecond latency is unlikely to make or break the business.

In light of this, it's easier to outsource the network and data infrastructure to a provider that actually does it day in and day out and put the limited engineering resources to something that is more impactful.

Just reaffirms that quite a few companies do not want to run their own servers unless they absolutely have to.

peteradio• 4 years ago

through the angle that their core value prop

Surely this is just one ingredient to a successful company and there are myriad factors involved. The core business of Boeing isn't to make planes its to sell them.

dend2• 4 years ago

That's the wrong analogy. In your example, the former is required for the latter. FedEx wants network stability and a data store, most likely, but the success of their business does not depend on them hosting their own infrastructure.

imperialdrive• 4 years ago

https://www.switch.com/colocation/ The Switch colos all appear to be rendered concepts, but impressively so.

dx03416• 4 years ago

I always wondered what latency requirements they must have in a distribution center that has to route packages dynamically and make those routing decisions within milliseconds. Surely there must be a lot of logic at the distribution center, they probably are their own data centers. Or can you really have 10+ms latency for each decision?

winddude• 4 years ago

Too bad it's not sooner, could use a flood of decent servers on the second hand market.

fuzzfactor• 4 years ago

Along with the data,

They. Own. The. Data center(s).

Well, in some sense or another. They might not be completely paid for but you get the idea, there's got to be a certain amount of worthwhile assets that could be leveraged.

Too bad there's not any business executives who have any experience at making money from a data center itself.

formerkrogemp• 4 years ago

From an managerial accounting perspective it makes sense. Reduce cost centers and focus on core competencies. But many things make more sense from a short-term, myopic, improve this quarter accounting sense that are actually bad ideas.

exabrial2• 4 years ago

"save"

In the very short term. Data centers are expensive as hell to operate, but so is the cloud. I think colo would have been the way to go.

intrasight2• 4 years ago

Most financial analysis are companies are short-term.

But labor costs keep going up and cloud computing costs keep going down.

So I don't see how this is short-termism.

hinkley4• 4 years ago

cloud computing costs keep going down.

For now.

I also have to keep reminding my coworkers that AWS gives us faster cores every few years but so far they always have the exact same amount of memory. So caching things (especially in process, but also on loop back) to save computation isn’t really a winning long term strategy. It’s never going to get better than what it does for you in the beginning. It will only decline in value.

exabrial2• 4 years ago

I see most places raising their prices as seed funding goes out.

wvh• 4 years ago

Here we are, meticulously crafting open-source programming languages, libraries, operating and distributed systems, only for people to squander the knowledge and freedom on closed ecosystems like Apple's and Google's, and, in the data-center world, the top three cloud behemoths. Assuming typical lock-in, this won't save anything in the long run, nor help with resilience and freedom of choice in the market.

Shorel• 4 years ago

Not saving, just giving those $400m to AWS or Azure, over some amount of time =)

citizenpaul3• 4 years ago

Long enough for this exec team to "prove" on paper they saved money during their tenure. Than hit the road and the next exec team to move it back because 'ooops'

jordemort• 4 years ago

Is there a website where I can bet on the outcome of this?

I've seen lots of companies announce their intentions to move off of their mainframes, but I'm not sure I've ever actually seen one pull it off.

schmeckleberg• 4 years ago

I think the thing you're looking for is a "prediction market". https://en.wikipedia.org/wiki/Prediction_market

Related to that, one funny thing I remember from Robert Heinlein's "The Moon Is A Harsh Mistress" is that the Lunie protagonist didn't understand why Terrans bothered with insurance companies. Are there no bookies?

rzz3• 4 years ago

It’s kindof shocking to see it take so long to move away from mainframes. I understand it a bit with regard to banking specifically, but I’m shocked to learn FedEx is such a dinosaur.

sithadmin2• 4 years ago

Guessing they're not counting stuff at their flight sim training center (12 bays) as a 'data center'. There is zero % chance that the workloads supporting simulators are gonna run in the cloud. They're incredibly latency sensitive, extremely finicky, and just getting the stuff virtualized continues to be a struggle for aviation OEMs and flight training centers (though FedEx was a pioneer in this regard).

Granted, the footprint per sim bay isn't huge (1-2 racks, usually, sometimes 3), but it can add up at the larger sim facilities. Several of the major airlines are running 30+ bays simultaneously, with United poised to come in with the most at 40+ in the near future.

dx03416• 4 years ago

They have up to three racks for a flight simulator? I would've never thought they're that complex. Do they need that much computing power or is it other devices that are needed?

sithadmin2• 4 years ago

2/3rd's ish compute, 1/3 other stuff from what I've seen. A good chunk of the compute are appliance-type boxes supporting simulation of particular embedded systems in a traditional environment. Most of the complexity comes from integration of these misc accessory systems.

In general aside from the boxes handling the graphics projection, most of the 'compute' is idle a lot of the time, even when the simulator is running. There's usually a big burst of work to prep and load the simulation into the simulator, then minimal (but very latency sensitive) chatter back and forth between the various simulator components and the backend compute as the simulation runs.

stjohnswarts2• 4 years ago

I think they will regret this in the long run when they have disagreements with their providers or their providers go bankrupt and fold overnight because of corporate mismanagement/thievery (ala Enron). Even Amazon and Google will go under eventually...

dx03416• 4 years ago

Or the amazon cloud becomes the new IBM mainframe and in 20 years they all work on getting off it, but can't because they are so deeply integrated.

stjohnswarts2• 4 years ago

it would be hilarious if IBM is the immortal one and eventually buy out a limping AWS and keep the software running for the next N decades.

Blackstrat• 4 years ago

Extremely shortsighted. But the cloud is still the big trend. It’s a “brilliant” move until it isn’t. And then the pain is monstrous.

qwertyuiop_• 4 years ago

This is because companies like AWS are able to perform Undifferentiated Heavy Lifting. There is no need for companies with undifferentiated workloads to keep investing to Capex that will depreciate as soon as they are stacked.

FedEx to close data centers, retire mainframes

Related Stories