Ask HN: What's behind all the UK IT failures this month?

https://news.ycombinator.com/item?id=39764874

by 0898 • 2 years ago

26 50 2 years ago

news.ycombinator.com

This month we've had McDonald's, Sainsbury's and now Greggs forced to shut temporarily because their payment systems have gone down.

Surely these can't be unconnected? Can anyone shed any light on what's happening here behind the scenes?

jjgreen2• 2 years ago

McDonalds, Sainsbury, Tesco, barely matters. But Greggs, that's some serious shit depriving people of pies. We need answers.

0898• 2 years ago

Is Putin hitting us where it hurts?

gertrunde• 2 years ago

Quite probably.

This could be an attack on national security, after all, Greggs sausage rolls are the fuel that runs the UK police forces, much like donuts in the US ;)

fifteen1506• 2 years ago

Please watch "The Undeclared War", on UK-Russia Cyberwar [fictional TV series] so the 2nd season gets produced.

gadders2• 2 years ago

The south is organising food drops to the North of England as we speak.

jjgreen2• 2 years ago

Poncey southern bastards. Still, pies come before pride ...

dsattt• 2 years ago

Probably leetcode.

i_have_an_idea• 2 years ago

Random chance and years of underinvestment and tech debt

abulman• 2 years ago

Maybe all the interns that were left in charge didn't properly do what the AI had said?

InsomniacL• 2 years ago

more likely all the interns that were left in charge DID properly do what the AI had said...

jesterson• 2 years ago

Most plausible explanation of all

Joeboy• 2 years ago

Maybe the dog[0] failed?

[0] https://quoteinvestigator.com/2022/01/30/future-factory/

usui• 2 years ago

McDonald's wasn't unique to the UK as it was global. McDonald's Japan was affected.

password54321• 2 years ago

Work is getting outsourced to India. Most CS grads don’t know how a computer works. People get hired for reasons other than merit. And ChatGPT…

smcl2• 2 years ago

Only 3 comments into a story about a UK problem and here's a comment blaming it on foreigners. Incredible.

soco• 2 years ago

I take the spirit of that comment that they didn't blame the foreigners, they blamed the outsourcing. Because even if the provider was local the result would have likely been similar (and we do have recent examples, Horizon and not only)

bilekas2• 2 years ago

A great quote I love.

Being British is about driving a German car to an Irish pub for a Belgian beer, then traveling home, grabbing an Indian curry or a Turkish kebab on the way, to sit on Swedish furniture and watch American shows on a Japanese TV.

All while being suspicious of foreigners.

dkdbejwi383• 2 years ago

I worked for one of the affected companies as a rare example of a permanent, in-house engineer.

They had a _lot_ of contractors, some independent based in the UK, some contracts with companies with a local presence who would work from the office alongside us, and some in India.

All were equally shit at worst, with only the independent contractors proving the exception on occasion.

I recall being stuck with “senior” colleagues who couldn’t manage to upgrade a dependency even while being handheld.

ozzcer• 2 years ago

Considering chat GPT is a few years old it seems unlikely those who used it in university and are now probably just Junior Developers are responsible for wide spread outages at major companies

jl6• 2 years ago

They could definitely all be unconnected. It’s not that suspicious. I guess we’ll find out if they ever publish post mortems.

ksec• 2 years ago

McDonald was worldwide. Sainsbury, Tesco and Greggs seems to be related somehow? At least they are all payment related.

buggeryorkshire2• 2 years ago

Just Eat went down Friday, too...

thinkingemote• 2 years ago

Sainsburys said it was due to an error with "an overnight software update". Perhaps they all use the same software but updated at different times?

https://twitter.com/sainsburys/status/1768972295622553900

nonrandomstring2• 2 years ago

Politicisation of technology.

It's a layer 8 problem. Human political expectations of technology are outpacing the engineering reality. Over the past two decades overselling of "utopias" mixed with professional management arrogance and ignoring engineers, experts and what people actually want.

That, plus a mixture of crony contracts and bad project management.

We've ended up with over-complex systems that we don't have the human capacity or money to maintain and secure.

A telling remark by a Labour politician to the question "What is the greatest concern about a 'cashless economy' was "Making sure people are not left behind".

In other words, we're forcing this on people whether they want it or not, and whether it works or not. The idea that there might be "risks" or necessary safeguards was simply not conceivable.

That's what happened with the Post Office Horizon system and it's what's happening with other new systems pushed out "for our own good"

That's not a remark against technology or modernisation, it's a criticism of bloody-minded recklessness and anti-democratic hubris.

Good technology requires care. We talk about it a lot here [0].

[0] https://cybershow.uk

bilekas2• 2 years ago

Politicisation of technology. It's a layer 8 problem. Human political expectations of technology are outpacing the engineering reality. Over the past two decades overselling of "utopias" mixed with professional management arrogance and ignoring engineers, experts and what people actually want.
That, plus a mixture of crony contracts and bad project management.
We've ended up with over-complex systems that we don't have the human capacity or money to maintain and secure.

That is a LOT of VERY sweeping statements to a pretty specific question it seems of payment processors, can you give some examples of what you're talking about ?

nonrandomstring2• 2 years ago

can you give some examples of what you're talking about ?

PLease see the podcast linked above where we have spent many hours researching, analysing, writing and presenting on specific cases. In episodes to come we're working on joining the dots, finding common factors in these cases. Enjoy, and please give feedback if you feel able.

damvigilante• 2 years ago

Mandatory cert rotation forced by a dependency that had a leak, and they all didn’t handle it properly. (Just a guess)

DarkNova6• 2 years ago

All of these enterprises are examples where IT is a "cost center", not a "value center". So you have this conundrum where everything goes smooth and the higher ups ask "why do we spend so much money on this without anything in return?". And when something does go wrong they are likely to repeat said question.

phtrivier• 2 years ago

Do you have sources about the "outages" ? (did not make the headlines internationally..)

It they all shut down because of "a payment system", then a simple explanation would be a failure of... the same payment system that they all happen to use ?

(It's not like Greggs is going to fully develop a payment system in house without relying on any infra, I suppose ?)

oakesm9• 2 years ago

Sainsbury's and Tesco: https://www.theguardian.com/business/2024/mar/16/sainsburys-...

Greggs: https://www.theguardian.com/business/2024/mar/20/greggs-shop...

The Sainsbury's and Tesco ones happened on the same day (last Saturday), but the scale was very different. Sainsbury's were unable to take card payments nationwide. My local one said it was due to the internet connection in their store not working. It also affected planned deliveries, likely because the scanners they use to fetch and pick the orders didn't have a connection.

Sainsbury's blamed this on a botched "software update".

Greggs happened today and was card payment related but doesn't seem to be nationwide (I've not been to one today so not 100% sure if that's true).

Ask HN: What's behind all the UK IT failures this month?

Related Stories