LSE Halts

http://www.londonstockexchange.com/exchange/news/sharecast/news-detail.html?newsId=4072679

by ashishb4u • 15 years ago

101 69 15 years ago

http://online.wsj.com/article/SB10001424052748704150604576165753761876730.html?mod=googlenews_wsj

famoreira2• 15 years ago

Does anyone know if this halt is related to the recent switch over to the new Linux based platform? (http://news.ycombinator.com/item?id=686116)

drstrangevibes5• 15 years ago

it is a new platform, hard to tell but I wouldnt go so far as to say linux is to blame, its more likely to be exchange software, they've gone from native to fix 5, so its using a new protocol, new os, new software stack and most likely new hardware stack

tallanvor2• 15 years ago

I don't think famoreira was trying to blame Linux. --That's just the part of the new system that you can reference easily.

IDisposableHero9• 15 years ago

I don't think that "linux is to blame" or even that any choice of programming languages or other tools are to blame.

It's a large system with extra-ordinarily tight performance requirements, and a lot of integration with external systems, and by nature hard to phase in gradually. That kind of system is very hard to get right first time regardless of your toolset. I'd be interested to hear more about thier methods to deal with these requirements and risks, less about thier platform choice. IIRC, both this and the previous failed system were outsourced, which doesn't exactly fill me with confidence.

known• 15 years ago

I believe this is due to a glitch in the Application and not in Linux per se.

ashishb4u• 15 years ago

Nothing official yet. (http://www.ft.com/cms/s/0/9bf8033a-40bb-11e0-9a37-00144feabd...)

drstrangevibes5• 15 years ago

this is the official place for updates, although its not very responsive at the moment,probably due to load http://liveservice.londonstockexchangegroup.com/en/incidents...

pnathan• 15 years ago

It's been archived - here's the link to this incident in the archive:

http://liveservice.londonstockexchangegroup.com/en/incidents...

JonnieCache2• 15 years ago

I wonder if it's HFT related? A group of rogue algorithms gone exponential? There are certainly people worried about that...

tpatke• 15 years ago

Where is the fail over? If ever a system needed 99.9999...

tallanvor2• 15 years ago

Unfortunately sometimes fail over isn't a solution. Until they've at least narrowed down the problem, a fail over system may make things worse.

Personally, I'm curious about the testing that was done - given the complaints and problems that have been reported so far, did they even try to run the new and old systems simultaneously for any period of time to verify that the various components could all keep up properly? Or at the very least, did they try to test the system at full load for a decent period of time?

JonnieCache2• 15 years ago

Failover in this scenario would require instant, 100% replication of a near-zero latency system.

I'm no expert, but I'm fairly sure having trades 'disappear' or any other loss of consistency would be worse than downtime in this case.

mtkd2• 15 years ago

"And as a final note, solely for the seriously nerdy techies, there's been some talk about the failure being due to F# error."

http://ftalphaville.ft.com/marketslive/

timrobinson7• 15 years ago

I've no idea where they got this from, although I mentioned it on Twitter as a joke: https://twitter.com/tim_g_robinson/status/41086872876892160

drstrangevibes5• 15 years ago

F# on linux?

timrobinson7• 15 years ago

http://www.mono-project.com/Release_Notes_Mono_2.10#Language...

viraptor3• 15 years ago

Via mono. If that's true... that would be an interesting news for the project (not because of the bug itself of course...).

mtkd2• 15 years ago

I doubt they'd be using Mono - if it really is an F# issue it maybe indicates it's a failure in some of the legacy system they are replacing.

alecco10• 15 years ago

I still don't understand how they choose languages with garbage collection for systems that should be real time or similar.

timrobinson7• 15 years ago

It's not F# or Java, it's C++. http://news.ycombinator.com/item?id=2261896

axod• 15 years ago

GC isn't a problem for realtime systems, if configured correctly. For example, have the GC running incrementally in a separate thread.

davidw• 15 years ago

Erlang is not "real time" in the hard, embedded systems sense, but it does pretty well and has GC.

Here's a bit more information: http://www.erlang.org/faq/academic.html#id55755

I don't know what sorts of 'real time' properties trading systems are required to have - can anyone talk about it with a bit more in-depth knowledge?

scott_s• 15 years ago

You may be interested in the Metronome project, which aims to provide a GC suitable for realtime use: http://domino.research.ibm.com/comm/research_projects.nsf/pa...

An academic paper about the project, Design and Implemention of a Comprehensive Real-Time Java Virtual Machine: http://domino.research.ibm.com/comm/research_people.nsf/page...

chollida15• 15 years ago

that link doesn't contain the text 'F#' any where on the page.

timrobinson7• 15 years ago

It's been archived to http://ftalphaville.ft.com/blog/2011/02/25/498596/markets-li...

alecco10• 15 years ago

What if all these systems were open sourced? Including all the load tests.

viraptor3• 15 years ago

At this size, some obfuscation might be good. If you find a bug and crash some browser, it's not a big deal. If you find some problem in LSE that can be triggered from outside, you're dealing with realworld $M transactions.

Also, unless you have a huge farm of servers it would be useless to you - and basically anyone really.

rbanffy2• 15 years ago

> What if all these systems were open sourced?

LSE considers its software to be a competitive advantage.

drstrangevibes5• 15 years ago

.....er sure

ig13• 15 years ago

The problem is that the LSE can't hire top tier talent. There are plenty of people who are experts in building hugely concurrent and fail-safe exchange software, and they pretty much all work for investment banks.

The LSE can't afford to compete with $1000/day salaries so they end up hiring second-tier developers. And unfortunately second-tier developers are a lot lot worse than first-tier developers.

laktek• 15 years ago

I don't think it's about talent, but more of a problem of proper planning and execution. It seems things had been rushed to meet the tight demands, causing the chaos.

When LSE acquired MilleniumIT, to redo the platform, they claimed to have developed world fastest trading platform[1] and had enough talent under their belt developing trading systems around the globe for last decade.

[1] http://www.wallstreetandtech.com/exchanges/showArticle.jhtml...

ig13• 15 years ago

None of the exchanges have the required talent. Being the best exchange technology company is like being the best team in a minor league. If you're good at extreme low latency stuff you're going to be working for a bank.

At a bank being great at low-latency can mean tens of millions of dollars of profit for your trading desk and a huge bonus for your personally.

At an exchange being great a low-latency means ... well not much, your exchange gets a bit more flow and you get a pat on the back.

alecco10• 15 years ago

That's very likely. Any programmer with even an entry level position at the LSE would get head hunted by multiple agents in the first month. And the investment banks do pay £700/day for talent.

fendale• 15 years ago

I know at least two guys who are getting paid closer to £1000 a day to work in London for a bank. The money is pretty much insane imho, and the guys tells me the 'talent' is just like anywhere else - ie most people are meerly compentent (but still getting paid £500 plus a day). Even graduates with zero experience are taken on at £300 a day or more.

He seems to better than competent, and so has managed to negotiate a much higher rate.

alecco10• 15 years ago

Sri Lanka? http://uk.reuters.com/article/2009/09/16/uk-lse-it-sb-idUKTR... (via /B @ftalphaville comments)

timrobinson7• 15 years ago

"Continuous trading has now resumed in all order driven trading services."

LSE Halts

Related Stories