Silo but deadly: Messy IT systems are a neglected aspect of the financial crisis

http://www.economist.com/businessfinance/displaystory.cfm?story_id=15016132

by vgeecc • 17 years ago

33 19 17 years ago

xelfer2• 17 years ago

At the company I worked at a sprinkler system failed on top of 2 racks of servers about a year ago, they were drenched. We dried them out, insurance was called, we got paid whatever they were insured for.

Finally, a time to upgrade this mass of beige boxes and randomly built servers.. or so I thought.

Instead, the insurance money was put towards paying salaries.. the servers that got flooded are still in production a year later.. and 30 of us were laid off.

giardini• 17 years ago

I believe that is insurance fraud and is criminally prosecutable. The insurance company involved, if not the local district attorney,would be interested in hearing this story.

anamax• 17 years ago

Where's the insurance fraud. Are you saying that they didn't have a loss or that they didn't use the insurance money appropriately?

If the latter, you're wrong. You're free to use the insurance money any way that you'd like.

For example, the cost to repair a car back to its "original condition" is often much higher than its value even though the car is still driveable. In that case, the insured is free to take the money, buy the car for its current value, which is significantly less, and pocket the difference.

If your house burns down, you're not obligted to rebuild.

xelfer2• 17 years ago

I made that exact comment to them at the time, they claimed it wasn't fraud.

swombat• 17 years ago

I worked in banking IT for 4 years, and yes, the systems are awfully messy there.

However, I think it isn't fair to solely blame the banks for it. If there weren't mountains of regulations to comply with all over the place, the systems would be a lot simpler, clearer, and more transparent. The first place to start, to make banking IT more transparent, would be simplifying and aligning national and international regulations. Make one set of regulations for the whole world, and make it clear and simple.

Of course, that might not be a realistic aim. First of all, it would make the international banking system even more vulnerable to a single loophole in laws. Secondly, it probably wouldn't be possible to do this in a way that follows new developments in financial products closely enough. Thirdly, the simplification itself would result not in all the current systems being decommissioned (that might happen in 10 or 20 years at the earliest), but in all the current systems becoming legacy but still active systems while a whole new spate of systems is built.

Maybe the current compromise is the one that works best, despite its flaws.

dhyasama• 17 years ago

While I agree banking and other financial regulations are onerous (and usually tacked on in reaction to a disaster rather than before to help prevent it), I don't agree with transferring blame because of it. If you chose to engage in a system, financial or otherwise, than you tacitly accept it's rules and regulations. Creating and maintaining solid systems around complex requirements is where the real skill comes in.

riffer• 17 years ago

http://news.ycombinator.com/item?id=978295

synnik• 17 years ago

I did work in banking for a few years. The article is accurate, especially about the detective work involved in tracking down data problems.

But there is yet another underlying problem behind the IT systems. The leadership of the banks I've worked with considered IT just a "cost center". They recognized the problems with the systems, but rather than investing in improvements, they threw blame towards IT, approving budgets only grudgingly.

The leaders in the industry need to start to respect technology and the people behind it before anything will change.

stcredzero• 17 years ago

Happenings at a multinational petroleum firm:

One process writes something to a database table. Another periodically queries it out of a table. This results in an email with a text attachment being generated. The text attachment is then retrieved from the inbox and FTP'd to another server, where a polling process retrieves it, does some processing, and ends up saving something into a database table.

I am not making this up, one distributed system was really put together like this. What's more, this Rube Goldberg monstrosity broke all the time, and when it got fixed, the techie doing the fix was lauded, since he was ensuring this link, which was vitally important to the business, continued to work.

ivenkys• 17 years ago

Data Problems in Banks are especially acute because the same "thing" is called by different names in different systems, no one can agree on its definition or its characteristics.

Added to that, the amount of time it takes for I.T to come up with a solution - due to the silo mentality - the Front-Office comes up with a patchwork mostly Excel-based solution "just for now" and well we all know how that goes.

SandB0x• 17 years ago

I've worked in financial software - this is spot on. It seems to be one of those unfortunate industries where nobody's interests are aligned with designing decent software:

Programmers don't use their software, users (often back office admin staff) don't have a say in the design, and the majority of programmer effort is spent on firefighting.

Due to the heavy regulation you can't really get in there with a lean, easy to use product. Even if you could, the data is often stored in impermeable proprietary mainframe systems and is garbled to the extent that importing and exporting data is handled by an army of consultants.

There's an enormous fortune awaiting companies who can create intuitive, well designed software for financial corporations. I, however, left because there more interesting things I'd rather be doing, and I guess most programmers feel the same.

coolnewtoy• 17 years ago

One of the obstacles is security. Programmers don't have the rights to even see their software in production environments. Designers and testers have access only to small (often not representative) subsets of data, and it's different subsets than the users see. Moving data from one system to another isn't easy when you don't have anyone who has enough access to look at the data in both systems.

yummyfajitas2• 17 years ago

Interestingly, I'm told one reason Goldman survived the crisis and came out on top is that their IT is excellent [1]. They invented their own dataflow programming language, which they use in most of their trading systems.

When the crisis hit, various higher ups were able to see their risk profile, run various scenarios, all in a matter of hours. I.e., the CEO could say "what if X happens", run the global system, and he would see the combined impact of X across ALL divisions.

In contrast, Citigroup simply does not have access to that information.

[1] Another reason is that their focus on short term profits meant they were not holding many bad assets at the wrong time (most mortgage products have long maturity times).

bwhite• 17 years ago

Goldman does have excellent IT vs other big i-banks. (Well, now they're all bank holding companies since they wanted (or were compelled to want, depending on who is telling the story) to gain access to the Fed's discount window.)

But slang, their proprietary language, has a pretty mediocre reputation. Every strat I know there hates it; anecdotal, I know, but still. It is old -- in use for at least 15 years as far as I know -- and cripplingly slow both to develop in and run. From what I hear, where speed matters a fair bit of C++ and even some Erlang (!) are in production use.

Goldman is doing well because it is full of bright people who are very good at their jobs. Last spring/summer/fall, however, Goldman had lost vast piles of money, particularly in their Global Alpha fund. It is true that Goldman had started to go short on housing as early as late 2007; this is why they didn't get pummeled as badly as their big-bank cohorts.[1] The nice rebound is due to a variety of factors, not the least of which is that part about bright people who are good at their jobs, plus a general market rise, plus govt rent seeking (you better believe it), etc.

[1] The mortgage-related products whose implosion wiped out the market and many players are all short-term vehicles. Indeed, the problem was specifically that mortgages were not treated like old school bonds with a long maturity, but that the income stream from these mortgages was sliced and diced into a wide variety of CDOs with a dizzying number of tranches such that they could be securitized and treated as virtually short-term instruments.

yummyfajitas2• 17 years ago

My understanding is that while slang the language sucks (where "sucks" sometimes means "is not C++" and sometimes is actually meaningful), slang the dataflow system is fantastic for the organization.

The dataflow system allows higher ups to simulate possible outputs and measure risks easily. I.e., you tweak the LIBOR, run each desk's slang code to see how it will affect them. Then you combine the results and see what happens firmwide.

In contrast, at most other places, you'd need to ask each individual MD to tweak the LIBOR in whatever spreadsheets or C++ models they are using, write a report, have your underlings combine the reports into their own spreadsheet, summarize, and explain to you.

CrLf• 17 years ago

This whole discussion just looks to me as IT everywhere, not just only on the financial "industry".

chubbard• 17 years ago

So I found this article very much with the way I see banks and financial systems. Being that I never worked on financial IT system this was just my hunch. But, hearing accounts of what was going on it sounded like people were making decisions on very little data, or that the data wasn't available to them. When the crisis hit I kept thinking over and over that this could all have been avoided if things were more transparent. The SEC is at best over burdened and probably more likely gamed. The oversight we have doesn't work, and every time we talk about reform bankers moan and complain you'll ruin everything. I call bullshit.

Given the proper financial data anyone could evaluate the risk of a company. It's not rocket science. It's all a bunch of electronic data. We need to rethink regulation. We need to transitions from auditors, committees and the SEC to financial transparency. We should be able to see every trade, every position, every loan, CDOs, and derivatives a company owns, trades, etc. Everything can be tracked and examined because it's just a bunch of electrons. If this sort of system existed it would be virtually free to conduct an audit, or monitor for manipulation. It benefits both the CEO and SEC. CEO's have better insight into what's going on inside their companies, and the SEC can make sure they aren't manipulating markets or doing risky things at any moment. Instead of sending out a fleet of auditors, relying on Sarbox, etc.

There's no excuse for bad loans being made. I kept hearing of loan evaluators being told to pass bad loans just because Wallstreet wanted to securitize them. That's ridiculous, and if they were doing that we should be able to see it in plain view. Loan packages can be graded by a computer, the only thing a human should be doing is checking for fraud at the most. And remember the bang up job Moody's did rating the pile of junk. Their failings seemed like just total lack of knowledge about what they were doing.

Which got me thinking why can't we automate the stress test on banks? If we had such a system then we could take all of the financial data from every institution into one big simulation. Simulate future financial products and their impact before we create them. Test out what could happen in the market if we introduce this. Play out what if scenarios. If we could do that then would we need an army of auditors, and more overhead the banks don't want. Light weight bureaucracy not over burdened with check marks, auditors, committees, and SEC filings.

johnohara• 17 years ago

I read this article 2 days ago on a flight from CA to Chicago. Sounded completely plausible.

All I could think was "if you think this is bad, wait until you see what's under the hood in healthcare management and healthcare reimbursement."

Silo but deadly: Messy IT systems are a neglected aspect of the financial crisis

Related Stories