Kraken Goes Down for 40 Hours, Drawing Mt. Gox Comparisons

http://fortune.com/2018/01/12/bitcoin-price-ripple-kraken-down/

by yonilevy • 9 years ago

84 33 9 years ago

fortune.com

verroq2• 9 years ago

Pretty irresponsible to completely brick production servers when doing an upgrade.

https://status.kraken.com

Sounds like their staff has being going without sleep for a few days now.

Shank• 9 years ago

Sounds like their staff has being going without sleep for a few days now.

Given that sleep deprivation has effects similar to being blackout drunk, I'd consider that equally irresponsible[0].

[0] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1739867/

switch007• 9 years ago

I've lost count the number of times "watched the sunrise" was the climax of an IT war story. Bonus points for two nights in a row ! :D

tzakrajs• 9 years ago

I can't help but feel less stressed about my own job after reading this status page.

bufferoverflow2• 9 years ago

It's not an upgrade, it's a bugfix. Probably an important bug, they don't want to risk tens/hundreds of millions of dollars.

verroq2• 9 years ago

They could have left the legacy system running before switching irreversibly to the new bugged system. That way they can rollback the change.

hal9000xp2• 9 years ago

Latest Kraken update says that:

We are still working to track down an elusive bug which is holding up launch. This bug did not appear in our many weeks of testing and only emerged in the production environment. Unfortunately, it is not consistently reproducible and we cannot launch until this issue is resolved.

Sounds pretty realistic to me. During my 8+ years of programming, I had a lot of pain and sleepless nights trying to catch elusive bugs which can be reproduced only in production servers and only sometimes. Many times, my clueless managers got nervous and demanded estimated time from me. Unfortunately, it's impossible to give realistic estimations. It could take hours, days or weeks. Also, searching for such bugs doesn't scale well. It means if you have 5 devs looking for elusive bug, it doesn't mean they will find a bug 5 times faster.

koonsolo4• 9 years ago

I even had clueless managers who demanded I explained the technically complex problem to them, because "maybe our insights could shed new light on the problem". They just don't get that best thing to do is leave us alone and let us do our job. They are just nervous because they are totally powerless in such situation.

In a factory, a manager might be smarter than the one doing manual labour. In tech, it's the other way around.

setr2• 9 years ago

If you assume potential malice (ie a lazy dev just not handling the bug at all) then a manager is required to ask about the bug, if only to try to verify that you've put real effort into the problem (based on say, confidence in whatever you spew). The worst possible scenario is that five months later, it turns out no one actually looked into properly: they just claimed to.

pdpi2• 9 years ago

Right idea, wrong reason. You have a tonne of assumptions about how the product works that I don’t. If the bug is that subtle, dollars to Donuts says that one of your assumptions is wrong, but you just haven’t thought to question it yet.

When you explain the issue to me, I’ll ask dumb questions about context I lack, so you’ll explain stuff you take for granted. And it’s answering those questions that will make you figure out where the problem is.

swalsh• 9 years ago

And this is why Revert first, continuous deployment with feature toggles is the most stable way to run production. Deploy the new stuff, add some feature toggles. If the last version is stable, and your code is easy to revert, then you don't have to worry about downtime. If you're doing a major deploy, you can use the feature toggle to slowly turn it on (or off) and even use it test your new code directly in production (the systems I've used in the past allowed a developer to specify a feature toggle to be on for just their session). Databases can be tricky to toggle, but you have to build that into your design.

hrpnk2• 9 years ago

They're back with free trading until end of January: https://blog.kraken.com/post/1449/kraken-returns-with-free-t...

jnordwick• 9 years ago

Down again. Amateur hour. Lol.

Bombthecat• 9 years ago

Still no withdraw..

azzhohle• 9 years ago

Kraken is notoriously slow, unresponsive and buggy exchange and frankly I dont know why they get away with it. For months or year(s) you often run into site unavailable errors (load balancing issues?). Or if you are lucky to get to the trading screen you are unable to get trades through because of engine issues (capacity issues again?). Sometimes all you see is an empty red background rectangle where the error text should be indicating no dice but then minutes/hours later you might get one or more of the positions you tried to open despite only having gotten (ambiguous) errors.

Let's hope they finally fixed their engine and purchased more capacity.

valdiorn• 9 years ago

Upgrade a 750 million dollar a day trading system with no rollback strategy.

Amateur hour, and if this were a "real" exchange, they'd lose their licence and have lawsuits and government agents knocking down their door (real-money MTFs have strict uptime and infrastructure requirements for anyone who didn't know)

arisAlexis• 9 years ago

everything is up and trading is free for the rest of the month. </drama>

lowlevel• 9 years ago

They say the next big thing is here, that the revolution's near, but to me it seems quite clear that it's all just a little bit of history repeating

rootsudo• 9 years ago

IT sounds like this week is going to be profitable on Kraken if you trust their status reports.

hndamien• 9 years ago

I tend to trust them for security because they are handling the Gox bankruptcy. For uptime... not so much.

Kraken Goes Down for 40 Hours, Drawing Mt. Gox Comparisons

Related Stories