3rd largest Bitcoin exchange lost its wallet.dat

by mef • 15 years ago

117 98 15 years ago

mef• 15 years ago

Looks like the admin stored the wallet.dat and backups on an ec2 instance, and then shut it down while doing a RAM upgrade without realizing he would lose everything on the disk.

Translation of post:

DECLARATION I hereby inform all users of the service www.bitomat.pl of system failure that occurred on 26 July 2011 and its consequences. At the outset I would like to apologize for such a long delay in publication of this statement. I explained that it was dictated only by the good service investigation conducted to determine the causes of failure and the people responsible for it. Unfortunately, to date, despite intensive efforts, could not determine these issues. However, I believe that the longer pause to disclose this communication at this stage would be unreasonable. I am also aware that service users are the appropriate explanation. On 26 July 2011 at about 23:00, I noticed that absorbs all Bitcoin server machine resources, and probably not used for making. So there was need to increase the amount of RAM in the server. As a result of that procedure - suddenly the whole virtual machine has been erased, all data stored on the server has been lost!, Including records concerning bitcoinowego portfolio and its backups (backups). I have taken action have established that the disappearance of the data was the result of the introduction of virtual server settings, which he never would have introduced. Amazon Web Services Company, which is located servers, website says that the machine that has been cleared has been set up in such a way as to be irretrievably destroyed automatically with the data on disks attached to it at the time of her arrest by the shutdown. We are constantly trying to determine who made changes to these settings and whether it will be able to recover lost data. Unfortunately partnership with Amazon Web Services, which was placed servers service is difficult. Once I realized that I deleted the machine have redeemed the biggest package of technical support, I talked to the manager, asked about the security of disk space, I explained, so far unsuccessfully. Still exerts pressure on the Amazon Web Services to accelerate their activities but without concrete results. At the moment I am unable to clearly determine the causes of crashes, I suppose that it is the result of actions of third parties, which are causing the server tried to cancel to hide their illegal activities, or intentionally wanted to website disappeared. If my suppositions are confirmed, the fact will let police and prosecutors. At the same time if possible take action through which it would be possible to recreate lost data. But what I need to interact with the server's owner, and that as I mentioned above is difficult. At this point I wish to inform and assure you that your cash deposited into your bank account service and not converted into BTC and unpaid cash from the sale of BTC remain safe and intact. Any further findings will you keep. At the same time I am counting on your help in solving the problem. I realize that the situation is very difficult, and you fear for the fate of their BTC. We are constantly working on a solution to the crisis, and I'm open to your suggestions. Currently going to: cancel all active orders so far, to restore service to operate to allow the performance of any operation (in particular the payment of PLN). Please your suggestions and ideas. I wish to inform you that I had several conversations with potential investors from home and abroad. Www.bitomat.pl service is on sale for EUR 17,000 BTC. If interested, please contact us at bartek@szabat.com. Best regards Bartek Shabbat Service Administrator www.bitomat.pl

StavrosK2• 15 years ago

> Www.bitomat.pl service is on sale for EUR 17,000 BTC

What?

WarDekar2• 15 years ago

That was my reaction as well. Not just the fact that it's ambiguous whether they're trying to sell for 17k EUR or 17k BTC, but the fact that they're even trying to sell it at all, and then the buyer is presumably on the hook for the BTC that were lost as well!

I currently had ~13 BTC in my Bitomat account- when I login they correctly show my balance, so I'm slightly confused as apparently their database wasn't lost, but their wallet data was? How does that happen? And why was the wallet data not backed up on a non-EC2 instance?

All of this is making me lose some faith in the exchanges- between the MtGox BS and this, is TradeHill the only exchange that can be trusted? Can they even be trusted? It seems they are by far the most competent from a technical perspective, but at this point who wants to trust any of these exchanges with any significant amount of money?

tshtf• 15 years ago

1. How do you "increase the amount of RAM" on an EC2 instance other than spinning up a new larger instance?

2. Why didn't they use EBS and take frequent snapshots to S3?

It sounds like they were running an instance store EC2, then fired up a larger instance and stopped the first one? This has to be a textbook case of how not to use EC2.

mh-2• 15 years ago

You can't. Instance types are fixed. He was likely using an EBS-root configured AMI, where the mount point for EBS is /. These are EBS-backed, but since the EBS block is automatically created when you spool up the instance; it is also automatically deleted when you terminate the instance, unless you specify otherwise.

jodrellblank• 15 years ago

Cynical translation of that: I didn't understand the setup and made a change which deleted everything. I haven't found anyone else to blame yet but I'm desperately trying.

There was need to increase the amount of RAM in the server. As a result of that procedure - suddenly all data stored on the server has been lost! [..] the disappearance of the data was the result of the introduction of virtual server settings, which he never would have introduced.

For Amazon persistent storage, you don't just change a setting, you need to pay extra for EBS storage and design your service to use it, so saying the following:

We are constantly trying to determine who made changes to these settings and whether it will be able to recover lost data.

Suggests a continuing lack of understanding, and that makes it seem more likely that the service was built on quicksand and is now sunk. Even if we assume that they did have EBS volumes available and someone had redesigned their service to not use them, that they have no access audits, no regular setup audits, no long term backups, no revision control history, no change procedure and no email records (and nobody who remembers anything about it) to give them a clue what happened, that's pretty damning in itself for a financial service.

Once I realized that I deleted the machine [..] At the moment I am unable to clearly determine the causes of crashes

Maybe an artefact in the translation, but you can't accept that you caused it while also being unable to determine the cause!

I suppose that it is the result of actions of third parties, which are causing the server tried to cancel to hide their illegal activities

If this is a serious admission that their Amazon account was hacked, the assurance that unpaid cash from the sale of bit coins is safe seems a bad idea, and hiding it at the bottom of another announcement is really poor form. If it's not a serious admission then it reeks of pointing the blame anywhere except 'here' and is also poor form.

Is anyone else reminded of Leafyhost about now?

nordsieck• 15 years ago

Am I the only one reminded of the Eve Investment Bank?

http://news.softpedia.com/news/Eve-Online-Economy-Suffers-70...

Does anyone know if they disclosed the address at which they stored their bit coins?

jpr• 15 years ago

There's a reason real banks use mainframes, not cloud services...

tghw2• 15 years ago

This is not why they use mainframes instead of the cloud. This is why they employ competent sysadmins.

otoburb• 15 years ago

Was this a relatively young BitCoin exchange? One would hope that the wallet.dat was backed up somewhere else (another EC2 instance, EBS, offline).

This doesn't bode well for establishing credibility in BitCoin exchanges and operators. Although nobody necessarily needs to use the exchanges, for better or worse, people use them to establish market values to more easily value their transactions.

It's obviously easy to criticize after the fact, but I am genuinely curious how backing up critical data wasn't considered. Unless, this was a very young / recently launched BitCoin exchange as a weekend hobby project that took off?

WarDekar2• 15 years ago

They've been around for a few months and were one of the higher volume exchanges. They were doing ~800 BTC in trades/day for the month prior to this, so ~$11k/day with their commission of .006*2 (I think it was .006, and on both sides) they were pulling in ~$130/day.

scotty795• 15 years ago

Everything (transactions, withdrawals, deposits) was free of charge.

Only way of profiting legally from the exchange was interest on the capital deposited by users on exchange owners bank account.

kiba6• 15 years ago

I guess the bitcoin economy is ruthless on its entrepreneurs and users.

Failure to backup, improper trust in individuals, improper understanding of the technology, poor security practice, and bad decisions will wipe out early adopters' wealth continuously and relentlessly.

I doubt we will see much of the early adopters retaining their bitcoin wealth from these early days. Those who did are either incredibly lucky, or incredibly competent.

I am an early adopter of bitcoin and I hope to survive or avoid the many dangers and poor decisions that ensnared many of peers on the road to bitcoin richness.

bcl3• 15 years ago

I think that's a bit over the top. It really isn't that hard to keep your coins in your own wallet and back it up.

What this demonstrates to me is that people are too trusting. You really shouldn't trust any of these services until they become more transparent in who is backing them, what their infrastructure looks like, their disaster plans, etc.

Duff2• 15 years ago

It isn't, but since the primary thing to do with Bitcoin is to speculate it, keeping it on your PC is problematic.

dedward• 15 years ago

Everyone keeping their bitcoins in exchanges/(think banks, or broker accounts) instead of their own wallets is completely subverting the point of the idea of bitcoin.

BTC has some great unique properties as a p2p cryptocurrency - which goes out the window when everything sits in a few exchanges rather than a bunch of wallets.

kiba6• 15 years ago

Find a way to easily trade bitcoin without centralizing wallets and you got the thousand bitcoin idea.

Next step is to turn the thousand bitcoin idea into an actual reality and get rich.

aw3c22• 15 years ago

I thought one can directly send bitcoin to another address without the need for a middleman?

timdorr• 15 years ago

You can always do that if you trust the person enough. The idea with exchanges is essentially to act as an escrow service, but one where the two parties don't have to directly know who each other is. I don't think you can decentralize trust, realistically. Someone ultimately has to be responsible for a bad transaction or handling of BTC.

rsynnott4• 15 years ago

Ah, but if the people running the _exchanges_ are too incompetent to safeguard their wallets, as in this case, how bad will the average user be?

omouse• 15 years ago

It's a free market, someone else will step in and create a more secure exchange.

dfc2• 15 years ago

What unique properties of a cryptocurrency are negated by a functional exchange?

One of the problems with ever realizing a "true free market" in bitcoin trading is the time spent waiting for block confirmations. There are barriers to entry in that you can not just transfer your coins to an exchange and sell instantly. Because of this a lot of people who trade bitcoins keep a certain amount in their exchange accounts.

scotty795• 15 years ago

Other thread about same subject here: http://news.ycombinator.com/item?id=2828091

jonknee5• 15 years ago

Sometimes you don't appreciate regulation until it's gone.

kiba6• 15 years ago

An exchange failed on its incompetency and now they are trying to liquidate and giving back their customers' money. The market works even if the consequence is quite sour and painful.

This will means that exchanges will tend toward competency rather than incompetency.

Regulations is something that you will have to design first or else you get regulatory capture or "TOO BIG TO FAIL" or some kind of unjust monopoly. Adding a regulator means that we will have to shift our focus in making sure that the regulators are doing its job rather than ruthlessly killing off companies that can't keep customers' money safe.

jonknee5• 15 years ago

Except with a lost wallet.dat there will be a lot of BTC that cannot be given back. Customers are likely going to take a haircut because an incompetent exchange lost their money. In the real world the people running the exchange would be going to jail.

bcl3• 15 years ago

I disagree. Regulation has given people a false sense of security, leading to them trusting these unknown and untested entities.

Instead of regulation what they need is more transparency. Tell your customers who you are, how you are building your service, what you will do in the case of a disaster.

In this particular case you have someone who is playing with the technology without having even a basic understanding of how to build the required infrastructure.

tghw2• 15 years ago

How do you ensure transparency without regulation?

jonknee5• 15 years ago

What's to prevent a firm from lying about who they are and what they are doing? Without regulations (and accompanying penalties for breaking the rules), it's the wild West. It sounds good on a libertarian check list, but in practice the people who can least afford it get absolutely destroyed.

sgornick• 15 years ago

> At this point I wish to inform and assure you that your cash deposited into your bank account service and not converted into BTC and unpaid cash from the sale of BTC remain safe and intact

The funds are likely held in aggregate in a single bank account under a single name. Without having the data showing which accounts hold which amounts, I wonder how those funds could be returned to the rightful owners.

sliverstorm3• 15 years ago

It sounds like they lost the bitcoins, not the database.

_delirium• 15 years ago

If this kind of thing keeps happening, isn't Bitcoin inherently deflationary? With gold, for example, only a small proportion is ever permanently "lost"; even shipwrecks can be recovered in the future. And of course more gold is always being dug up. But if bitcoins from a fixed set are slowly being lost in various ways, irreparably shrinking the money supply, it seems like it'd have trouble being a viable long-term currency.

wmf• 15 years ago

The 21M BTC limit already made it deflationary, so lost BTC just increases the rate slightly. The early adopters tend not to complain.

rsynnott4• 15 years ago

In practice, when used to back a currency, gold is usually also inherently deflationary; mining of new gold generally cannot keep up with real economic growth.

SkyMarshal2• 15 years ago

Yes, BTC is inherently deflationary, but one of the fascinating things about it is that since it can be infinitely subdivided, this may not be as much a problem as it for meatspace currencies.

Currently BTC is divisible down to 8 decimal places, enabling a maximum possible 2.1 quadrillion atomic units. But that is only an artifact of the data structure used in the current implementation, and could potentially be modified to allow even more granularity.

So, as the value of BTC rises against goods, services, and other currencies, the market can reprice in smaller and smaller increments of BTC, and the incentive to save/hoard that is typically associated with deflationary monetary systems may be weaker with BTC.

jahnu• 15 years ago

> the incentive to save/hoard that is typically associated with deflationary monetary systems may be weaker with BTC.

I don't see how that is possible. The coins may undergo a process similar to a stock split but any holders of the coins end up with equal purchasing power. The incentive to hoard does not go away if the currency is deflationary.

sliverstorm3• 15 years ago

On the bright side, if the wallet has truly been lost and not stolen, everybody else's BTC got just a bit more valuable.

In other news, funneling thousands of trades through one wallet.dat... how on earth do you successfully back that up when it has new data every minute? If exchanges are going to become a "thing", I'd think you'd want something like multiple redundant networked databases storing the BTC.

doublec3• 15 years ago

The individual trades in exchanges don't hit the wallet. Only deposits and withdrawal's do. The trades themselves are handled by the exchanges database.

You can pre-generate addresses in a wallet and then do a backup. This backup is valid and can be used to recover funds as long as that pre-generated pool doesn't run out.

There's also a 'backupwallet' RPC command that the bitcoin daemon provides to safely copy/backup the wallet. So lots of options available.

joe_the_user• 15 years ago

Not necessarily,

The hit to confidence in bitcoin might decrease the value of the remaining bitcoins more than the loss of these bitcoins increased it. I suppose we'll have to look at the exchanges...

tedjdziuba• 15 years ago

And nothing of value was lost.

drivebyacct22• 15 years ago

I can sell my BitCoins immediately for about $250 (what I originally put in). How can you sit and say it has "no value"?

astrodust• 15 years ago

Theoretically you can sell them, but what if nobody's buying?

zb• 15 years ago

Some people believe that price and value are the same thing. Others, not so much. It's not inconceivable that Ted here holds the latter opinion.

nradov• 15 years ago

Ha ha! A fool and his money are soon parted. Anyone stupid enough to waste real currency on bitcoins deserves what he gets. Bitcoins are the modern equivalent of speculating on "limited edition" Beanie Babies. The truth hurts so I expect all the bitcoin fanboys will downvote me now. Go ahead do your worst. :-)

noonespecial• 15 years ago

The time seems right for the first bitcoin insurance company.

SODaniel• 15 years ago

Only problem. Who would insure a data file of unknown value?

weavejester• 15 years ago

That's not really a problem. You'd insure individual bitcoin addresses, and the amount of bitcoins assigned a bitcoin address is public.

The real problem is figuring out whether an address has been actually stolen, or whether the user is indulging in a spot of insurance fraud.

3rd largest Bitcoin exchange lost its wallet.dat

Related Stories