Instapaper Server Update

http://blog.instapaper.com/post/6854208028

by psychotik • 15 years ago

163 47 15 years ago

blog.instapaper.com

ck2• 15 years ago

Assume all data on the drives has been cloned.

There's little chance they didn't.

m0nastic3• 15 years ago

He should absolutely assume that the data has been cloned, but there is about a 99% chance that it wasn't.

If the data was cloned on-site, then we wouldn't have heard about the search warrant (there'd have been a gag order).

And as this happened yesterday, there's almost no chance that the server could have been taken to the lab, imaged, and then returned to be reconnected today.

I would bet 100 dollars that the actual target server of the raid is still sitting on a dolly in the forensics cage while a tech is waiting for someone to tell him what he's supposed to be looking for.

Skroob• 15 years ago

You'd be surprised. I've worked with forensic law enforcement in the past, and they were equipped to clone drives quickly and easily. The idea is to create a clean image of the drive that they can use for evidence, and they don't boot the machine so if there's any trapdoors or whatever they won't hit them. And this was a local police force over 5 years ago, I can't imagine what the FBI would be capable of these days.

mike-cardwell2• 15 years ago

99%? I'd assume that the first thing they do on seizing a machine is pull the hard drive out and drop it in a disk cloning machine?

ConceitedCode• 15 years ago

I like to always follow the saying "Hope for the best. Plan for the worst".

Assume the FBI has a copy of everything on the drives and assume that they will see every file on the drive, even though odds are they don't and they won't.

TheCondor• 15 years ago

What do you base this upon?

He should contact the FBI and continue to beat on Digital One and determine if it was seized or not. It very well may have simply been turned off, is that not a valid possibility?

ichilton• 15 years ago

I hope he securely erased the data rather than just deleting it, otherwise it might not be just the FBI who have potentially got a copy...

nettdata• 15 years ago

Would there not also be backups at DigitalOne?

SeoxyS2• 15 years ago

With dedicated servers, the responsibility is on the client to back up what matters to him. Ditto with VPSs.

cheald• 15 years ago

Out of curiosity, does anyone know how volume-level encryption (like dm-crypt) holds up against this sort of thing? I find myself wondering what I'd tell customers if my server were seized as collateral damage like Instapaper's were. Will that sort of encryption serve as a plausible safeguard of customer data, or is it more of a padlock (easily broken with the right tools)?

psychotik• 15 years ago

You tell them what Marco told them.

Tharkun• 15 years ago

It would certainly make it much harder to steal the data -- and at the same time it would make it much harder to boot the machine, because at some point you'd have to enter a passphrase. It's possible to do so over SSH, but it's still a bit tricky. Rebooting the server while you're away from an internet connexion will leave it in a useless state.

That being said, even with volume encryption, the key is in memory somewhere. RAM isn't wiped as quickly as you might think, and it's apparently possible to extract keys from memory after the machine has been powered down. So be warned, and be paranoid!

JoachimSchipper• 15 years ago

Indeed, the encryption keys can be recovered from memory fairly easily. http://www.usenix.org/events/sec08/tech/full_papers/halderma... is very readable; http://www.youtube.com/watch?v=JDaicPIgn9U is a well-made 5-minute introduction by the authors.

bhousel• 15 years ago

You don't necessarily need to encrypt the boot volume.

tswicegood• 15 years ago

+1 to the attitude. It's all about getting the service back up and making it better. We (collectively) tend to spend too much time trying to figure out who to blame instead of just doing the cool shit that needs to be done. Kudos!

SeoxyS2• 15 years ago

I wish more people thought the way you do. You, sir, are an inspiration.

Gorbzel• 15 years ago

Sadly, I simply cannot agree. I absolutely understand Marco's desire to focus on developing Instapaper, but there's quite a bit more at stake here, and his willingness to move on without even a bit of protest is concerning.

He claims that there's nothing that can be done, but there are definitely legal actions one can take in this scenario. Sure, they might not ever fully compensate him, nor convince the skeptics, but it will at least reveal the details behind the seizure, and it will send a message to both the FBI and to others like DigitalOne that could prevent further "seize first and ask questions later" style operations.

statictype3• 15 years ago

So for a one-man software shop, given the choice between:

a) Spend money and time on legal action against the FBI for holding onto his servers for a couple of days while in the process of an investigation, in order to hopefully try and make a larger point.

and

b) Spending his time and no money by actually improving his bottom line by enhancing the features of his software,

you expect any rational person to choose (a) ?

idlewords• 15 years ago

I was more severely affected than marco and I'm certainly going to pursue this matter to find out what happened. I imagine the Curbed team might share my curiosity.

jrwoodruff• 15 years ago

Agreed on this point. You have to choose your battles, and this is a lose-lose battle. Even if he would fight, win and receive some type of compensation, that's unlikely to be greater than what you had to give up in the fight.

gojomo• 15 years ago

Should services consider storing all user data on encrypted volumes?

Pro: after a powerdown, seizure/theft/cloning of the volume won' reveal user data

Con: need a secure way to supply decryption passphrase on each reboot

Supermighty• 15 years ago

I think a neat idea would be to create 1U - 4U lock doors that could be bolted in front of a server in a rack.

I wonder if this would be enough to give each server it's own legal "separate space" insomuch as to prevent a whole rack from being seized due to warrants like this.

jsatok• 15 years ago

The server that was taken from Marco was a blade server, and he just rented a single node in the blade. Locking the single blade he was on wouldn't have prevented the situation that occurred, as the FBI seized a number of enclosures, rather than servers. Most likely, the agents who took the servers didn't understand a blade server, and took the entire enclosure, rather than pulling a couple nodes out of the blade.

patio11• 15 years ago

I think you're more than justified in assuming no compromise happened. Someone with legitimate authority had physical access to your server. That has happened before and will happen again. Your server's cage opens several times a month, probably, as techs and whatnot do their icky atom-touching sorcery. Physical access to your server by the adversary means untraceable compromise, sure, but if that is true, hasn't it happened before? (Sure, the techs might be more trustworthy than the FBI... Unless they were the FBI, which is paranoid squared, but if you think that agents executing a warrant routinely grab servers in the vicinity because, hey, computers tend to have juicy stuff on them and we have way too much free time, then assuming infiltration seems almost reasonable by comparison.)

In a threat environment this bad, you would personally be scrutinizing sign-in lists every month. You have better things to do, right? Right. Because this is really not Instapaper's threat environment. You got mildly inconvenienced as a side-effect of routine government operations. You can pick whether you want to escalate that to A Full Crisis. That seems like a bad call, which you recognize on the legal side. Do it on the security side, too.

You may think that vocal parties will disagree with "This was a non-event for us.". They might. Screw them. You're a CEO; dealing with paranoid ramblings does not generate business value like two months of normal operations does.

IgorPartola• 15 years ago

Isn't there a way to get information from HDD's about how long they have operated from when they were manufactured using SMART? Not that anyone keeps those records on hand once the HDD is no longer in their possession, but if you are paranoid, you could start offloading those records every minute to a different server. That way if yours is seized, you could at least know if someone had turned on your HDD's and for how long.

It'd be cool if HDD's/SSD's started compiling rough stats about numbers of reads and writes for this purpose.

richardblair• 15 years ago

Man, you have an awesome service. It served me greatly through my post secondary education. I am glad you got your server back, and you can keep on keepin on.

Again, thank you for the services you offer.

nanoanderson2• 15 years ago

Tangent alert! Marco makes me feel great.

One of my favorite things about Marco is that he gives me confidence that there is not a gaping wide chasm between the skills of the best engineer-entrpreneurs and me. This time, it was his frank admission that he was taught by commenters and emailers than his current password encryption was unsafe and that there were better options out there.

When you realize that someone as successful as Marco has the same learning to do that "the rest of us" do, it makes me feel great.

sneak• 15 years ago

News flash: Security breaches happen because almost everyone in the industry sucks at this stuff, including (most importantly) those who SHOULD KNOW BETTER, yourself and Mr. Arment included.

nanoanderson2• 15 years ago

And now we do. :-)

absconditus• 15 years ago

Why do you assume that Marco is one of "the best engineer-entrepreneurs"?

cosgroveb• 15 years ago

"I’m not convinced that they did everything they could to prevent the seizure of non-targeted servers, and their lack of proactive communication with the affected customers is beneath the level of service I expect from a host."

I can't speak to the level of service you did or did not receive but I seriously doubt DigitalOne could have done more to prevent "non-targeted" servers from being seized. If it was in the same rack as the "targeted" server I'm sure that 100% of the time LE would take it and find out whether it was related later. It's their job to take every step necessary to preserve evidence first and foremost when executing a search warrant like this.

I agree with you that the data was almost certainly not compromised and I think it's great that you're so proactive about protecting your customers by moving to bcrypt.

suking• 15 years ago

This is the exact opposite approach dropbox takes - well done with good communication. Props.

smackfu• 15 years ago

This is all just assumptions, since amazingly the hosting company still has no comments at all. Who exactly rebooted the server? Not the FBI, for damn sure.

Instapaper Server Update

Related Stories