SendGrid employee’s account was compromised and used to access internal systems

https://sendgrid.com/blog/update-on-security-incident-and-additional-security-measures/

by lxm • 11 years ago

110 53 11 years ago

nwenzel• 11 years ago

Two Factor Authentication. Please, all companies that have any of my data, don't just give me the ability to use 2FA, please force all of your employees to use it on their email, GitHub repos, everything. It should be a requirent right alongside https.

My personal favorite service is Authy, now part of Twilio.

troycarlson2• 11 years ago

This. It's not unreasonable to flat-out require 2FA.

atmosx• 11 years ago

2-factor auth might be good for SendGrid but depending on another device (be it your phone or a token-device) sucks big time in the real world.

toomuchtodo3• 11 years ago

https://twofactorauth.org/

tomjen32• 11 years ago

You should never use an external 2 factor app, use the Google app since it implements a standard and works offline. You will want this eventually.

jsmthrowaway2• 11 years ago

TOTP is fairly weak. Convenient, but weak, due to the shared secret. I've seen proofs of concept that trivially lift the secrets from Google Authenticator. On the server side, the service is probably storing your TOTP secret one column adjacent to your hashed password. When done poorly (and most are), TOTP is basically no additional factor. Also backup codes, app specific passwords, etc, etc.

Certain classes of non-shared-secret hardware token are waaaaaaaaay better. Just less convenient. You should look into DoD CAC, for example. Google Authenticator is nice but will never protect Secret information. I know this sounds way out of startup league, but it shouldn't; we should instead study what we can learn from such things instead of blanket advice like yours.

Reiterating: none = bad, TOTP = better, strong tokens/biometric/etc = best.

dorfsmay• 11 years ago

Absolutely, on the condition that you encrypt the device where you have installed gogle auth.

The reason people use authy and the like is because they make it simple to recover if you lose access to your device (lost, stolen, broken). My work around this is to take a screen cap of the QR codes and encrypt them with gpg. You could also save them inside a password manager (especially for less technical people).

shard972• 11 years ago

Two Factor Authentication. Please

My understanding is that Sendgrid isn't too keen on dongles.

hueving• 11 years ago

too soon :)

mtrimpe• 11 years ago

Maybe they should just publicly fire someone to make this all go away.

biafra2• 11 years ago

A service for 2FA?

I prefer a stand alone app or device for that. Why would Authy demand that I give them a phone number and an email address if not for monetizing (as in selling to anyone) my personal information?

tracker13• 11 years ago

Secondary factor for authenticating your authy account? In addition to your own data key. 2FA for your 2FA.

famousactress• 11 years ago

Yes please. One of the thing that annoys me about these notices is that they all recommend that we turn on 2FA, when the fucking root cause was that THEY DIDN'T.

rubbingalcohol• 11 years ago

evidence suggests that the cyber criminal accessed servers that contained some of our customers’ recipient email lists/addresses and customer contact information. We have not found any forensic evidence that customer lists or customer contact information was stolen.

Which doesn't mean the customer list data wasn't stolen. The wording on this is too cute. Accessing data from privileged accounts doesn't necessarily leave evidence, depending on the type of system access or server logging involved.

I think SendGrid should clarify what could have happened here, worst case.

noir_lord• 11 years ago

Thought exactly the same, they lost me as a customer over the wording on their released statement and email.

Hell it wasn't instantly clear why you needed to reset your password and the link to make it clear was way smaller than the link to do anything else, pretty dirty pool.

dantiberian• 11 years ago

What was their password hashing algorithm? The lack of specificity isn't encouraging. The vague "hashed and salted" could mean anything from MD5 upwards.

compbio2• 11 years ago

I'd hope it would be PBKDF2. But then again, they could have just specified that. The lack of specificity makes me think otherwise.

kordless2• 11 years ago

Time to decentralize our data, people!

bdcravens2• 11 years ago

Which means what in terms of using a service for transactional email?

davideous2• 11 years ago

My company, www.drh.net, provides an SAAS or on-premises Mail Transfer Agent, GreenArrow Engine.

You can run it in your network and tightly lock down the system. We can take care of all of the email deliverability setup and operations tasks, if you want.

One advantage of running licensed software on your network is that you don't pay per-message fees, so for higher volume it's really economical compared to a service like SendGrid.

kordless2• 11 years ago

"Our data" does not exclusively mean data we share with others. There are many use cases that don't require our data to live on a centralized system. It is my belief talking about this subject causes a good degree of cognitive dissonance in people who are intent on building systems which rely on centralized storage. The fact is, the only value in centralized storage of data that isn't entangled with other's data is that it brings 'value' to the company and its shareholders. It's not bringing long term value to the users, that's for damn certain.

mr337• 11 years ago

Did anyone get an email about this from SendGrid. We have been searching and haven't found a "We got hacked, please reset your passwords".

I had to find out on HN, also on twitter they seem to be downloading it too.

akarambir• 11 years ago

I got an email on 28th and a reminder yesterday on both of my account emails to reset my password.

ZoF• 11 years ago

Same here.

bdcravens2• 11 years ago

No, but this was a visited link for me - I changed our SendGrid password a couple of days ago. I believe I saw this on /r/bitcoin, as exploiting Coinbase was impetus for hack.

c0achmcguirk• 11 years ago

I didn't get it from Sendgrid, but Microsoft sent it out to their Azure customers using Sendgrid on April 30th.

dirtae• 11 years ago

I am a SendGrid customer and received no notification from them about this. I only found out about it via Hacker News. Very poor form.

yuvadam• 11 years ago

CYBERCYBERCYBER

How about you just give me the details with resorting to propaganda?

sneak• 11 years ago

Spread the meme: Saying "cyber" means "I have no idea what I'm doing or talking about".

specto• 11 years ago

Because that's what the government and DoD calls it?

colinbartlett• 11 years ago

The customer targeted has got to be Coinbase, right? There was an email sent around a few weeks ago, clearly spam/scan but seemingly sent by legitimate coinbase.com servers. I received something like this:

  subject:      colin, We've got a message for You
  mailed-by:    em.coinbase.com
  signed-by:    coinbase.com

ikeboy• 11 years ago

Yes. See http://bits.blogs.nytimes.com/2015/04/09/sendgrid-email-brea...

compbio2• 11 years ago

used to access several of our internal systems on three separate dates in February and March 2015.
On April 8, the SendGrid account of a Bitcoin-related customer was compromised

If I can gather this right: SendGrid was fully hacked for 3 months on end. At least that is what they were able to recover from forensics, it may have been longer.

This sounds illogical:

We have not found any forensic evidence that customer lists or customer contact information was stolen. However, as a precautionary measure, we are implementing a system-wide password reset.

How would a password reset help combat information that was stolen before the reset? The password reset is because the systems accessed contained password hashes. Also it may be to upgrade the hashing mechanism to be more secure than "salted and iteratively hashed".

Sendgrid's privacy policy is cookie cutter, but it contains this:

For example, our policy is that only those individuals who need your personally identifiable information to perform a specific job are granted access to that personally identifiable information.

Apparently the employee that was hacked needed access to the data of all his colleagues and all users.

Upon discovery, we took immediate actions to block unauthorized access and deployed additional processes and controls to better protect our customers, our employees, and our platform.

Then the Privacy Policy again:

We will use at least industry standard security measures on the Site to protect the loss, misuse and alteration of the information under our control. While there is no such thing as "perfect security" on the Internet, we will take all reasonable steps to insure the safety of your personal information.

So apparently there were still some reasonable steps left to take, which were forced by this hack, not by 'industry standard security measures'.

Two-Factor Authentication: We encourage all of our customers to enable two-factor authentication, which can effectively prevent unauthorized logins.

I think you can better encourage (or force) your employees to enable this, so you can prevent unauthorized logins into superuser accounts.

From the Privacy Policy you'd expect they already did this:

Likewise, all employees and contractors are kept up-to-date on our security and privacy practices.

Then the unspecified hashing mechanism. Should you worry about the chance of account compromise again?

salts and iteratively hashes passwords

It would be a breath of fresh air if these companies would just say 'We use bcrypt' in their privacy policy.

Our Ongoing Commitment to Security

Your 3 month struggle with hackers. Also, three reasonable steps follow that could have been taken before this hack, like your Privacy Policy promised us.

NOTE: We require passwords to be a minimum of 8 alpha-numeric characters. Make sure any new passwords you set conform to this requirement.

Before or after this hack? Why should the customer make sure his password conforms to this requirement? Is it even possible to set a shorter password?

Security update: Please reset your SendGrid account passwords today. Beginning today, and in line with standard practice, we are requesting that all of our customers reset their passwords to all of their SendGrid account access points.

Why not force this? Asking nicely? Standard practice would be to force this upon next log-in and temp disable accounts that have not changed their password yet.

SendGrid employee’s account was compromised and used to access internal systems

Related Stories