Sendgrid under siege from hacked accounts

https://krebsonsecurity.com/2020/08/sendgrid-under-siege-from-hacked-accounts/

by elsewhen • 6 years ago

199 146 6 years ago

Well, I've tried contacting sendgrid about phishing emails that look like they're from @sendgrid.com , using their own server. The SPF records match, so it isn't marked as junk mail.

It's similar to the github pages exploit from years ago, which is why we now have github.com and gitub.io

The purpose of these mails is of course to get credentials and spam some more.

At first I checked for a bounty, but as this is a "configuration error" (SPF), there is no bounty.

Then I sent it by mail, but yeah.. No reply.

nodesocket2• 6 years ago

Can you elaborate how a 3rd party is able to send e-mail from @sendgrid.com?

jbverschoor2• 6 years ago

Simple. Their ip addresses of their client/guest-servers are whitelisted in the SPF records. So anyone can impersonate them.

Not 100% sure if that was the case now, as I can't find the email anymore, but I've been getting them from both sendgrid and mailgun.

Mailgun's SPF (was at least):

  v=spf1 include:spf1.mailgun.org include:spf2.mailgun.org include:_spf.google.com include:aspmx.pardot.com include:mail.zendesk.com include:spf.mailjet.com ~all

  spf1.mailgun.org: v=spf1 ip4:104.130.122.0/23 ip4:146.20.112.0/26 ip4:141.193.32.0/23 ip4:161.38.192.0/20 ~all

  v=spf1 ip4:209.61.151.0/24 ip4:166.78.68.0/22 ip4:198.61.254.0/23 ip4:192.237.158.0/23 ip4:23.253.182.0/23 ip4:104.130.96.0/28 ip4:146.20.113.0/24 ip4:146.20.191.0/24 ip4:159.135.224.0/20 ip4:69.72.32.0/20 ~all

In the mailgun case I received mails from for example 198.61.254.24 with from: invoice@mailgun.com

This is actually part of a bigger issue, where lots of people use the same email service to send out emails. In theory that should work with gmail or office365, but iirc gmail forces your from sometimes, and google+ms probably thought about this, right? ;)

XCSme2• 6 years ago

Someone sent spam emails using my domain name, so I looked a bit into it and made a summary of what I learned: https://www.usertrack.net/blog/stop-others-use-your-domain-e...

perennate2• 6 years ago

Same -- I got several e-mails with subject "The following services failed to auto-renew and are about to expire" that I almost believed were real because they came from an actual sendgrid server (wrqvpkzw.outbound-mail.sendgrid.net [149.72.49.233]).

If SendGrid can't develop simple filters to deter phishing e-mails sent through their platform targeting their own service, then obviously they're going to end up with a lot of users being hacked.

I mean seriously, why would an e-mail provider like this allow e-mails to be sent from "renewaldepartment@sendgrid.com" and "support@Sendgrid.com"??

moomin• 6 years ago

I still remember when SendGrid got DDoSed after an incident at PyCon* and they caved. This told me two things: 1) they weren’t prepared to stand up for their own employees when the chips were down 2) their engineering wasn’t good enough to cope with a bunch of amateurs attacking them.

* Let’s not derail this into a discussion of whether or not they should have done it in the first place. They didn’t, they only acted after they couldn’t cope with the DDoS.

Drdrdrq• 6 years ago

For those of us who don't know / remember the incident: https://venturebeat.com/2013/03/21/sendgrid-under-ddos-attac...

bytecycle• 6 years ago

My university straight up blocks any mail coming from a SendGrid IP because of the massive amount of phishing.

What would be a huge UX and security improvement is showing which team members are using 2FA. Just like GitLab shows a little blue label next to the username.

pmlnr3• 6 years ago

That's actually good. Sendgrid & the rest are excluded from the paranoid gmail, m$, etc spam filters, while small senders suffer from being sent to spam immediately.

There should be no exceptions.

stefan_• 6 years ago

Yes, time for GMail et al to block them or admit it's just more anti-competitive nonsense.

gruez• 6 years ago

But is sendgrid "too big to fail"?

calmworm• 6 years ago

Not even close.

dimitrios12• 6 years ago

Too big to fail also was meant to communicate the vast amount of effort, time, and money that would be involved in rebuilding the financial infrastructure if indeed it had collapse, not to mention the immense short term pain and suffering that would occur in the interim.

A SendGrid sized void could be filled in a matter of weeks to months by teams of highly motivated engineers looking to fill a market gap. The cost of starting up there is virtually nonzero (compared to say, starting a bank), and the regulatory burden is virtually nonzero (Email regulations are arguably some of the most clear and concise).

All that to say, drawing a comparison with the "too big to fail" bailout of major US lenders is, frankly, silly.

greggh• 6 years ago

Wait, you mean the small timers can't afford $15,000 for services like Return Path?

The entire thing is a protection racket. There are super expensive services that can get you onto those same whitelists.

hansjorg2• 6 years ago

If you're not whitelisted there are stricter rules and some non obvious pitfalls, but small senders with correctly configured servers does not get sent to spam in my experience.

When you pay Sendgrid or one of their competitors, you pay to not spend time dealing with this.

The overall result (in general) is to make spamming harder.

pmlnr3• 6 years ago

If you're not on the contact list of a personal gmail account, it's nearly certain you'll land in spam if you cold contact them. Happened to me so many times; sending a mail to the person sitting next to me who I hadn't ever contacted form my own mail server (dmarc, dkim, spf, even effing google mta-sts is configured) and I still landed in spam.

tunesmith• 6 years ago

I was under the impression that individual servers should be fine as long as their ip isn't on a recognized blacklist and they use free things like SPF and DMARC - is that not true?

BenjiWiebe• 6 years ago

It's mostly true.

jabroni_salad• 6 years ago

So long as you do not also send spam, or send mail that someone might think is spam. I'm a mail admin and can see my users's spam reports. Lots of those messages are authenticated and there's actually not that much malicious stuff being delivered in the first place. It is just that the messages aren't wanted.

Once your messages start to get reported, the sender's reputation goes down until its spamscore qualifies for the filter.

bleepblorp• 6 years ago

It's not generally true unless you're a Capital-E grade Enterprise customer running a single server as a side component of a much larger Internet presence.

Finding connectivity that's not already blacklisted by the GOOG-MSFT email protection racket and will allow individual servers to send outgoing SMTP is basically impossible for sub-enterprise scale organizations. The major cloud services block outbound SMTP for non-Enterprise customers and minor cloud services will already be blacklisted.

Getting your mail delivered by GOOG-MSFT also means that you need to have matching forward and reverse DNS, and ideally you should have your own ASN that has never been associated with any addresses that have ever been accused of sending spam. This is an impossible hurdle except for large organizations.

The entire email ecosystem is stacked for force smaller players to pay a head tax either directly to GOOG-MSFT, in the form of using their mail services and paying extortionate per-user mailbox costs, or by using paid mail relay services.

citrin_ru• 6 years ago

It still possible to send mail from a small mail server, but there no grantee at all that messages will not end up in a spam folder. Most likely Gmail uses some ML based spam filter which has relatively high rate of false positives. For big senders they can do something manually, but they don't care about smaller ones. Gmail is too big and have little incentives to improve they spam filter (until users will start to leave, which will not happen anytime soon).

walrus01• 6 years ago

Sendgrid and its cohorts have always been a source of spam. The spamassassin setup on my postfix SMTP gateway ranks all of the bulk email newsletter sending services with +2.5 points or higher.

This is like a sewage treatment plant complaining someone has come at 3am, cut the lock off their gate and dumped a tanker truck full of additional sewage.

techsupporter• 6 years ago

Sendgrid and its cohorts have always been a source of spam.

Sendgrid is the source of a lot of e-mail spam and their parent company Twilio is the source of a lot of SMS spam. It's really a synergy match made in MBA heaven.

aspectmin• 6 years ago

Sigh. Same happened to me. A few months ago. Sendgrid account hacked, someone set up a campaign and sent a huge amount of spam mail. Big bill on my credit card. Contacted Sendgrid, they blamed it on me. I cancelled my account, never going back. Screw them

jeffbee3• 6 years ago

Was it not your fault?

PakG12• 6 years ago

Is that sarcasm?

jlgaddis6• 6 years ago

Right!? He should have known better than to go out by himself, late at night, dressed like that! /s

ratww• 6 years ago

The same thing happened to my previous employer a few months ago.

Our SRE had used a very long novel password and access was very limited, but 2FA wasn't enabled because it lacked some features (I'm guessing it was the SMTP, as other poster mentioned).

We were able to catch it pretty fast and never asked for refunds or anything, but Sendgrid's support was very standoffish and combative when we asked for help with our "investigation" of how the account was compromised, to the point the company started seeking a competitor.

jacquesm2• 6 years ago

Don't forget to charge back.

crizzlenizzle• 6 years ago

This has been ongoing for months already. Nobody at Sendgrid cared. We had several issues with them leading to penalize mails from their IP space.

I’m actively discouraging my customers from using their service.

jlgaddis6• 6 years ago

... over the past few months there has been a marked increase in malicious, phishous and outright spammy email being blasted out via Sendgrid’s servers.

Fortunately, I hadn't noticed the recent increases.

After it became apparent that they didn't bother looking ibto -- or even reading -- abuse reports, we blacklisted basically their entire IP ranges a long time ago and then added exceptions when they were requested.

I certainly did notice the marked decrease in spam then, though.

urda• 6 years ago

E-mail is a hard problem to solve, and I'm honestly really glad I do not have to manage it day-to-day myself.

pmlnr3• 6 years ago

Enough with this "hard to solve" nonsense; email has been around since the 70s.

Google, Yahoo!, and Microsoft are f*king it up, that's all.

statquontrarian3• 6 years ago

I've been a paying customer for years on their shared server plan ($15/mo) and I've had terrible customer service and lots of emails marked as spam (e.g. to my Mom) due to a bad reputation of the shared box. I haven't migrated out of laziness and inertia.

This article and the fact that other comments mention that SendGrid is seen as spammy is the final straw and I'm going to migrate. What paid email senders do people recommend that aren't excessively marked as spammy by gmail, etc.?

throwaway200829• 6 years ago

Throwaway since I’m a former Twilion.

Even though people love Twilio for their developer friendliness and nice APIs, what no one wants to talk about publicly is their disproportionate amount of revenue that comes from spammy behavior. Either as a company or it’s customers.

I was in sales. I saw the revenue numbers and sold these deals. It’s staggering.

Yet wall street looooooves Twilio because of their revenue growth.

Note: Twilio owns Sendgrid.

lowdose• 6 years ago

What I would like to know from a Twilion:

Is Twilio used by commercial parties in the past to participate in voting in the European Song Festival?

Shared404• 6 years ago

You may have luck with protonmail's paid tiers. I use the free tier currently and have been pretty happy.

statquontrarian3• 6 years ago

Thanks but I host my own email so I'm only looking for an outbound SMTP server. I went to ProtonMail's business page and the pricing plan only shows full email hosting which I don't need.

statquontrarian3• 6 years ago

After doing some more research, it seems the major alternatives are Postmark, MailGun, and Amazon SES. I liked one comment left by Postmark on another HN post where they say they want high quality senders, so I think I'm going to Postmark...

n_u_l_l2• 6 years ago

Same here regarding the support. I helped a company migrate to Sendgrid, and right after paying and integrating, we were locked out of the customer panel.

We created a ticket, and a few hours later we were also locked out of the support area. All in all, it took them 7 days from locking us out before we received a human response and they fixed it.

We recently couldn't deliver to Hotmail. It took them 27 days before the first response (P2 Degraded). They didn't fix it, and wanted us to switch plans.

XCSme2• 6 years ago

Are you still using them? Why?

sweeneyrod• 6 years ago

I've also experienced my IPs getting blacklisted (albeit on the free plan). I can understand this happening (you get what you pay for) but what I can't understand is that their support response was saying that I just had to deal with it rather than switching my IP. I've also used MailGun and not had that problem (although I'm not a fan of the way you can't export logs from them for some reason).

afrcnc• 6 years ago

how I hate this kind of reporting

3 people complain on reddit and it's now "[COMPANY X] under siege"

jlgaddis6• 6 years ago

Between the comments on the article itself and this HN thread, there's already more than 10 times that. It's not being exaggerated quite the way that you "hate".

To further prove how bad SendGrid is, Invaluement recently spent a ton of time and energy to create an entirely new "Service Provider DNSBL" [0] -- just for SendGrid! In fact, they just announced it about a week ago [1].

[0] https://www.invaluement.com/serviceproviderdnsbl/

[1] https://www.mail-archive.com/mailop@mailop.org/msg11403.html

krebsonsecurity• 6 years ago

Spend a few minutes looking at the spam list threads linked in the article. This is not just a few people complaining. E.g.:

https://www.mail-archive.com/search?l=mailop%40mailop.org&q=...

There was a ton of material I did not include in the story, including a story from a company that had a client have 40 million phishing emails sent through their Sendgrid account, which it turns out was set up by an employee long ago who was no longer with the company and had not turned on 2FA (and probably was re-using passwords).

I started reporting this story almost a month ago after receiving more than 3 emails from different IT experts who were really frustrated with the amount of malware and phishing coming from Sendgrid accounts. They were frustrated because they couldn't block Sendgrid outright because too many companies they were expecting regular emails from used the platform.

The day before I published the story, I head from someone else who was getting phishing attacks spoofing Aruba Networks.

tobltobs3• 6 years ago

You also could have a look on twitter how many people with hacked accounts try to reach sendgrid support via this channel.

jlgaddis6• 6 years ago

The CSO said:

"... 2FA for customer accounts is the right thing to do, and we’re working towards that end ... This is part of the reason we acquired Authy ..."

Cool, it sounds like they're making a effort to implement -- and require -- 2FA... oh wait, FIVE YEARS have passed since then!

nodesocket2• 6 years ago

Twilio supports and soon will require TFA. I personally don't use SendGrid (use Mailgun), but I have to believe it should be coming soon (if not already supported).

maerF0x03• 6 years ago

see my comment here: https://news.ycombinator.com/item?id=24319007

jeffbee3• 6 years ago

If you run a service that can be abused like this, you're going to need objective account abuse metrics to warn you before these things happen. Specifically you're going to need a Russian speaker to prowl around on Russian forums buying your own hacked accounts, so you can figure out the market price. If accounts on your system are selling for ten cents, might be time to reflect on account security.

neolog• 6 years ago

Do big companies actually do this (check market price by buying their own accounts on forums)?

timClicks• 6 years ago

Absolutely. Edit: Although it's rarely direct. Normally big corp will engage a security consultancy and discuss type of reconnaissance as one part of a wider strategy.

PakG12• 6 years ago

I run IT for my organization and I am seeing a lot of phishing emails from Sendgrid. I haven't counted or done data analysis for proportion, but it's enough volume for me to really notice it.

jlgaddis6• 6 years ago

This list is probably incomplete / out-of-date but ...

  149.72.0.0/16
  167.89.0.0/17
  168.245.0.0/17
  192.254.112.0/20
  198.21.0.0/21
  198.37.144.0/20

jeffbee3• 6 years ago

Isn't their SPF record authoritative?

ip4:167.89.0.0/17 ip4:208.117.48.0/20 ip4:50.31.32.0/19 ip4:198.37.144.0/20 ip4:198.21.0.0/21 ip4:192.254.112.0/20 ip4:168.245.0.0/17 ip4:149.72.0.0/16

jlgaddis6• 6 years ago

Usually, yeah, but not necessarily (e.g., for those with dedicated IP addresses).

Even then, it usually will be since most customers will just add the (customer-specific) "include:nnn.wl.sendgrid.net", or even just "include:sendgrid.net". But, for whatever reason, I've come across several that add an "a:" or "ip4:" for their dedicated IPs instead (I assume there's a good reason for that, but I don't know what it is).

Farfromthehood• 6 years ago

Sendgrid user for many, many years.

I believe we disabled 2FA because it broke /prevented our client's ability to send via SMTP. No idea if they've fixed the issue (and honestly don't care). Make security easy to adopt of you want us to implement it.

Also, organization-wide, we don't use 2FA that requires a mobile phone number/SMS (because we don't trust vendors with our phone number).

*IIRC, Sendgrid initially only offered 2FA via SMS, but now you can also use an authenticator app.

dylz2• 6 years ago

Phone 2FA is broken by default.

Yes, it broke SMTP and it also broke all API access until they kinda-fixed API keys. Pretty much anything you used basic auth for at a touch point.

maerF0x03• 6 years ago

Neil Schwartzman, executive director of the anti-spam group CAUCE, said Sendgrid’s 2FA plans are long overdue, noting that the company bought Authy back in 2015.

It's worth noting that this mixes up history. Authy was acquired by Twilio in 2015. Sendgrid was acquired by Twilio in 2019, there's no reason to assume in the 4 years interim that SG should have used Authy.

dylz2• 6 years ago

I think the point was that they own a literal 2FA company, but don't have 2FA

tialaramex8• 6 years ago

They do have 2FA they just don't require 2FA.

psanford• 6 years ago

For a company that went out and acquired a 2FA subsidiary, Twilio really doesn't seem to have their 2FA story together.

For example you'd think a company with an division dedicated to authentication would support what the industry considers the best 2FA solution: WebAuthn. Sadly that is not an option for your Twilio account (other Auth providers like Duo support WebAuthn).

Twilio has also been emailing me the past few months telling me I must turn on 2FA on my account. This is weird since I have TOTP enabled.

I feel like Twilio must be internally conflicted about this. As a company dedicated to phone services they really want to push SMS solutions even though SMS based 2FA is vastly inferior to WebAuthn.

kelnos• 6 years ago

Twilio has also been emailing me the past few months telling me I must turn on 2FA on my account. This is weird since I have TOTP enabled.

There's been a bit of confusion about this email: you can turn on 2FA separately for your Twilio account (identified by a 32-char hex string prefixed with "AC") and your user account (your email address). The email is talking about your user account. Even if you have 2FA turned on for your Twilio account, the email is asking you to turn it on for your user account.

For some context for those who may not be familiar: the Twilio account is essentially a billing/project unit that is a container for Twilio resources you've purchased or configured. Multiple user accounts can have access to a single Twilio account, and a single user account can have access to multiple Twilio accounts. Enabling 2FA on the Twilio account means all users who want to sign into that account (regardless of their user account setting) will have to use 2FA. Similarly, enabling 2FA on your user account will require 2FA whenever you sign in, regardless of the settings on any accounts you may have access to.

(Source: I work there, got confused about this email myself, and managed to clarify internally what was going on.)

hnzix• 6 years ago

even though SMS based 2FA is vastly inferior to WebAuthn.

SMS based 2FA is worse than useless thanks to the rise of SIM Swapping hacks, it adds an easy attack vector into your account.

u801e2• 6 years ago

you'd think a company with an division dedicated to authentication would support what the industry considers the best 2FA solution: WebAuthn

How would WebAuthn work with someone using a mail user agent connecting via SMTP?

taf2• 6 years ago

I don’t get this sense at all... as a long time Twilio customer... they have since on the Authy purchase been pretty consistently pushing it as the solution for two factor... sms is a medium for retrieving the 2nd code by not pushed in anyway...

jaxn2• 6 years ago

They don't seem that conflicted to me. I have always assumed their SMS-only 2fa was an attempt to sell people on SMS as 2fa. But all I want is to use my Yubico.

adrianh• 6 years ago

What service do people recommend in place of SendGrid?

Personally I need a service to send transactional emails from my company website with a low chance that they’ll be marked as spam. Tracking is not necessary (I disabled that in SendGrid).

xenospn• 6 years ago

Amazon SES can work for that.

nucleardog• 6 years ago

And depending how much you need to send, if you’re already in AWS the first 62k messages every month are free.

The only annoying thing is their sandbox and usage limits. You’re supposed to accept a lower limit, send high quality mail, and then request a higher limit once they’ve seen that you’re not generating a bunch of abuse reports.

Slowly on boarding senders is good for delivery but makes it difficult to just do a simple cut-over if you’re currently sending more than whatever quota they’ll give you right now. You need to continue sending through your old service and slowly move your usage over.

moltar• 6 years ago

Postmark has been great for us for ages. They are strictly transactional and will flag accounts quickly that don’t follow this rule. This detracts spammers.

tobltobs3• 6 years ago

Sadly they do now also allow bulk emails and therefore will soon be as shitty as all other emails sender services.

social_quotient• 6 years ago

Maybe Twilio will use that run up (bubble) in stock price to allocate to some tech debt like this.

169% YTD 33.8% last 3 months

I’ve always looked at SendGrid as a premium provider for transactional mail. Why recommendations for something better but similarly priced?

a2tech• 6 years ago

We saw our deliverability go way up with SES. It’s pricey for what it is but we’ve had really good luck with it.

cortesoft2• 6 years ago

Unless a company is selling new stock, an increase in stock price doesn't give them any extra money to spend. It just means their shareholders are worth more.

darkport• 6 years ago

I developed a tool called shhgit.com that watches code commits across GitHub, Gitlab and Bitbucket in real time. In a ~24 hour period you will find 100s of valid SendGrid credentials/API keys so this does not surprise me. They really need to enforce 2FA and IP whitelisting for API key use.

edoceo• 6 years ago

Oof. So,I tried sending mails from our own domain and our mail server - but Google tagged us as spam. We migrated into Google hosting and it was fixed, now back to Spam. So we move to a provider like sendgrid, but then they have a hack. Damnit!

When we hosted our own, we had DKIM, SPF, DMARC and such, clean IP and clean messages, we only sent account create and transaction notices - low volume, relevant messages to folks who opt-in. We had our little box setup right and we're polite actors but got tagged which killed our user experience (password resets went to Spam and folk called support)

But it feels the "big-web" forced us into vulnerable providers for the spam game.

We need more self hosted email and less centralized control.

smnplk• 6 years ago

`We need more self hosted email and less centralized control`

We need to start over and build a truly open internet.

Reelin• 6 years ago

We need to start over and build a truly open internet.

I want to agree, but email is actually an example of an open and federated protocol achieving success. It's pretty much everything that could have been hoped for as far as openness and interoperability go at the protocol level. Unfortunately, balkanization still occurs at the social (ie network) level.

dredmorbius2• 6 years ago

The open email Internet was buried in an avalanche of spam.

Open alone is insufficient.

Silhouette• 6 years ago

We need more self hosted email and less centralized control.

This is true, from both sides of the equation. The biggest single problem with email today, IME at least, is the number of major mail providers who are acting as gatekeepers for their users and doing a bad job of it. In particular, they block way too much legitimate mail, and often do it silently, so the sender is not even aware of the problem or able to contact them about getting it fixed.

Then the sender of the missing mail gets the customer support requests about the missing password reset emails, or the complaints that someone didn't know they were still subscribed despite the receipt emails being sent for each payment, or...

At this point, there really ought to be a blacklist for unreliable mail services on the receiving side analogous to the spam blacklists, so businesses can warn their users if given an address on a bad service and invite them to choose another.

jacquesm2• 6 years ago

At this point in time I think I'd rather deal with the spammers than with Google, Microsoft etc. Unfortunately they've managed to lock up email to all intents and purposes, good luck reversing the situation to the point where they have to play nice.

TylerE2• 6 years ago

To be honest, at this point I wish email would just die and get replaced with an actual modern protocol. It's a nightmare. Everything about it sucks.

samat• 6 years ago

Please don't. Anything modern would another lock-in. I can read decade old emails, switched multiple providers. Messages in those 'new protocols' are long unaccessible and protocols themselves are dead.

smabie2• 6 years ago

Such as? How would it work?

chromedev• 6 years ago

There are plenty of people successfully hosting their own email. If you're getting flagged as spam, then you need to take a few steps every once in a while but self-hosted email is still totally doable.

foepys• 6 years ago

As somebody who is running their own mail sever I'd love to agree with you but GMail, Hotmail, and all the other large email providers make it extremely hard to do so.

If you only send a few emails per month, you are fine. When you host an email server for others, even if it's only your family and friends, it can quickly become a nightmare. And if I ran a business, I'd never do it myself.

Landing on a blacklist of those large providers effectively shuts your email server down and it's way too easy to get blacklisted. There are so many techs around email now (DKIM, TLS, SPF, HELO testing), that coordinating them becomes a huge responsibility. Misconfiguring a single one can lead to rejection at the large providers that deal with billions of incoming emails a day, with hyper aggressive spam filters. The worst thing that can happen is that your emails were to be classified as spam but you didn't know it. A flat out rejection is relatively rare so you are often not notified that your configuration is incorrect.

Getting off those blacklists is extremely hard. Especially if you don't land on GMail's blacklist but on those community driven ones that only allow your IP to be removed from it once. If you land on them a second time, you need a new IP and hope that this IP isn't already blacklisted.

Many large providers are blacklisting entire ISP IP ranges due to past so waves. E.g. GMail is blocking the entire AWS EC2 range, I believe.

tialaramex8• 6 years ago

Keep in mind that some users almost certainly believe your email is spam. Even if they're explicitly paying you to send them that email.

For an end user "spam" is "email I don't want".

"But" I hear you screaming, "That's their credit card bill, they even confirmed their email address and specifically opted into receiving it by email - that's not spam". It's unwanted, they don't have $850 to pay off the balance and they don't want to think about the interest charges, so it's spam as far as the user is concerned, some non-trivial fraction of users will click "Spam".

You've probably seen people who have piles of unopened real world postal mail, they feel the same way about that. Unsolicited catalogue. Bank statement showing I am overdrawn. Nagging letter from my mother. All unwanted: Spam.

Worse, some of these same users will anyway be angry that the important message they marked as spam is treated as spam. Why are facts true? They shouldn't be true if I don't want them to be. Is that irrational? Sure, that's humans for you.

This is a losing proposition, and the end-user facing providers are aware of that, but "spam blocking" is something users demand from such providers even while being very angry that it doesn't do what they want. "Humans are irrational" is not actionable.

DyslexicAtheist2• 6 years ago

For an end user "spam" is "email I don't want".

last time I checked (been a few years since I used Google) BigTech is perfectly able to distinguish a bad actor from an opt-in type of service. They even presented me with a little notification whenever I moved stuff into SPAM that didn't belong there asking "do you want Gmail to unsubscribe you from X" ... I can't exactly recall the text - but all their lofty claims about their "AI" should account for something?

I agree with you that the _user_ is the biggest wildcard. But it's not that we should allow Google/Microsoft get way with this as an excuse. They are currently right in the middle of killing open protocols like imap in favor of their own flavored shit (probably something AMP like but for email). They do not deserve the benefit of a doubt. They do not even deserve to have their side of the argument heard at this stage.

tialaramex8• 6 years ago

I am not a Sendgrid customer, however examining their documentation what I think the "attacks" look like is this:

14 year old Jimmy guesses that Foo Corp has the Sendgrid username "FooCorp" and the password "FooCorp0" obeying a company requirement to use "At least 8 characters" and "At least one digit". Very secure. Much password.

Jimmy connects to Sendgrid's old API and literally just provides username "FooCorp" password "FooCorp0" to send as much email as Foo Corp have paid to be able to send. No other steps needed.

Insisting customers get 2FA would in practice mean:

1. Get all customers to sign up to Authy's phone-based 2FA and use it for Sendgrid. Accept that this will have a revenue impact, you're essentially firing customers who either were too lazy to sign up for this or regard this type of security as a joke.

2. Deprecate older APIs that allow username + password auth

3. Thus force users into APIs that rely on a bearer token they can only get using 2FA.

And yet after all this work if Jimmy instead gets a copy of your bearer token (e.g. because some idiot commits it to github) then he can once again send emails as Foo Corp until somebody disables the stolen bearer token (and then generates a new one and updates all their running code).

It's a poor show that in 2020 this is their goal, it's not even the thing they have and need to improve upon, it's just a future goal.

cnst3• 6 years ago

The whole idea behind Sendgrid is a joke.

It's become popular for some providers since the move to the Cloud to block outgoing port 25, but apart from that, you literally don't need Sendgrid. Any senior engineer can write a local "Sendgrid" in about half a day. Sendgrid doesn't even support all the RFCs properly, so, you'll likely get dropped mail if you move to Sendgrid from Postfix/Sendmail. Besides that, Sendgrid's UI/UX was terrible the last time I've played with it. It's literally the lamest startup there could possibly be.

As a customer of some unrelated service, I've already converted said service away from Sendgrid in just a couple of emails to the owner. (Sadly, they've moved to one of the many Sendgrid's competitors, not to Postfix/Sendmail, but the switch was obviously painless.) That's how horrible Sendgrid is, and how easy it is to make other people switch.

icedchai• 6 years ago

You can set up Sendgrid as a Postfix or Sendmail relay host.

manquer• 6 years ago

Any sort of fixed token without expiry is poor security .

No security protocol is perfect, however using a private key aws style , or oauth2 short expiry token like google , GitHub and many others is vastly better than Basic auth like system sendgrid is using in the older API.

There are many ways without changing API you could have improved security, Rate limit , IP whitelist, force frequent password changes, force use pass phrase etc and other techniques could have mitigated this issue

cnst3• 6 years ago

The whole idea behind Sendgrid is a joke.

It's become popular for some providers since the move to the Cloud to block outgoing port 25, but apart from that, you literally don't need Sendgrid. Any senior engineer can write a local "Sendgrid" in about half a day, if not less. Sendgrid doesn't even support all the RFCs properly, so, you'll likely get dropped mail if you move to Sendgrid from Postfix/Sendmail. Besides that, Sendgrid's UI/UX was terrible the last time I've played with it. It's literally the lamest startup there could possibly be.

As a customer of some unrelated service, I've already converted said service away from Sendgrid in just a couple of emails to the owner. (Sadly, they've moved to one of the many Sendgrid's competitors, not to Postfix/Sendmail, but the switch was obviously painless, and actually improved reliability.) That's how horrible Sendgrid is, and how easy it is to make other people switch.

P.S. I often send mail through shell scripts. Try selling me Sendgrid when my laptop already comes with Postfix/Sendmail, with spool support. Does Sendgrid even work when you have no internet connection? (That was a rhetorical question; the answer is, No, Sendgrid does not work when internet is not available; but Postfix/Sendmail works just fine even without any internet at all, and Postfix/Sendmail will spool all your mail until you get connected to an internet with port 25 not blocked.) Sendgrid is a downgrade however you look at it. When they move to MFA, it'll probably be easier to just switch to one of their plentiful competitors than deal with any new restrictions; that's why Sendgrid will probably be very careful around any new requirements; because their service is 100% disposable.

cortesoft2• 6 years ago

You seem to not understand the sendgrid service at all. They aren't selling SMTP service, they are selling deliverability (along with tracking and metrics)

You seem to not understand the actual problem at all if you think you will be able to successfully deliver email sent directly from a local sendmail client on any random IP that lets you connect over port 25. No major email provider will accept your email coming from a residential IP.

cnst3• 6 years ago

If that's what they're selling, they're not very good at it. Sendgrid doesn't support all applicable RFCs, and they don't support spool, so, they have exactly 0% deliverability if the recipient host is down for any reason, or uses greylisting.

I've never had any issues with mail from my WiFi IP accepted. You'd normally send it just to your own server from your local laptop w/o any extra configuration, so, it's up to you how you configure your own server. I just tried sending to Gmail directly from Postfix on my MacBook, without any extra configuration, and it's been accepted by Gmail as well. Zero configuration on the laptop. Postfix and the mail CLI clients come preinstalled, and the mail just works.

tgsovlerkhgsel• 6 years ago

You're mainly paying to deal with deliverability (IP reputation, proper "ramping" of mail servers, and the increasing number of technical requirements for correctly operating a mail server).

sarreph2• 6 years ago

At work we had a Sendgrid account that got compromised, after which all of our comms were (predictably) being marked as spam. After back and forth with support, we were told to continue sending as normal and that "eventually" our genuine emails would stop getting sent to spam.

That lacklustre support response was enough to put me off the service. And then, funnily enough, a few days ago I received a reply from Sendgrid support about another account I had opened for a side project. This reply came in ~2 months late, where I was asking if I could be moved to a different IP range as all my emails were getting blocked by Outlook services due to the shared IP I was assigned being (seemingly) misused by spammers. Again, the response was "just keep trying as normal" and eventually my emails would work... Apparently. That's obviously not a viable business communications strategy.

The funniest part about both these support responses was that despite one of them taking several weeks to get a response, I'm being pinged every few days asking me if my issue was solved. It's all well and good for me to wait months when I need a response as to why all my emails are getting sent to spam, but as soon as the ball is in their court, they don't mind pestering me for a response.

I don't have enough bandwidth for a dedicated IP, so I signed up for a service that requires a credit card even for its free plan, which I believe is a decent enough barrier against spamming.

old-gregg• 6 years ago

That lacklustre support response

Frankly, this makes my blood boil. Full disclosure: I used to work at Sendgrid's competitor (Mailgun) and I'm quite familiar with what's happening under the hood. You allowed spam to be sent from your account. You poisoned the IP space (not just one address you were sending from, but the IP block, affecting others) with spammy reputation. These blocks are expensive and hard to acquire (due to IPv4 shortage) and at this point you have caused more damage to Sendgrid and their customers, than your account value can probably compensate for. What kind of "support" do you expect at this point? You rented a hotel room, gave the key to someone who erected a meth lab there, and you're unhappy with the hotel's reaction?

Sending email on behalf of other people is hard work. The margins are not great, the ratio of spam-to-ham for new accounts is insane, and they're constantly under pressure between their customers who pay very little but want to spam the entire universe (I am sorry but most of you do!) and ESPs not wanting that traffic.

sarreph2• 6 years ago

I don't think I did a great job of making clear that my main gripe with their support (taking months) was from a separate, personal account which — from day one — was unable to send non-spam, transactional email to anyone with an Outlook address. In my first example, the account's (for previous employer — not "me") right to support I guess is more debatable...

But I don't think your analogy of the hotel room is totally representative here. Without knowing how someone has had their credentials hacked, it's much more prudent to assume that in your analogy that the room key was pickpocketed / stolen. And then it becomes more of a grey-area as to how much support one can expect.

That being said, I do appreciate your insight on account value, as compromised accounts clearly do constitute a burden that don't end up paying for themselves (even if they aren't "to blame").

3np2• 6 years ago

I really don’t understand this - I’ve read the parent comment 3 times and can’t see how you you come to the “gave someone the key to someone else” analogy.

Are we interpreting OP differently? From all I can read of available information, they seem to have been reasonable. Sounds more like their neighbor or the previous tenant had a meth lab.

ukyrgf• 6 years ago

I was sure this was satire of the kind of response you'd get from any run of the mill mail provider. I can't believe you're being serious...

te_chris• 6 years ago

As a sendgrid customer, this explains why, even after upgrading to dedicated ips and setting all our auth/security stiff correctly we still have emails being blocked inexplicably. Top off the rubbish customer service - they didn’t respond to me about getting blocked by outlook on our private IP until I tweeted them... - and next week looks like my job is to move to postmark.

redm• 6 years ago

Our customers are also getting our emails flagged as malware. Great way to onboard a service and verify your email, with a Google Malware alert. :/

Its Google's world and Twilio is living in it. :|

_nickwhite• 6 years ago

Sendgrid have a HUGE problem with spammers using their service. I've probably reported 20 SPAM campaigns using their official channels, but I think they route such e-mails to /dev/null. Never gotten a response- just more SPAM through their service. And I can't block their IPs, as a lot of legitimate sites utilize their services.

PaulBGD_• 6 years ago

Related, 2 months ago my sendgrid account gets hacked and a stolen (I presume) credit card gets added. They buy a plan, then proceed to send spam/phishing. I of course contact their support within minutes of receiving an email that $750 was spent on my account. They've now spent $1500 through my account on these spam emails, yet sendgrid finally dealt with the issue yesterday by deleting my account.

https://mobile.twitter.com/PaulBGD/status/129591880033708441...

tobltobs3• 6 years ago

This article sounds like sendgrid is the victim. But in reality this is going on since months. My account was hacked three months ago and abused by spammers. There was no way to stop it in the UI and the support didn't react for days. In the end I had to pay ~1k$.

I guess Twilio will close down sendgrid soon anyway and this is a way to monetize the last few months.

Stay away from Sendgrid and Twilio, they are incompetent and/or aholes.

rootsudo• 6 years ago

Yep, this happened to our ORG a while back, and Sendgrid took no responsibility. Wish I could go into details, but... they're finally implementing 2FA and revamping how they deal with identities.

Sendgrid under siege from hacked accounts

Related Stories