Google App Engine Silently Stopped Sending Email 5 Weeks Ago

https://code.google.com/p/googleappengine/issues/detail?id=12786

by markdown • 10 years ago

286 112 10 years ago

code.google.com

markdown11• 10 years ago

No change at https://status.cloud.google.com/

tedmiston• 10 years ago

I've set this thread to not be visible by anyone but yourself and Google.

Or not.

Klathmon• 10 years ago

a few comments later he asked for it to be made public again.

cyberpanther• 10 years ago

I had a Google engineer tell me once that the App Engine email api was never meant to send a lot of mail. That is why the quota is so low. I would recommend using a third party service to do e-mail.

Maxious• 10 years ago

I've been getting bounces on Google Apps hosted gmail for calendar notifications coming from calendar-notification@google.com

Email is hard.

pmlnr3• 10 years ago

For programmers, maybe. Running mail servers is a sysadmin task. (I mean no attack here, this is plain statement)

tshtf2• 10 years ago

Running a mail server is simple, even for programmers.

Running a mail server with reliable outbound delivery, and inbound spam filtering, is anything but simple.

chris_wot• 10 years ago

Which is why you pay Google to do it for you via AppEngine!

mikecb• 10 years ago

Sendgrid is a great alternative and not hard to set up.

SuperKlaus• 10 years ago

Also: mailgun

codenhagen• 10 years ago

Also: Mandrill.

foxylad2• 10 years ago

Yeah, but pricing is ten times Appengine's (actually more - recently we don't seem to be being charged for email at all). Amazon SES is the only service with a comparable cost that I've found. I'd love to hear of other alternatives at the 1c/100 level.

daveguy2• 10 years ago

At least a few reported that it was bouncing at the destination once they set up an appropriate handler. They are only "not being sent" when there is a url in the message and that is a big red flag on the receiving side (not much text but a url). It doesn't look like an App Engine problem. It looks like messages being identified as spam problem.

markdown11• 10 years ago

Not the case for me. No bounces being reported. In any case, gmail and outlook don't bounce messages for spam; They put them in the spam bin.

Nothing whatsoever changed in my code. I haven't touched it in a year, and all of sudden email stopped on the 1st of April.

There's nothing in the changelogs to indicate there was a change that should cause this behaviour. It's absolutely an App Engine problem.

https://cloud.google.com/appengine/docs/python/release-notes

daveguy2• 10 years ago

Interesting. I have been somewhat weary of the "machine learning for everything!" approach that google has taken recently -- i just don't think it's there yet. I wonder if a learning algorithm on the google side just started deciding outgoing spam and not giving correct feedback. It is very curious that no url works and with url doesn't. That should give straightforward repeatability (if they can simulate as your account). Have you tried sending to different target e-mail addresses during the test? I saw once it just worked and a bunch of stuff flew through. Also saw there was some question as to quotas. Did you get it clarified? Are you maybe above usage quotas with the regular e-mails so test e-mails are being stopped/delayed? Are test and production separate? It seems like if it were a google side issue there would be an uproar from the users.

djsumdog2• 10 years ago

No, gmail constantly drops messages without putting them in spam. That's actually a huge problem. Microsoft does the same. That's for e-mails that don't even contain any links!

jlees• 10 years ago

I ran into a similar problem a while back (using Heroku) --- after a brief foray into the world of spam filtering, it turned out to be caused by our domain being identified by SpamAssassin (IIRC) as a spam signal. We changed the wording of our emails and moved to a different subdomain and suddenly all the email got through. It was.. interesting to debug.

fencepost• 10 years ago

Reading through the thread, it looks like in the last few days it's been narrowed down to "messages which contain the URL of the App Engine instance silently disappear with no bounces or errors."

achant• 10 years ago

And only sometimes. The issue resolved itself on my app two weeks ago. My guess is there is an outgoing spam filter run amok in some of the clusters.

zitterbewegung• 10 years ago

I think there is not a spam filter but this is a rule to prevent people from setting up phishing scams or other deceptive emails and they are using an appengine instance to host their scam.

djsumdog2• 10 years ago

I wrote about this last year, but I feel it's still relevant. Spam filters, especially Google and Microsoft's (whose e-mail servers don't play by any of the standard rules nor do they send DMARC reports), are horrible over aggressive.

http://penguindreams.org/blog/how-google-and-microsoft-made-...

I have seen more phishing/ransomware e-mails come through mine (lots of malicious Javascript made to look like Excel files), so I can understand how great the risk is. The fact still remains though. E-mail is broken and is less reliable than an envelop dropped in a letterbox.

markdown11• 10 years ago

That seems to be the case for my app.

It was sending an HTML email with the appspot domain used for the logo: https://example.appspot.com/images/logo.png

Changing that to http://www.example.com/images/logo.png fixed the problem, but now I have to host the image elsewhere on a HTTPS enabled domain.

A separate question is what has google done with all the messages that users of this app sent? Are they gone for good, or stuck in a queue somewhere?

waleedka• 10 years ago

Must be an isolated issue. I run two App Engine apps and send 30K emails a day and I haven't noticed any change in my email open/click stats, which implies that emails are being delivered as usual.

Ensorceled• 10 years ago

Well, that is one of the problems I kept hitting when I was on App Engine ... issues that "only" impact 10% of customers.

markdown11• 10 years ago

An isolated issue that's been allowed to continue for weeks.

mattbillenstein• 10 years ago

Don't use appengine for anything -- ever. It's a terrible platform/product and google would rather you move on to GCE anyway...

enneff• 10 years ago

google would rather you move on to GCE anyway...

That's not true. We continue to invest heavily in App Engine as a platform.

groks• 10 years ago

Appengine's whack pricing says: we'd like to kick you off appengine onto GCE but that would scare away all the enterprise users from GCE. If we announce regular Moore's Law following price cuts for GCE and nothing for appengine, well hey, we can't stop you leaving. We're totally not twisting you're arm! (aside: we are)

"A commitment to Moore's Law: https://cloud.google.com/pricing/philosophy/ "

levemi2• 10 years ago

Isn't SnapChat built entirely on AppEngine? Why haven't they moved to GCE if it's so terrible?

jethro_tell2• 10 years ago

Jeez, if the Snapchat backend looks anything like the ux works this isn't much of an endorsement. Just because some company moves a lot of bits doesn't mean they know what they are doing. The easiest way to move a lot of bits is to be bad at infrastructure and have investors with deep pockets. Fuck just run a rack of open relays and bad NTP/DNS servers.

bkovacev• 10 years ago

Why? When you make a bold statement like this - provide some context and proof.

justinblat• 10 years ago

Greetings folks! One of the App Engine PMs here. That couldn't be further from the truth - we absolutely want people to use App Engine. If you want to manage your own infrastructure, Google Compute Engine is great. If you want something that scales automatically, and requires less management - App Engine is a great option. We've actually made a ton of changes recently, specifically on our support for docker based runtimes, new docs, and new runtime support (like nodejs, ruby, python3, etc).

https://cloud.google.com/nodejs https://cloud.google.com/ruby https://cloud.google.com/python .... etc

If you have any questions at all about the service, feel free to ask me anything.

breakingcups• 10 years ago

When will App Engine finally upgrade Go to 1.6? It's currently stuck at 1.4.

Also, why the long delay? Is it because of the switch from C to Go in 1.5?

martindelemotte3• 10 years ago

I wonder if people at Google know how awful it is to report an Appengine bug. You have to battle for days to get your bug acknowledged.

I've noticed that they've added more support people since the early days but it's still a pain as they seem to have an incentive to answer the tickets as fast as possible without doing any real investigation.

The linked issue is a good example of this behavior.

kordless2• 10 years ago

They should be working on a migration tool for users of AppEngine to migrate to Google Container Engine.

foxylad2• 10 years ago

We wouldn't use it.

To me, the whole point of Appengine is to abstract out system administration. GCE (like AWS) abstracts out the systems, but not the administration - you suddenly become responsible for all the scaling, fault-tolerance, and all the subsystems that Appengine manages for you.

stcredzero• 10 years ago

I wonder if people at Google know how awful it is to report...

No. (Or they don't care.)

spicyj• 10 years ago

They have paid support if you want it.

nikanj• 10 years ago

Paying for the service should be enough to get support.

jpatokal2• 10 years ago

Please remember that Google App Engine is entirely free up to (roughly) 5 million page views/month: https://cloud.google.com/appengine/kb/#quota

If you're running production services, you should probably sign up for a paid support package, which have guaranteed response times down to 15 minutes: https://cloud.google.com/support/

Disclaimer: I work in Google Cloud Support.

martindelemotte3• 10 years ago

I can't comment on the paid support. I've just noticed that the non-paid support engineers seems to be under pressure.

I feel little bit bad for them sometimes. They don't seem to have the time to look into problems. It must be quite a boring job for those assigned on the public issues.

Maybe their higher-ups aren't aware of that.

inopinatus• 10 years ago

I think this is a terrible attitude. I will never implement it in my own support model. People shouldn't have to pay me in order to report a bug in my system. Arguably I should pay them, because one report is the tip of an iceberg of silent affected customers.

More often than not, what most people want is simply an acknowledgement. They're not seeking a bug bounty. And definitely not the brush-off you just gave the GP: when someone's house is on fire, you don't demand prepayment for the water.

Animats• 10 years ago

This is what happens when you buy a service from an organization oriented toward ad-supported services. The mindset is that it doesn't matter if the user isn't being served.

ajackfox• 10 years ago

I'm pretty surprised by the long wait times between "updates" from support... that's pretty rough. I understand it can be hard to get a good signal to noise ratio with support, but when so many other people are reporting issues, perhaps it's time to take it a bit more seriously?

This thread is a great example of why support is so hard.

user_0001• 10 years ago

not really. I wish I could get away with providing such crap support as the big players do.

If my customers don't get a reply within a few hours, then all hell breaks loose. Even if, there request comes in at 3am / 0300 on a Sunday morning my time.

To be fair, when I point this out to them they do apologise, didn't realise time difference, are in a different part of the world where Sunday is not a day off etc. I then ask what they expect from some billion dollar companies support wise and they say they are happy with a reply within a week. Asking them why a one man shop doing quite a bit less than billions of dollars is expected to provide so much more.

Anyway, rant over. Support isn't hard. You hire enough competent people as your customers require. The end. Happy customers

Godel_unicode2• 10 years ago

Oh, is that all? Someone should let Google know.

awinter-py2• 10 years ago

The problem is that pesky humans can't decide what they want. Not sending mail is a bug but not receiving mail (i.e. spam filtering) is a feature? I can appreciate how this is confusing.

markdown11• 10 years ago

Not sending is a bug when it's undocumented, when it happens without notification, when there's no error message, and when there's no outgoing-spam bin.

The GAE docs say "do x and we'll send your mail". If we do x and mail isn't sent, then that is a problem.

Godel_unicode2• 10 years ago

Except as soon as you tell the spammer what is blocked and what isn't, they just add that to the playbook.

I agree with you that some measure of notification that there's spam filtering would be good, but it's Google we're talking about. There's spam filtering.

tominous• 10 years ago

Maybe I'm old-fashioned but if I was affected by this issue (and it had persisted for so long) I would set up an SMTP server and an appropriate MX record, then demonstrate using mail logs or port snooping that this was definitely not a receive-side issue.

Of course the support organisation here (Google) should be well equipped to set up this kind of testing themselves and work with a customer to root cause the issue. In my experience though these issues are rarely taken seriously until there is no-one else left for support to blame.

jpatokal2• 10 years ago

Google Cloud Support here. To correct the misleading title, this is not a generalized outage of any kind, but an issue with some applications sending mail that trigger spam/malware/etc filters. Since spammers generally do not comply with RFC 3514, filtering is unfortunately not an exact science, so false positives happen and we're working with the customers in question on a case-by-case basis.

Also, App Engine is integrated with both Sendgrid and Mailgun, and we strongly recommend using these if you're planning on sending larger quantities of mail:

https://cloud.google.com/appengine/docs/python/mail/sendgrid

https://cloud.google.com/appengine/docs/python/mail/mailgun

markdown11• 10 years ago

5 weeks ago your infrastructure stopped sending out mail. Our apps didn't change, your infrastructure did.

Whether it's your outgoing spam filter that's overzealous, or if a datacenter is on fire is irrelevant. Your infrastructure stopped doing what your docs said it would.

Paying customers like myself have apps running on your infrastructure. Your infrastructure stopped delivering the exact same emails that it had been delivering for years. It did this without notification and without triggering any errors.

Approximately 35 quote requests that should have been delivered have disappeared into thin air. You've cost me money, and I'm disappointed to find out that you'd been notified of this issue weeks before it began affecting me.

moondev2• 10 years ago

Why do you think services like sendgrid, mailgun etc exist? It's pretty common knowledge that if you need to send mission critical email, you should use a service specifically designed for that. Sending email programmatically directly from a web server has ALWAYS been a gray area.

At the VERY least you could have easily set up a gmail account and sent them from smtp. Choose the right tool for the job.

awjr• 10 years ago

I really have to highlight how dangerous an approach this is. You have no idea if the previous user of that IP was spamming. You should seriously consider mailgun or sendgrid. They also bring a lot of features that enable you to track when an email was read etc.

Confusion• 10 years ago

Infrastructure always changes. You need to have something in place to detect failure of important features of your apps, that get checked every day, every hour or even every minute, depending on how costly an outage is for you. There are always doublechecks possible.

cramsdale2• 10 years ago

Hey Folks,

The issue reported here is linked to App Engine and Gmail tightening up their spam filters. The root cause was an increase in organizations sharding out their spam systems to utilize App Engine’s free tier in such a way that is (a) in direct violation of our ToS and (b) making all of our lives suck a bit more (raise your hand if you want spam). It’s unfortunate that while App Engine is trying to provide a free tier that enables developers to easily use our platform, others see it as an opportunity for exploitation. Even more unfortunate is that it has a negative effect on legitimate users. It’s a fine balance that has been highlighted by several users within this thread.

Spam filtering is not a perfect science, and we’re constantly tweaking things -- with our customers in mind. This issue should be limited to new applications where the trust signal might be a bit lower. Thus existing apps / customers shouldn’t be experiencing issues (which was also highlighted by a few within this thread). If this isn’t the case email me: cramsdale@google.com. For those asking, “hey, why am I being penalized for being a new customer?” See my previous comment about spam filtering not being a perfect science. Then email me.

We’re here and we want to help.

-- Chris (Lead PM for App Engine)

je42• 10 years ago

Isn't there an API to find out why a message was SPAM filtered ?

Or are you afraid that "rogue" applications will use it to produce messages that are SPAM but yet not trigger the SPAM filter ?

jcrites2• 10 years ago

The recent post about Candy Japan's attempt to combat credit card fraud gives a good sense of what fighting fraud and abuse is like: https://news.ycombinator.com/item?id=11431881

As an email service provider it's like that and more, since (1) once an abuser uses your service, they've gotten the benefit immediately and keep it even if their account is discovered as fraudulent later, e.g. stolen CC number & chargeback. (2) Abusive users can directly harm good users such as by harming the deliverability of the overall platform. It's not just bad debt, it's bad experience too. (3) Unlike Candy Japan where fraudsters mostly just wanted to check CC numbers and not actually buy product, email abusers really want to send emails (4) It can be hard to tell good and bad senders apart because some companies with an internet presence aren't email savvy and might make mistakes or might get hacked.

Spam filters are always tough because if you give someone transparency into which actions of theirs that you consider abuse, then they will quickly detect and route around your attempt to block them. (See Candy Japan article) It's pretty easy for a human to guess what might be the sign of their fraud and run a few experiments to see what gets flagged e.g. By comparison a machine learning system might be hard to outsmart, but then it's also challenging to explain and troubleshoot false positives. Hence what's effective is often a combination of machine-learned filters and heuristics along with manual overrides by human judgment.

All other things equal, new users are a lot more likely to engage in fraud than existing ones, and so tend to be under more suspicion. Aside from B2B fraud where companies take out lines of credit and then go bankrupt intentionally, it's uncommon for existing established customers to turn fraudulent - they're already vetted. (Consider: who is more likely to be fraudulent. The first time subscriber to Candy Japan, or a subscriber who has been using it for 12 months and is about to buy their 13th month?) It's not a great experience as a new user to be under suspicion, but if it's temporary and easily overridden by a human it can be a decent trade-off - the need to reach out acts a deterrent to spammers but does not deter legitimate users as much (speaking generally).

fpgaminer• 10 years ago

Just a quick thank you for the free tier that App Engine offers. I use it to provide an educational app for my wife, who's a teacher, that she uses as part of her classes. It's nice to be able to provide that to her and her class, without hitting the bank (the school does that enough...). So, thank you.

brianmcconnell• 10 years ago

Long time App Engine user here. If you're sending email as part of your app, I'd recommend using Mailgun (or similar services). You can set up DKIM, inbound routes, etc, and generally have more control over how your mail is processed. I never got the impression App Engine really wanted to be in the email delivery business, so if that's important to you, there are tools that are focused on that (and free up to a monthly quota). My $0.02.

siva7891• 10 years ago

I completely agree with your recommendation on Mailgun. Using it for past 4 months, never had a problem with their service.

wastedhours• 10 years ago

+1 as well, after the changes to Mandrill recently, was looking for a new provider and can also recommend Mailgun (at least, for ease and configurability in set up and testing so far, not yet in production).

metachris• 10 years ago

I've recently compiled a list of free transactional email services with a feature comparison: https://www.metachris.com/2016/03/free-transactional-email-s...

manyxcxi• 10 years ago

Love Mailgun. You can configure it to just be a very easy mail provider or use the APIs to send mail and turn on all kinds of things like campaign monitoring, tags, click opens, etc.

The best part for me when using it for customers has been the bounce/click through rate tracking. When someone asks me, "how do I know they got it?" It's incredibly nice to point them to the dashboard and show a less than 1% bounce rate (people putting in bad email is almost always the reason) with a log of every single email sent.

Most my clients get this service for free because their volume us low enough. They have quite a generous free tier.

elbigbad• 10 years ago

Thanks for the suggestion! I was having issues with SendGrid and had never heard of Mailgun. Took literally 2 minutes to get set up on Mailgun, add the credentials to my app, and send my first email! It's a super small app so don't need anything fancy, Mailgun is perfect.

edit: Not to say Mailgun isn't fancy, I have no idea if it is or not, I can say that it works in minutes though.

jcrites2• 10 years ago

Possible reason that email service providers and other email sending services might see an increase in new signups and free tier use recently: http://blog.mandrill.com/important-changes-to-mandrill.html Mandrill in essence raised their price, eliminated free tier, and required prepurchased blocks of sending capacity IIRC

The numbers Mandrill released about that business suggest that they had a large number of low volume senders, who may now be looking for a new home.

markdown11• 10 years ago

Hi Chris,

This issue should be limited to new applications where the trust signal might be a bit lower. Thus existing apps / customers shouldn’t be experiencing issues.

1. My app is not new. It's been running without issue for 2 or 3 years.

2. My app is not on the free tier.

3. On average, my app sends out under 8 emails a day, and it's done the same for over 2 years.

How your algorithm considers my app a spam risk sure beats me.

I'm missing 11 days worth of quote requests from customers. Are these recoverable? (is there a hidden outgoing-spam bin?)

My app started sending mail again today after I changed the src of an image in it from https://example.appspot.com/images/logo.png to http://www.example.com/images/logo.png

How well do you think email clients are going to like emails with images embedded in them without HTTPS?

tim333• 10 years ago

You might look at Sendgrid. It's free for those sort of volumes

klewandowski• 10 years ago

Hi, if you haven't already sent Chris an email, can you send me the details of your account and I'll get answers for you today? lilkim@google.com

JackWebbHeller• 10 years ago

Chris, you say Gmail tightened up their spam filters... Did this take place in the last ~6 weeks or so?

I've noticed the spam filter on Gmail for Google Apps has gotten significantly less accurate recently, resulting in far more false-positives than usual. Any ideas if this is a known issue? I can only presume it's more accurate overall, but it was definitely a noticeable change for our organisation.

emdd• 10 years ago

Yep, this has been an unfortunate thing for us as well.

hartator2• 10 years ago

This is a documented API endpoint: https://cloud.google.com/appengine/docs/python/mail/ with quota.

samskeller• 10 years ago

How could you loudly stop sending email?

Google App Engine Silently Stopped Sending Email 5 Weeks Ago

Related Stories