Heroku Postgres Official Maintenance

https://status.heroku.com/incidents/510

by rdegges • 13 years ago

50 49 13 years ago

status.heroku.com

falcolas• 13 years ago

A good reason to take a few minutes and make sure your backups are up-to-date and working, and that your application won't blow up in unexpected ways when the DB is unreachable.

neya2• 13 years ago

I'm yet to see an update/fix on Heroku's recent PR nightmare on the random routing issue. Anyone?

coolsunglasses• 13 years ago

I'm not sure why this point isn't being made, but they're not going to change anything about how the routing works, least of all for Bamboo.

Heroku is past the point of caring about pushing the envelope, stateful routing is an infrastructure risk and overhead that primarily their least informed customers need or want.

It's a high margin operation is a low margin business, they're trying not to fuck it up.

So stop asking.

neya2• 13 years ago

As a customer, no matter what they do, whether it's a high margin risk or low margin profits, I'm supposed to be informed of their decision - whether they try to fix it or not, because I was one of the affected. They promised they would do so in their previous blog posts, hence the curiosity.

>So stop asking.

I have every right to ask and I will rightfully so. A disguised internet profile about sunglasses means nothing to me.

joevandyk• 13 years ago

They are about to push an update that lets you run larger dynos, which reduces the risk of having all the requests to a dyno stuck doing something long-lived.

redguava• 13 years ago

I believe the larger dynos are mostly to better handle using Unicorn as the webserver. Unicorn allows multiple workers to run per dyno, and has it's own smart routing, so it compensates well for Heroku's routing system. The downside to Unicorn was that the workers share a dyno's memory and often you couldn't have more than 2 per dyno with 512mb memory. This change helps that a lot.

dserodio2• 13 years ago

They've put together a FAQ about routing: https://blog.heroku.com/archives/2013/4/2/routing_and_web_pe...

pilif• 13 years ago

So. Us normal users will get the security fix on Thursday, but Heroku is somewhat special and gets early access? What does one have to do to become special?

I don't like how this is being handled, no matter how serious the issue. I could kind of understand what the postgres team was doing until this point where they show that while everybody is equal, some are more equal than others and get early access to security fixes.

Doing this does nothing but motivate the bad guys even more to find the hole on their own.

The initial announcement got me scared already, but this just made it worse.

mrkurt2• 13 years ago

Heroku is special, they host a huge number of Postgres setups that are publicly accessible and somewhat easily discovered. It's not a matter of equality, though, it's one of trust/privacy. Heroku is an easy company for the Postgres guys to trust since they have a pre-existing, healthy relationship.

tensor4• 13 years ago

Meanwhile they are holding back a security fix for numerous other companies that also take security extremely seriously. This creates a situation where companies considering posgresql will now have to ask "will I get security fixes as soon as they are ready or will I intentionally be left vulnerable while more privileged users get early access?"

Not a good precedent as far as I'm concerned.

bdittmer• 13 years ago

iirc Heroku has several full time employees who have commit access to PG. ie they're paying for some of postgres's development.

fdr• 13 years ago

False (I work there), although some of my colleagues have submitted patches.

Commit access is handed out very selectively at postgresql.org, and committing patches is not the bottleneck in making features (reviewing and writing the patches is).

seany• 13 years ago

There are similar problems with XEN. aws and linode getting early access etc.

benmmurphy• 13 years ago

this is happens with other opensource projects as well. you just don't hear about it. if you have contacts in the project then you have a good chance of getting early patches.

ig1• 13 years ago

This is true for most software vendors. In particular companies which make IDS software, control large number of deployments or are in highly-targeted sectors (defence, finance, etc.) will tend to have arrangements with software vendors to get early access to patches/vulnerability information.

Typically to get that type of relationship you need to show the vendor that you have a real need for early access and that you have strong internal controls that would prevent any information gained from early access from leaking.

breser3• 13 years ago

This is not an uncommon practice at all. There are other projects that provide early notifications to large vendors. Part of the point of that early notice is so binary packages can be available. Packages can't get built, tested and released alongside security announcements without someone getting early notification. Users are in a much better position if the binary packages are available when the announcement hits than if the announcement hits and the binaries are available a week later.

If the issue leaks out early or someone independently finds it they're just in that much better of a position to release the fix. Vendors will be closer to having packages ready.

Yes, it's probably not entirely fair. But there's no way to tell everyone without telling the bad guys too. The risk of telling select people to get the ball rolling is worth it if the entire ecosystem is ready when the announcement comes out.

Dealing with these issues is always going to be about managing the risk. You can't draw a line in the sand.

tensor4• 13 years ago

I thought the point of the delay was so that people could not reverse engineer the patch through the source or binaries? I suppose with Heroku being a service it might be the case that people don't have access to the new patched binaries. If that's the case, then there is no issue.

vetrom• 13 years ago

Responsible versus full disclosure is a debate rooted in problems far older than HN. At least we get to see security issues in the light these days!

sneak• 13 years ago

You are not entitled to information just because you think you are, or because they have a tradition of publicizing information.

Lose the sense of entitlement.

MichaelGG• 13 years ago

> Doing this does nothing but motivate the bad guys even more to find the hole on their own.

Please explain the logic here. How does Heroku getting early access provide more motivation to discover vulnerabilities?

gcr• 13 years ago

How do we know that this "brief but important update" is related to the security fix?

dserodio2• 13 years ago

Timing

gingerlime2• 13 years ago

I was just recently considering whether we should move our postgresql over to Heroku.

On one hand, it sounds promising that they deal with important security updates, and can do it even before the fix is officially announced. Very impressive.

On the other hand, not being able to schedule the downtime to fit with your app/userbase is quite annoying. I understand this could be a potentially big security vulnerability that needs urgent fixing. But each organization is different and their risks are therefore different. Some people might prefer to take the risk of a few hours delay to apply the fix (being exposed) in exchange for not having to experience unscheduled downtime of this nature... It seems a little awkward to me that Heroku is taking this decision for its entire customer-base in such a way.

ryan_f2• 13 years ago

This is rather disconcerting. They are given us a window of 48 hours where our site will be down for 60 seconds. On top of that we cannot schedule the time with them.

60 seconds may seem like a small inconvenience but it is one still the same. I host a few clients on their servers and I need to relay this information. I sound like a jackass being that I can't give more specifics on the timing of the server being down to my clients. How does Heroku not feel the same?

I am appreciative of the update but that is also what I pay them to do.

fein• 13 years ago

> How does Heroku not feel the same?

Because they, as a hosting provider, have several orders of magnitude more clients to please than yourself. The only way to be fair to everyone is that everyone gets treated like shit (obviously embellished).

All you should be doing to CYA is telling your clients exactly why this is happening and that its Heroku, and not you, that holds the blame at this point.

The caveat would be those difficult customers (suits, exec managers, etc) that you have to handle with kid gloves. For these people... I don't think there's a graceful way to present the upcoming issue.

gingerlime2• 13 years ago

I doubt customers typically know or care whether you use heroku, Amazon or any other provider. What they want is to see their system up and running.

That said, I wonder if with the heroku postgresql setup, this vulnerability could mean that if even one db is compromised, it could allow access to other databases too. Maybe heroku runs several postgresql db's on the same virtual/physical hosts?. If that's the case, then heroku simply can't afford to let even one database stay exposed because it would risk all others.

Whether or not this is the case, I don't know, but it kinda feels like running your app on a shared-hosting account...

ryan_f2• 13 years ago

I understand they have more clients to please than myself. Even a smaller window of time would be helpful or to know it is going to be done in off hours. There's no reason I should be fine with this as a paying customer.

thruflo• 13 years ago

It may be de-rigueur to bash Heroku in the wake of the Rap Genius drama/revelations but this kind of "downtime to apply critical security patch" announcement is precisely why I choose them.

I spotted the announcement of the postgres vulnerability and thought "sounds serious". Now I read that my production databases will be fixed as soon as the patch is available without me having to engage with the details.

nevinera• 13 years ago

>Due to the nature of this update, a scheduled time is not possible.

What, it takes place outside of linear time?

This crap is unacceptable. Even my $2.00 shared host tells me when they're going to take down my site without asking.

viraptor• 13 years ago

More likely it takes around 5min per host if everything goes right. If they have 100 hosts and get delayed on the 10th and 15th for some reason, all other scheduled times would get shifted... And the upside is that they spot failures as they appear, rather than "oops, the last 5 hosts we did at the same time do not start up, there's some big issue".

</speculation>

JuDue2• 13 years ago

Just as an aside, how good is Heroku's Postgres performance, speed wise?

guywithabike• 13 years ago

Don't ask for anecdotes -- test it yourself with conditions similar to those that you'd be operating under.

JuDue2• 13 years ago

Hey, reviews and anecdotes save time. Everyone does it.

Heroku Postgres Official Maintenance

Related Stories