Today's Email Incident

https://github.com/blog/1440-today-s-email-incident

by buttscicles • 13 years ago

256 171 13 years ago

jstanley• 13 years ago

I'm impressed. I thought it much more likely that somebody carelessly pasted a bunch of addresses into a "To" box instead of "Bcc".

sergiotapia• 13 years ago

This is bad - real bad. Goes to show you if the 'golden boys' can make mistakes such as these, what about the other 96% of companies handling your info?

Trufa• 13 years ago

I don't see what's bad about it, well except for the bug itself.

Computer software is bound to have bugs, and if you ask me. a company that discloses those bugs, thoroughly explains the cause, addresses how to avoid them the future and apologises to their customers is on the right track.

Don't get me wrong, you software should be very stable, but when the inevitable happens, well this is the way you address it.

markmac• 13 years ago

Bad incident, great transparency and handling. Just don't do it again please... :)

RexM• 13 years ago

Would it make sense to just always put the recipients in the BCC field to ensure you don't accidentally leak email addresses if/when something like this were to happen again?

swanson• 13 years ago

"putting them in the To: field to enhance deliverability"

famousactress• 13 years ago

Yeah. Is there a strong argument against sending separate emails though? Selfishly curious because that's one of the things we've been doing to prevent this sort of mistake.

polemic• 13 years ago

Not really. They thought they were emailing administrators at the one organisation. From the recipients point of view, it would be much easier to deal with if you saw who it was going to. A bunch of BCC recipients within the same organisation is going to lead to confusion.

pud• 13 years ago

I previously assumed the cause was some marketing intern pasting email addresses into the wrong form. Better training (and a slap on the wrist) would prevent it from happening again.

But it's somewhat scarier knowing this was due to a bug triggered by a security patch. Who knows what yet-undiscovered bugs are still in there.

That said, there's no such thing as bug-free software and I still find GitHub to be trustworthy.

727374• 13 years ago

I don't think anyone doubts github's trustworthiness, but as an enterprise customer, I do wonder if they have tight enough controls. Marketing emails are an area that companies should be spending extra time testing as those screwups tend to be irreversible.

chadscira2• 13 years ago

a little more on the incident http://chadscira.com/post/5148b1f0f93e934b550112be/Github-le...

dbpatterson• 13 years ago

atlassian? :)

chadscira2• 13 years ago

i bet github grinned when they saw that email popup

jschumacher• 13 years ago

Atlassian builds a JIRA connector for Bitbucket and Github:Enterprise. To test the connector, an installation of Github:Enterprise is required.

Internally we use Stash of course ;)

blarghnox• 13 years ago

http://pastebin.com/JX1PdiMB

susi22• 13 years ago

You'd think the .edu addresses would go to bitbucket and get free unlimited hosting.

amadeuspzs• 13 years ago

Non-technical companies which do this tend not to apologise - it draws attention to the mistake which most of their (non-technical) customers never noticed in the first place.

I guess github doesn't qualify for the non-technical crowd :)

ghuntley5• 13 years ago

Very concerning - they accidentally distributed their entire customer list and now I have to handle the internal PR fall out re: my email address and that those of non-technical co-workers (board, accounts, finance, engineering managers) have been distributed globally and then subsequently pastebined. Times like this I'm glad that our code is hosted internally and access to it is controlled internally.

https://dl.dropbox.com/u/109905580/Screenshots/qg~abn4fl601....

https://dl.dropbox.com/u/109905580/Screenshots/73dw6%7Eq_h4h...

eli2• 13 years ago

I wouldn't normally consider my email address a secret. I assume the concern is over spear phishing?

ghuntley5• 13 years ago

Email address isn't secret, the use of certain tools for some however is. Enterprise is a different game than your usual early stage web/product startup. Politics are real, exist and credibility is on the line when you stick your ass out to vouch for a company/product outside of the company standard. Base cost for this software is $5k/yr for one seat pack (20 devs), whereas the competition - Atlasssian is incredibly price competitive (10 users for $10, 25 users for $1,800) - keep in mind competition also exists in the form of TFS (ergh) which is free for MSDN subscribers and included in most enterprise licensing agreements.

robryan• 13 years ago

If a competitor is looking to market their version control solution to enterprise this email list is going to be a very good place to start.

robinson-wall• 13 years ago

I've already received an "invite to check out Dropbox" to an email address that only GitHub knew.

revelation• 13 years ago

It looks like you printed that email and then splashed some water on top. Why would you link to that in the first place?

ghuntley5• 13 years ago

Proof that I was one of the enterprise customers who had their information disclosed and thus is directly affected by this unlike 99% of the other posters in that run-away lol ruby thread which is unfortunately at the top of HN.

badgar5• 13 years ago

Rails security point release breaks apps by changing the ORM.

News at 11.

stanleydrew2• 13 years ago

I'm not sure why you found it necessary to post this comment. I found the original article to be a well-written warning about a rather serious bug that might affect a lot of people. But your comment seems to be sarcastically suggesting "move along, nothing to see here."

mbetter• 13 years ago

I'm not sure why you found it necessary to post that comment, or why I found it necessary to post this one.

jleader2• 13 years ago

I don't think the GP is criticizing the article, he's using the article to criticize Rails. Not so much "nothing to see here" as "well, what did you expect from Rails?".

(I'm explaining what I think the GP meant, not my position; I'm an old Perl guy, still undecided on the overall trade-offs offered by Rails)

obstacle1• 13 years ago

I took it more like "oh look, another Rails security flaw, big surprise". Not "move along, nothing to see", but rather "take pause and reconsider using Rails".

rtwtlkj5• 13 years ago

I wonder if this is Karma for all the hyperbole and bad nature comments that the Rails community has thrown at Java. This is some really good schadenfreude. Please keep them coming Rails/Ruby.

bhauer• 13 years ago

Recognizing that it is not politically correct in these parts to pose this question, I sometimes wonder if the Github team (which seems like a highly-skilled, competent team) regrets building on Rails.

It seems they have been able to use their high skill and competence to laboriously steer a bulky and (dare I say it?) heavily-engineered framework to serve Github as the modestly fast and fairly reliable site that it is.

But would not they be happier had the efforts exhausted on scale, reliability, and repair been available to spend building features had a faster, less confusing platform and framework been selected?

I know, it's antithetical or heretical to imply that Rails is anything but a model of efficiency.

sitharus4• 13 years ago

I've stopped building on Rails because of things like this. Point releases shouldn't break functionality. I got tired of regression testing my apps for framework changes.

voidlogic3• 13 years ago

When I read the blog, the first thing I thought was "boy am I glad I don't use rails". I realize all frameworks and toolkits have issues, but Rails seems to be particularly bad.

I think the big take away here is the importance of apparently redundant sanity checks....

CamperBob22• 13 years ago

I got tired of regression testing my apps for framework changes.

I get tired of paying taxes and brushing my teeth, too, but I still do it, because life will get worse in a hurry if I don't.

33degrees• 13 years ago

To be honest, I've had a lot more problems with gems external to rails than problems with rails itself; and this particular case seems like an issue that should have gotten caught via testing.

niggler3• 13 years ago

"regrets building on Rails."

I'm surprised they haven't tried to build their own framework / programming language (like Facebook with HipHop). It wasn't a bad idea to start with rails, but now they subject themselves to the issues of Ruby and Rails without an alternative

nonamegiven• 13 years ago

I sometimes wonder if a good alternative to django/rails is to use the most minimal framework that's reasonable to start with, and then gradually customize it away.

jaggederest2• 13 years ago

Why would bringing a framework in house be better than using open source?

I'm confused by all these people that thing 'Rails is crap, clearly people should roll their own everything since that will be more reliable and better engineered'

If you think Rails is crap, the only answer is to look to 'more mature' platforms, not roll your own. Spring/Hibernate might be a reasonable answer. Django is not, being at a very similar level of maturity. Nothing in PHP even begins to approach the level of maturity.

Software engineering is hard. Really hard.

tptacek5• 13 years ago

When Github started, they didn't just use Rails: they used Rails and shelled out to command line programs to integrate with git. If you have misgivings about engineering at Github now (I don't; I have the opposite of misgivings about they way they handle software), imagine a site built in the equivalent of shell scripts.

I think the short answer is probably, nobody at Github has a lot of macro-level regrets at this point.

I think the real issue is that the Rails community should be supremely grateful that they have a team like Github's building on the framework they've chosen. They all get to draft off the deployment/security/development/reliability work Github sinks into the platform.

And, painful though it will surely be for HN to hear this, snags like the one today are how frameworks eventually get to be solid. Spent much time looking at Java Spring's track record?

phillmv• 13 years ago

I think people forget how transformative Rails was. When it came out it blew people away. We all make fun of the 15 min blog today, but at the time it was really impressive. Nowadays it's not so amazing by comparison because everyone's caught up.

Honestly, I think it's great that this all gets aired out in the open. The amount of transparency behind all of this is a breath of fresh air.

Rails has had its share of growing pains; it really sucks that a security patch had a regression and I think this gave people a lot of flash backs to the pre Rails 3 days. By the end of this, however, we'll end up having a solid security infrastructure.

I'm willing to wager that 90% of the time these kinds of issues just get swept under the rug and silently patched.

dubcanada2• 13 years ago

Github is not a website in the normal sense. Github is a layer over top of Git. One of the best wrappers for libgit/what ever there is (regardless of Language) is the Ruby rugged library which Github pretty much built. Before that was grit which was also Ruby based. And what Github was built on.

Everything else is just extra they added on. Their core product is a Web based Git.

But I sometimes wonder this as well, I love Ruby a ton. But I wonder how Github would have done had they used another language PHP, Python, Lua what ever.

But then again the same thing can be said about Facebook, would they need less servers if they switched to another language?

danenania3• 13 years ago

It's become fashionable to hate on rails, and honestly I understand the motivation--it isn't cool and cutting edge anymore. It's slow. It doesn't do concurrency well. Ruby is nice for what it is, but it lacks the power and stability of functional languages, and lacks the performance of go or javascript on v8.

But rails is just doing what it's always done: trying to optimize developer productivity for front end web dev and routine web tasks. In this it's far ahead of the pack, and for many businesses, both small and large, this is exactly the right thing to optimize. Say what you want about github, but they push a lot of features and push them fast.

Rails is not for crafting algorithms, but when you need to crank out 6 new pages with various complex crud views and forms by Friday, and by the way can it be exportable to cvs and oh can you also do paypal and oauth integration and user avatars with thumbnails, and keep it all reasonably readable and organized... the sad fact is that for all of rails' heft and lack of cs cred, trying the above with any other framework after you know rails will drive you insane, because despite how great the language is or how many connections your server can handle, you are again and again spending days on things that take an hour in rails.

Ultimately, I don't think the way forward from rails is to search for a partial reimplementation of rails in some other language. Rails is the best we've got for rapid front end and shallow back end work. So use it for what it does well. For what it doesn't do well, use something else. Personally, I think it's silly to have an ORM written in ruby. ActiveRecord is for the most part a lovely api, but its guts should be a separate service written in go or something. Building a complex recommendation engine or game AI? Write clojure or haskell services. But when the data comes back and you need to dice it up and lay it out as html, with a hundred little ohs and ifs and buts, it's not a fun time without rails.

eru• 13 years ago

Intereting. Your arguments sound a lot like the usual arguments in favour of PHP.

rtwtlkj5• 13 years ago

>>It's become fashionable to hate on rails, and honestly I understand the motivation

Well, they were pretty vocal about calling everybody, in no uncertain terms, for using frameworks other than Rub/Rails, an idiot. There is plenty of bad will that they've been accumulating because of that and this is simply the backlash.

If you really think you are the best there is no need to insult developers who do not use your frameworks. Unfortunately that is exactly what they did. People have long memories.

And in my opinion they deserve every bit of it.

The comment below by user mhartl demonstrates the attitude of pretending everybody else is shit.

>>Ah, so following Greenspun's tenth rule [1], perhaps we

>>have this:

>> Any sufficiently complicated web application contains

>> an ad hoc, informally specified, bug-ridden, slow

>> implementation of half of Ruby on Rails.

>>[1] http://en.wikipedia.org/wiki/Greenspun%27s_tenth_rule

The nerve to call other frameworks bug-ridden as if their shit didn't stink. Specially in light of all the security problems with Ruby/Rails that have come to light recently.

tomjen33• 13 years ago

I don't care about cool or cutting edge or even performance. I hate rails because it is built by people with no understanding of security.

But most of all I hate it because it uses so much magic.

sc0rb5• 13 years ago

All that stuff by Friday... I can do that in Java / Spring. It will also be high performance, extendable and maintainable.

Rails brought very little to the table in that respect. Not to people who actually know what they're doing anyway.

jrockway4• 13 years ago

Rails isn't the problem. If it wasn't this bug that caused the emails to be exposed, it would have been some other bug. The problem was that there wasn't enough test coverage, and there weren't enough runtime assertions to shut off failing software. (if to_emails > 10: logging.fatal("Sending message to too many emails %s", to_emails))

kaoD2• 13 years ago

So if I'm driving my car and the accelerator pedal gets stuck... it's my fault for using the pedal without proper extensive testing? Even if the pedal got stuck after fixing my door locks because the manufacturer stuck the pedals as a side effect with no previous notice?

There is no excuse: change of behavior shouldn't happen when installing security patches, specially if it's not mentioned anywhere in the patch release notes (which were reviewed by the GitHub team).

Okay, I get your point, they should have tested it, you're right, but saying that Rails is not the problem is kind of being in denial.

tedchs• 13 years ago

FWIW, popular advice for TDD is "don't test the framework, test your own code".

ecopoesis• 13 years ago

And this is why you shouldn't use ORMs. Magic blackboxes always bite you in the ass eventually.

drivebyacct24• 13 years ago

Yeah, why use SQL or language bindings, just write into the B-trees themselves.

thrownaway2424• 13 years ago

Because there's no middle ground between ORMs and packing your own varints into registers.

AaronBBrown• 13 years ago

https://yourlogicalfallacyis.com/slippery-slope

tptacek5• 13 years ago

Yes I would definitely trust an application much more if each and every one of its SQL queries was lovingly hand crafted by a human being.

Because if there's one thing computers are truly bad at, it's transforming high-level things into lower-level things.

martinced• 13 years ago

As an expert on security, what makes you think that in 2013 every webapp out there should still be backed by SQL and running SQL queries?

I mean is SQL something inherently so secure that all webapp devs should use?

If you're not a DB expert, what would be your rationale for using SQL instead of other alternatives?

It's not like if every website out there was using SQL. Heck, not even all sites are using OO languages.

So while I agree that an ORM is nice if you're stuck that particular Java/C# + SQL hell I fail to see how exactly OP did imply that he recommended objects and SQL...

He said ORM sucked. Not that he was using "objects" and SQL...

JamisonM• 13 years ago

If there is one thing computers are in fact bad at, it is understanding intentions, an ORM in many cases is a translation of an intention from one level to another.

In my experience lovingly hand-crafted SQL for your application does not take as long as you think it will and does not subject you to the interpretations of an intermediary. ORMs are useful but there is something to be said for writing the SQL you want to do what you want with a database.

cgh2• 13 years ago

Until someone gets the bright idea to push all your complex queries down into stored procedures. Then it becomes a mess, the dba quits, you go back to an ORM, have some weird issue like this Rails bug happen, then someone says, "Hey, what about using raw SQL...we'll just make the gnarly bits into stored procedures..."

Circle of life, man.

banachtarski• 13 years ago

Is this sarcasm?

chj• 13 years ago

Frameworks are innocent.

modarts• 13 years ago

Seing as how this list of customers are about to run out of their trial for Github Enterprise, it'd be funny if atlassian or any other hosted Git providers were to make use these emails for their own gain.

erikpukinskis• 13 years ago

This is potentially a pretty big security hole for apps, right? For example:

    posts = current_user ? current_user.posts : Post.public
    posts = posts.where(:user_id => params[:user_id]) if params[:user_id]

In the old Rails, that code is guaranteed not to show any private posts. But with the new Rails it'll show you everything written by a user.

Granted, it's pretty awkward code, but I can imagine situations where people might be inclined to write something like that.

pjungwir6• 13 years ago

I think that sort of code is extremely common. It's a bit less awkward if `current_user` is never nil, but sometimes a non-saved Guest record, letting you write code like this:

    @posts = current_user.posts
    @posts = @posts.where(user_id: params[:user_id]) if params[:user_id]

swampthing• 13 years ago

In case anyone's interested, I think this is a Github Issue on the same bug:

https://github.com/rails/rails/issues/9813

pjungwir6• 13 years ago

Thank you for posting that (one of the only informative contributions on this whole thread). That issue description is worrisome because the problem actually isn't limited to `default_scope`, which many people avoid, but also affects just `scope`.

IgorPartola• 13 years ago

How would one unit-test an app like this, where you have to talk to an external database and over SMTP at a specific time of day? Seems like in a case like this thorough unit testing needs to be very extensive to catch a bug like this, yet it would have been worthwhile.

boyter• 13 years ago

You would only need to unit test the programmer written ORM code against a seeded database to be sure you are getting just the data you expected.

You probably wont catch new issues with the ORM (which you should not be testing) but you will ensure you wont put this bug back into production.

losvedir2• 13 years ago

Wait, so what exactly is the situation that's at issue here? We upgraded yesterday, too, and have both scopes and class methods that we append onto has_many/belongs_to relations, but haven't experienced anything out of the ordinary. Now I'm terrified we have some unexpected behavior lurking somewhere that's not tested or something.

> In this case, when using the scope method to define an Arel scope on Organization, the where clause of the scope is overriding the condition imposed by the Organization#teams association. The part of the WHERE clause meant to restrict the query to Team records related to the Acme organization was dropped.

This doesn't happen in all cases -- all the cases where I've done this appears to be fine. And a scope just blowing away an association in all cases would be too obvious to not be caught.

Is it because both the scope and the relation select by the same attribute? Does that attribute have to be `id` for this bug to occur?

Or does it have to do with the fact that they're nesting a query within the where clause to determine the IDs? (I don't see why this would matter).

I guess I'm off to a REPL unless anyone here knows more concretely what the issue is.

EDIT TO ADD:

I can't reproduce this behavior. I just spun up a new Rails 3.2.13 app, and created those two models. Going through what they do in the console, my results differ on the potentially dangerous part:

     1.9.3-p0 :015 > teams = acme.teams.using_octocats_scope
     Team Load (0.3ms)  SELECT "teams".* FROM "teams" 
     WHERE "teams"."organization_id" = 2 
     AND "teams"."organization_id" IN 
       (SELECT id FROM "organizations" 
        WHERE "organizations"."has_octocats" = 't')
     => []

I do get an empty array, as is supposed to be the case. I notice that my nested select remains a SQL statement, rather than being evaluated to '(1)' as in their example.

So... I don't know if I'm doing something wrong, or if GitHub has other code / configuration settings interacting strangely here.

Can anyone else reproduce?

EDIT AGAIN:

Here are the four files you need to reproduce yourself. https://gist.github.com/losvedir/5202121

ONE MORE EDIT:

https://github.com/losvedir/test_github_thing

There's a repo with my history in the console. Dunno why I'm seeing the behavior I am.

pjungwir6• 13 years ago

I can reproduce it. I'm very worried also because I love using scopes. Here is a repo that shows the problem:

    https://github.com/pjungwir/scope-error

I think it is actually a mite simpler than the gist in your "EDIT AGAIN."

losvedir2• 13 years ago

Yeah, very strange. I saw your comment so I decided to put mine on github, too.

    https://github.com/losvedir/test_github_thing

I'm going to bed now, so I can't really compare mine to GitHub's to yours right now, but it seems to me that sometimes it's a problem and sometimes not. I dunno...

grey-area2• 13 years ago

SELECT "projects". FROM "projects" WHERE "projects"."user_id" = 1 AND "projects"."user_id"=5

This expected SQL doesn't really make sense to me. It would be better if your example didn't include selecting twice on the same column (not a join) with an equals clause, as I'd expect that to be removed (it's redundant and would always produce zero results). So the resulting sql makes more sense (replace where clause when new where clause uses same col):

SELECT "projects". FROM "projects" WHERE "projects"."user_id" = 5*

Looking at the github example, they used xx=1 AND xx IN(1,2,3), which makes more sense as it could possibly produce results, and I see why they were doing that.

pjungwir6• 13 years ago

I whipped up a quick example Rails project to demonstrate the issue. It's a lot like the Github post, but a little simpler: for instance, there's no real reason to talk about class methods:

https://github.com/pjungwir/scope-error

I'm pretty concerned about this bug, because scopes are one of my favorite Rails features.

teyc2• 13 years ago

A modern mail server should have features to prevent too many users from being present in the To: field. This will also prevent a lot of the Reply All problems we see regularly. This can be implemented as part of a Quota management system.

vinhboy• 13 years ago

Wow, I had to scroll a long long long way to finally get a reasonable suggestion on how to fix problems like this. A problem that potentially EVERY software has, regardless of framework or language. Kudos to you.

pilif• 13 years ago

Exim does this since forever (http://www.exim.org/exim-html-current/doc/html/spec_html/ch-... - look for max_recipients). I would assume other MTAs do so too.

Problem is, options like these feel nice until they annoy you because you do in fact need more recipients. At that point you build a workaround into your software to send a new mail for every max_recipient users which brings you right back to square one with the additional technical dept caused by potential bugs in your batch sending code (off-by-one errors for example)

Be careful what you wish for :)

teyc2• 13 years ago

Just like one can have safe_parse() and unsafe_parse(), there could be an elevated API that could be safely used when vetted. For instance, when the SMTP server rejects a request it could put it in a special queue which requires manual confirmation.

This type of problems happen so regularly. In fact, a Japanese trader accidentally punched one too many zeros and lost a lot of money for the company. This type of problems can get very expensive in an instant. As an industry, we tend to think of input validation in passing because we don't pay enough attention to the problem domain.

Nazzareno• 13 years ago

Sending emails always requires more attention than expected. Everyone, even Apple in 2007, made some severe mistakes in delivering emails. By using a professional SMTP relay (disclaimer: we provide SMTP+) this wouldn't happen since, for example, every A:/CC:/CCN: recipient is splitted from the original message by default.

isarat• 13 years ago

The excuses are on an on. I admire Github guys. They give a super cool impression to the other outside world and they're doing a great job. But the recent outages and the problems are giving high attention to Github. It's giving negative sign for the people who are interested go for Github way of hosting code. Either their own premise or on github. giving detailed analysis is a good tech thing... it fires developers heart and we are still going admire what they did to resolve this issue but definitely this isn't a good sign for github.

robertsosinski• 13 years ago

For quite some time, I have been using Sequel (http://sequel.rubyforge.org) as an ORM with Postgres instead of ActiveRecord, and have never been happier. Overall, Rails has been a great web-framework, but ActiveRecord has been it's sore spot, at least for me.

Also, using the sequel_pg (https://github.com/jeremyevans/sequel_pg) on Postgres ensures that even raw SQL queries are type-casted properly, plus it supports streaming and cursors, while also being very very quick.

If you want to stick with Ruby/Rails and relational databases, but think ActiveRecord might not be your bag, give Sequel a shot. It also fits nicely into lighter frameworks such as Sinatra.

jrochkind1• 13 years ago

I'm trying to figure out how likely my apps are to be effected by this same bug/behavior change as Github's, and if I should delay installing the security update without more investigation.

Not a great situation to be in with a security update.

Today's Email Incident

Related Stories