GitHub hit by DDoS attack

http://status.github.com/

by theyeti • 11 years ago

405 198 11 years ago

status.github.com

jdsnape• 11 years ago

Looks like another case of Chinese traffic being tampered with to load resources from another domain - in this case Baidu searches: http://insight-labs.org/?p=1682

nickysielicki• 11 years ago

This is far more interesting than the OP. Thanks for sharing.

plq• 11 years ago

You mean TFA.

fibo• 11 years ago

Sad but true, it is better to use tcpwrappers to block China bot nets, see for example http://g14n.info/2015/03/server-hardening-tips/#restrict-ssh...

nickodell2• 11 years ago

So who's doing this? The Chinese government? Baidu themselves?

toong2• 11 years ago

I was going to make point that Baidu should serve his analytics-js over https only.

But thinking about it, there are fails on so many levels.

_RPM• 11 years ago

def noticed this like 5 minutes ago. DNS completely failed for a second.

ptr• 11 years ago

Still getting lots of instability in spite of the status page.

jhildings2• 11 years ago

A good warning sign for companies that only have their codebase at Github which seem more common to me nowdays. If you run your own server at least you can like physicaly restrict the access to local network only.

reidrac• 11 years ago

I'm not sure if I understand what you're saying.

They're using git, don't they? Obviously you miss the web interface to issues, pull requests, etc; but if you don't have git repositories distributed in your own infrastructure you're doing it wrong.

inglor2• 11 years ago

Why though? The whole point of git is that it's distributed source control - you _always_ have a local copy of the source and the history. That's one of the biggest wins in using git and not a centralised source control in the first place..

butwhy• 11 years ago

But you don't necessarily have an up to date copy. If you had a local mirror, you could queue collective commits until the github server becomes available again.

rodgerd2• 11 years ago

If people have their whole workflow in git kernel-dev style, sure. But the reality is that for a lot of people their workflow is github. Leaving them with raw git is equivalent to leaving them with nothing.

jhildings2• 11 years ago

Well yes, but if you need to setup the codebase on another computer for example or are very used to use GitHub to do some "extra things" like browsing and follow commits, it will disrupt the workflow.

thejosh• 11 years ago

Or run something like https://toranproxy.com/ if you use composer, and also mirror your code remotely.

onion2k• 11 years ago

First of all, physically restricting access to a local network pretty much makes the benefits of using Git redundant. Ignoring that though..

I don't think there are many companies who have the depth of info security knowledge that Github can draw upon. You might believe that running your own server and locking it down to your local network (as best your team can do that) is better, but I'd rather trust Github even if that means my code is available online. While the chance of being victim to a non-specific attack is far higher (Github is a much bigger target; I'm affected if they're attacked), the chance of someone targeting my code and actually getting it are far, far lower because Github is better at making things secure than I am, and they have people who are paid to make sure things stay that way.

A policy of only having your code on Github has it's flaws but making sure it's secure isn't one of them.

maaaats• 11 years ago

First of all, physically restricting access to a local network pretty much makes the benefits of using Git redundant.

No, it doesn't. Why should it?

flyinglizard2• 11 years ago

I don't think there are many companies who have the depth of info security knowledge that Github can draw upon

That's true and I believe in that as well. However, GitHub also poses a much larger attack surface than any single company and it's safe to assume that once someone is in, they're going to get _everything_. Apart of that, you're also vulnerable to having GitHub disgruntled employees accessing your data, and the inherent vulnerability in having a remotely accessible repo in the first place.

Confusion• 11 years ago

Parent post is not concerned with security, but with availability.

amirmc• 11 years ago

"I don't think there are many companies who have the depth of info security knowledge that Github can draw upon."

I recall at least two vulnerabilities that GitHub was exposed to. The mass-assignment one from Rails (which they didn't fix until after they were the poster-child for it), and the cross-site one, which prompted them to use github.io.

I suspect their security teams are better now but you're still taking it on faith. By using 3rd-party services, you are increasing your exposure, not diminishing it. Someone who wants to attack you specifically will do so regardless.

ojilles• 11 years ago

GitHub enterprise

oelmekki3• 11 years ago

And what when your own server hosting repos is down ?

The problem is not having a third-party host, it's having no redundancy.

We use this, in our capistrano tasks :

    set :repository,  ( ENV[ 'FALLBACK' ] ? fallback_url : github_url )

If github is down, we just push on that secondary repos (which is indeed on one of our servers) and just process business as usual.

simon_vetter• 11 years ago

If the reason you're using github is its tools and (nice) interface, why not start your deploy by syncing github to your private repo, then have capistrano always pull from there? Less code paths to test and no need to define env[FALLBACK]. Also, it's probably faster.

lmm3• 11 years ago

Internal servers might theoretically be under your control, but in practice your sysadmin team probably isn't better than Github's and you're not going to have 100% uptime. If you're spending time and effort doing stuff in-house, be sure you're getting real value and not just the feeling of being in charge.

semi-extrinsic• 11 years ago

No, but even if your sysadmin team is crap and you have a 95% uptime, that's a 95% chance that your solution will be a backup on the fairly rare occasions that github is down. Unless you have some reason to have your downtime correlated with github downtime.

batou3• 11 years ago

No.

Our internal source control server is more reliable than our local shared ethernet with cable backup combined with the reliability of github.

We've had 43 minutes (scheduled and unscheduled combined) downtime in the last 11 years across three VCS technologies hosting internally.

If it was github and remote, we're talking about 7 days of downtime in the last 2 years alone.

We're definitely in charge and it has consumed virtually no admin effort in that time. The VCS server currently has an uptime of 3 years, 5 days, 20 hours and 38 minutes. It's also shifting 10mbit peaking at 200mbits pretty much all day every day 24/7/365.

(CentOS 6, SVN, HP DL380 for ref)

emodendroket4• 11 years ago

I don't know. If you're a really small shop the odds are frankly higher that poor administration will lead to your internal servers going down than that your (or Github's, really) server is down because of an attack by a state actor or someone with similar resources.

darkarmani3• 11 years ago

If only git was a decentralized rcs, you could have both.

maxhodges• 11 years ago

because GitHub's uptime of 99.9912% isn't good enough for you? Even during this attack we were able to push and pull to our projects off and on. https://status.github.com/graphs/past_month

alexchamberlain• 11 years ago

Making the not unrealistic assumption that the data is _not_ normally distributed, mean is a useless average here. You should be looking at median, which will be much less distorted by the long tail.

hayd2• 11 years ago

Would be interesting to see the 10/90% too.

TeMPOraL• 11 years ago

This, or if you are evil and know that your competitor has a stupidly-designed build process that depends on GitHub being available - by DDoSing GitHub itself you'll make your competition unable to work.

edwintorok• 11 years ago

the page says that everything is operating normally, but the main github.com page doesn't even load for me...

Edit: works again now

mwadams3• 11 years ago

It seems fine for me - and http://www.downforeveryoneorjustme.com/github.com

mwadams3• 11 years ago

However, I still prefer http://gitlab.com :-)

dengnan4• 11 years ago

This is an article [0] summarizes what happened. It is however in Chinese. So let me put a simple summary here:

Baidu has Baidu Analytics, a service similar to Google Analytics. In short, a website includes a javascript file from Baidu and Baidu will report some basic analytics to the site manager like how many visitors per day, how much time they spent on average per page etc.

Someone in the middle between a client outside China and Baidu, allegedly it should be the Great Fire Wall, changed the javascript file from Baidu and added some code so that any client executing the javascript file will periodically access https://github.com/greatfire/ and https://github.com/cn-nytimes/. This means any user who is accessing a site using Baidu Analytics will be an attacker to github.

Here is a simple solution: Block any javascript from Baidu if you do not use it. For chrome users, add the pattern [*.]baidu.com. See here[1].

Edit 1: Added a solution.

Edit 2: Format.

Edit 3: Oh, it's not only Baidu Analytics. Baidu Ads' javascript is also being hijacked and changed [2]. Imagine that all sites containing Google Ads use their visitors as attackers to attack github. Now it is literally what is happening to Baidu and its customers (and their customers' visitors.) The javascript is only changed for visitors outside China. This is why people believe that is done by Chinese government --- the only entity who has total access to all out-going routers in China. Since many Chinese users use VPN or other types of proxy to access Internet, they are all considered as visitors outside China.

0. http://drops.wooyun.org/papers/5398

1. http://www.howtogeek.com/tips/how-to-block-javascript-and-ad...

2. http://www.solidot.org/story?sid=43489

orf2• 11 years ago

According to this[1] post GitHub (or someone else in between) started changing the responses to alert("Malicious Script Detected")[2]. That's an awesome counterattack - this stops the script from looping indefinitely and annoys the users.

1. http://insight-labs.org/?p=1682

2. https://github.com/greatfire/

dengnan4• 11 years ago

For github, this is a smart move.

But, really, you can hardly negotiate with Chinese government. I'm pretty sure that they will deny this attack and re-emphasize their so-called Internet policy.

If I were github, instead of a warning message, I would redirect the workload to some Chinese government's website and let them suffer what they've created. Let's face it, they are waging a war on the Internet first.

Edit: Disclaimer: I know that my post is quite biased, especially this one. I'm not suggesting that people should wage a war to Chinese government. Please take my words just as a (biased?) sample from an ordinary Chinese citizen who is really tired of the government's censorship.

NoCowLevel• 11 years ago

Very interesting defense. It seems that it works because the attacking AJAX call is done with content dataType 'script'. I don't think it'll be too hard for the attacker to fix that.

jfroma• 11 years ago

Why the attacker will run the content loaded as an script instead of just dumping what they get?

Edit: I think is the dataType: "script" part. From jquery docs:

"script": Evaluates the response as JavaScript and returns it as plain text.

imaginenore• 11 years ago

Instead of that "script detected" they should convert the whole page to something the Chinese government really hates, like Tiananmen Square massacre.

JasuM• 11 years ago

If the message had some information about the Tiananmen Square massacre or some other censored information, the attack would probably stop. At least temporarily.

afarrell• 11 years ago

I wonder what percentage of Baidu users read English rather than just Chinese.

dheera• 11 years ago

So Baidu is using eval() instead of JSON.parse()? What kind of engineers did they hire?!

jle• 11 years ago

i feel like this alert should be in chinese for greatest effect

toxicFork2• 11 years ago

Asked in another thread but it went down because the post linked was taken down it seems[0]:

Would it have been prevented if Baidu served the .js files only over https? Are there any reasons of using http for anything that Baidu serves?

[0] https://news.ycombinator.com/item?id=9275201

dengnan4• 11 years ago

Probably, yes. But considering that CNNIC, a root CA from China, is issuing unauthorized certificates [0], I cannot help to connect these two events together. I won't be surprised that Chinese government is using unauthorized certificates to initiate MITM attack specifically targeting TLS traffics. If that is the case, there will be really bad days for the whole Internet.

0. http://googleonlinesecurity.blogspot.com/2015/03/maintaining...

toong2• 11 years ago

For this to work properly, it requires https only, to prevent downgrade attacks (you stated that).

Google analytics is also served both over http & https ? Can anyone shine a light on that ?

teknologist7• 11 years ago

What I don't get is why they didn't inject a script into all html passing through the firewall. That would have achieved a much greater effect if they really wanted to take out GitHub - the Baidu Analytics tracker is just a single script.

dengnan4• 11 years ago

Easy to implement?

This is just my theory: I think that GFW is currently entering its next stage, which probably includes MITM attack to TLS traffic and some attack specific to websites outside China. I suppose that since everything now is in a "research" stage, they are just trying to see if the technique works and how much it could go.

Disclaimer: I was an user inside China and being blocked from the real Internet. So please take my words with a grain of salt.

vinceyuan3• 11 years ago

Can we report it to Baidu and ask Baidu to clean up the scripts immediately?

hk__22• 11 years ago

If that’s a MITM attack they can’t do anything I guess.

im2w1l• 11 years ago

Wow this must be a major blow to Baidu. This time their scripts was hijacked to DDoS, not that bad.

Since we all know that the Chinese government would never do a thing like this, it must mean that there is a very powerful hacker group behind this. And they are probably DDoSing for profit. Who knows what they may do for profit next? Spy on users? Steal passwords? Credit cards? Impersonating users?

Until Baidu implements a secure crypto solution that can prevent this malevolent hacker gang from sending corrupted scripts, it would be very irresponsible to use baidu analytics!

hk__22• 11 years ago

Since we all know that the Chinese government would never do a thing like this

Why?

Rooster61• 11 years ago

Something bothers me about this post. It's obvious astroturfing, but this user has a large amount of legit comments. They are in perfect English, while this is clearly not, and uses a totally different tone and voice than their previous comments.

coldpie2• 11 years ago

"Gee coldpie, why do you use NoScript? All you're doing is breaking every website you visit!"

Shoe's on the other foot now, hahaha!

jdjb• 11 years ago

I use NoScript as well (and I wish more people would) but to be fair I doubt the users who were part of the botnet even noticed it at all. It's only github who would've benefited from these users running NoScript.

teknologist7• 11 years ago

Having no sites on the internet actually work finally paid off huh

teknologist7• 11 years ago

Using jQuery to send this request is really kind of amateur. They could just append a <script> tag to the page which is effectively what that $.ajax call is doing.

r3bl2• 11 years ago

Why the hell would anyone launch a DDoS attack against GitHub? Seriously, the only point I see in DDoS-ing GitHub is to prove yourself that you can DDoS it.

brador• 11 years ago

There is the Github you see and use, then there is the Github you don't see. It's a service that allows files/information to be uploaded, downloaded, shared, by pretty much anyone. That's something some governments sadly don't like.

rodgerd2• 11 years ago

Note that it seems to be attacking two specific projects, according to the details above, one of which is to get around the restrictions of the Great Firewall of China. It's most likely an threat about the advisability of hosting something unapproved-of.

mirhagk• 11 years ago

Interestingly enough they have succeeded in pulling those projects down.

dagw2• 11 years ago

Given that the DDoS wasn't targeted at "https://github.com/" but rather "https://github.com/greatfire/" and "https://github.com/cn-nytimes/", two projects that can reasonably be described as not being pro Chinese government, it seems that github was targeted for hosting anti-Chinese 'propaganda'.

r3bl2• 11 years ago

Thanks for the clarification!

mitkok• 11 years ago

How is this news ?

bitinn• 11 years ago

I have written a brief summary of issues, as a tweetstorm: https://twitter.com/bitinn/status/581350026217013248

tsheeeep2• 11 years ago

We use bower and npm for our project. Every couple of months github is under attack by a DDoS or not working correctly leaving us with broken deploy scripts. What is the best way to fix this? We don't like the idea of commiting the node_modules or bower_components folder. Is there a tool which will cache the npm and bower sources so they only have to be downloaded if something changes?

kowdermeister• 11 years ago

Committing downloaded packages is not a bad practice. Yes, it can be a bit big, but otherwise I don't see much problem with it. You will be always sure that the installed packages are compatible with each other.

tsheeeep2• 11 years ago

To be sure everything that we know works together we use things like npm-shrinkwrap files. We don't like it because it makes the git changelogs a lot bigger and almost unreadable if you want to compare a pull request.

astrodust• 11 years ago

That works great until you have compiled extensions. Then it's misery.

realityking• 11 years ago

We currently only use it for private modules, but [private bower](https://www.npmjs.com/package/private-bower) also allows caching bower packages.

s_kilk2• 11 years ago

We don't like the idea of commiting the node_modules or bower_components folder

The bower documentation actually advises checking bower_components into source control.

http://stackoverflow.com/a/22328067

Cthulhu_• 11 years ago

You could install something like Angry Caching Proxy (https://www.npmjs.com/package/angry-caching-proxy) to cache commonly downloaded packages in your local network. That should also please whoever's paying for the NPM repository.

Another thing I used to do was have a simple script that tried to do npm install, if that failed, npm install using the NPM Europe mirror - although I see now that that one's been deprecated because NPM's primary repository has become more reliable and hosted at a CDN with multiple locations across the world, etc.

xtrumanx• 11 years ago

Another option is Sinopia (https://www.npmjs.com/package/sinopia) which is a private npm repository server. After installing it, you'll have to change the npm registry url to your new private server and any npm requests (install, update, etc.) will go through your private server and fallback onto the official server if a package is unavailable.

darkarmani3• 11 years ago

Don't use npm for deployment. that's even the recommendation by joyent.

ssube• 11 years ago

I recently rolled out a Sonatype Nexus server at work to handle both NPM and Maven artifacts. Allows us to publish internally, and a combined internal/proxy repo group allows us to serve both our packages and cache global registry stuff. Took ~3 minutes off some of our CI builds.

ef4• 11 years ago

It's much more robust to have a dedicated build step that is separate from your deploy step.

The build step is where you install deps, do compilation, etc. Then it saves the output. The deploy step can just copy all the files from the last known good build to the new/updated servers.

This way you can always deploy a last-known-good release quickly and without external dependencies.

It also lets you take advantage of package caching on the build server so that as long as your deps don't change, you can deploy new releases to old or new servers without hitting any external services.

lilred• 11 years ago

Git submodules are designed for your use case, but I don't know how well they handle it.

rdl• 11 years ago

Someone turning a widely-used third-party-hosted JS into "evil" seems like an incredibly difficult layer 7 DDoS to address. Assuming you have great capacity to filter on the edge (CloudFlare, being Google, etc.), but a limited backend, it's still very hard to identify legit vs. non-legit traffic and do filtering.

(Obviously if the attack is against, say, Chinese users, and your site's legitimate users are mainly in Estonia, you can do filtering, or if the attack only hits an obscure URL, but the attack doesn't have to be weak in that way.)

There are a bunch of potential ways to address it, but they all work best if you have a site with a defined community of users. If you're a large public site, without login, it's hard. Some of the better techniques are in-browser challenges (JS, CAPTCHA, etc.), but it's conceivable with enough endpoint with real browsers and real humans on them, these could get defeated.

nickodell2• 11 years ago

GitHub seems to have done just this. Both attack URLs return

alert("WARNING: malicious javascript detected on this domain")

They can probably serve that without hitting their database servers.

zhufenggood• 11 years ago

Baidu's javascript cdn is being Hack by national firewall, inject these JS attack script. If other webseit include some javascript library from Baidu's javascript CDN will automatically run JS script that will DDOS attack Github. The attack JS script is here:

https://gist.github.com/zhufenggood/7bb040b1effb71d14bcc

Here is deobfuscate version using http://jsbeautifier.org/

https://gist.github.com/zhufenggood/6a38c2a2b2185977b3cb

Github notice that, it replace that DDOS http request respond with a alert("WARNING: malicious javascript detected on this domain"). That is why some Chinese guy gets a weird pop-up with English text when visiting Chinese websites.

talnet3• 11 years ago

clever move we should say ? or any better idea ?

andao• 11 years ago

maybe the point is to scare people into preemptively blocking Chinese IPs so the Chinese gov doesn't have to swat flies with the Great Firewall. it's good marketing too: "look at all those foreigners who refuse to let us access the free internet!" "anti-Chinese prejudice!" etc etc

dEnigma2• 11 years ago

If that is their plan it seems to be working, judging by some of the comments here.

mahouse• 11 years ago

I wonder what's the positive effect of sharing the same Internet with the chinese. I never visit nor I know someone who visits sites from China.

Major ISPs from the west should definitely consider blackholing all traffic coming from there to avoid DoS attacks, spam, etc. – From my experience, this would mitigate spam by a 50% or even more.

danuker• 11 years ago

I assume Github could block it themselves if it were that simple.

chrisBob• 11 years ago

So you agree with the Chinese government and think we should just censor the Internet for all Chinese citizens?

s_kilk2• 11 years ago

I wonder what's the positive effect of sharing the same Internet with the French?

Or the Americans? How about the Scottish? I know I personally want rid of the Irish from the internet, so we should black-hole all traffic from there too while we're at it.

GeneralMayhem• 11 years ago

Not that I agree with the parent, but you're being disingenuous. The Americans, Sottish, French, and Irish are not actively trying to wreck the Internet, certain US Congress members notwithstanding. There's a difference between throwing someone out of your store because you don't like the look of them and throwing someone out of your store because they shit on the floor every time they come in.

emodendroket4• 11 years ago

Read a little more closely. The way this DDoS was achieved only endpoints from outside China are being used to carry out the attacks.

mikeash5• 11 years ago

Yeah, screw all those Chinese people living overseas who want to keep in contact with their relatives back home. Screw everybody who visits China and doesn't want to be cut off from the world while they're there. Screw the hundreds of billions of dollars in trade to and from China which the internet facilitates.

talnet3• 11 years ago

You my friend really know how to poke.

lmorchard• 11 years ago

I wonder what's the positive effect of sharing the same internet with you? I never visit nor do I even know you

neo2001• 11 years ago

That's a pretty stupid idea. Lot's occidental businesses are (or area trying) to expand their market to china. Give it a look to this map [1], and you'll understand why China is a massive opportunity.

[1] http://en.wikipedia.org/wiki/List_of_countries_and_dependenc...

hokutosei• 11 years ago

oh boy, think we have to go home early today and its tgif.

vinceyuan3• 11 years ago

Fxxx GFW!

vinceyuan3• 11 years ago

To the guys who gave me -1: If you cannot visit Google, Facebook, Twitter, Pinterest, and many other websites, you will say "Fxxx GFW!" too.

ramchip• 11 years ago

I didn't downvote, but I can understand this getting pushed down as it doesn't bring anything to the discussion.

mirekrusin• 11 years ago

Could github whitelist ip addresses who did commit to protect normal users from DDoS effects (splitting traffic to two sets of servers during DDoS etc)?

zer0defex• 11 years ago

Seems like a reasonable strategy to me, but probably very infeasible for an attack already in progress if this tactic weren't planned and ready to go in advance. It would be something I'd investigate post-attack however to see if it's a viable strategy for mitigating future attacks.

dataker• 11 years ago

Is it confirmed the Chinese government is behind this? Could it be possible to also be a competing company?

mikekchar• 11 years ago

Given Snowden's recent allegations that the Canadian government is engaging in false flag operations (causing havoc and placing the blame on other nations), it could even be another country that just wants to make China look bad. To be honest, I would expect an attack from China to be a little bit more subtle... Of course it could be a double bluff... but then... Basically, it's pretty hard to know what the heck is going on.

mwadams3• 11 years ago

Apologies - it seems my earlier comment was made during a brief respite.

teknologist7• 11 years ago

And we're down again

yAnonymous• 11 years ago

They should redirect the attack to the server hosting the script as a friendly encouragement to use encryption.

Ethan_Mick• 11 years ago

What we really need is a free and open source distributed version control system.

sgarrity• 11 years ago

This is a reminder that having a single service, like Google Analytics or Google Fonts, injected into just about every major site on the web might not be a great idea.

omgitstom• 11 years ago

This!

Even though I don't think this would have proactively helped this DDoS, but it can't be closer to the truth.

CDNs hosting libraries / fonts / resources for the web are going to be targeted more and more, it is just too attractive to malicious people.

kalleboo• 11 years ago

This is why I wish we could have a file hash attribute added to certain tags (such as script). It could improve caching across domains and validate the content you're serving up. I proposal was posted here a while ago.

sreya• 11 years ago

If someone wouldn't mind explaining, what could the motive possibly be for the Chinese government to be doing this?

touristtam• 11 years ago

Maybe because the github repo is for this website: https://zh.greatfire.org/ ?

b123400• 11 years ago

Did Baidu or its employee said anything about their script being used to attack Github? Would like to know how they think about it

TACIXAT• 11 years ago

The internet is so cool. I hope no one ever fixes the ability for shit to go crazy online. It makes me so happy to be alive in the age of data leaks, ddosses, and malware. It's all the more awesome that it isn't just individuals but entire nation states fucking shit up. This is a really neat attack. I hadn't thought of a MITM being used on a such a massive scale, and to leverage uninfected computers as a botnet is pretty great.

Props to China. 很好!

jmakov• 11 years ago

What I think is most interesting of all is not that a foreign country is attacking a US company nor that the company has no support from the wast pool of three letter agencies but the fact that github as a company designed their architecture in such a way that a sub site is allowed to eat all of the resources bringing the whole company down. Kudos to all the engineers and architects with +100k salary over there.

jeremybass• 11 years ago

Question, way can't `code.jquery.com` help here? or has the attack moved passed this MIM attack?

Irish• 11 years ago

Anyone in europe having trouble with github this morning? I cant get bower to install and it fails with cannot connect to github error, status page seems to suggest everything is working

GitHub hit by DDoS attack

Related Stories