Pin-pointing China's attack against GitHub

http://blog.erratasec.com/2015/04/pin-pointing-chinas-attack-against.html

by zmanian • 11 years ago

305 138 11 years ago

blog.erratasec.com

finnn• 11 years ago

That's cool, I didn't know how traceroutes work. Is he planning on releasing the http traceroute tool?

cleverjake• 11 years ago

you should comment there to ask them

itistoday2• 11 years ago

Here ya go: https://github.com/robertdavidgraham/masscan

    @collinrm I just took masscan, changed the HTTP request,
    then tweeked the code to generate a small TTL.

Source: https://twitter.com/ErrataRob/status/583433175302479872

blacksmith_tb• 11 years ago

On OSX, it's available via Homebrew, brew install masscan

zombination• 11 years ago

This is important evidence for our government. It'll be interesting to see how they respond to these attacks -- attacks by a nation state against key United States Internet infrastructure.

It seems a bit of a stretch to say that Github is "key US Internet infrastructure"...

yeskia• 11 years ago

Isn't GitHub hosted on AWS, which as a platform is pretty significant.

klaustopher• 11 years ago

GitHub used to be hosted on EngineYard in the beginning, but later moved to bare metal servers at RackSpace: https://github.com/blog/493-github-is-moving-to-rackspace

andyfleming• 11 years ago

I don't know that it is "key US Internet infrastructure", but it's important to a lot of developers and companies. Thankfully GitHub handled the attack pretty well.

Disclaimer: I'm not sure what actual usage stats are like for GitHub.

Daishiman• 11 years ago

If GitHub were down for two days that's a metric shitload of projects that can't get deployed. There's a significant number of software projects with dependencies on Github-hosted stuff.

a3n• 11 years ago

If github were down for two days, you're probably right, those projects can't be deployed.

But if github were down for two months, the nature of git suggests that deployment for those many individual projects would shift either to the originators' infrastructure, or some other aggregating service.

mfkp• 11 years ago

While this is a very interesting read (learned a thing or two), the author's conclusion is a bit suspect.

Using my custom http-traceroute, I've proven that the man-in-the-middle machine attacking GitHub is located on or near the Great Firewall of China.

Although suspicious, it seems one would need to know a lot more about China Unicom and their infrastructure to say this conclusively.

mikekchar3• 11 years ago

I apologize because I have not had time to really looking into this closely, but I think you are correct that the author's conclusions need to be reviewed. He repeatedly says that it is a man in the middle attack, but the link he points to says it is a man on the side attack. This is a very different beast: a man on the side attack does not rewrite packets, but rather uses it's position on the backbone to send replacement packets that will arrive before the real ones do. Also, the referenced link states that these packets are being sent on only 1% of responses. So if this is the case a traceroute is probably not going to find the issue. The offending node does not have to respond at all -- especially if it sends the bogus packets by a different route.

Again, I have not looked at this in detail yet and probably won't have time for a few days. I would welcome a discussion on this point.

makomk• 11 years ago

Because it's a man on the side attack, it sends its attack packets when it sees the original request. He's taking advantage of this by lowering the TTL so that the request never actually reaches the destination server and only the system doing the attack is able to respond.

tedunangst3• 11 years ago

The man in the middle attack was injecting javascript to recruit unwitting man on the side attackers against github. github was not being MITM'd, but its "attackers" were.

edit: above may be imprecise. I went back and read the original more closely. they note that if they artificially drop an injected packet, it doesn't get resent (and hence the conclusion that it's man on the side), but they don't mention whether they get the original packets or not. If something is blocking the original baidu packets, it would have to be in the middle, not just on the side.

samcrawford• 11 years ago

My first thought was that at the very least he should be comparing it to a tcp-traceroute (i.e. traceroute -T ...) rather than an ICMP one. Other routers before them may make different routing decisions based upon the type of IP traffic, so assuming both take the same path is unsafe.

snowwolf2• 11 years ago

I think the thing everyone is forgetting is that if it isn't state sponsored/approved then why hasn't it been turned off (as far as I am aware the attack is still ongoing).

If it was a hack, surely China Unicom should have fixed it by now?

moe9• 11 years ago

Has someone contacted China Unicom about this?

Perhaps they don't even know something is going on?

SyneRyder• 11 years ago

Hasn't it stopped as of March 31? Github system status now shows green, whereas they previously said "We will keep our status at yellow until the threat has subsided": https://status.github.com/messages

whoopdedo• 11 years ago

To say that the "Chinese government" is involved I think understates the situation. We know as fact that their army has invested considerable time and money in a cyberwarfare unit. And that the company that operates the Firewall is a military contractor.

When Sony was hacked a few months ago, the media couldn't wait to label it a "terrorist act" by North Korea.

I just now searched Google News for "github terrorism".

1. http://www.itpro.co.uk/security/24319/github-falls-victim-to...

2. https://grahamcluley.com/2015/03/github-ddos-attack/

That's all, even though the evidence appears more clear that the Chinese government is involved. Whereas North Korea's responsibility was in doubt. The silence speaks volumes.

But is "terrorism" even the correct word for this? When Saddam invaded Kuwait, was that a terrorist act? Consider this quote written by a Chinese military analyst 30 years ago:

  those who take part in information war are not all soldiers. Anybody who
  understands computers may become a "fighter" on the network. Think tanks
  composed of non-governmental experts may take part in decision-making;
  rapid mobilization will not just be directed to young people; information-
  related industries and domains will be the first to be mobilized and enter
  the war..

(From http://fmso.leavenworth.army.mil/documents/chinarma.htm)

However, the Chinese may respond by claiming this was a preemptive defense. That GreatFire.com was designed to weaken their security and they were justified in taking action to protect the sovereignty of their computers. Haven't western nations done the same when they were threatened by terrorism or nuclear arms?

It seems to me that we have officially entered the era of a weaponized internet.

methou3• 11 years ago

It seems to me that we have officially entered the era of a weaponized internet. Couldn't agree more.

And GFW is indeed a WMD that must be stopped.

1stop• 11 years ago

... I can't tell if this is sarcasm or serious. But seriously... What "Mass destruction" can the GFW cause? ... Like Amazon/Azure/Rackspace are WMDs under your current definition?

maybeathrowaway• 11 years ago

Is terrorism the correct word for the US government infiltrating, manipulating and subverting technology and services all over the world? Is spreading fear about being monitored by a seemingly lawless entity terrorism?

psykovsky• 11 years ago

Terrorism is like a catch-all word.

tedunangst3• 11 years ago

Not terrorism. The attack was somewhat narrowly targeted (at least as narrowly as technically possible, given it's all on one domain) and also tried to achieve the desired end goal. They weren't blowing up random parts of github infrastructure.

jacquesm10• 11 years ago

I wonder if the same terminology would have been used if Iran or North Korea had conclusively perpetrated this attack.

JonnieCache2• 11 years ago

But is "terrorism" even the correct word for this?

Don't use that word. It's barely even a word any more, its become one of those weaponized magic symbols used for mind control. See also "freedom", "globalization", "sharing", "choice" and so on.

Instead, you can just use words like "murder", "destruction of infrastructure" and the like.

jacquesm10• 11 years ago

You forgot 'communism'.

hyperpape3• 11 years ago

Makes sense, once you can come up with a single word to refer to targeting civilians with violence to try to effect political change. That's what "terrorism" meant and that's what it means, even if our news media and government have decided that it can only ever be applied to Muslims.

wongarsu2• 11 years ago

With any meaningful definition of the word neither the github attack nor the Sony hack was terrorism. Nobody was trying to spread terror among civilians, nobody tried to influence government policy.

frozenport• 11 years ago

>have officially entered the era of a weaponized internet.

Nope, the State department has yet to respond.

belorn• 11 years ago

If we are going to use terms like "cyberwarfare", it make logical sense that other military tactics will also be ported over to digital versions. One of those is the classic weapon test, where a nation demonstrate their military power to the world.

China used 1% of the available traffic from a single CDN. Their choice of target might just have been randomly picked from low priority list for the dual purpose of sending a political message.

whybroke• 11 years ago

There is never going to be serious public criticism of China in the west because there are vast numbers of western organizations making vast sums there. The public will see little to nothing that might ask for trade restrictions as a response.

If decades of continuous trade deficit and serious attacks on places like Los Alamos go under or unreported, don't hold your breath for GitHub.

Of course if it were North Korea (or Iran) it would be headlines.

moe9• 11 years ago

"github terrorism"

the era of a weaponized internet

Is HN turning into Fox News now?

guscost2• 11 years ago

The Internet has always been weaponized (in the sense that it was designed to be a countermeasure against a nuclear first strike).

addingnumbers• 11 years ago

That's not true, pure urban legend.

http://en.wikipedia.org/wiki/ARPANET#Misconceptions_of_desig...

Even if it were true, designing something to withstand attack doesn't make it a weapon. Is a kevlar vest a weapon?

phelmig• 11 years ago

That the server is located in China doesn't proof anything.

simonh• 11 years ago

This analysis doesn't just prove the attack orriginated in China, it shows that it takes place immediately inside the first Chinese network the connection reaches on it's way into China. The second piece of analysis shows that this is the same network layer in which the network blocking performed by the Great Firewall occurs. So the great firewall and this attack are both being implemented at the same point in the network infrastructure.

maxerickson• 11 years ago

I think an interesting question is how likely it is that some of the great firewall is compromised.

(It could be by some party outside of China, or by some group inside of the Chinese government that does not have an official mandate to use it for things like the Github attack)

dexcs• 11 years ago

Is there a statement from github what they think who did the ddos?

kiliankoe• 11 years ago

No and there probably won't be. Them publicly saying they were being attacked by the Chinese government would put them on some seriously questionable legal ground. They definitely went the right route by not saying anything.

stevenh• 11 years ago

Which laws would they be violating by announcing they were attacked by China?

nickodell2• 11 years ago

When TPB was pretending to be in North Korea, someone proved that they weren't, [1] because of how quickly they responded to a ping. Could someone narrow down the physical location of the firewall similarly?

[1] https://rdns.im/the-pirate-bay-north-korean-hosting-no-its-f...

mirashii2• 11 years ago

You can really only use a method like this to say where a server is not (it can't be halfway around the world because the speed of light limits it), but this is assuming you're communicating directly with the server. The method used to inject these packets on the wire makes this sort of analysis even harder to do this sort of analysis (and if there was concern, appropriate amounts of random delay and noise could be added).

nickodell2• 11 years ago

The method used to inject these packets on the wire makes this sort of analysis even harder to do this sort of analysis

I was under the impression that this was a man on the side attack, so they'd sent a bogus SYN-ACK back to you the moment that they saw a SYN. Theoretically, you should still only be dealing with one RTT.

(and if there was concern, appropriate amounts of random delay and noise could be added).

I don't think China cares if it gets traced back to them.

Ashwinning• 11 years ago

"blocking GitHub is not really a viable option" he said. Tell that to the world's craziest democracy - India, which banned GitHub, Vimeo, Pastebin and a bunch of others in December last year. Some bans were lifted later. Source : http://www.zdnet.com/article/india-blocks-32-websites-includ...

ptaipale4• 11 years ago

I think that action was somewhat different in nature, though. It's an instance of Vogon-type bureaucracy that sends blocking orders to Internet service providers without an understanding of the issues. I presume the Indian government is not actively trying to shut down VPNs that people use to access this kind of resources.

In short, I believe the Chinese government knows what it is doing, while the Indian government does not.

enupten4• 11 years ago

"Vogon-type": that's mind-bogglingly accurate!

dimmuborgir• 11 years ago

Only Gist was blocked, not Github. It was a knee-jerk reaction to a court order after ISIS made a series of threats and some Indians were found to be in contact with the ISIS. All those sites were unblocked soon after.

imron2• 11 years ago

Agree with this. Anyone who thinks blocking GitHub is not really a viable option has never been on the other side of the GFW.

The Chinese government will happily block any site they want to and they have little/no regard for the popularity or usefulness of the site in question, and often they will block popular foreign sites to help copycat local versions thrive.

Off the top of my head they block Facebook, Twitter and Youtube entirely and Wikipedia selectively (used to be permanently also). I can tell you they don't care about blocking GitHub.

The also have the ability to dynamically block sites based on page content rather than just entire domains, so it would be perfectly feasible for them to block just the project pages and not the entirety of GitHub.

mckoss• 11 years ago

You can't selectively block content on an SSL connection w/o having a back door to the encryption keys used to secure the connection. A man in the middle attack would be detectable unless the root certificates were compromised.

est• 11 years ago

Agree with this. Anyone who thinks blocking GitHub is not really a viable option has never been on the other side of the GFW.

Before long time ago, people hosting tons of anti Chinese government stuff on Google, especially Common Storage Service, yes people say China wouldn't dare block Google.

LOL

enupten4• 11 years ago

"World's craziest democracy" - that really made me smile :)

I don't think the Indian Government's competancy in IT (and many other things) come close to that of China; but hey atleast we don't have a muzzle over our mouths (that's partly a lie).

Wasn't the "ban" actually a DNS entry removal ?

eaxitect2• 11 years ago

I don't agree with author's conclusion, having a server in a chinese ISP does not mean it's orchestrated by chinese gov. Also, I don't suppose MITM attack organizers have no clue about how TTL works.

orf• 11 years ago

They didn't have a clue about $.ajax with a 'script' data type did they?

eaxitect2• 11 years ago

So they should be chinese gov, right?

methou3• 11 years ago

I have a question with the method, hypothetically, if I am the attacker, I know the ttls of each packets tha tis passing through, right?

So when I get a packet with ttl so small that won't survive long enough to reach the target, instead of altering, I just leave it along. So the probe will never know where I am in the route.

jacquesm10• 11 years ago

But upstream providers within the TTL range will. And during a DDoS like this you can bet that everybody in the chain that is on the good side is in constant communication.

brudgers9• 11 years ago

A sufficiently sophisticated man in the middle can be anywhere between origin and destination and have arbitrary distribution. Proving that a particular node is responsible for a particular alteration requires using a trusted trust computer to send packets into the great wall on their first hop.

The experiment in the article required trusted trust of packets destined for the great wall passing through US infrastructure. That this infrastructure can generally be considered neutral is no guarantee that it was in this case. Any router or switch can use arbitrary tables and conditional logic on any packet. The purpose of the experiment was prosecuting a particular suspect not arm's length analysis.

simula673• 11 years ago

China is the second most powerful country in the world. Do people really think they are that stupid ? They are not going to attack an American company using infrastructure that anyone can track back to them. This Github DDoS has got to be the work of someone trying to frame the Chinese government. Has anyone considered that angle ?

prawn• 11 years ago

What's going to happen if they are known to be the attackers but can issue denials? No one is going to pull them up on it.

stefantalpalaru4• 11 years ago

Take a look at this brand new executive order authorizing economical and financial sanctions against global "cyber threats": https://www.whitehouse.gov/blog/2015/04/01/our-latest-tool-c...

This GitHub business looks more and more like a false flag operation.

jacquesm10• 11 years ago

How does 'economical and financial sanctions' equate to 'Whitehouse gives order to start false flag operation against github'?

The more that comes out and the more silence there is from the Chinese government the less it looks like a false flag operation. Usually the victim of a false flag operation (China in this case, not github) would be very adamant about its non-involvement and would work very hard to expose the originator.

enupten4• 11 years ago

Wouldn't financial sanctions hurt the US a lot more than it does China ?

I mean they've Tibet for the trump card after all; Cyber-attacks are kid-stuff.

(P.S: I sympathize with the Tibetans; but there is really no political will to resolve this, not in the US, not in EU, not in India).

nmc• 11 years ago

Maybe they do not care that "anyone can track [the infrastructure] back to them".

jacquesm10• 11 years ago

Do people really think they are that stupid ?

That's not really a defence. They could easily be that stupid, bureaucracies tend to do extremely stupid stuff when looked at from the outside but every cog on the inside thinks that its action makes perfect sense.

This Github DDoS has got to be the work of someone trying to frame the Chinese government.

Evidence?

Has anyone considered that angle ?

Sure, but so far the evidence is that that is not the case.

If it is someone trying to frame the Chinese government I'm sure they'll tell us all about it soon.

simula673• 11 years ago

bureaucracies tend to do extremely stupid stuff when looked at from the outside but every cog on the inside thinks that its action makes perfect sense.

If this is Chinese doing, the likely ones responsible are the Chinese Intelligence, not their bureaucracy.

Evidence?

Occam's Razor. I find it hard to believe that a society with sufficient level of sophistication to obtain $9 trillion GDP[1] would 'accidentally' go on to declare cyber war on US. Especially considering the fact that the attack itself was pretty sophisticated.

[1] http://en.wikipedia.org/wiki/List_of_countries_by_GDP_%28nom...

wongarsu2• 11 years ago

Powerful countries doing a show of force is nothing unusual. They know they can get away with it. If they had severely impacted github it would have sent a strong message not to do buisiness with people who circumvent the Great Firewall. It makes complete sense to me, they just failed.

On the other hand, why would anybody go to this length to frame to Chinese government? What would they try to accomplish? Obama sending a strongly worded letter to the Chinese government?

laumars• 11 years ago

The problem with that kind of speculation is you can equally use the same argument against itself. ie maybe they're bluffing people into thinking that the Chinese military / government have been framed as it seems too obvious that it would have been them.

We could also speculate from the angle that the officers in charge of making these decisions are not tech-savvy themselves (or at least not to the level that they might realise just how traceable these attacks), which isn't a huge assumption to make when you look at how incompetent many government officials are who have serious influence over technology policies (eg http://www.bbc.co.uk/news/technology-23437473)

So anyway, my point is it's better to look for a little evidence to support a hypothesis rather than blindly speculate.

jokoon• 11 years ago

In the end, I wonder if the purpose of the great firewall is not for China to defend itself against foreign cyber attacks or because "free internet" might not benefit China currently.

I'm sure computers are now mainstream enough that it would matter for any country to put cyber warfare as a key strategy. The US and the west dominate through open trade and easy communications and free speech. Maybe that makes China vulnerable, and they're trying to defend themselves economically.

You can accuse China all you want, but if you're an american, it's harder to listen to those accusations.

brudgers9• 11 years ago

The US spent more than 50 years protecting its residents from Cuban sugar and Cuban cigars and sunburns on Cuba's beaches. Such are the absurdities of powerful sovereigns.

brudgers9• 11 years ago

The article is titled:

   Pin-pointing China's attack against GitHub

baconschizer• 11 years ago

Is this April fool joke? Or are you guys really taking this whole Chinese government theory seriously? If you were leading a 1.6bn populated country how much you would care about a programmer's code site?

To give all those conspiracy theorists a clear picture, what really happened is merely the scale of problem you have never worked on or dreamed to be working on outside China.

This happened year ago when a Chinese state funded train ticket booking website accidentally deployed to production with a opensourced Javascript vendor file still linked to github. And first day that site went live, 30 billion visitors tried to secure a ticket for coming Chinese New Year, when took down github for a good while. Yes it was a DDOS attack from China, by train ticket buyers.

Last November, Chinese online c2c marketplace TaoBao.com, saw 16.7bn transactions in one day, with more than 1 billion CNY settled in a minute. If any of the web dev responsible for even a small promotion page left a link of cool jquery plugin from GitHub, you could have written another holy crap evil government attack post here.

snowwolf2• 11 years ago

I made a similar comment elsewhere in the thread, but lets assume it's not the Chinese government and it's something accidentally deployed or a hack by some unknown entity.

Why is the code still running a week later? It doesn't take that long to find the offending server/s code and remove it. Especially as it is making the Chinese government look bad, there would be added incentive to fix this pretty quickly.

weberc2• 11 years ago

If it was an "accident" that made the Chinese government look bad, we would be hearing about the "unrelated" murder of magistrates by now.

peterwwillis• 11 years ago

While many explanations are possible, such as hackers breaking into these machines, the overwhelmingly most likely suspect for the source of the GitHub attacks is the Chinese government.

Correlating the circumstantial facts that China has a giant firewall, the content being blocked is getting around China's firewall, and an attack came from somewhere deep in one of China's largest backbone providers, does not make an 'extreme likelihood'; it makes a weak correlation. Likelihood requires reviewing known outcomes to determine a likely result. What other known evidence of specifically these three behaviors by the Chinese government are you basing this conclusion on?

This is important evidence for our government.

You've taken a massive leap in logic from a machine inside China manipulating global traffic to attack servers in the US, to conclude that it is more likely to have been the Government than anyone else. This is exactly the same as saying any attack originating from the US which appears to be related to US interests must be from the US government. If this was the basis for how we concluded all investigations into illegal actions, anyone who 'looked like' they did it would be found guilty, sans evidence. That may be how other nations' justice system works, but not ours.

Furthermore, in no way is either Github or Baidu's analytics considered 'key US Internet infrastructure'. I mean, Git is even a decentralized system - people can still get work done if it's down!

This is not evidence of the Chinese government's complicity, and pretending it is creates a dangerous logical fallacy that could improperly shape public opinion.

moe9• 11 years ago

This mouth-breathing Bullshit really needs to stop.

the overwhelmingly most likely suspect for the source of the GitHub attacks is the Chinese government.

Why would the "Chinese government" carry out an open attack against an american company for absolutely no potential gain at all?

Do you really think they are stupid enough to believe such an attack could remove these two software packages from the internet?

wnevets3• 11 years ago

What is the diplomatic downside for china exactly? If China can occupy & claim other countries territory in the face of international pressure without a problem, what makes you think a convert op to DDoS github is out of the question?

moe9• 11 years ago

what makes you think a convert op to DDoS github is out of the question?

The fact that this attack doesn't pass even a most cursory risk/reward analysis.

Anyone with the technical smarts to carry it out must be well aware that there is zero upside potential for China. The targeted projects are not gonna disappear, github is not gonna disappear.

All possible outcomes are negative; The targeted projects get extra media attention (Streisand effect), the "Cyberwar" narrative in the west is fueled (cf. this HN thread), in the worst case there could even be a minor diplomatic quarrel with the US.

What do they have to win here?

pekk3• 11 years ago

Why would the "Chinese government" carry out pervasive domestic censorship? They know information is still going to get around on back channels. That was never the point.

The potential gain is that they can extend this capability (and chilling effect) to GitHub and other sites that aren't under the sovereign control of China. Nobody wants to go out of business just because a user uploaded a file which is politically controversial in China. Now every company will think twice and come up with some weaselly reason why they can't have text files about Falun Gong or whatever.

moe9• 11 years ago

Now every company will think twice and come up with some weaselly reason why they can't have text files about Falun Gong or whatever.

You mean like Github did in this very case? Except wait, they didn't.

So who is this "every company" that will now "think twice"?

BhavdeepSethi• 11 years ago

I don't get it. Why doesn't GFW just block those github pages in particular? A lot of people can fork these repos but they won't be anywhere popular to the current pages.

shaftoe• 11 years ago

HTTPS. They can't tell what URL a user is requesting.

y0ghur7_xxx• 11 years ago

HTTPS. They can't tell what URL a user is requesting.

I am sure they have a private key of some of the CAs shipped with major browsers lying around somewhere...

cschramm• 11 years ago

There's absolutely no proof in this.

The fact that a request with a TTL smaller than 12 does not trigger a response does not mean the responder is the host after 12 hops. Assuming none of the previous hosts misbehaves (they could be increasing or at least not decreasing the TTL) you can only conclude that it is none of the later hosts, but it can still be any of the previous ones.

That leaves you with Comcast AND China Unicom hosts and, considering that the replies you see in the traceroute results can easily be spoofed, it can be any third party as well.

Possible scenarios include (I don't say they are more likely):

1. Comcast is producing the responses, but only does so if the request TTL is large enough to make you blame China Unicom.

2. China Unicom hands the packets over to a third party after just a few hops in their backbone. The third-party sends ICMP Time exceeded messages looking like they are from other China Unicom hosts to make you blame China Unicom.

Conclusion: This is either an obvious attack from within the China Unicom backbone OR a more sophisticated attack where the attacker wants to a) hide his identity and b) blame China Unicom for it (I can't think of a scenario where b) would be a by-product and not on purpose).

Just saying. The sentences in the post that include the words "prove" and "proven" are simply wrong.

Pin-pointing China's attack against GitHub

Related Stories