We may have witnessed a NSA "Shotgiant" TAO-like action

http://blog.erratasec.com/2014/03/we-may-have-witnessed-nsa-shotgiant-tao.html

by julespitt • 12 years ago

235 138 12 years ago

blog.erratasec.com

mjolk• 12 years ago

In 2012, during an incident, we watched in real time as somebody logged into an account reserved for Huawei tech support, from the Huawei IP address space in mainland China.

I'm a little skeptical.

I wonder what they mean by "watched," because I doubt that they guessed the tty for reading or that the hacker joined a screen session. What is the likelihood that one would just "happen" to be staring at that server during an "incident."

TrainedMonkey• 12 years ago

From encrypting and emailing to a hotmail account, it sounds like they connected with Teamviewer/VNC type of software and author was watching what was going on the screen.

I am somewhat skeptical as well.

willvarfar• 12 years ago

Unless the support staff use screen sharing software for doing support, and what was witnessed was an unannounced support session?

dmix• 12 years ago

Google "live incident response" software. The author works for a security company, they get hired to detect malware and intruders.

You can remotely connect to machines and analyze memory, commands, etc. It doesn't matter what TTY it was when you have full system access.

http://digital-forensics.sans.org/blog/2011/07/21/live-mem-f...

https://code.google.com/p/grr/

https://www.mandiant.com/resources/download/redline

jessaustin• 12 years ago

I wonder what they mean by "watched"...

My interpretation was that after IDS had identified a particular host, they had tailed syslog (or the equivalent) on that host. The observation that they would have missed it if they hadn't been watching seems to imply that normally their logs wouldn't have retained the level of detail needed to see either the event or the deletion of the logs of the event.

robertgraham8• 12 years ago

It was an internal system. We noticed with 'netstat' that it had a connection to an outside system. 'who' told us it was the account setup for Huawei remote support, and the IP address told us indeed that it was from a Huawei network.

The SQL query took 15 minutes to run. We saw it using 'ps'.

We then kept dumping their '.bash_history'.

peterwwillis5• 12 years ago

That's weird. Bash_history doesn't usually get flushed for every command you run; only when you exit an interactive shell. If you `kill -9 $$` or erase the .bash_history file and create it as a directory, it loses the history. The exception is if you create a custom PROMPT_COMMAND="history -a; history -n", which would append on each new bash prompt. (You'd think a hacker would know these things...?)

As an alternative to dumping history, if your system has perl and strace and you want to watch a live ssh or bash session, I wrote a script that will do that. https://github.com/psypete/public-bin/blob/public-bin/src/sy...

gwu78• 12 years ago

It seems hard to believe someone doing something like this would not at least try to cover their tracks.

  export HISTSIZE=0?

And how do you know these were unmodified versions of netstat, who and ps that you ran?

Do they have mtree in this OS?

I'm no security expert but this little story just sounds very unsophisticated given the seriousness you are attributing to it.

jnbiche• 12 years ago

So what was the SQL query?

noir_lord2• 12 years ago

SELECT * FROM USERS WHERE TERRORIST='1'

LukeWalsh• 12 years ago

SELECT * FROM HIDE_FROM_AMERICANS; ?

notastartup• 12 years ago

SELECT * FROM Foreign_Corporations fc INNER JOIN Foreign_Corporation_Employees fce ON fc.FCID = fce.FCID WHERE fc.Country IN ('Pakistan', 'Afghanistan', 'Iran')

robertgraham8• 12 years ago

I can't reveal the exact SQL query because that's customer private information.

However, it had both a subject and a timeframe that were peculiar. Googling the subject revealed news stories about it -- making it clear this was something the U.S. was interested in, but which would be no particular interest to anybody else.

Zhenya• 12 years ago

Joining in the from statement - thank you.

at-fates-hands2• 12 years ago

SELECT * FROM PLANS WHERE HOW_TO_RULE_WORLD='1'

peterkelly2• 12 years ago

They already have that information

noir_lord2• 12 years ago

The normal guidelines for developing a security strategy is to estimate the resources and capabilities ranged against you and the probability they will be levelled against you and then develop a strategy for mitigation (absolute security is impossible).

The capabilities the NSA and GCHQ have developed are scary enough in and of themselves but the sheer breadth and depth of what they have achieved is far more horrifying, If I was the CTO for a large multi-national or a foreign government I'm not even sure where I'd start protecting against them.

jobu• 12 years ago

What I don't understand is why the US government would point fingers at the Chinese for putting backdoors in Huawei devices when it was really the NSA all along. It seems like they're shooting themselves in the foot by giving pointing out the backdoors. My best guess is that they assumed someone would figure it out eventually and they wanted to spread misinformation to get out ahead of that.

Has anyone else come up with a better reason?

UVB-76• 12 years ago

I don't think they are referring to the same vulnerabilities.

The US government was publicly accusing the Chinese government of inserting backdoors in Huawei products, while at the same time seeking and exploiting vulnerabilities in Huawei products themselves.

Perhaps this was an attempt to cover tracks by pre-emptively blaming the Chinese government for backdoors installed by the US government, should these backdoors ever be discovered.

Personally I'm inclined to call Hanlon's razor on the hypocrisy of it all.

Ihmahr• 12 years ago

I think an important part of the mission of the NSA is to spread fear, so that people are more likely to consent. This means terrorist (which of course isn't really a danger if you look at the numbers), nuclear Iran, cyber China, etc.

at-fates-hands2• 12 years ago

Two things.

First, if its fear they're trying to spread, it's working. No way I'd ever use a Huawei device, ever.

Secondly, do you think a nuclear arms race in the middle east would be a good thing??

stephengillie• 12 years ago

One of the best ways to distract others from blaming you is to publicly accuse them of doing what evils you're secretly doing.

ypeterholmes• 12 years ago

And thus the war on terror was born

Tobu2• 12 years ago

It's also a good way to drum up support; start by accusing the "other side" of doing whatever it is you want to do. You're basically coordinating with the other team with the full cooperation of the people you've scared.

jordan0day• 12 years ago

Why couldn't it be both? To me, the least-surprising explanation would be that both the Chinese government and the NSA were trying to exploit Huawei. Any finger pointing could be chalked up as typical "left hand doesn't know what the right hand is doing."

frandroid• 12 years ago

Protectionism.

zokier• 12 years ago

You are assuming that the left hand knows what right hand is doing.

VintageCool• 12 years ago

According to the article, it wasn't the NSA that put backdoors in Huawei devices, it was Huawei. It sounds like like any intelligence agency that can bribe a Huawei support engineer has access to any Huawei networking device. That definitely includes the Chinese and the American intelligence agencies.

The US intelligence agencies can simultaneously act to protect the information security of American businesses by warning people of the vulnerability AND act to exploit the vulnerability for their own intelligence-gathering goals.

mattkevan• 12 years ago

I imagine it's because they had already compromised the Huawei networks and knew full-well what the Chinese government were up to.

ww520• 12 years ago

It's a political way to keep the Chinese firms out of the U.S. market.

paul• 12 years ago

Where there are security vulnerabilities, I'd rather it be the NSA exploiting them than someone else. The fact that Huawei support engineers have so much power is much more troubling.

Nacraile• 12 years ago

I think it's naive to assume that the NSA is the only entity that is likely to be able to exploit vulnerabilities. This is the crux of the controversy around NSA's attacks on web security.

I only see a difference between an opaque, unaccountable organization in the USA and an opaque, unaccountable organization in China when I look through a nationalistic lens.

lukeschlather• 12 years ago

Qualitatively speaking, I think as opaque as the NSA and CIA are, they're more accountable to the average US Citizen than their Chinese equivalents are to the average Chinese citizen.

They're less accountable than I would like, but they are accountable for their actions.

insuffi12• 12 years ago

Why would you prefer NSA exploiting them?

bpodgursky• 12 years ago

Because I'm American, and while I'm not really a fan of most of the shit they're pulling, it's still more likely to be in my interest (or less against my interests) than whatever the PLA has in mind.

robertgraham8• 12 years ago

No. It was a username/password assigned to Huawei tech support.

tptacek5• 12 years ago

This is so obviously true that I found the tone of the post confusing. It's similar to people's reaction to the comparative threat of keeping their email on Google Mail or some random webmail provider that's likely to lose their mail spool to SQL injection. I'm not arguing that the NSA threat isn't worrisome; it is. But other threats are in fact even worse!

rhizome2• 12 years ago

I'm not arguing that the NSA threat isn't worrisome; it is. But other threats are in fact even worse!

As HN's possibly-most strident torchbearer for the measurement of organizational dynamics, can you quantify this statement? How are your ranking?

robertgraham8• 12 years ago

There was no "tone" to the post. It's not pro or anti NSA. Something was in the news. I saw something related to that. So I reported it.

ig1• 12 years ago

Criminals by and large just want money, governments want power. That makes them a far more serious threat.

A government is far more likely to oppress you, deny you rights, blackmail you for political reasons, etc. because it has the resources to do so.

baddox2• 12 years ago

Sure, you're both saying that the least bad option is the best option, which is obviously true. But that's not really the argument, or at least not the end of it. Some of us would like to have options that are significantly better than the best existing option.

monkeynotes2• 12 years ago

You are assuming that an NSA employee is more responsible than a Huawei employee. I am equally uncomfortable with either.

peterwwillis5• 12 years ago

Probability dictates it's more likely Huawei will abuse its power than the NSA will abuse Huawei's power. So be equally uncomfortable if you want, but it's Huawei you should be scared of (in this case).

madaxe_again• 12 years ago

You are assuming that the individuals are distinct - it is FAR more likely that a TLA agency has implanted support engineers who operate on their orders.

ohwp• 12 years ago

There will always be a person with top level access. In networking these people are chosen by trust.

For example. You can become a Juniper Networks Certified Internet Expert but that doesn't mean you can get a job. People still need to trust you.

And a good spy is someone people trust.

fit2rule• 12 years ago

Isn't this a little bit like being glad that American phone records aren't being processed in Israel?

RankingMember• 12 years ago

It's scary to think that a third of the internet relies on any one company's backbone products, regardless of the country that company calls home. Way too many eggs in one basket, but much easier for the humans involved compared to having a ton of different manufacturers who would have their own individual issues. Find an exploit once, employ it (most) everywhere (appropriation of old Java tagline).

italophil• 12 years ago

It's one thing getting spied on by the US government, but one would hope they'd use something more sophisticated than Hotmail to move the information around.

makomk• 12 years ago

Trouble with that is, the more sophisticated and unusual the mechanism you use, the more likely it is that someone will (a) notice you, and (b) be able to identify you based on it.

robertgraham8• 12 years ago

That's my theory. Any monitoring of outgoing information would just see a typical attachment to a hotmail address.

diminoten2• 12 years ago

Well, it was encrypted, and they can't exactly send it to "nsa.gov", can they?

dantiberian• 12 years ago

Hotmail seems like a reasonable first link in the chain to extract the information. I assume they would log in through Tor or another anonymous method before extracting it to its final location. They're not going to email it to joe.spy@nsa.gov directly.

pkinsky• 12 years ago

It's a dead drop. If this was cold war era Moscow, they'd use a drainage ditch or a loose brick. Hotmail is just the digital equivalent.

brown9-2• 12 years ago

A backdoor or 0day for a Huawei router would be of limited use to the NSA, because the control ports are behind firewalls. Hacking behind firewalls would likely give full access to the target network anyway, making any backdoors/0days in routers superfluous.

But embedding themselves inside the support infrastructure would give the NSA nearly unlimited access to much of the world. Huawei claims that a third of the Internet is running their devices. Almost all of it is under support contract. These means a Huawei support engineer, or a spy, can at any time reach out through cyberspace and take control of a third of the Internet hardware, located in data centers behind firewalls.

So the companies that use Huawei's products put the control ports behind their firewalls, but somehow are allowing unrestricted access through that firewall to/for Huawei's support mechanism?

Is that common?

ds9• 12 years ago

I was skeptical of that too - however, another part of the article implied another scenario, leaving it unclear whether the writer confused them or refers to both. The second scenario is, company may control its own gear well enough, but relies on a service provider that uses Huawei devices, and the Huawei support people can access comms thru the latter. Which begs questions about encryption, so the implications are murky.

robertgraham8• 12 years ago

Extremely common.

It's the norm today that companies have firewall/VPN holes allowing support engineers from other companies to have access to their networks, to manage things as simple as the HVAC system, or things as complex as their entire routing infrastructure.

Throughout the world, most Huawei routers come with such support contracts.

eropple• 12 years ago

to manage things as simple as the HVAC system

Hello, Target breach. =)

sliverstorm5• 12 years ago

I'm guessing what that means is something like this:

1. Huawei has support contracts

2. Huawei needs to be able to interact with their hardware to execute those support contracts

3. Companies don't want to expose routers

4. Huawei routers "phone home" (i.e. query Huawei) and in this fashion allow Huawei support to establish a connection

danielweber3• 12 years ago

Firewalls can (and should) block outbound connections, as well. (Although trying to ultimately clamp down on this in any normal business environment where humans want to web browse is a battle you will ultimately lose if you are trying to completely stop all of it.)

diminoten2• 12 years ago

I'm not sure if this is in any way useful, but consider that Ed Snowden himself was in a "support"/administrator role and that's what gave him access to the documents he later then leaked.

jontas• 12 years ago

I dont understand why this level of access (if it is accurately described in the article) would only be of use to American intelligence, and "would['t] interest other intelligence services -- except to pass it on to the Americans."

It seems like something that powerful would be of interest to any intelligence service (or group of any sort), anywhere.

sliverstorm5• 12 years ago

You misunderstand. The author is saying that the particular SQL query they saw executed, would only return data interesting to Americans

danielweber3• 12 years ago

That's leaving a lot to his interpretation.

Chinese intelligence might be interested in something simply because they (correctly or not) deduce that American intelligence will be interested in it.

vampirechicken• 12 years ago

How does the support login have the privileges to delete all of the activity log files, and why is a login with enough privilege to delete logs allowed to perform SQL queries?

malandrew• 12 years ago

What we really need is a new agency just like the NSA except for it's only mandate is closing holes everywhere even if those holes are actively being exploited by the NSA and CIA. Such an agency would actively discover holes, patch them when possible or disclosing the vulnerabilities to the engineers responsible for the software or hardware in question. Furthermore, the NSA and CIA would need to be barred from trying to get any access to this organization for its own use.

danielweber3• 12 years ago

This blog post is trying to say something tremendously important but it also is not giving us any information to evaluate it. Apparently everything is on fire but they can't tell us how.

eli• 12 years ago

I don't understand why there's a sharp distinction between installing a backdoor and using stolen support access as a backdoor.

tzs• 12 years ago

TAO?

Edit: finally found it, with some Googling. There are a lot of things with TAO as their TLA leading to a lot of false leads. TAO in this story means "Total Access Operations".

Edit 2: "tailored", not "total".

mikecb• 12 years ago

Tailored.

MatthiasP• 12 years ago

Tailored Access.

tptacek5• 12 years ago

Among vulnerability research people, TAO is practically slang for "the branch of NSA that hacks into Chinese computers". Robert Graham comes from those circles. I think that's what he's trying to evoke by referencing TAO.

throwaway7767• 12 years ago

I was not aware that the NSA had decided to restrict TAO's operations to China. That must be a very recent thing if so. Can you provide any references to that?

sgt101• 12 years ago

Witnessing in this way runs counter to my experience of system management. How can you see (in real time) a query, the encryption, the email and the log deletion? I have run sql monitors and I see queries appear and then disappear... but my brain doesn't allow me to understand what the user is "up to" without lots of investigation and so on.

peterkelly2• 12 years ago

One of the biggest ironies of the Huawei hacking case is that now every time someone detects an attack from a Huawei device or the company itself, they can never be sure if it's China or the US that's behind it.

einhverfr3• 12 years ago

This sort of thing is significant. It puts remote support for systems in a very different light. At Efficito, we have plans to release on-premise appliances as well as our cloud hosting options. This sort of story makes me think about how to avoid this sort of problem.

Here are rules I am suggesting.

1. The on-premise appliance should not be directly accessed from the network unless folks at the local environment enable contact.

2. Everything else, regarding services, should be loosely coupled and designed not to give significant access to either party over the other.

This sort of thing strikes me as an area where the industry is going to have to evolve. The danger of "we can connect to your systems" is becoming clearer to a larger section of the market.

We may have witnessed a NSA "Shotgiant" TAO-like action

Related Stories