SSH Key Audit on Github (required)

https://github.com/settings/ssh/audit

by ericelias • 14 years ago

292 105 14 years ago

github.com

memset• 14 years ago

Here is the command you use to obtain your fingerprint for this audit:

`ssh-keygen -lf ~/.ssh/id_rsa.pub`

zdw• 14 years ago

Some people might be running DSA keys - this covers that case:

`ssh-keygen -lf ~/.ssh/*.pub`

masklinn• 14 years ago

> Some people might be running DSA keys

Yep, or use a non-default key name.

cschmidt3• 14 years ago

Thanks for this. I found it kind of weird that the email they sent didn't include any instructions on what to do. The linked webpage didn't have any guidance either. I guess they assume we all know what to do.

charliepark• 14 years ago

This. In fact, it creates an even more dangerous situation, as users could go to the site, see their keys, and say "I dunno. Looks fine?" and approve all of the keys, without actually confirming that the keys are legitimate.

Not giving instructions on the page on how to verify the info was weak. Github people, if you're reading this, please update that page.

justincormack• 14 years ago

When I did it just now they did have instructions for Mac OS, Windows and (default) Linux linked off the page. Maybe they added them after the original email?

The Linux instructions were

ls ~/.ssh/*.pub | xargs -L 1 ssh-keygen -l -f

mise• 14 years ago

Very weird. What I consider to be a "public key" is one of those long strings of characters in my `.pub` files. What they asked me to verify turns out to be a fingerprint (from what I understand elsewhere in HN comments).

I'll be honest - I didn't understand what those MAC address-type fingerprints were, and I accepted each key. Their email made me trustful, saying that probably no account was affected. I do feel bad saying this in public, but it is what I did when there was not more information immediately available to me.

larelli• 14 years ago

I mailed them as soon as I received their notification to check my keys, that any kind of help on how to actually view the fingerprint would be useful. Also I suggested as a bonus, that they detect the OS from the browser and display the appropriate information. I assume, that I wasn't the only one asking for more detailed guidance. Funny enough, four hours later, they did the exact thing and notified me again via E-Mail. What a great response!

keeperofdakeys• 14 years ago

If you actually look around the page, there is the link "Need help verifying fingerprints?", which takes you to the correct page.

js2• 14 years ago

You can also use 'ssh-add -l' to see all the keys you've added to ssh-agent.

k33l0r• 14 years ago

You can also iterate over all the public keys in ~/.ssh:

    for f in ~/.ssh/*.pub; do ssh-keygen -lf $f; done

tonyarkles• 14 years ago

I had the same problem. I emailed them to invite them to add that exact command to the audit page. Glad to see I wasn't the only person who had trouble.

graywh• 14 years ago

There's a help page now: http://help.github.com/linux-verify-ssh/

finnh• 14 years ago

Sadly the link in the email isn't direct (it's a tracking link through "news.github.com"), so Thunderbird flags it as a possible phishing attempt =(

Edit: github send out an email with a link to the ssh audit page; that's the email to which I refer

spullara• 14 years ago

I guess this answers my questions about how long this vulnerability existed (a long time) and whether or not they could verify no other accounts were compromised (no).

zby• 14 years ago

I've just seen it and I headed to Hacker News to verify if it was legit :)

oelmekki• 14 years ago

Same here, wasn't sure seeing no reference to github.com in mails' header, even if the link pointed directly on their domain.

i guess that's the precise moment when I should feel too much paranoid.

Ecio78• 14 years ago

I've just registered yesterday on Github (it's suggested for Coursera's Saas Class i'm attending) but they've sent it to me too, even though the vulnerability has already been resolved before my account was created. Maybe they've not checked account age..

sophacles• 14 years ago

I suspect that since they only closed the hole late Sunday or early Monday, that they decided the number of new accounts was so small, it wasn't worth introducing the complexity (and potential bug/insecurity sources) to handle such a small percentage of the user-base. A reasonable tradeoff since getting this audit done in a timely manner is more important than waiting to handle edge cases such as yours.

jgrahamc• 14 years ago

It would be interesting to know the details of the vulnerability. Given that they've patched it, it would be good to see what the error was in case others are affected.

Was this Rails-related and what was it?

chimeracoder• 14 years ago

There are a number of articles that surfaced on Sunday/Monday, but in short, yes - it was a Rails mass-assignment vulnerability.

oscardelben• 14 years ago

I'd like to point out that this is not a rails vulnerability, but a mistake Github engineers made, which happens to the best of us. Mass assignment is a feature and I guarantee the problem has been know for years and Github engineers were probably well aware of it.

technoweenie5• 14 years ago

It was a mass assignment vulnerability in our code.

andrewjshults• 14 years ago

They also did a notification when you tried to push:

ERROR: Hi andrewjshults, it's GitHub. We're doing an SSH key audit. Please visit https://github.com/settings/ssh/audit/<removed>; to approve this key so we know it's safe. Fingerprint: <removed> fatal: The remote end hung up unexpectedly

A little weird to see when you're doing a push but good that they put it in there. Their email got flagged as bulk in gmail so until I saw this I didn't know they were doing the audit.

kapitalx• 14 years ago

> "good that they put it there"

I think this was handled very badly. This broke alot of automated jobs for many people. The email actually arrived after some of our automated jobs had stopped running. There should have been an advanced warning that such an audit would take place. I'm glad we had monitoring on our automations, but many people won't have those and might still be unaware that they are not running until they authorize their keys.

brazzy3• 14 years ago

AND when you tried to pull, which broke my current project's build. First time I've seen current news directly affect my work...

pilif2• 14 years ago

What makes me the most happy about this is that they ask for the password in order to add a key now.

I was always very afraid of XSS attacks (I know - there shouldn't be any - but there could and were, though not for this) that would add another key, so I always hoped they would add that additional bit of protection.

As such: Another huge thanks to @homakov for forcing the issue.

natrius• 14 years ago

This change is orthogonal to @homakov's attack. Silently changing the user id on a key wouldn't generate an email.

ajross• 14 years ago

The change is good and was quite clearly precipitated by Homakov. Thus it seems entirely reasonable to thank him. So I don't see what your point is beyond irrelevant pedantry.

homakov4• 14 years ago

You are right. Same with asking password. But these additions are very sane.

nitrogen• 14 years ago

Note to sibling comment by SaltwaterC: a previous negatively-voted comment (https://news.ycombinator.com/item?id=3593799) got your account auto-killed.

SaltwaterC 5 hours ago | link [dead]

The API token is still there, in the "plain": https://github.com/settings/admin

Fetching it via XSS should be fairly trivial. Via a simple script in that page is straight forward. Still have to see if I can get it via XHR :).

niels_olson2• 14 years ago

Um, is anybody else having the experience that their keys really do seem to be different?

rmc• 14 years ago

You should probably tell GitHub

niels_olson2• 14 years ago

I thought I deleted this comment. Sorry. Turns out I had an old key still in my .ssh folder. Two current and valid keys and no others. My bad.

jdludlow• 14 years ago

Each fingerprint listed on my audit page was exactly what I expected it to be.

brown9-2• 14 years ago

Sure you aren't confusing the fingerprint with the actual key contents?

tclem• 14 years ago

Did you contact support@github? We definitely want to know about that.

alastairpat• 14 years ago

I'm missing a whole bunch of keys – I previously had about 4 keys registered, now it's just one.

spicyj• 14 years ago

The accompanying email:

  A security vulnerability was recently discovered that made it possible
  for an attacker to add new SSH keys to arbitrary GitHub user accounts.
  This would have provided an attacker with clone/pull access to
  repositories with read permissions, and clone/pull/push access to
  repositories with write permissions. As of 5:53 PM UTC on Sunday,
  March 4th the vulnerability no longer exists.

  While no known malicious activity has been reported, we are taking
  additional precautions by forcing an audit of all existing SSH keys.

  # Required Action

  Since you have one or more SSH keys associated with your GitHub
  account you must visit https://github.com/settings/ssh/audit to
  approve each valid SSH key.

  Until you have approved your SSH keys, you will be unable to
  clone/pull/push your repositories over SSH.

  # Status

  We take security seriously and recognize this never should have
  happened. In addition to a full code audit, we have taken the
  following measures to enhance the security of your account:

  - We are forcing an audit of all existing SSH keys
  - Adding a new SSH key will now prompt for your password
  - We will now email you any time a new SSH key is added to your
    account
  - You now have access to a log of account changes in your Account
    Settings page
  Sincerely, The GitHub Team

  --- https://github.com support@github.com

rwmj• 14 years ago

http://unix.stackexchange.com/questions/2116/given-keys-in-s...

This script is very useful when doing this audit, because you can turn your .ssh/authorized_keys file into a list of key names and fingerprints to check against what github is showing you.

homakov4• 14 years ago

you got balls guys. It is hard to force everyone to do something but you did it. Kudos

also, if we go back few years ago this way would be a bit secure to handle keys @key.body = params.. @key.title = params.. I am sure update_attributes is good choice when you got 5+ fields and update database scheme pretty frequently. Just my 2 cents

technoweenie5• 14 years ago

We sent someone back in time to make this change. Somewhere there's an alternate universe with a secure GitHub by default. Thanks!

rdl6• 14 years ago

Why are ONLY keys at risk, which this implies?

Presumably someone could have added a key, done evil, then removed the key. Evil includes all sorts of interesting things, like checking in code under the name of an existing contributor. This could potentially be really subtle and would be difficult to find in an audit later.

(Remember the stink over OpenBSD potentially having backdoors in the IPsec stack, revealed in late 2010? http://blogs.csoonline.com/1296/an_fbi_backdoor_in_openbsd)

ivix• 14 years ago

My guess is the last thing github want to do is make its users aware of that distinct possibility.

gcr2• 14 years ago

I disagree. It is github's duty to provide a realistic view of the situation.

Such an attack is unlikely, yes, but still possible.

archivator• 14 years ago

I thought git itself provides authentication and integrity? In fact, if someone modifies the history of a branch that will raise all sorts of red flags the next time a legitimate user pushes.

rdl6• 14 years ago

Not modifying stuff on disk outside git, but using substituted keys to masquerade as a legitimate user and contribute evil code. It leaves an audit trail, but means you need to audit all contributes even those ostensibly from project owners.

gcr2• 14 years ago

Not really. If you make a new commit on top of the history, then nobody will notice unless you sign every commit (which i don't think is common practice).

Only modifying (existing) history will cause git to complain.

homakov4• 14 years ago

I don't see any way to remove key after you dropped it to target.

rdl6• 14 years ago

Ah, I didn't check that. It still means a bad key could have been used for evil, just that there would be clear evidence left behind.

The issue is that I've had lots of ssh keys, and might not have my fingerprints for all of them. If I see a bad fingerprint, it's entirely probable it's just an old key of mine from an old laptop, cellphone, script, or whatever, and NOT an attacker's key.

But, in light of this attack which you revealed, now any account which contains keys which aren't 100% accountable could have been compromised by an attacker. (in fairness, someone who stole the github user password could have done the same thing too, but that's an obvious attack route)

Key management is such a pain!

technoweenie5• 14 years ago

There's no way to delete keys on another account. The bug only let someone move their key to another account.

tomjen32• 14 years ago

Then you move the key back afterwards?

DannoHung• 14 years ago

You could just take a gander at your activity and see if there are any commits that you didn't make.

famousactress• 14 years ago

It's possible that there's a plan to review or alert repo-owners to commits introduced by keys that are rejected in this process.

pak2• 14 years ago

As an interesting side effect, they will have pretty exact stats on how many active users they have; might help them sunset old accounts or move them to the slowest servers.

(Because of the offline nature of most git actions and different habits on pushing/pulling, it's probably hard to otherwise estimate how much a user cares about their github.)

arohner• 14 years ago

Why is this more effective than say "number of push / pulls in the last month"?

pak2• 14 years ago

Some people try to pool local commits into larger, less frequent pushes and pulls so the number of push/pulls is perhaps less relevant than their cumulative size. But pushes/pulls will never correspond well with user involvement because people use github for all kinds of scenarios. For instance, I might be developing branches that I don't want to push or pull to github yet--maybe I don't even intend to ever make them public. However, I may still want people to clone from my github repo and report issues to me.

The amount of time between someone getting an email from github and re-activating their account is probably the best metric github will ever have on users' attachment to their accounts.

icefox• 14 years ago

From the way their servers are laid out in the presentation I saw they wouldn't want to move them all to a slow server, but would want them evenly distributed to help decrease load on the servers.

technoweenie5• 14 years ago

We have stats on this from other sources, actually. But it did provide some interesting graphs this morning :)

levigross• 14 years ago

They also added a audit log so you will be able to track and address any future issues.. https://github.com/settings/security

avar2• 14 years ago

Correct me if I'm wrong but the nature of the vulnerability was that someone who's not you had to submit a page with certain POST variables they could have determined after the fact to be malicious while logged in.

So the fact that they're sending out this E-Mail tells us that they either don't keep logs on requests + POST contents, or that they haven't had the time or inclination to analyze this data if they have it.

chrisguitarguy• 14 years ago

Or they realize that psychology is just as important to security (and customer confidence) as logs, analysis, and good code.

Every person that got this email now feels more secure about Github. They audited their own private keys. They were reminded that they can remove keys at will. And they know Github has improved its code and given users more power (email alerts, etc) to be in control of content.

avar2• 14 years ago

I for one am feeling way less confident in them with every announcement they make. They clearly have no idea if and how they were exploited, and their communication with their users only asks their users to check for one attack vector, while in actuality the attack was not limited to just adding ssh keys.

maratd• 14 years ago

> So the fact that they're sending out this E-Mail tells us that they either don't keep logs on requests + POST contents, or that they haven't had the time or inclination to analyze this data if they have it.

No.

Github is primarily a B2B company. They're not making their big bucks off of individuals.

Businesses understand that problems arise. What they want to see is immediate action taken to rectify the problems.

Business 101. Even if the problem can be easily fixed by flipping a switch on your end that the customer never has to know about, always show the client "you did something to fix the problem". This is an in-your-face-we-are-taking-charge action. Even though it is completely unnecessary from a security standpoint, it is necessary from a business one.

They get it.

philjr• 14 years ago

Do people keep POST contents?

there• 14 years ago

Rails logs POST and GET parameters by default.

technoweenie5• 14 years ago

This isn't the only thing we're doing. Most of the work is going on behind the scenes though.

homakov4• 14 years ago

Is it a good idea to check created_at != updated_at ?

People update public keys very rarely. I would even say NEVER.

Just make an sql against your table to see what are the most possibly are malicious keys.

(i see no reason to update timestamps doing 'the trick'. I believe attackers didn't)

aparadja• 14 years ago

The vulnerability probably also allowed the attacker to edit the created_at and updated_at columns in the ssh key table.

benatkin• 14 years ago

It was easier for me to just delete all of the keys. I had some I didn't need anymore. I also didn't pick great names for the keys I had. It's easy to add a key so instead of checking the fingerprints I can just create a new key.

joshklein3• 14 years ago

Several comments below praise the Github team response to this vulnerability. I agree. But it should also be mentioned that the first email I sent to my company this morning read, "should [our product] source code be in the cloud?"

rdl6• 14 years ago

At the very least, I think it is irresponsible for your product's source code to ONLY be in the cloud. Luckily git provides an easy way to keep a mirror (it's kind of automatic), but some kind of regular off-github backup, signing, etc. would make a lot of sense.

joshklein3• 14 years ago

This is a vulnerability where an attacker would be able to add his SSH key to a private repository and pull proprietary source code he was not authorized to see. This is why we pay for Github, not use a free account. We don't want people to be able to walk off with the intellectual property of our company.

That being said, we don't have the resources to deploy a more secure alternative without hamstringing our development capabilities (e.g. no internet connectivity).

ricardobeat• 14 years ago

Did this change just disable re-use of deploy keys across multiple repos?

my8bird• 14 years ago

while this was a good response to their security issue a little heads up would have been good. they broke all of our auto builds and by the time we figured it out the guy who's key was used for the builds was gone on his vacation. luckily, we got ahold of him prior to him turning his phone off.

skrebbel• 14 years ago

Damn I envy the GitHub guys. They can send a mail to their users about SSH Keys and nearly all users simply understand it and get it over with.

In any other business, the result of a similar mail would be an overloaded helpdesk, a significant reputation hit and a massive bucketload of competitor FUD.

tomjen32• 14 years ago

Why are you guys praising GitHub? They basically screwed up thrice: first by not catching such an obvious flaw (granted it should have been changed in Rails, but still), second by breaking half the scripts that rely on their service and finally by sending such an obnoxious email (really required action? Who the hell to do you think you are?).

Anyway it is pretty moot at this point since I have long ago forgotten my password and changing the orgion to somebody else is pretty easy.

That said, can anybody recommend alternatives? I know Bitbucket and they seem pretty great, especially as they allow private repositories, but it seems the consensus here doesn't like them for some reason?

unoti• 14 years ago

I love Bitbucket and have been using them exclusively for a long time. I have many private repos histed there for free. I have several private repos there shared between a few users, also free. I'm unnerved by how they havent asked me for money, but grateful. Free jira, wiki, and hipchat, too.

I'm a Mercurial guy, but they do git too.

SSH Key Audit on Github (required)

Related Stories