Weak passwords brute forced

https://github.com/blog/1698-weak-passwords-brute-forced

by olefoo • 13 years ago

203 143 13 years ago

github.com

voltagex_2• 13 years ago

Good reminder to add 2FA via https://github.com/settings/two_factor_authentication/config...

voltagex_2• 13 years ago

If you use GitHub via SourceTree, this may not work though...

Solution: https://confluence.atlassian.com/display/SOURCETREEKB/Two-Fa...

allochthon2• 13 years ago

I think two-factor auth should be the new basic auth. Every site should provide it, either directly or through an integration with a trusted OAuth provider.

I'm waiting for the day that the security people work out a solid federated single-sign-on system that gains widespread adoption.

Goopplesoft• 13 years ago

Shameless plug: to get alot of sites using 2FA in a standardized manor is exactly why I made GAuthify (https://www.gauthify.com)

forrestthewoods2• 13 years ago

How is this possible? I would expect repeated failed login attempts would force a timer making a brute force attack impossible? Or at least more difficult. Is that not the case or am I missing a key piece of info? (Serious question, not trying to troll or be smug.)

ajtaylor• 13 years ago

The article did mention the use of 40k unique IP addresses, so I'm guessing their rate limiting is based on IP address and not the username or IP+username combination.

X-Istence• 13 years ago

Why not both?

pavs• 13 years ago

From the blog post:

"While we aggressively rate-limit login attempts and passwords are stored properly, this incident has involved the use of nearly 40K unique IP addresses. These addresses were used to slowly brute force weak passwords or passwords used on multiple sites. We are working on additional rate-limiting measures to address this. "

Pxtl2• 13 years ago

I've seen this mentioned before - using a tremendous number of IP addresses across a tremendous number of sites and over a massive length of time to perform a very low-grade incessant brute-force attack. It means that any weak password anywhere is always vulnerable because everybody is always under low-grade brute force attack.

ChuckMcM• 13 years ago

This is one of the more insidious scraping techniques. At Blekko if we see the same exact same query coming in from a bunch of different addresses, differing only by which page of the results they want we temporarily ban the entire lot of them.

I'm sure a similar situation would work here, especially with a decent geoip database. If you get 3 - 5 failed attempts on an account separated by more than a few miles (and not along an interstate :-)) you can probably assume someone is using a botnet to try to break in.

berlinbrown• 13 years ago

Seems a bit excessive for a open source repo site.

ye2• 13 years ago

Looks like they didn't rate-limit per-user, but only per-IP.

Now that is dumb. I expected more from GitHub.

beering• 13 years ago

You may not be trying to troll, but either you didn't bother reading the article or they added details since you read it (doesn't seem likely). They answer this near the end of the not-very-long article.

forrestthewoods2• 13 years ago

Hrm, so they do! I swear I read the article a second time and even re-skimmed looking for that detail before making my post. My brain really, really wanted to not read that next to last paragraph. Stupid brain.

jcampbell1• 13 years ago

The attack was done using 40,000 IP addresses. You can only force a timer per IP address.

muglug• 13 years ago

You can also force a timer on a per-account basis - e.g. twenty failed logins for one account from five different IP addresses -> set a timer on those IP addresses and a timer on that account. It's not simple, but it would help mitigate a mass-brute-force effort.

rurounijones3• 13 years ago

Rate-limiting based purely on username/password is a great way for me to lock other people out of their accounts (Or force them to do a password reset, or wait for a timeout) by just failing to login to theirs a few times.

The way around this problem is to add source IP into the rate-limiting algorithm so that I cannot mess someone else up.

Unfortunately that can be worked-around somewhat by using 40k different source IP addresses.

No perfect answer. Personally I think GH did ok here.

ams6110• 13 years ago

I suppose if you see repeated failures across a range of IP addresses you could temporarily block the entire account and send a reactivation email to the registered email address.

That would prevent a practical joker at work from locking you out unless he went to a lot of extra trouble.

evanspa2• 13 years ago

See "Part VII" of this SO answer from Jeff Atwood describing mitigation techniques for "distributed brute force attacks"

http://stackoverflow.com/questions/549/the-definitive-guide-...

TL;DR - compute the average number of system-wide failed password attempts, and if it's over the norm, impose small delay on all users (except those that login via a persistent login cookie).

ajcarpy2005• 13 years ago

Couldn't the server check for a "trusted IP(s)"?

Two-Factor Authentication is great but even without, it would seem that they could more aggressively lock out any new IP address which is failing password retries several times. (but continue allowing log-in from trusted IP's.)

annnnd• 13 years ago

Rate limiting is not the only thing they could have done. They could also notify users of failed attempts and ask them to confirm it was their attempt (once they were logged in of course) - the offending IPs could then be blacklisted.

Of course, this works only if the number of attacks is small enough.

shill• 13 years ago

I have a single failed login from an IP address in Brazil 16 hours ago. And another single failed attempt from an address in Thailand three days ago. No suspicious activity had been logged before that.

The attacker probably tried a password like 'github' on the first pass and 'password' on the second pass.

stokedmartin• 13 years ago

One could attempt to log in from multiple IPs over a long span of time, let's say weeks. This scenario is difficult to trace using timers. Alternatively, sending email alerts on each failed log in attempt may not very optimal and generate a lot of false positives - person A (not an intruder) randomly tries logging in by impersonating multiple github users.

IMO something similar to SSH security policies like allowing logging in from a set of IPs and/or without password i.e. using public key, may be a good idea.

scovetta• 13 years ago

What if the attackers were brute-forcing the username given a common password? Instead of:

  foobarjones:123456

  foobarjones:passwd

  foobarjones:test

Do:

  foobarjones:123456

  drsmith:123456

  ilikeorangejuice:123456

  ...

(some time later)

  foobarjones:passwd

  drsmith:passwd

  ...

Wingman4l711• 13 years ago

What was the purpose? Were the repositories private? Was the attack targeted toward known accounts working on new projects -- i.e., industrial espionage?

Fargren• 13 years ago

If you get a user-password combo, you can try it in more sensitive sites that may have stronger protection against brute force attacks. People reuse passwords a lot, specially the kind of users that use weak passwords on the first place.

If the account has an email address connected to it, you can also try the password against the e-mail account. That can be worth quite a lot if it works.

Wingman4l711• 13 years ago

Ah yes, what with the usual "email address being the skeleton key to your online identity" and all. I keep forgetting about this, and find myself wondering why the heck people are phishing me for my LinkedIn account login.

rurounijones3• 13 years ago

I just checked my security section (Didn't know about that!) and there were 12 attempts on my private, totally non-corporate, account.

My password has not been reset by github but I have updated it anyway.

On a sidenote I think that is a very clear and well written overview with clear details and a nice reminder about a section of the site that I was not familiar with.

Wingman4l711• 13 years ago

I just checked too (https://github.com/settings/security) and I had 6 attempts, all from unique IPs. Asked a friend of mine and he had 6 attempts (all from unique IPs) as well.

patio113• 13 years ago

I'm going to guess it was simple spray-and-pray. (My account took its fair share of compromise attempts, too, probably because I use the same username everywhere so it's going to be in many modern dev-heavy password dumps.)

Github is, for better or worse, a high-value target these days. Compromise of a Github account will often get you credentials which you can use to compromise high-value targets, for example production machines.

(Example from the Rails world: if someone gets read access to your git repository, expect them to figure out your secret token which you use for HMACing sessions to prevent forgery. If an attacker can forge sessions, they have remote code execution on your production servers. You can assume they will go directly from code execution to root on those servers, and from root on those servers to a systemic compromise of your entire network (if they want to do that).)

Use your imagination on what a mass compromise of accounts gets you. The simplest possible example is grepping for /bitcoin/, finding the full source code and credentials for a Bitcoin exchange, and rooting them then emptying the hot wallet. That results in fairly obvious economic advantage, right? You could also use scriptable attacks to root thousands of big-n-beefy production machines on fairly trusted IP addresses, then add them to a botnet, which you'd use for spamming or various other nefarious activities.

Then, even farther down the list, you have a dedicated adversary actually go through every repository they got access to and look for something uniquely fun to do with that particular target.

If you were looking to do industrial espionage on a particular target, you'd probably pick a way that was less likely to be detected and cause a company-level security response. (Use passive recon to identify one or a few likely target email addresses, find one or a few likely passwords for them, and try them first. Heads you win, tails you just left evidence in a log of perfectly normal user behavior. You might follow up this perfectly normal user behavior with a social engineering attack.)

Wingman4l711• 13 years ago

Don't forget API keys in commits from users foolish enough to trust the security of private repos.

Yeah, spear phishing / social engineering makes more sense for industrial espionage.

tedunangst• 13 years ago

Is there a setting I can change to turn off this "session -> code execution" feature?

Bluestrike2• 13 years ago

On the Rails note, you should be setting an ENV variable rather than hardcoding it as is the default. Same goes for any other keys/tokens/credentials that you're interacting with in your code. With these sorts of things, it's best to rely on pessimistic assumptions and take proactive steps accordingly.

borski2• 13 years ago

From a Github email to a friend to whom this happened:

"We have reviewed our logs and it doesn't appear that any actions were taken by the attacker other than to authorize the 'GitHub XRP Giveaway' application against your account.

You should be able to find the OAuth events for that application in your account's security history:

https://github.com/settings/security

We do not believe that the application's authors were responsible for the break-in, rather that the attackers were attempting to game the giveaway.

Ripple's explanation of the giveaway can be found here: https://ripple.com/blog/git-in-the-game-2020-xrp-giveaway-fo...

Wingman4l711• 13 years ago

Wow... a botnet-orchestrated massive brute force attack to try and game a cryptocurrency giveaway? o.O The future was yesterday.

As of this comment, 2020 RXP is worth ~ 0.03 BTC. Multiply by ~$500USD/BTC and you get ~$16 USD (over $20 when BTC was peaking $800+USD/BTC in the last couple days). Multiply that by the number of compromised accounts that meet the cutoff date criteria, and you get the take.

Potentially some good money depending on your success rate, but maybe not worth the cost of renting a botnet?

Wingman4l711• 13 years ago

Wow, ~40,000 unique IP addresses were used to circumvent rate-limiting.

Thankfully that even with that many unique IPs and presumably a few tries per IP, you still only have enough tries to crack weak passwords.

null_ptr• 13 years ago

Interesting, my account's Security History shows a few failed login attempts in the last 3 days from Eastern Europe and Southeast Asia, and my account is not even popular.

doughj3• 13 years ago

Your comment prodded me to check my own Security History, and I have five failed login attempts within the past 3 days from unrecognized IPs also on a not-popular account.

city41• 13 years ago

I do as well. One from Korea and one from Venezuela.

mikexstudios• 13 years ago

I have 12 failed attempts in the past 4 days, and I don't have a popular account either.

allochthon2• 13 years ago

Yep -- in my case from Turkey, France and Venezuela. And I'm in California.

Wingman4l711• 13 years ago

A friend and I both showed 6 login attempts from unique IPs and neither one of us are very active. Seems like they hit a lot of people.

imdsm2• 13 years ago

They did the OAuth token with the Ripple give away 16 days ago, and before that, I had no failed attempts, just a single successful attempt. I too used the same password for a few sites.

oneeyedpigeon3• 13 years ago

I've seen a small amount of suspicious activity (possibly 3 failed logins that weren't mine); I'm wondering how relevant various attributes of the username are. I'm starting to move to unique usernames for various services: not anything vastly secure, but something that's at least harder to automatically cross-reference. Does anyone know if using much longer usernames is a worthwhile investment?

pimeys• 13 years ago

I had 13 login attempts from South America, Asia and Middle East. My login details have been leaked in several leaks, latest being the Adobe leak. And even though I've had strong password, my fault's been that I've used the same password in lots of places.

Luckily the Adobe leak was the final straw and I finally started to use a good password manager and super strong, distinct passwords in all the services I'm using. It wasn't that bad at all, just selecting a good password manager[1], strong encryption and an easy way to sync your passwords between devices (git), I've had no trouble at all with strong passwords.

Although it would be nice to have an Android app for the password manager. I think it's a nice excersize for writing my first Android app at some point...

[1] http://zx2c4.com/projects/password-store/

andrewaylett• 13 years ago

I use Keepass, which has an Android client. Sync is via Dropbox, soon (I hope) to shift to git-annex.

kmfrk• 13 years ago

India here. Don't underestimate the long tail of user distribution.

S4M• 13 years ago

Same here, from Venezuela and Turkey. My github isn't affiliated with any company, so I don't really see the point in trying to hack into it.

captn3m0• 13 years ago

If these attackers had tried targeted brute-force on some limited accounts, they might have been successful without tripping over the GitHub security team. Nonetheless, kudos to GitHub for handling this perfectly.

benaiah• 13 years ago

Had five attempts over the last 2 days, four from Venezuela and one from Bangladesh. All unique IPs.

samweinberg• 13 years ago

Same here. 5 attempts over the last 2 days all from unique IPs. Four from Venezuela and one from Beijing.

uladzislau• 13 years ago

Can Fail2ban http://www.fail2ban.org help in this case?

nobodyshere• 13 years ago

Not really. Some ISP provide same external IP to multiple users and you have to take that into account. I think otherwise a good part of github's users would never be able to access it.

cft• 13 years ago

Just show captcha randomly on login, with the frequency proportional to login failure rate for an account.

eliteraspberrie• 13 years ago

Only allow logins from country/continent: X

With a feature like that, if an IP address from country C attempts to log in to accounts of users in separate countries A and B, they are quickly spotted.

itchitawa• 13 years ago

Gmail does something a bit more sophisticated than this but it's a real pain trying to check your mail from the airport in another country when your phone doesn't work either to confirm that way. Also annoying for VPN users who may appear to be in another country.

ismail• 13 years ago

Single failed login attempt from china. Though could this be any way related to the adobe leak? If you have the list of emails and password, you could try a bunch of commonly used services and see if that works.

objclxt• 13 years ago

GitHub do say in their post:

These addresses were used to slowly brute force weak passwords or passwords used on multiple sites.

..."multiple sites" could possibly refer to a compromised list, like Adobe? Of course, it doesn't have to be Adobe - Macrumors was compromised recently, etc etc.

arasmussen• 13 years ago

I'd love to know just how weak "weak" is. Something like [A-Za-z0-9]+ and <= 6 characters, or [a-z]+ and <= 8 characters?

FiloSottile2• 13 years ago

Here, it is weak as in "password", "qwerty" or "{your Lifehacker leaked password}"

codygman2• 13 years ago

my password was \w{6}\d{2} and it was compromised.

ya• 13 years ago

ripple.. you bastard,.

Wingman4l711• 13 years ago

It wasn't Ripple, it was someone trying to game their giveaway: https://news.ycombinator.com/item?id=6766293

fphilipe2• 13 years ago

Considering that the code in a GitHub organization or in a repo with collaborators is only as safe as the weakest link in the chain of users that have access to it, it would be nice if the admin of that org/repo could set a policy that all members need to have 2-factor authentication activated before they get full access to the codebase. Enforcing strong and unique passwords is hard, but enforcing 2-factor is easy and the bump in added security is high. At least I would sleep better knowing that, for an attacker to access the codebase, brute forcing would not be possible.

csmuk2• 13 years ago

Enforcing unique and strong passwords is not hard. We do it.

fphilipe2• 13 years ago

How do you enforce a unique password? With unique I mean that the user has not used that password for any other website.

mtrimpe• 13 years ago

Unique across all other services the user has accounts at? If you did that it would be both creepy and impressive. :)

welder• 13 years ago

This is a security log of important events involving your account.
18 hours ago user.failed_login: Originated from 178.245.129.47 (Istanbul)
3 days ago user.failed_login: Originated from 180.250.45.186 (Indonesia)
3 days ago user.failed_login: Originated from 123.119.141.184 (China)

Is this a distributed brute force attack against my account?

masklinn• 13 years ago

It's part of the brute-force effort against all of github, not specifically your account. I have 13 such events logged.

dbaupp• 13 years ago

I see similar a thing; I'd guess that they just tried a few common passwords against millions of accounts, rather than targeting some specific accounts.

Wingman4l711• 13 years ago

I wonder why Github didn't publicly post the details behind the attack -- namely, that is was an attempt to game a Ripple (cryptocurrency) giveaway. They mentioned it in their "heads up Github user, your account was compromised" emails, apparently: https://news.ycombinator.com/item?id=6766293

nilved4• 13 years ago

That wasn't in my email.

dm2• 13 years ago

Along with 2-factor auth, what about the option to block all IPs from outside of a certain country (if you live/work in the US, only allow US logins).

This obviously won't work for everyone, but might prevent a decent amount of brute force successes.

rythie2• 13 years ago

Facebook detects this when you move countries and makes you do a quiz about your friends pictures - though it can be pretty annoying [though I haven't seen it recently so maybe they've stopped].

krlkr• 13 years ago

I just dealt with this today with my own account using a proxy, quite a clever security approach IMHO

nilved4• 13 years ago

The attack must have been done over Tor. My account was flagged and locked but it's not reasonable to say the password was cracked. However, I do my day-to-day browsing over Tor and the matching IP addresses caused a false positive.

Its worth mentioning, though, that the rate limiting is at the account level and not the IP level. When I was trying to get access to my account before the email was sent out, my account was locked and remained locked even after developing a new Tor circuit. How were the attackers able to circumvent the aggressive rate limiting? Three password attempts a minute is a generous estimate and it would take longer than the expected life of the universe to crack even a moderately secure password at that rate (my old password had 704 decillion permutations.) Perhaps some part of GitHub isn't being rate limited?

ushi• 13 years ago

> The attack must have been done over Tor.

Nope, none of the failed attempts on my account originate from a tor exit node.

nilved4• 13 years ago

I only have Tor logins recorded and nothing indicates that my account was compromised, so one of the exit nodes I used must also have been used by the attackers. I guess it doesn't necessarily mean that the entire attack was done over Tor. Perhaps a zombie in the botnet was running it. :)

imdsm2• 13 years ago

Looking at my security log, it looks like they gained access over two weeks ago. No failed login attempts, just a successful login. No brute-forcing about it.

oauth_access.create: GitHub XRP Giveaway - 16 days ago

user.login: Originated from 23.29.121.166 - 16 days ago

andyhmltn• 13 years ago

Logging into my account, I can see quite a few failed attempts from a variety of IP addresses. All of them are from the same ISP in Venezuela. Quite scary, but my password is 40+ characters long and I have 2FA enabled. How weird!

Damin0u• 13 years ago

I wish the weak password blacklist was public, so we can all implement password check like this easily, and maybe get rid of them forever (even if banning "chicken" as password is only a way to have "egg" popping up).

shubhamjain• 13 years ago

This reminds me the #1 tip of web development: Never trust the user. I can't imagine there is a person out there who has all the time in the world to brute force accounts with common phrases with 40,000 proxies (or whatever). Seriously, is there anything worth gaining if you get hold of even say 100s of accounts.

gambler• 13 years ago

You can spread malware through Open Source projects.

Wingman4l711• 13 years ago

Well, first of all, it's very likely they used a botnet and some automated attack software. Secondly, there was a financial incentive[1] -- my back-of-the-napkin calculation was ~$15-25 per compromised account. That of course is ignoring secondary values like API keys, reused passwords allowing access to accounts on other sites, etc.

[1] https://news.ycombinator.com/item?id=6766293

cik• 13 years ago

I'm very impressed by GitHub's response - what a prime example of how you should handle an intrusion attempt! 11/10 for style from an IT governance and disclosure point of view. They've successfully: identified the nature of the attack, notified customers in a polite manner, publicly disclosed the goings on, and exercised added paranoia (which in this case == security) by disabling related accounts. Kudos! The only thing they missed is disclosing that they're working with local law enforcement - something I'd presume they're doing.

More interesting is the attempt. I frequently scan customer systems on their behalf, by spinning up clouds of resources to hammer their login APIs. By using Digital Ocean $0.05/hour boxes, and massive threading, tests tend to cost me ~$50/hour (yes, that's 1000 boxes orchestrated).

Given the real world cost of cloud computing, and the ease of libraries such as libCloud (or ruby's: FOG), the cost of doing this in terms of time and real money has dropped dramatically.

Weak passwords brute forced

Related Stories