1.4B records from “Have I been pwned” for analysis

https://www.troyhunt.com/heres-1-4-billion-records-from-have-i-been-pwned-for-you-to-analyse/

by runesoerensen • 10 years ago

165 37 10 years ago

minimaxir3• 10 years ago

Hmm. Given the amount of information removed (justifiably so), there isn't much left to analyze other than aggregates. (Which the dashboard mockup in the post already does)

What might be interesting is graphing a network of the relationship between services which suffered a breach. Fortunately, I have the proper workflow prepared to process the data in this manner, so I'll take a look.

drvdevd• 10 years ago

So it would appear the most interesting data point on these records is the date and time of the breach discovery, or using that to estimate a breach date? Correlating that with some other publicly available data sets (e.g.: historical routing data) might yield some useful results?

thaumaturgy2• 10 years ago

Yeah. I would've rather had a dump of just the passwords -- no usernames, site names, service names, counts, deduplication, just 1.4 billion passwords, one password to a line.

I'd feed it into my bad-password-indexing program (https://github.com/robsheldon/bad-passwords-index) and it would spit out a file that could be used to improve password security in a not-entirely-stupid way somewhere.

ComodoHacker• 10 years ago

Also that's exactly what malicious actors would want to improve their dictionaries, to brute-force poorly-hashed dump from the next breach.

NKCSS2• 10 years ago

It's not all plain-text pwd's ofcource; a lot of the records (thankfully) have had hashed passwords.

minimaxir3• 10 years ago

Alright, made a first draft illustration of the relationship: http://i.imgur.com/FpLiFGk.png

I might be onto something. It appears the communities algorithm correctly detected the Russian and China communities.

flashman2• 10 years ago

You know, this is exactly where my mind went, too: http://i.imgur.com/fME7MK7.png

The difference between your first draft[1] and mine is that I've only included links where each site had at least 1% of users in the other site. This helps make links more meaningful.

For instance, about 37k users were members of both Tumblr and Plex. Tumblr had the largest number of Plex users, so it should be a strong link, right? No: because Tumblr has such a large userbase, this is a relatively insignificant relationship; 0.14% of Tumblr users had Plex accounts, which ranks only 20th for Plex.

Compare this to Plex and Xbox-Scene. 5.6k users had Plex and Xbox-Scene accounts, which is about 2% of each site's userbase. I believe this link is much more meaningful than the Plex-Tumblr link, and I believe my visualisation bears this out.

[1]http://i.imgur.com/FpLiFGk.png

* Note: since I ignored all lines with only one site, this "1%" only includes users who were in more than one breach!

minimaxir3• 10 years ago

Yes, that sounds like a sane optimization compared to my black-gray mess.

jasonallen2• 10 years ago

I'm always amazed at how efficient AWS is at detecting private keys on the internet (checked into github, etc..), and then proactively locking down accounts. I wonder how long it will be until we see a similar service from consumer accounts, like github or twitter. Seems like "have I been pwned" might offer a commercial API for such benefit...

symlinkk• 10 years ago

HIBP is simply a database of email addresses that have been associated with leaks. I don't see how that would help him identify private keys.

johncolanduoni• 10 years ago

Well AFAIK Twitter doesn't give users keypairs, so I think they meant do the same with leaked credentials.

imtringued• 10 years ago

If the bots can find the api keys then amazon can just run their own bots.

gkafkg8y8• 10 years ago

I used to be somewhat interested by stats on passwords, etc. from breach data dumps.

haveibeenpwned is a helpful and legit site, though I think it should have used email confirmation instead of requiring only an email address.

I also respect Troy along with many other security researchers. Even those that are up to no good in the security world in some ways have contributed good things; after all, the rest of us are stronger and more vigilant now than we used to be because of their work.

However, this anonymized data will almost certainly be used by black hats more than white hats, and I don't see how this release is good for the majority of those that were affected by these breaches.

Arcsech• 10 years ago

It does require email confirmation before showing "sensitive" data, like if an email was in the Ashley Madison or AdultFriendFinder breaches.

problems2• 10 years ago

Which in and of itself is silly. The raw dumps are already available to everyone, blackhats included. Personally I'm tempted to make a site that just lists it by domain and stuff. I found several people at my company with Ashley Madison accounts using a quick grep.

Gaelan• 10 years ago

How will black hats use the data?

Systemic• 10 years ago

I'm with you that haveibeenpwned should require email confirmation before listing the breaches an email address was in. As it is, it has become an easy to use dictionary of places to find personal data for any given email address. Yes it doesn't list the breaches that Troy has deemed "sensitive", but if you're trying to commit identity theft or fraud using personal data, it doesn't really matter if these search results don't include websites that deal with porn/adultery/children.

Doubtless there are black market tools that provide such a service but expose far more data, but haveibeenpwned lowers the barrier to entry significantly by being far more available to the public.

jxramos• 10 years ago

Good distinction, data for analysis/trends, data for nefarious victimization.

ksec• 10 years ago

I have wanted to ask this for a long time. What do / should you do once you have been pwned?

You definitely have to change password, or even use password manager. But your record is now widely available it feels as you have been naked on the Internet.

You may likely change your email address ( Login Name ). But opening another email account is such an hassle. An opening yet another account on those of your favorite site means you lost all of your pass record.

hughes2• 10 years ago

What are you worried about here? Further breaches? If you've changed your password, there's not much more you can do. Getting a new email address doesn't make you more secure. If you're trying to conceal the fact that you have an account somewhere, you should not be using your email address there in the first place.

awad• 10 years ago

Saw this on HN earlier in the year and have been slowly phasing into my life. I have mail setup for my domain and assign per-site aliases (hn@mail.com, fb@mail.com, etc). All of the mail just forwards to my gmail that I've had forever, though that is more of a matter of convenience and habit. The key thing is, I can spin up a new email whenever I want since I am the wildcard, and I control the host and am not tied to the whims of gmail so I am covered on both ends.

FT_intern• 10 years ago

Be careful running your own email server. Some domain hosts have terrible security (easily socially engineered).

I would love to know which DNS provider has the best security if anyone has done the research.

kumarski• 10 years ago

The data can all be found here as well though: https://www.thecthulhu.com/

I've come to assume some non-zero percentage of sales/marketing/lead gen companies are using this as a go-between for an SMTP validator.

gnu8• 10 years ago

I still want the dumps for myself, so I can determine if any of my accounts are compromised. I understand why it's not possible to distribute these dumps, but for the same reason, I'm not going to search someone else's database. How can we have a database of dumps that people can query safely, without revealing their information to the people with the dumps?

mdrzn• 10 years ago

Search the dump yourself and look for your address.

Or even better, build the service you're asking for! Clearly, there's a need.

meneses• 10 years ago

Awesome Analysis of the data - https://www.sizzleanalytics.com/Boards/sizzle/Data-Breaches-...

1.4B records from “Have I been pwned” for analysis

Related Stories