1.5 TB of Dark Net Market scrapes

http://www.gwern.net/Black-market%20archives

by gwern • 11 years ago

364 73 11 years ago

b6• 11 years ago

gwern, you are an absolute force of nature when it comes to generating and collecting and presenting information in a useful way. Thank you.

icpmacdo• 11 years ago

I hope a he gets some good donations from HN, Ill send him a few dollars in BTC.

curiousg2• 11 years ago

What amazing work! I am very interested in doing research with Tor and a dataset like this could make my job a heck of a lot easier. I have a legal question though: Are your scrapes text only? Before I work with this dataset, I want to make sure that there's no possibility it contains illegal images (child porn).

mslev• 11 years ago

Lower down on the page, he says he did scrape at least one site with such images, although he specifically only took text. Can't verify that this was the case for all scraped sites.

fineman3• 11 years ago

How about ascii art?

Actually, this is an interesting topic. Poisoning a dataset. CP would work for private security investigators, and to poison against government investigators you could use leaked classified secrets.

Could you work around this by operating on the files on VPS you don't own, streaming a very low-res ('Basilisk'-proof - https://en.wikipedia.org/wiki/BLIT_(short_story) ) remote desktop image.

curiousg2• 11 years ago

Possession laws are pretty strict and hard to decode. I wouldn't want to be the test case in court. The idea of "poisoning" a dataset is an interesting theoretical. But in practice, I just want to judge the likelihood that the dataset is poisoned by the presence of images. If it is then there's not much I can do with it.

njloof• 11 years ago

Does ASCII art have victims? Other than its audience, I mean.

gwern6• 11 years ago

They are generally not text only. I feel that images are useful to allow browsing the markets as they were and may be highly valuable in their own right as research material, so I tried to collects images where applicable. (The forums usually did not support any kind of image upload other than avatars, so this is more relevant to the markets than forum scrapes.)

As far as CP goes, there should be essentially zero CP anywhere in the archive. DNM users almost universally loathe CP, and no market has ever dared to permit sales. (You may find this funny: CP is so taboo, on the DNMs like elsewhere, that it's been used in at least one attack - SR2's DoctorClu/Brian Farrell infamously attacked a rival market's forum by posting CP to it.)

userbinator• 11 years ago

DNM users almost universally loathe CP, and no market has ever dared to permit sales.

These users are willing to do so many other illegal things, but the thought of being known as a pedophile or supporting pedophilia in any way, is completely abhorrent to them? Interesting datapoint.

joshmn3• 11 years ago

This is so cool. Thanks gwern.

If someone's feeling bored, you're welcome to put the entire archive on a web server for us to look at......

Or maybe I'll just do it.

fractalcat• 11 years ago

You mean like this? https://archive.org/download/dnmarchives

gwern6• 11 years ago

I don't think that works. It's not remotely browsable or searchable. It would be quite challenging to put these scrapes up, anyway. They're regular wget crawls with a regular directory/file structure, the problem is that there's so much material and so many files that it can be almost impossible to find what you are looking for... (Plus you need to rewrite links into relative links to make everything render properly.)

ryanlol2• 11 years ago

collating and creating these scrapes has absorbed an enormous amount of my time & energy due to the need to solve CAPTCHAs,...

Have you considered automating this?

smeyer• 11 years ago

The whole point of CAPTCHAs is to be difficult to automate. Or are you suggesting automating farming out the CAPTCHA solutions to cheap workers?

jMyles• 11 years ago

I assumed it was a joke. But maybe not?

ryanlol2• 11 years ago

There's a plenty of service providers that sell APIs to captcha solving services at reasonable prices.

davidgerard2• 11 years ago

The main problem is that most CAPTCHAs are terrible, and do a better job of keeping humans out than robots.

timboslice• 11 years ago

Yes. Quick list of providers I've worked with, in no particular order:

http://www.imagetyperz.com/

http://expertdecoders.com/

http://www.deathbycaptcha.com/user/login

https://de-captcher.com/

http://bypasscaptcha.com/

gwern6• 11 years ago

I did, but the problem is that I never expected to be scraping for so long and it was always easier to just solve by hand than do a complete rewrite and allow for using CAPTCHA libraries. If I had known I would be scraping for ~3 years, I would have done many things differently: http://www.gwern.net/Black-market%20archives#how-to-crawl-ma...

branchless2• 11 years ago

This is why I love the internet. This article has given me a fascinating glimpse into a world I have no idea about.

Author: thank you so much for taking the time to document this.

ethanbond• 11 years ago

You should take the time to read his other posts. Gwern is one of the most fascinating people/websites I know of.

branchless2• 11 years ago

Thanks, I will read at least another by him.

jmduke• 11 years ago

Unrelated to the darknet, but this Twitter account evoked a very similar feeling in me (random snippets from userboards in the 80's and early 90's):

https://twitter.com/wwwtxt

oxide• 11 years ago

what a great twitter account. thanks for bringing this to my attention.

WalterGR2• 11 years ago

"I heard about the Apple Watch recently and was going to check it out—but not now. It can't even transmit or input data. ☯93JAN"

Prescient.

acosmism• 11 years ago

you could also use a library to handle captchas. have a look at 'tesseract'

rcpt2• 11 years ago

I've had a very slow-moving hobby project to parse and analyze a subset of this data: https://github.com/rcompton/black-market-recommender-systems

So far I've had some ok results along the lines of "91.7% of vendors who sold speed and MDMA also sold ecstasy" http://ryancompton.net/2015/03/24/darknet-market-basket-anal... I am working on extending this to markets besides evolution now.

ez123• 11 years ago

Just FYI, MDMA is ecstasy.

rcpt2• 11 years ago

I chose that statistic to be a bit tongue-in-cheek, cf

While 'ecstasy' is the popular name for MDMA, the functional definition of ecstasy is any pill represented as MDMA on the street. Ecstasy pills are notoriously unreliable in content, more so than most other street drugs, and commonly contain either caffeine, ephedrine, amphetamines, MDA, MDE, DXM, or--in rare cases--DOB, and don't necessarily contain MDMA or any psychoactive.

https://www.erowid.org/chemicals/mdma/mdma_basics.shtml

gwern6• 11 years ago

The system works!

woah• 11 years ago

What people think of as ecstasy (the pill that makes you dance all night) is usually mdma mixed with some kind of speed. That's why this statistic was amusing.

chm3• 11 years ago

I don't do MDMA personally but I have some friends who like going to raves with some chemical supplies. On the streets, some make it a point of honor to sell/buy only "MDMA" because "ecstasy" is "crap".

The only precise name is MDMA. In my opinion the name users give to what they're ingesting is inconsequential. Most users have no idea what MDMA means anyway, nor would they know how to chemically identify if what they bought might contain MDMA.

Moshe_Silnorin• 11 years ago

Gwern has a Patreon page now for anyone interested in supporting his research: https://www.patreon.com/gwern?ty=h

mapt• 11 years ago

Gwern does more interesting things with lower monetary burn rate than most anyone I've met.

stevewepay• 11 years ago

What is the legality with respect to downloading this file? Could it contain material that would put us at legal risk?

NhanH• 11 years ago

I really don't know the answer. But I just want to remark that it seems to be a terrible situation that more than one of us have to wonder about the legality of downloading the dataset (there was a comment thread below on the same topic)

brandon272• 11 years ago

I wouldn't download it unless you want a visit from the feds.

belorn• 11 years ago

Since when would it be illegal to posses statistics. Are we afraid that dark markets might claiming proprietary ownership of that data?

SergeyHack• 11 years ago

"HOW TO CRAWL MARKETS" section has good tips for general crawling as well.

1.5 TB of Dark Net Market scrapes

Related Stories