The Wise Way to Crowdsource a Manhunt

http://www.newyorker.com/online/blogs/comment/2013/04/reddit-tsarnaev-marathon-bombers-wisdom-of-crowds.html

by scholia • 13 years ago

67 34 13 years ago

kanamekun• 13 years ago

The Snapshot Serengeti project is trying to do something along the lines of NASA Clickworkers: to use groups of amateurs to "identify the thousands of animal photographs collected from 225 motion-activated cameras in the nature preserve": http://connect.dpreview.com/post/3745415348/sanpshot-serenge...

It's a lot of fun and it feels great to contribute to a noble project. Plus it's surprisingly addictive: http://www.snapshotserengeti.org/

incision• 13 years ago

Very nice, hadn't heard of that one. I've spent some time on SETILive [1] which does something very similar for signal classification and found it to be almost meditative.

It's not quite the same thing, but also I get the same kick out of foldit [2] and eterna [3].

1: http://setilive.org/

2: http://fold.it/portal/

3: http://eterna.cmu.edu/web/

corresation• 13 years ago

In the not so distant future, visualizing and "walking through" the past -- ala the movie Deja Vu (but minus the wormholes) -- will become a reality. It was the lack of even a rudimentary version of this that made the Reddit sleuthing so hopeless, with most media having zero context, location, or even time taken, that metadata not available.

But imagine photo synth/streetview with a time slider: Thousands of pictures and overlaid videos time synchronized and placed into a scene. Such a tool could be incredible for such investigations.

hornbaker• 13 years ago

Google will almost certainly add that functionality to Street View in the future. The coming ubiquity of always-on Glass-type cameras will turn a significant fraction of the public into walking surveillance cameras.

podperson• 13 years ago

The first key point of this article was that the footage showing the actual bombers wasn't released until the FBI identified them as suspects, therefore the "crowd" was on a wild goose chase.

Also pointed out in the article, having the "crowd" incorrectly identify a suspect isn't the big problem, the police are perfectly capable of persecuting the wrong people without the help of a crowd.

danso2• 13 years ago

This was one of the most insightful, thoughtful pieces on crowdsourcing that I've ever read. Reddit's an easy target to bash, but the OP is right that the expectations, on both sides, were too high given the disparity of information between the crowdsource and traditional investigators. Until there's a situation in which both groups have roughly the same access to available information, there's not much point in arguing which way is more effective.

This, of course, is a different argument than whether crowdsourcing, in its pursuit, ends up causing too much collateral damage. The voting scheme proposed by the OP seems like a possible way to mitigate the herd mentality...but I also agree with the OP that the feeling of community is what makes Reddit so engaging, for better or worse.

davidw• 13 years ago

The author is the guy who wrote the "Wisdom of Crowds" book.

maxerickson• 13 years ago

I wasn't watching terribly closely, but it was a bit disappointing to see people circling backpacks, rather than say, establishing a set of photos/videos that were in close proximity to the bombs, in both time and location.

Much of the time, the context people were putting around photos they were linking was observably false (that is, contradicted by glancing at the photos). Stuff like saying a backpack was gone, when a strap was clearly visible...

A WaPo article [1] about the bombings mentions that negative consequences from internet speculation factored into the decision to release the photos of the suspects.

The potential to do some good certainly seems to be there, but I'm not sure the efforts to do things in a careful and organized manner will ever be able to keep up with the excited speculation. Of course, the careful and organized effort may still useful if it can be separated from the excited speculation.

[1] http://www.washingtonpost.com/world/national-security/inside...

danso2• 13 years ago

I didn't read that WaPo article but that's an interesting assertion...because the release of the photos almost certainly spurred the bombers into action. Tragically, it resulted in the death of the police officer, but had they slipped away, the damage may have been worse (I'm assuming that even if photos were distributed to officials only, the bombers could have had an opportunity to escape notice).

stordoff• 13 years ago

>Much of the time, the context people were putting around photos they were linking was observably false

The most obvious case I saw of that was the number of times photos like this[1] were posted (and others of the same people without one of the backpacks) as 'proof' of their involvement, despite one of the photos being from after the explosions[2]

[1] http://i.imgur.com/GZgmb3L.jpg [2] (warning: shows bomb site. Not too graphic, but some may find it unpleasant) http://i.imgur.com/iVzG7MK.jpg

sodomizer• 13 years ago

"this method of sleuthing would be far less exciting for users, and would probably generate less traffic, than its current free-for-all approach."

When people do things for their own entertainment, they're always thoughtless.

When they do them out of deep conviction that they're contributing something positive, as in the open source movement, they are more effective.

Ironically, terribly and not-funny-at-all, the Boston bombers were working with the latter mindset. This enabled them to put a city on standstill and terrify a nation using bombs made from household items and black powder (normally considered too slow to be useful as a modern weapon).

salman89• 13 years ago

I thought about this too, ideally you would want to use crowdsourcing to just gather evidence and let professionals who are also liable to sift through the evidence. Another angle to work is to allow the crowd to analyze the evidence but not make that analysis public. Problem boils down to incentive - if your contributions are not made public and there is no attribution, is the motivation still there? 4chan is probably more interesting to look at in this regard, since all posts are anonymous.

prof_hobart• 13 years ago

Some of this could potentially be solved by having the crowdsourcing contributions filtered/moderated by the professionals (e.g. not getting posted publicly until a pro has read it, and "voting" done or at least swayed, by the pros).

The question is whether the time spent doing this moderation would be better spent actually doing the investigations themselves.

macspoofing• 13 years ago

I do have to say that for an internet witch hunt, this one was very restrained.

kstenerud2• 13 years ago

I for one am glad that the Reddit effort turned into the fiasco it did.

Criminal investigation is a profession. Asking a bunch of amateurs to come up with a solution is akin to asking a bunch of amateurs to build an air traffic control system. Sure, there might be a few people in the crowd who can actually do it, but most people can't. Worse, they're likely to suffer from the Dunning-Kruger effect. And with everyone given equal voice regardless of ability, and a popularity driven forum, you have a recipe for disaster every time. The only difference between this and Salem is the crowd size.

The FBI played it right by not releasing anything important and letting the public have its little playtime. It's unfortunate that this caused some innocent people to be flagged by the mob, but I think that given the circumstances of our more connected lives, this was the best play to make in a bad situation. The mob would move regardless, so why not slow then down by sending them chasing their tails for a bit while the real professionals got down to business?

darkarmani2• 13 years ago

> I for one am glad that the Reddit effort turned into the fiasco it did.

I don't know if i really believe that it was that much of a fiasco. All the photos i saw where pictures of people with backpacks near the area of the bombings. That definitely gave the FBI leads.

kstenerud2• 13 years ago

Meanwhile every person singled out by the mob was quickly alerted to that fact. A big part of professionalism (and law enforcement in particular) is knowing when to be discreet.

nsns• 13 years ago

I think it is worth mentioning that once the bombers were identified, Reddit did an amazing job covering the manhunt in real-time, I think it was a first, and a great achievement. With live reporting, pictures, videos and police scanner streams, it almost felt like actually being in Boston, and a new form of journalism.

nissimk• 13 years ago

I agree. Even before the police scanner feeds, the news posts about the bombing were kind of like a filtered twitter search. While twitter is a better source of realtime information than cable news, it is very hard to filter out all the noise, but it seemed like during this story that reddit was able to do that.

Androsynth• 13 years ago

reality tv + vigilantism = a new form of journalism?

im being glib, but there are some serious ethical journalism questions that deserve to be brought up. First of all, does every step of the investigation need to be broadcast? Is this reality tv (in internet form) or actual journalism? Can rampant vigilantism on the internet be stopped? Can real journalism prevail in the face of the instant-gratification ppl can get on reddit? Do people want real journalism?

The redditors are basically using the scorched-earth method of journalism, with no regard to who they accuse. Theyre causing considerably more harm than help, which is typical of vigilantism.

An interesting question is this: why is having reporters in vietnam a good thing while redditors following every move of the bombing investigation a bad thing (considered so by some of us at least)? (please dont answer that with a smug, poorly thought-out answer. sometimes its better to think things through and not just go straight to whipping out the epeen)

darkarmani2• 13 years ago

> The redditors are basically using the scorched-earth method of journalism, with no regard to who they accuse. Theyre causing considerably more harm than help, which is typical of vigilantism.

What? The difference is that redditors don't claim to be journalists, so why would anyone put any weight in their accusations? No one was lynched by redditors (or in danger of being lynched), so let's not call this vigilantism.

zeteo• 13 years ago

So the "wisdom of the crowd" turns out to work best when it's not allowed to be a crowd. Taking independent votes from non-communicating individuals is in line with Surowiecki's original example (estimating the weight of an ox), but far from the common understanding of a "crowd" [1] or from Reddit's core mechanisms. These latter are more about emotional involvement and early consensus (both of which seem to be harmful for the accuracy of conclusions).

Infrastructure operating close to Surowiecki's specifications does exist: Mechanical Turk. I'm surprised he's not mentioning it. After some initial hype, its actual uses have turned out to be far less glamorous and revolutionary than originally hoped for.

[1] http://en.wikipedia.org/wiki/Crowd#Terminology

joekrill• 13 years ago

Everyone is quick to blame only Reddit -- and they certainly deserve a great deal of it, collectively. But I think we should look at laying part of the blame on the professional investigators themselves, too. They clearly knew who they were looking for long before they released any useable information, and I think they should have taken more initiative to explicitly eliminate Reddit suspects they knew were not involved. They could have easily issued a brief statement definitely stating they cleared Sunil Tripath, for example, without releasing information on the identities of the actual bombers. Same goes for the two innocent guys that were featured on the front page of the NY Post.

The investigators did a terrific job, and I don't want to put down anything they did. And maybe I'm being naive and this would be a slippery slope. But I still think some of these specific false accusations could have been mitigated very easily.

ProblemFactory• 13 years ago

I don't think that's a workable solution.

The professionals had identified the brothers as suspects - but only as very probable suspects. They could still have been wrong. Publicly definitely clearing someone who might later turn out to be the actual bomber or an associate would have enormous political (and perhaps legal) consequences.

I believe that any form of crowdsourced investigation is reasonable only if there just isn't enough professional interest or resources to pursue a case. For example finding your stolen laptop. If there are thousands of FBI agents working on a priority case, with additional private data feeds from security cameras, telephone logs and wiretaps, public "help" is of no use beyond stroking the ego of the participants.

sp332• 13 years ago

The "clickworkers" comment at the bottom reminds me of https://www.zooniverse.org/ which is still ongoing. You can sign up, and they quickly train you to sort and catalogue various kinds of images including moon craters, weather patterns, galaxies, etc.

juskrey• 13 years ago

Rubbish in == rubbish out. Crowdsourcing or not.

Exposure to bad photos and news is not the same as exposure to crime scene (and police data).

brown9-2• 13 years ago

A lot of this article's advice refers to Reddit as a singular entity, as if Reddit The Company was behind the online investigation:

For example, in a situation similar to the marathon bombings, Reddit could simply post the available images without comment (save, perhaps, for time stamps and where the pictures were taken), and allow users to sift through them, looking for both incriminatory and exculpatory evidence (like, say, people carrying black backpacks near the bomb sites, or still carrying them once the bombs had already gone off), while creating a voting system that would let people independently vote on which images they thought were worth further investigation. Once a critical mass of users had perused the images, Reddit could aggregate the results, and pass them on to the authorities. (The results of the voting wouldn’t even have to be made public, if people were concerned about false accusations.) This would take advantage of the crowd’s ability to filter through lots of images quickly, while also preserving the diversity and independence that make crowds smart. The point of this wouldn’t be to compete with, let alone replace, the work of official investigators, but rather to provide, as it were, another set of eyes, one that might see something that investigators missed.

If Reddit were looking for a model to follow, it could use NASA’s Clickworkers experiment...

For any of this to work, "Reddit" would need to be an organization with a top-down command and control hierarchy, not an online forum where any user can create a new subforum at any time.

In this idealized world of Reddit investigations, who is overseeing the crowd-sourcing and handling the aggregation?

Vivtek2• 13 years ago

This is a thoughtful article, except for the basic premise that if Reddit wants to be useful at crowdsourcing of this nature, it needs to be something other than Reddit.

Reddit is a social forum. It allowed people to feel part of things, and that's great. Expecting your hammer to be good at driving screws, though? That's just doomed to failure.

Moral of this story for me: just because you have a crowd doesn't mean you're crowdsourcing. Use tools appropriate to the task and you might actually get results.

Sure didn't stop the national media - who actually get paid to know better - from blowing Reddit crowdsourcing up as some kind of magical deus ex machina, though.

pseut• 13 years ago

That was the last paragraph of TFA:

The problem from Reddit’s perspective, of course, is that this method of sleuthing would be far less exciting for users, and would probably generate less traffic, than its current free-for-all approach. The point of the “find-the-bombers” subthread, after all, wasn’t just to find the bombers—it was also to connect and talk with others, and to feel like you were part of a virtual community....

Your last two sentences are points made by the article.

Vivtek2• 13 years ago

No, I think these are afterthoughts. The point of the article as I read it is that Reddit needs to change, that these effects are only due to its current design. But I think a saner takeaway is that Reddit is simply the wrong tool for crowdsourcing.

The Wise Way to Crowdsource a Manhunt

Related Stories